[jira] [Commented] (PHOENIX-938) Use higher priority queue for index updates to prevent deadlock
[ https://issues.apache.org/jira/browse/PHOENIX-938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14349091#comment-14349091 ] Jesse Yates commented on PHOENIX-938: - Yup, that's a bug. Over on PHOENIX-1676 we are tracking down all the issues with the index priority - mind adding your comment over there? At some point I was trying to refactor HBase Rpc Schedulers to handle generic queues so scheduler impls wouldn't have to actually manage their own queues, but alas, that started to get very convoluted and was never finished. Use higher priority queue for index updates to prevent deadlock --- Key: PHOENIX-938 URL: https://issues.apache.org/jira/browse/PHOENIX-938 Project: Phoenix Issue Type: Bug Affects Versions: 4.0.0, 4.1 Reporter: James Taylor Assignee: Jesse Yates Fix For: 5.0.0, 4.1 Attachments: PHOENIX-938-master-v3.patch, phoenix-938-4.0-v0.patch, phoenix-938-4.0-v0.patch, phoenix-938-master-v0.patch, phoenix-938-master-v1.patch, phoenix-938-master-v2.patch, phoenix-938-master-v4.patch, phoenix-938-master-v5.patch With our current global secondary indexing solution, a batched Put of table data causes a RS to do a batch Put to other RSs. This has the potential to lead to a deadlock if all RS are overloaded and unable to process the pending batched Put. To prevent this, we should use a higher priority queue to submit these Puts so that they're always processed before other Puts. This will prevent the potential for a deadlock under high load. Note that this will likely require some HBase 0.98 code changes and would not be feasible to implement for HBase 0.94. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (PHOENIX-938) Use higher priority queue for index updates to prevent deadlock
[
https://issues.apache.org/jira/browse/PHOENIX-938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14348299#comment-14348299
]
Jeongdae Kim commented on PHOENIX-938:
--
Jesse Yates - I found something strange in your codes, that
callExecutor.start() do not be called anywhere in PhoenixIndexRpcScheduler as
below.
@Override
public void start() {
delegate.start();
}
i think callExecutor.start() must be called in PhoenixIndexRpcScheduler.start()
as below to work fine as you intended.
@Override
public void start() {
delegate.start();
callExecutor.start(port);
}
could you explain your intention or some codes that i might not be seen before,
if your implement is intended?
Use higher priority queue for index updates to prevent deadlock
---
Key: PHOENIX-938
URL: https://issues.apache.org/jira/browse/PHOENIX-938
Project: Phoenix
Issue Type: Bug
Affects Versions: 4.0.0, 4.1
Reporter: James Taylor
Assignee: Jesse Yates
Fix For: 5.0.0, 4.1
Attachments: PHOENIX-938-master-v3.patch, phoenix-938-4.0-v0.patch,
phoenix-938-4.0-v0.patch, phoenix-938-master-v0.patch,
phoenix-938-master-v1.patch, phoenix-938-master-v2.patch,
phoenix-938-master-v4.patch, phoenix-938-master-v5.patch
With our current global secondary indexing solution, a batched Put of table
data causes a RS to do a batch Put to other RSs. This has the potential to
lead to a deadlock if all RS are overloaded and unable to process the pending
batched Put. To prevent this, we should use a higher priority queue to submit
these Puts so that they're always processed before other Puts. This will
prevent the potential for a deadlock under high load. Note that this will
likely require some HBase 0.98 code changes and would not be feasible to
implement for HBase 0.94.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (PHOENIX-938) Use higher priority queue for index updates to prevent deadlock
[ https://issues.apache.org/jira/browse/PHOENIX-938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14073405#comment-14073405 ] James Taylor commented on PHOENIX-938: -- [~jesse_yates] - that'd be great if you could update the website docs. The website lives in SVN and in general is based on a set of markdown files. Here's the process: - checkout the website: svn checkout https://svn.apache.org/repos/asf/phoenix - modify the appropriate markdown file: ./site/source/src/site/markdown/secondary_indexing.md - generate the html: ./build.sh - checkin the modified files (including the generate html file(s)): svn commit [~mujtaba] - would you mind adding this info to the How to Contribute page? Use higher priority queue for index updates to prevent deadlock --- Key: PHOENIX-938 URL: https://issues.apache.org/jira/browse/PHOENIX-938 Project: Phoenix Issue Type: Bug Affects Versions: 4.0.0, 4.1 Reporter: James Taylor Assignee: Jesse Yates Fix For: 5.0.0, 4.1 Attachments: PHOENIX-938-master-v3.patch, phoenix-938-4.0-v0.patch, phoenix-938-4.0-v0.patch, phoenix-938-master-v0.patch, phoenix-938-master-v1.patch, phoenix-938-master-v2.patch, phoenix-938-master-v4.patch, phoenix-938-master-v5.patch With our current global secondary indexing solution, a batched Put of table data causes a RS to do a batch Put to other RSs. This has the potential to lead to a deadlock if all RS are overloaded and unable to process the pending batched Put. To prevent this, we should use a higher priority queue to submit these Puts so that they're always processed before other Puts. This will prevent the potential for a deadlock under high load. Note that this will likely require some HBase 0.98 code changes and would not be feasible to implement for HBase 0.94. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (PHOENIX-938) Use higher priority queue for index updates to prevent deadlock
[ https://issues.apache.org/jira/browse/PHOENIX-938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14072103#comment-14072103 ] Hudson commented on PHOENIX-938: SUCCESS: Integrated in Phoenix | Master | Hadoop1 #289 (See [https://builds.apache.org/job/Phoenix-master-hadoop1/289/]) PHOENIX-938: Use higher priority queue for index updates to prevent deadlock (jyates: rev 1954c717a12561bdc2184ba23c53afae3f900084) * phoenix-core/src/main/java/org/apache/phoenix/hbase/index/builder/BaseIndexBuilder.java * phoenix-core/src/main/java/org/apache/phoenix/hbase/index/IndexQosRpcControllerFactory.java * phoenix-core/src/it/java/org/apache/phoenix/end2end/index/IndexHandlerIT.java * phoenix-core/src/main/java/org/apache/phoenix/hbase/index/IndexQosCompat.java * phoenix-core/src/main/java/org/apache/phoenix/util/IndexUtil.java * phoenix-core/src/main/java/org/apache/phoenix/hbase/index/table/CoprocessorHTableFactory.java * phoenix-core/src/main/java/org/apache/phoenix/hbase/index/builder/IndexBuildManager.java * phoenix-core/src/main/java/org/apache/phoenix/hbase/index/ipc/PhoenixIndexRpcSchedulerFactory.java * pom.xml * phoenix-core/src/main/java/org/apache/hadoop/hbase/ipc/PhoenixIndexRpcScheduler.java * phoenix-core/src/test/java/org/apache/hadoop/hbase/ipc/PhoenixIndexRpcSchedulerTest.java * phoenix-core/src/main/java/org/apache/phoenix/hbase/index/builder/IndexBuilder.java * phoenix-core/src/test/java/org/apache/hadoop/hbase/regionserver/PhoenixIndexRpcSchedulerFactoryTest.java Use higher priority queue for index updates to prevent deadlock --- Key: PHOENIX-938 URL: https://issues.apache.org/jira/browse/PHOENIX-938 Project: Phoenix Issue Type: Bug Affects Versions: 4.0.0, 4.1 Reporter: James Taylor Assignee: Jesse Yates Fix For: 5.0.0, 4.1 Attachments: PHOENIX-938-master-v3.patch, phoenix-938-4.0-v0.patch, phoenix-938-4.0-v0.patch, phoenix-938-master-v0.patch, phoenix-938-master-v1.patch, phoenix-938-master-v2.patch, phoenix-938-master-v4.patch, phoenix-938-master-v5.patch With our current global secondary indexing solution, a batched Put of table data causes a RS to do a batch Put to other RSs. This has the potential to lead to a deadlock if all RS are overloaded and unable to process the pending batched Put. To prevent this, we should use a higher priority queue to submit these Puts so that they're always processed before other Puts. This will prevent the potential for a deadlock under high load. Note that this will likely require some HBase 0.98 code changes and would not be feasible to implement for HBase 0.94. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (PHOENIX-938) Use higher priority queue for index updates to prevent deadlock
[ https://issues.apache.org/jira/browse/PHOENIX-938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14071063#comment-14071063 ] Jesse Yates commented on PHOENIX-938: - Now that HBase 0.98.4 has been released (thanks again [~apurtell]!), looks like its time to commit this guy too. I plan on committing tonight/early tomorrow, unless there are any objections. Use higher priority queue for index updates to prevent deadlock --- Key: PHOENIX-938 URL: https://issues.apache.org/jira/browse/PHOENIX-938 Project: Phoenix Issue Type: Bug Affects Versions: 4.0.0, 4.1 Reporter: James Taylor Assignee: Jesse Yates Fix For: 5.0.0, 4.1 Attachments: PHOENIX-938-master-v3.patch, phoenix-938-4.0-v0.patch, phoenix-938-master-v0.patch, phoenix-938-master-v1.patch, phoenix-938-master-v2.patch, phoenix-938-master-v4.patch With our current global secondary indexing solution, a batched Put of table data causes a RS to do a batch Put to other RSs. This has the potential to lead to a deadlock if all RS are overloaded and unable to process the pending batched Put. To prevent this, we should use a higher priority queue to submit these Puts so that they're always processed before other Puts. This will prevent the potential for a deadlock under high load. Note that this will likely require some HBase 0.98 code changes and would not be feasible to implement for HBase 0.94. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (PHOENIX-938) Use higher priority queue for index updates to prevent deadlock
[ https://issues.apache.org/jira/browse/PHOENIX-938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14071074#comment-14071074 ] James Taylor commented on PHOENIX-938: -- +1, as long as you've tested the new Phoenix snapshot jar against a pre-0.98.4 release to make sure we'll still run (albeit without your fix). Use higher priority queue for index updates to prevent deadlock --- Key: PHOENIX-938 URL: https://issues.apache.org/jira/browse/PHOENIX-938 Project: Phoenix Issue Type: Bug Affects Versions: 4.0.0, 4.1 Reporter: James Taylor Assignee: Jesse Yates Fix For: 5.0.0, 4.1 Attachments: PHOENIX-938-master-v3.patch, phoenix-938-4.0-v0.patch, phoenix-938-master-v0.patch, phoenix-938-master-v1.patch, phoenix-938-master-v2.patch, phoenix-938-master-v4.patch With our current global secondary indexing solution, a batched Put of table data causes a RS to do a batch Put to other RSs. This has the potential to lead to a deadlock if all RS are overloaded and unable to process the pending batched Put. To prevent this, we should use a higher priority queue to submit these Puts so that they're always processed before other Puts. This will prevent the potential for a deadlock under high load. Note that this will likely require some HBase 0.98 code changes and would not be feasible to implement for HBase 0.94. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (PHOENIX-938) Use higher priority queue for index updates to prevent deadlock
[ https://issues.apache.org/jira/browse/PHOENIX-938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14071140#comment-14071140 ] Andrew Purtell commented on PHOENIX-938: Confirmed that artifacts for 0.98.4-hadoop1 and 0.98.4-hadoop2 are available now on repository.apache.org Use higher priority queue for index updates to prevent deadlock --- Key: PHOENIX-938 URL: https://issues.apache.org/jira/browse/PHOENIX-938 Project: Phoenix Issue Type: Bug Affects Versions: 4.0.0, 4.1 Reporter: James Taylor Assignee: Jesse Yates Fix For: 5.0.0, 4.1 Attachments: PHOENIX-938-master-v3.patch, phoenix-938-4.0-v0.patch, phoenix-938-master-v0.patch, phoenix-938-master-v1.patch, phoenix-938-master-v2.patch, phoenix-938-master-v4.patch With our current global secondary indexing solution, a batched Put of table data causes a RS to do a batch Put to other RSs. This has the potential to lead to a deadlock if all RS are overloaded and unable to process the pending batched Put. To prevent this, we should use a higher priority queue to submit these Puts so that they're always processed before other Puts. This will prevent the potential for a deadlock under high load. Note that this will likely require some HBase 0.98 code changes and would not be feasible to implement for HBase 0.94. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (PHOENIX-938) Use higher priority queue for index updates to prevent deadlock
[ https://issues.apache.org/jira/browse/PHOENIX-938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14061040#comment-14061040 ] Andrew Purtell commented on PHOENIX-938: Ok. If you think of anything else that needs to go in here before the .4 release is official, please let me know. We can pull the RC for further work if needed. Use higher priority queue for index updates to prevent deadlock --- Key: PHOENIX-938 URL: https://issues.apache.org/jira/browse/PHOENIX-938 Project: Phoenix Issue Type: Bug Affects Versions: 4.0.0, 4.1 Reporter: James Taylor Assignee: Jesse Yates Fix For: 5.0.0, 4.1 Attachments: PHOENIX-938-master-v3.patch, phoenix-938-4.0-v0.patch, phoenix-938-master-v0.patch, phoenix-938-master-v1.patch, phoenix-938-master-v2.patch With our current global secondary indexing solution, a batched Put of table data causes a RS to do a batch Put to other RSs. This has the potential to lead to a deadlock if all RS are overloaded and unable to process the pending batched Put. To prevent this, we should use a higher priority queue to submit these Puts so that they're always processed before other Puts. This will prevent the potential for a deadlock under high load. Note that this will likely require some HBase 0.98 code changes and would not be feasible to implement for HBase 0.94. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (PHOENIX-938) Use higher priority queue for index updates to prevent deadlock
[ https://issues.apache.org/jira/browse/PHOENIX-938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14061177#comment-14061177 ] Ted Yu commented on PHOENIX-938: bq. +Class.forName(org.apache.hadoop.hbase.ipc.RpcExector); Looks like there was a typo above: RpcExector Use higher priority queue for index updates to prevent deadlock --- Key: PHOENIX-938 URL: https://issues.apache.org/jira/browse/PHOENIX-938 Project: Phoenix Issue Type: Bug Affects Versions: 4.0.0, 4.1 Reporter: James Taylor Assignee: Jesse Yates Fix For: 5.0.0, 4.1 Attachments: PHOENIX-938-master-v3.patch, phoenix-938-4.0-v0.patch, phoenix-938-master-v0.patch, phoenix-938-master-v1.patch, phoenix-938-master-v2.patch With our current global secondary indexing solution, a batched Put of table data causes a RS to do a batch Put to other RSs. This has the potential to lead to a deadlock if all RS are overloaded and unable to process the pending batched Put. To prevent this, we should use a higher priority queue to submit these Puts so that they're always processed before other Puts. This will prevent the potential for a deadlock under high load. Note that this will likely require some HBase 0.98 code changes and would not be feasible to implement for HBase 0.94. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (PHOENIX-938) Use higher priority queue for index updates to prevent deadlock
[ https://issues.apache.org/jira/browse/PHOENIX-938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14061358#comment-14061358 ] Jesse Yates commented on PHOENIX-938: - and linking this to HBASE-11513 so we an use the BalancedRpcExecutor in Phoenix as well, rather than having to make our own. Use higher priority queue for index updates to prevent deadlock --- Key: PHOENIX-938 URL: https://issues.apache.org/jira/browse/PHOENIX-938 Project: Phoenix Issue Type: Bug Affects Versions: 4.0.0, 4.1 Reporter: James Taylor Assignee: Jesse Yates Fix For: 5.0.0, 4.1 Attachments: PHOENIX-938-master-v3.patch, phoenix-938-4.0-v0.patch, phoenix-938-master-v0.patch, phoenix-938-master-v1.patch, phoenix-938-master-v2.patch, phoenix-938-master-v4.patch With our current global secondary indexing solution, a batched Put of table data causes a RS to do a batch Put to other RSs. This has the potential to lead to a deadlock if all RS are overloaded and unable to process the pending batched Put. To prevent this, we should use a higher priority queue to submit these Puts so that they're always processed before other Puts. This will prevent the potential for a deadlock under high load. Note that this will likely require some HBase 0.98 code changes and would not be feasible to implement for HBase 0.94. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (PHOENIX-938) Use higher priority queue for index updates to prevent deadlock
[ https://issues.apache.org/jira/browse/PHOENIX-938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14057462#comment-14057462 ] James Taylor commented on PHOENIX-938: -- How about this as a plan, [~jesse_yates] and [~apurtell]? - check-in your patch and document that it fixes the issue for 0.98.3 only (assuming it doesn't break Phoenix for 0.98.2- and 0.98.3+). - work with HBase community to make these APIs public and evolving for 0.98.next release. - implement solution on top of these public APIs. I don't think transactions is really going to help with this, so it'd be good to get an as-permanent-as-possible solution IMO. Use higher priority queue for index updates to prevent deadlock --- Key: PHOENIX-938 URL: https://issues.apache.org/jira/browse/PHOENIX-938 Project: Phoenix Issue Type: Bug Affects Versions: 4.0.0, 4.1 Reporter: James Taylor Assignee: Jesse Yates Fix For: 5.0.0, 4.1 Attachments: phoenix-938-4.0-v0.patch, phoenix-938-master-v0.patch, phoenix-938-master-v1.patch With our current global secondary indexing solution, a batched Put of table data causes a RS to do a batch Put to other RSs. This has the potential to lead to a deadlock if all RS are overloaded and unable to process the pending batched Put. To prevent this, we should use a higher priority queue to submit these Puts so that they're always processed before other Puts. This will prevent the potential for a deadlock under high load. Note that this will likely require some HBase 0.98 code changes and would not be feasible to implement for HBase 0.94. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (PHOENIX-938) Use higher priority queue for index updates to prevent deadlock
[ https://issues.apache.org/jira/browse/PHOENIX-938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14056785#comment-14056785 ] Jesse Yates commented on PHOENIX-938: - Started looking at this today at first, 0.98.3 is pretty simple. Then it starts to get complicated as 0.98.4 completely changed how rpc scheduling is implemented (HBASE-11355). I don't know if we have the bandwidth to continually monitor all the possible changes to the scheduler code to support this. Further, as we look to real transactions, this implementation becomes somewhat moot; maybe we just leave the code as-is? The nitty-gritty of the details is that 0.98.4 introduced the idea of an RpcExecutor (which is a great improvement over the current munging), but that isn't in 0.98.3, so we would either need to port that class to phoenix (losing any updates from the HBase community), but that's kinda already what I was doing with this patch, so maybe that's alright for now. Now, we could have a whole reflection framework to support the different HBase versions we are running (which becomes a testing pain, but doable) and then pick the most optimal one (0.98.4+ just uses RpcExecutor as-is, 0.98.3 uses the copied code, =0.98.2 ignores). Or we can copy the changed implementations back and just use the same thing everywhere, but we loose out on changes... There really isn't a clean solution here :-/ Really, this stems from the RpcScheduler code being a private interface in HBase but wanting to leverage it outside HBase. thoughts [~jamestaylor]? Use higher priority queue for index updates to prevent deadlock --- Key: PHOENIX-938 URL: https://issues.apache.org/jira/browse/PHOENIX-938 Project: Phoenix Issue Type: Bug Affects Versions: 4.0.0, 4.1 Reporter: James Taylor Assignee: Jesse Yates Fix For: 5.0.0, 4.1 Attachments: phoenix-938-4.0-v0.patch, phoenix-938-master-v0.patch, phoenix-938-master-v1.patch With our current global secondary indexing solution, a batched Put of table data causes a RS to do a batch Put to other RSs. This has the potential to lead to a deadlock if all RS are overloaded and unable to process the pending batched Put. To prevent this, we should use a higher priority queue to submit these Puts so that they're always processed before other Puts. This will prevent the potential for a deadlock under high load. Note that this will likely require some HBase 0.98 code changes and would not be feasible to implement for HBase 0.94. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (PHOENIX-938) Use higher priority queue for index updates to prevent deadlock
[ https://issues.apache.org/jira/browse/PHOENIX-938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14056818#comment-14056818 ] Andrew Purtell commented on PHOENIX-938: I linked over to HBASE-11355 with a comment pointing here so watchers on that issue can find their way over. Use higher priority queue for index updates to prevent deadlock --- Key: PHOENIX-938 URL: https://issues.apache.org/jira/browse/PHOENIX-938 Project: Phoenix Issue Type: Bug Affects Versions: 4.0.0, 4.1 Reporter: James Taylor Assignee: Jesse Yates Fix For: 5.0.0, 4.1 Attachments: phoenix-938-4.0-v0.patch, phoenix-938-master-v0.patch, phoenix-938-master-v1.patch With our current global secondary indexing solution, a batched Put of table data causes a RS to do a batch Put to other RSs. This has the potential to lead to a deadlock if all RS are overloaded and unable to process the pending batched Put. To prevent this, we should use a higher priority queue to submit these Puts so that they're always processed before other Puts. This will prevent the potential for a deadlock under high load. Note that this will likely require some HBase 0.98 code changes and would not be feasible to implement for HBase 0.94. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (PHOENIX-938) Use higher priority queue for index updates to prevent deadlock
[ https://issues.apache.org/jira/browse/PHOENIX-938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14056848#comment-14056848 ] Jesse Yates commented on PHOENIX-938: - Thanks Andy! Perhaps just a bit tired today to find my way to the community solution :) Use higher priority queue for index updates to prevent deadlock --- Key: PHOENIX-938 URL: https://issues.apache.org/jira/browse/PHOENIX-938 Project: Phoenix Issue Type: Bug Affects Versions: 4.0.0, 4.1 Reporter: James Taylor Assignee: Jesse Yates Fix For: 5.0.0, 4.1 Attachments: phoenix-938-4.0-v0.patch, phoenix-938-master-v0.patch, phoenix-938-master-v1.patch With our current global secondary indexing solution, a batched Put of table data causes a RS to do a batch Put to other RSs. This has the potential to lead to a deadlock if all RS are overloaded and unable to process the pending batched Put. To prevent this, we should use a higher priority queue to submit these Puts so that they're always processed before other Puts. This will prevent the potential for a deadlock under high load. Note that this will likely require some HBase 0.98 code changes and would not be feasible to implement for HBase 0.94. -- This message was sent by Atlassian JIRA (v6.2#6252)
