[jira] [Updated] (HIVE-15710) HS2 Stopped when running in background

2017-02-15 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15710?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-15710: -- Attachment: HIVE-15710.3.patch Update to address comments. I think CLI also needs this logic. [~Ferd] let me

[jira] [Commented] (HIVE-15926) Hive 2.1.1 is not supporting any version of Spark

2017-02-16 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15869485#comment-15869485 ] Rui Li commented on HIVE-15926: --- Hi [~hrishidypim], Hive-2.1.1 doesn't support Spark-2.0.0 or later.

[jira] [Resolved] (HIVE-15946) Failing test : TestCliDriver cbo_rp_auto_join1

2017-02-21 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15946?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li resolved HIVE-15946. --- Resolution: Resolved Fix Version/s: 2.2.0 Fixed in HIVE-15948 > Failing test : TestCliDriver

[jira] [Commented] (HIVE-15859) Hive client side shows Spark Driver disconnected while Spark Driver side could not get RPC header

2017-02-22 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15878013#comment-15878013 ] Rui Li commented on HIVE-15859: --- Another way to fix (and also mentioned in the Livy PR) is to combine the

[jira] [Commented] (HIVE-15893) Followup on HIVE-15671

2017-02-20 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15893?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15874676#comment-15874676 ] Rui Li commented on HIVE-15893: --- Hi [~xuefuz], our RPC channel has a handler that monitors the channel

[jira] [Commented] (HIVE-15948) Failing test: TestCliDriver, TestSparkCliDriver join31

2017-02-20 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15875267#comment-15875267 ] Rui Li commented on HIVE-15948: --- Thanks for the patch [~stakiar]. I noticed there're some other places that

[jira] [Commented] (HIVE-15948) Failing test: TestCliDriver, TestSparkCliDriver join31

2017-02-20 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15875334#comment-15875334 ] Rui Li commented on HIVE-15948: --- [~stakiar] thanks for the update. +1 pending test > Failing test:

[jira] [Updated] (HIVE-15710) HS2 Stopped when running in background

2017-02-16 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15710?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-15710: -- Resolution: Fixed Fix Version/s: 2.2.0 Status: Resolved (was: Patch Available) Committed to

[jira] [Commented] (HIVE-15710) HS2 Stopped when running in background

2017-02-16 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15871059#comment-15871059 ] Rui Li commented on HIVE-15710: --- The age1 failure can't be reproduced locally. I'll commit shortly if no

[jira] [Commented] (HIVE-15882) HS2 generating high memory pressure with many partitions and concurrent queries

2017-02-23 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15881733#comment-15881733 ] Rui Li commented on HIVE-15882: --- Thanks [~mi...@cloudera.com] for the benchmark results. It looks promising.

[jira] [Commented] (HIVE-15882) HS2 generating high memory pressure with many partitions and concurrent queries

2017-02-23 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15882157#comment-15882157 ] Rui Li commented on HIVE-15882: --- The patch looks good to me overall. I left some minor comments on the RB.

[jira] [Commented] (HIVE-15859) Hive client side shows Spark Driver disconnected while Spark Driver side could not get RPC header

2017-02-23 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15880098#comment-15880098 ] Rui Li commented on HIVE-15859: --- I think I have managed to reproduce the issue by introducing some sleep

[jira] [Commented] (HIVE-15859) Hive client side shows Spark Driver disconnected while Spark Driver side could not get RPC header

2017-02-22 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15879995#comment-15879995 ] Rui Li commented on HIVE-15859: --- Hi [~xuefuz], netty's channel is thread safe. We can write to it

[jira] [Commented] (HIVE-15882) HS2 generating high memory pressure with many partitions and concurrent queries

2017-02-22 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15879917#comment-15879917 ] Rui Li commented on HIVE-15882: --- Hi [~mi...@cloudera.com], I guess one possible issue of String::intern is

[jira] [Commented] (HIVE-15893) Followup on HIVE-15671

2017-02-13 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15893?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15864892#comment-15864892 ] Rui Li commented on HIVE-15893: --- My hunch is HIVE-15860 doesn't help here. It's only intended for RSC

[jira] [Updated] (HIVE-15859) Hive client side shows Spark Driver disconnected while Spark Driver side could not get RPC header

2017-02-13 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15859?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-15859: -- Attachment: HIVE-15859.1.patch Patch v1 based on the Livy PR. [~KaiXu], could you test if the patch fixes your

[jira] [Updated] (HIVE-15859) Hive client side shows Spark Driver disconnected while Spark Driver side could not get RPC header

2017-02-13 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15859?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-15859: -- Status: Patch Available (was: Open) > Hive client side shows Spark Driver disconnected while Spark Driver side

[jira] [Commented] (HIVE-15859) Hive client side shows Spark Driver disconnected while Spark Driver side could not get RPC header

2017-02-13 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15865239#comment-15865239 ] Rui Li commented on HIVE-15859: --- Hi [~vanzin], thanks for providing the Livy PR. That's exactly the same

[jira] [Comment Edited] (HIVE-15859) Hive client side shows Spark Driver disconnected while Spark Driver side could not get RPC header

2017-02-13 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15865239#comment-15865239 ] Rui Li edited comment on HIVE-15859 at 2/14/17 7:11 AM: Hi [~vanzin], thanks for

[jira] [Commented] (HIVE-15710) HS2 Stopped when running in background

2017-02-14 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15865389#comment-15865389 ] Rui Li commented on HIVE-15710: --- Ping [~xuefuz], [~mohitsabharwal] again. > HS2 Stopped when running in

[jira] [Assigned] (HIVE-15859) Hive client side shows Spark Driver disconnected while Spark Driver side could not get RPC header

2017-02-13 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15859?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li reassigned HIVE-15859: - Assignee: Rui Li > Hive client side shows Spark Driver disconnected while Spark Driver side > could not

[jira] [Commented] (HIVE-15671) RPCServer.registerClient() erroneously uses server/client handshake timeout for connection timeout

2017-02-09 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15859267#comment-15859267 ] Rui Li commented on HIVE-15671: --- Hi [~xuefuz], can you get a thread dump of HS2/CLI when you hit the 2nd

[jira] [Commented] (HIVE-15887) could not get APP ID and cause failed to connect to spark driver on yarn-client mode

2017-02-14 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15865384#comment-15865384 ] Rui Li commented on HIVE-15887: --- In the AM log, the "failure to connect to driver" happened around 5:05.

[jira] [Commented] (HIVE-15912) Executor kill task and Failed to get spark memory/core info

2017-02-14 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15912?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15867198#comment-15867198 ] Rui Li commented on HIVE-15912: --- The failure to get mem/core info is just a warning. So it's not related to

[jira] [Commented] (HIVE-15887) could not get APP ID and cause failed to connect to spark driver on yarn-client mode

2017-02-14 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15867150#comment-15867150 ] Rui Li commented on HIVE-15887: --- The symptom is quite similar to that of HIVE-12650. Is this a busy cluster?

[jira] [Updated] (HIVE-15710) HS2 Stopped when running in background

2017-02-15 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15710?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-15710: -- Attachment: HIVE-15710.2.patch Update patch v2 based on Ferdinand's comments. [~Ferd] please let me know if it

[jira] [Updated] (HIVE-15710) HS2 Stopped when running in background

2017-02-14 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15710?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-15710: -- Status: Patch Available (was: Open) > HS2 Stopped when running in background >

[jira] [Commented] (HIVE-15710) HS2 Stopped when running in background

2017-02-14 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15867372#comment-15867372 ] Rui Li commented on HIVE-15710: --- Thanks [~Ferd] for your suggestions. I guess the redundancy is not a big

[jira] [Updated] (HIVE-15859) Hive client side shows Spark Driver disconnected while Spark Driver side could not get RPC header

2017-02-15 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15859?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-15859: -- Attachment: HIVE-15859.2.patch Fix test. > Hive client side shows Spark Driver disconnected while Spark Driver

[jira] [Updated] (HIVE-15710) HS2 Stopped when running in background

2017-02-14 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15710?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-15710: -- Attachment: HIVE-15710.1.patch Move the check to bin/hive in patch v1. Also pinging [~Ferd] for reivew. > HS2

[jira] [Commented] (HIVE-15710) HS2 Stopped when running in background

2017-02-15 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15867495#comment-15867495 ] Rui Li commented on HIVE-15710: --- Sorry I meant to start HS2, user can either call {{bin/hiveserver2}}, or

[jira] [Updated] (HIVE-15860) RemoteSparkJobMonitor may hang when RemoteDriver exits abnormally

2017-02-12 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-15860: -- Attachment: HIVE-15860.2.patch > RemoteSparkJobMonitor may hang when RemoteDriver exits abnormally >

[jira] [Commented] (HIVE-15860) RemoteSparkJobMonitor may hang when RemoteDriver exits abnormally

2017-02-13 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15863228#comment-15863228 ] Rui Li commented on HIVE-15860: --- Latest failure not related. > RemoteSparkJobMonitor may hang when

[jira] [Updated] (HIVE-15887) could not get APP ID and cause failed to connect to spark driver on yarn-client mode

2017-02-13 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15887?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-15887: -- Description: {noformat} 2017-02-13T03:10:01,639 INFO [stderr-redir-1] client.SparkClientImpl: 17/02/13

[jira] [Updated] (HIVE-15860) RemoteSparkJobMonitor may hang when RemoteDriver exits abnormally

2017-02-13 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-15860: -- Resolution: Fixed Fix Version/s: 2.2.0 Status: Resolved (was: Patch Available) Committed to

[jira] [Commented] (HIVE-15887) could not get APP ID and cause failed to connect to spark driver on yarn-client mode

2017-02-13 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15863160#comment-15863160 ] Rui Li commented on HIVE-15887: --- Hi [~KaiXu], the failure is just a warning and it shouldn't make the query

[jira] [Updated] (HIVE-15860) RemoteSparkJobMonitor may hang when RemoteDriver exits abnormally

2017-02-10 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-15860: -- Status: Patch Available (was: Open) > RemoteSparkJobMonitor may hang when RemoteDriver exits abnormally >

[jira] [Updated] (HIVE-15860) RemoteSparkJobMonitor may hang when RemoteDriver exits abnormally

2017-02-10 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-15860: -- Attachment: HIVE-15860.1.patch Actually Rpc's listener can detect the remote end becomes inactive and mark

[jira] [Updated] (HIVE-15860) RemoteSparkJobMonitor may hang when RemoteDriver exits abnormally

2017-02-09 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-15860: -- Summary: RemoteSparkJobMonitor may hang when RemoteDriver exits abnormally (was: RemoteSparkJobMonitor may

[jira] [Assigned] (HIVE-15860) RemoteSparkJobMonitor may hang when RemoteDriver exists abnormally

2017-02-09 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li reassigned HIVE-15860: - > RemoteSparkJobMonitor may hang when RemoteDriver exists abnormally >

[jira] [Updated] (HIVE-15860) RemoteSparkJobMonitor may hang when RemoteDriver exits abnormally

2017-02-09 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-15860: -- Description: It happens when RemoteDriver crashes between {{JobStarted}} and {{JobSubmitted}}, e.g. killed by

[jira] [Commented] (HIVE-15860) RemoteSparkJobMonitor may hang when RemoteDriver exits abnormally

2017-02-09 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15859410#comment-15859410 ] Rui Li commented on HIVE-15860: --- We cannot simply extend the monitor timeout to cover {{JobSubmitted}},

[jira] [Commented] (HIVE-15860) RemoteSparkJobMonitor may hang when RemoteDriver exits abnormally

2017-02-10 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15861036#comment-15861036 ] Rui Li commented on HIVE-15860: --- A more specific way to fix it is just add the check when job has started

[jira] [Commented] (HIVE-15860) RemoteSparkJobMonitor may hang when RemoteDriver exits abnormally

2017-02-10 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15861020#comment-15861020 ] Rui Li commented on HIVE-15860: --- [~xuefuz] - yeah the monitor loops forever in that case. For the monitor,

[jira] [Updated] (HIVE-15860) RemoteSparkJobMonitor may hang when RemoteDriver exits abnormally

2017-02-10 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-15860: -- Attachment: HIVE-15860.2.patch Patch v2 adds the check only when job is in STARTED state and we can't get the

[jira] [Updated] (HIVE-15859) Hive client side shows Spark Driver disconnected while Spark Driver side could not get RPC header

2017-02-13 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15859?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-15859: -- Description: Hive on Spark, failed with error: {noformat} 2017-02-08 09:50:59,331 Stage-2_0: 1039(+2)/1041

[jira] [Commented] (HIVE-15859) Hive client side shows Spark Driver disconnected while Spark Driver side could not get RPC header

2017-02-13 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15863622#comment-15863622 ] Rui Li commented on HIVE-15859: --- I noticed writing to the channel is protected by channelLock in

[jira] [Commented] (HIVE-15887) could not get APP ID and cause failed to connect to spark driver on yarn-client mode

2017-02-13 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15863645#comment-15863645 ] Rui Li commented on HIVE-15887: --- Seems the application is ACCEPTED but didn't reach RUNNING. YARN log may be

[jira] [Commented] (HIVE-15882) HS2 generating high memory pressure with many partitions and concurrent queries

2017-02-17 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15871561#comment-15871561 ] Rui Li commented on HIVE-15882: --- I also noticed lots of duplicated Strings in {{ListSinkOperator::res}} when

[jira] [Updated] (HIVE-15519) BitSet not computed properly for ColumnBuffer subset

2017-01-18 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-15519: -- Attachment: HIVE-15519.5-branch-1.patch > BitSet not computed properly for ColumnBuffer subset >

[jira] [Updated] (HIVE-15519) BitSet not computed properly for ColumnBuffer subset

2017-01-18 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-15519: -- Attachment: (was: HIVE-15519.5-branch-1.patch) > BitSet not computed properly for ColumnBuffer subset >

[jira] [Commented] (HIVE-15519) BitSet not computed properly for ColumnBuffer subset

2017-01-18 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15829151#comment-15829151 ] Rui Li commented on HIVE-15519: --- Hi [~thejas], could you have a look at the latest patches? FYI v6 is for

[jira] [Updated] (HIVE-15519) BitSet not computed properly for ColumnBuffer subset

2017-01-17 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-15519: -- Attachment: (was: HIVE-15519.6.patch) > BitSet not computed properly for ColumnBuffer subset >

[jira] [Updated] (HIVE-15519) BitSet not computed properly for ColumnBuffer subset

2017-01-17 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-15519: -- Attachment: HIVE-15519.6.patch > BitSet not computed properly for ColumnBuffer subset >

[jira] [Commented] (HIVE-15519) BitSet not computed properly for ColumnBuffer subset

2017-01-18 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15828082#comment-15828082 ] Rui Li commented on HIVE-15519: --- The last test is for v6 patch. The patch is also in RB. Test failures with

[jira] [Commented] (HIVE-15580) Replace Spark's groupByKey operator with something with bounded memory

2017-01-18 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15828143#comment-15828143 ] Rui Li commented on HIVE-15580: --- Hi [~xuefuz], I'd like to check my understanding too. Before the patch, we

[jira] [Commented] (HIVE-15580) Replace Spark's groupByKey operator with something with bounded memory

2017-01-18 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15828249#comment-15828249 ] Rui Li commented on HIVE-15580: --- Hmm Spark does make it clear groupByKey can't handle large keys:

[jira] [Commented] (HIVE-15580) Replace Spark's groupByKey operator with something with bounded memory

2017-01-18 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15828183#comment-15828183 ] Rui Li commented on HIVE-15580: --- [~xuefuz], thanks for your explanations. It makes sense. So in general, the

[jira] [Commented] (HIVE-15519) BitSet not computed properly for ColumnBuffer subset

2017-01-20 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15831399#comment-15831399 ] Rui Li commented on HIVE-15519: --- The last test is for v5 patch for branch-1. I tried some of the failures

[jira] [Commented] (HIVE-15671) RPCServer.registerClient() erroneously uses server/client handshake timeout for connection timeout

2017-01-20 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15831452#comment-15831452 ] Rui Li commented on HIVE-15671: --- Hi [~xuefuz], I tried your patch locally.

[jira] [Commented] (HIVE-14554) Hive ptest should delete the itests/thirdparty directory everytime it builds hive

2016-08-18 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-14554?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15427464#comment-15427464 ] Rui Li commented on HIVE-14554: --- The checksum way sounds good to me. > Hive ptest should delete the

[jira] [Commented] (HIVE-14554) Hive ptest should delete the itests/thirdparty directory everytime it builds hive

2016-08-18 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-14554?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15427348#comment-15427348 ] Rui Li commented on HIVE-14554: --- Oh sorry I misunderstood it. Just thought more about this, maybe it's

[jira] [Updated] (HIVE-14595) TimestampWritable::setTimestamp gives wrong result when 2nd VInt exists

2016-08-21 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-14595?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-14595: -- Status: Patch Available (was: Open) > TimestampWritable::setTimestamp gives wrong result when 2nd VInt exists

[jira] [Updated] (HIVE-14595) TimestampWritable::setTimestamp gives wrong result when 2nd VInt exists

2016-08-21 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-14595?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-14595: -- Attachment: HIVE-14595.1.patch > TimestampWritable::setTimestamp gives wrong result when 2nd VInt exists >

[jira] [Commented] (HIVE-14595) TimestampWritable::setTimestamp gives wrong result when 2nd VInt exists

2016-08-21 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-14595?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15430063#comment-15430063 ] Rui Li commented on HIVE-14595: --- Failures are not related. [~xuefuz], could you take a look at this? It's

[jira] [Updated] (HIVE-14595) TimestampWritable::setTimestamp gives wrong result when 2nd VInt exists

2016-08-22 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-14595?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-14595: -- Resolution: Fixed Status: Resolved (was: Patch Available) Committed v2 to master. Thanks Xuefu for the

[jira] [Updated] (HIVE-14595) TimestampWritable::setTimestamp gives wrong result when 2nd VInt exists

2016-08-22 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-14595?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-14595: -- Status: Patch Available (was: Reopened) > TimestampWritable::setTimestamp gives wrong result when 2nd VInt

[jira] [Reopened] (HIVE-14595) TimestampWritable::setTimestamp gives wrong result when 2nd VInt exists

2016-08-22 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-14595?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li reopened HIVE-14595: --- > TimestampWritable::setTimestamp gives wrong result when 2nd VInt exists >

[jira] [Updated] (HIVE-14595) TimestampWritable::setTimestamp gives wrong result when 2nd VInt exists

2016-08-22 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-14595?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-14595: -- Attachment: HIVE-14595.2.patch > TimestampWritable::setTimestamp gives wrong result when 2nd VInt exists >

[jira] [Commented] (HIVE-14595) TimestampWritable::setTimestamp gives wrong result when 2nd VInt exists

2016-08-22 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-14595?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15430600#comment-15430600 ] Rui Li commented on HIVE-14595: --- I'm sorry but I missed a safety check in v1 patch. Attach v2 to amend this.

[jira] [Updated] (HIVE-14595) TimestampWritable::setTimestamp gives wrong result when 2nd VInt exists

2016-08-22 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-14595?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-14595: -- Resolution: Fixed Fix Version/s: 2.2.0 Status: Resolved (was: Patch Available) Thanks Xuefu

[jira] [Commented] (HIVE-14412) Add a timezone-aware timestamp

2016-08-22 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-14412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15430384#comment-15430384 ] Rui Li commented on HIVE-14412: --- I think make HiveTimestamp a subclass of Timestamp can be problematic. E.g.

[jira] [Updated] (HIVE-16047) Shouldn't try to get KeyProvider unless encryption is enabled

2017-02-27 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-16047?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-16047: -- Attachment: HIVE-16047.2.patch Thanks guys for your suggestions. Patch v2 uses reflection to check whether the

[jira] [Commented] (HIVE-15882) HS2 generating high memory pressure with many partitions and concurrent queries

2017-03-01 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15891439#comment-15891439 ] Rui Li commented on HIVE-15882: --- I'll commit shortly if there's no further comments. > HS2 generating high

[jira] [Comment Edited] (HIVE-16071) Spark remote driver misuses the timeout in RPC handshake

2017-03-01 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-16071?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15891686#comment-15891686 ] Rui Li edited comment on HIVE-16071 at 3/2/17 6:50 AM: --- Hi [~ctang.ma], for the

[jira] [Updated] (HIVE-15882) HS2 generating high memory pressure with many partitions and concurrent queries

2017-03-01 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15882?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-15882: -- Resolution: Fixed Fix Version/s: 2.2.0 Status: Resolved (was: Patch Available) Committed to

[jira] [Commented] (HIVE-16071) Spark remote driver misuses the timeout in RPC handshake

2017-03-01 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-16071?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15891686#comment-15891686 ] Rui Li commented on HIVE-16071: --- Hi [~ctang.ma], for the issue mentioned in this JIRA, I think you can try

[jira] [Updated] (HIVE-15859) HoS: Write RPC messages in event loop

2017-02-26 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15859?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-15859: -- Summary: HoS: Write RPC messages in event loop (was: Hive client side shows Spark Driver disconnected while

[jira] [Updated] (HIVE-15859) HoS: Write RPC messages in event loop

2017-02-26 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15859?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-15859: -- Resolution: Fixed Fix Version/s: 2.2.0 Status: Resolved (was: Patch Available) Committed to

[jira] [Updated] (HIVE-15859) HoS: Write RPC messages in event loop

2017-02-26 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15859?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-15859: -- Affects Version/s: (was: 2.2.0) 2.1.1 > HoS: Write RPC messages in event loop >

[jira] [Updated] (HIVE-15859) Hive client side shows Spark Driver disconnected while Spark Driver side could not get RPC header

2017-02-26 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15859?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-15859: -- Attachment: HIVE-15859.3.patch Thanks [~KaiXu] for the clarifications. Update patch v3 to make sure we log the

[jira] [Commented] (HIVE-16071) Spark remote driver misuses the timeout in RPC handshake

2017-02-28 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-16071?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15889474#comment-15889474 ] Rui Li commented on HIVE-16071: --- [~ctang.ma], you might want to take a look at HIVE-15671. I'll also have

[jira] [Assigned] (HIVE-16047) Shouldn't try to get KeyProvider unless encryption is enabled

2017-02-27 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-16047?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li reassigned HIVE-16047: - > Shouldn't try to get KeyProvider unless encryption is enabled >

[jira] [Updated] (HIVE-16047) Shouldn't try to get KeyProvider unless encryption is enabled

2017-02-27 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-16047?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-16047: -- Attachment: HIVE-16047.1.patch The change is trivial. But I'm not sure if there's compatibility concern,

[jira] [Commented] (HIVE-16046) Broadcasting small table for Hive on Spark

2017-02-27 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-16046?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15885458#comment-15885458 ] Rui Li commented on HIVE-16046: --- Details why we didn't choose broadcast for map join can be found in

[jira] [Updated] (HIVE-16047) Shouldn't try to get KeyProvider unless encryption is enabled

2017-02-27 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-16047?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-16047: -- Status: Patch Available (was: Open) > Shouldn't try to get KeyProvider unless encryption is enabled >

[jira] [Updated] (HIVE-16047) Shouldn't try to get KeyProvider unless encryption is enabled

2017-02-28 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-16047?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-16047: -- Resolution: Fixed Fix Version/s: 2.2.0 Status: Resolved (was: Patch Available) Committed to

[jira] [Commented] (HIVE-16071) Spark remote driver misuses the timeout in RPC handshake

2017-02-28 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-16071?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15889695#comment-15889695 ] Rui Li commented on HIVE-16071: --- My understanding is the timeout on SparkClient side is longer because it

[jira] [Commented] (HIVE-16047) Shouldn't try to get KeyProvider unless encryption is enabled

2017-02-28 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-16047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15889415#comment-15889415 ] Rui Li commented on HIVE-16047: --- Failure with age 1 can't be reproduced. > Shouldn't try to get KeyProvider

[jira] [Commented] (HIVE-14686) Get unexpected command type when execute query "CREATE TABLE IF NOT EXISTS ... AS"

2016-09-05 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-14686?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15466173#comment-15466173 ] Rui Li commented on HIVE-14686: --- Thanks [~fanyunbojerry] for the update. Seems some test failures are

[jira] [Updated] (HIVE-14412) Add a timezone-aware timestamp

2016-09-06 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-14412?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-14412: -- Attachment: (was: HIVE-14412.1.patch) > Add a timezone-aware timestamp > -- > >

[jira] [Updated] (HIVE-14412) Add a timezone-aware timestamp

2016-09-06 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-14412?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-14412: -- Attachment: (was: HIVE-14412.1.patch) > Add a timezone-aware timestamp > -- > >

[jira] [Updated] (HIVE-14412) Add a timezone-aware timestamp

2016-09-06 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-14412?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-14412: -- Attachment: HIVE-14412.2.patch I chose to make the tz-aware timestamp a new type in v2 patch, named

[jira] [Updated] (HIVE-14412) Add a timezone-aware timestamp

2016-09-06 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-14412?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-14412: -- Attachment: HIVE-14412.3.patch > Add a timezone-aware timestamp > -- > >

[jira] [Commented] (HIVE-14686) Get unexpected command type when execute query "CREATE TABLE IF NOT EXISTS ... AS"

2016-09-01 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-14686?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15457213#comment-15457213 ] Rui Li commented on HIVE-14686: --- I think the patch doesn't apply to latest code. [~fanyunbojerry], could you

[jira] [Updated] (HIVE-14719) ASTNode rootNode is not maintained properly when changing child/parent relation

2016-09-08 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-14719?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-14719: -- Attachment: HIVE-14719.1.patch > ASTNode rootNode is not maintained properly when changing child/parent >

[jira] [Commented] (HIVE-14719) ASTNode rootNode is not maintained properly when changing child/parent relation

2016-09-08 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-14719?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15474031#comment-15474031 ] Rui Li commented on HIVE-14719: --- In {{ASTNode::setChild}}, if I clear the {{rootNode}} field from the

[jira] [Updated] (HIVE-14719) ASTNode rootNode is not maintained properly when changing child/parent relation

2016-09-08 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-14719?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-14719: -- Status: Patch Available (was: Open) > ASTNode rootNode is not maintained properly when changing child/parent

[jira] [Commented] (HIVE-14412) Add a timezone-aware timestamp

2016-09-11 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-14412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15482896#comment-15482896 ] Rui Li commented on HIVE-14412: --- The failure don't seem related. [~xuefuz] could you take a look? I've put

[jira] [Commented] (HIVE-14719) ASTNode rootNode is not maintained properly when changing child/parent relation

2016-09-10 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-14719?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15479295#comment-15479295 ] Rui Li commented on HIVE-14719: --- Latest failures are not related. > ASTNode rootNode is not maintained

<    1   2   3   4   5   6   7   8   9   10   >