[jira] [Comment Edited] (HIVE-15682) Eliminate the dummy iterator and optimize the per row based reducer-side processing

2017-02-06 Thread Xuefu Zhang (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15855133#comment-15855133 ] Xuefu Zhang edited comment on HIVE-15682 at 2/7/17 1:45 AM: I

[jira] [Commented] (HIVE-15682) Eliminate the dummy iterator and optimize the per row based reducer-side processing

2017-02-06 Thread Xuefu Zhang (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15855133#comment-15855133 ] Xuefu Zhang commented on HIVE-15682: I did some performance measurement query order by

[jira] [Commented] (HIVE-15682) Eliminate the dummy iterator and optimize the per row based reducer-side processing

2017-02-06 Thread Xuefu Zhang (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15855125#comment-15855125 ] Xuefu Zhang commented on HIVE-15682: I took a look at the code block again, and it doe

[jira] [Commented] (HIVE-15815) Allow to pass some Oozie properties to Spark in HoS

2017-02-04 Thread Xuefu Zhang (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15853125#comment-15853125 ] Xuefu Zhang commented on HIVE-15815: +1. I assume that spark.hadoop.oozie properties w

[jira] [Updated] (HIVE-15749) Add missing ASF headers

2017-02-01 Thread Xuefu Zhang (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15749?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuefu Zhang updated HIVE-15749: --- Resolution: Fixed Fix Version/s: 2.2.0 Status: Resolved (was: Patch Available) Comm

[jira] [Commented] (HIVE-15749) Add missing ASF headers

2017-02-01 Thread Xuefu Zhang (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15848196#comment-15848196 ] Xuefu Zhang commented on HIVE-15749: +1 > Add missing ASF headers > -

[jira] (HIVE-15485) Investigate the DoAs failure in HoS

2017-01-29 Thread Xuefu Zhang (JIRA)
Title: Message Title Xuefu Zhang commented on HIVE-15485

[jira] [Commented] (HIVE-15485) Investigate the DoAs failure in HoS

2017-01-28 Thread Xuefu Zhang (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15844215#comment-15844215 ] Xuefu Zhang commented on HIVE-15485: Sorry for my late reply. (I'm currently OOO.) The

[jira] [Commented] (HIVE-15580) Eliminate unbounded memory usage for orderBy and groupBy in Hive on Spark

2017-01-24 Thread Xuefu Zhang (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15836414#comment-15836414 ] Xuefu Zhang commented on HIVE-15580: Hi [~Ferd] and [~dapengsun], I'm wondering if you

[jira] [Commented] (HIVE-15671) RPCServer.registerClient() erroneously uses server/client handshake timeout for connection timeout

2017-01-20 Thread Xuefu Zhang (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15832749#comment-15832749 ] Xuefu Zhang commented on HIVE-15671: [~vanzin], could you please review the patch? Tha

[jira] [Comment Edited] (HIVE-15671) RPCServer.registerClient() erroneously uses server/client handshake timeout for connection timeout

2017-01-20 Thread Xuefu Zhang (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15832582#comment-15832582 ] Xuefu Zhang edited comment on HIVE-15671 at 1/20/17 11:12 PM: --

[jira] [Commented] (HIVE-15671) RPCServer.registerClient() erroneously uses server/client handshake timeout for connection timeout

2017-01-20 Thread Xuefu Zhang (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15832582#comment-15832582 ] Xuefu Zhang commented on HIVE-15671: Patch #1 followed what [~vanzin] suggested. With

[jira] [Updated] (HIVE-15671) RPCServer.registerClient() erroneously uses server/client handshake timeout for connection timeout

2017-01-20 Thread Xuefu Zhang (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15671?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuefu Zhang updated HIVE-15671: --- Attachment: HIVE-15671.1.patch > RPCServer.registerClient() erroneously uses server/client handshake t

[jira] [Resolved] (HIVE-15527) Memory usage is unbound in SortByShuffler for Spark

2017-01-20 Thread Xuefu Zhang (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15527?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuefu Zhang resolved HIVE-15527. Resolution: Not A Problem > Memory usage is unbound in SortByShuffler for Spark > --

[jira] [Updated] (HIVE-15527) Memory usage is unbound in SortByShuffler for Spark

2017-01-20 Thread Xuefu Zhang (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15527?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuefu Zhang updated HIVE-15527: --- Status: Open (was: Patch Available) The issue was addressed in HIVE-15580. Cancel the patch here and

[jira] [Updated] (HIVE-15580) Eliminate unbounded memory usage for orderBy and groupBy in Hive on Spark

2017-01-20 Thread Xuefu Zhang (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15580?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuefu Zhang updated HIVE-15580: --- Resolution: Fixed Fix Version/s: 2.2.0 Status: Resolved (was: Patch Available) Comm

[jira] [Commented] (HIVE-15580) Eliminate unbounded memory usage for orderBy and groupBy in Hive on Spark

2017-01-20 Thread Xuefu Zhang (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15832383#comment-15832383 ] Xuefu Zhang commented on HIVE-15580: Thanks, Chao! I will commit this first and create

[jira] [Commented] (HIVE-15671) RPCServer.registerClient() erroneously uses server/client handshake timeout for connection timeout

2017-01-20 Thread Xuefu Zhang (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15832245#comment-15832245 ] Xuefu Zhang commented on HIVE-15671: [~vanzin], thanks for your insight. I think we ar

[jira] [Commented] (HIVE-15580) Eliminate unbounded memory usage for orderBy and groupBy in Hive on Spark

2017-01-20 Thread Xuefu Zhang (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15832194#comment-15832194 ] Xuefu Zhang commented on HIVE-15580: RB: https://reviews.apache.org/r/55776/ > Elimin

[jira] [Commented] (HIVE-15678) Hive On Spark queries fail when HBase is configured and down even when the query doesn't rely on HBase

2017-01-20 Thread Xuefu Zhang (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15831733#comment-15831733 ] Xuefu Zhang commented on HIVE-15678: Hive on Spark doesn't attempt to make any HBase c

[jira] [Comment Edited] (HIVE-15671) RPCServer.registerClient() erroneously uses server/client handshake timeout for connection timeout

2017-01-19 Thread Xuefu Zhang (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15831178#comment-15831178 ] Xuefu Zhang edited comment on HIVE-15671 at 1/20/17 4:33 AM: -

[jira] [Commented] (HIVE-15671) RPCServer.registerClient() erroneously uses server/client handshake timeout for connection timeout

2017-01-19 Thread Xuefu Zhang (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15831178#comment-15831178 ] Xuefu Zhang commented on HIVE-15671: Actually my understanding is a little different.

[jira] [Commented] (HIVE-15671) RPCServer.registerClient() erroneously uses server/client handshake timeout for connection timeout

2017-01-19 Thread Xuefu Zhang (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15831131#comment-15831131 ] Xuefu Zhang commented on HIVE-15671: Thanks, [~vanzin]. To confirm, when you say "the

[jira] [Updated] (HIVE-15671) RPCServer.registerClient() erroneously uses server/client handshake timeout for connection timeout

2017-01-19 Thread Xuefu Zhang (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15671?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuefu Zhang updated HIVE-15671: --- Description: {code} /** * Tells the RPC server to expect a connection from a new client. * ...

[jira] [Commented] (HIVE-15671) RPCServer.registerClient() erroneously uses server/client handshake timeout for connection timeout

2017-01-19 Thread Xuefu Zhang (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15831072#comment-15831072 ] Xuefu Zhang commented on HIVE-15671: [~vanzin]/[~lirui], could you please review? cc:

[jira] [Updated] (HIVE-15671) RPCServer.registerClient() erroneously uses server/client handshake timeout for connection timeout

2017-01-19 Thread Xuefu Zhang (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15671?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuefu Zhang updated HIVE-15671: --- Status: Patch Available (was: Open) > RPCServer.registerClient() erroneously uses server/client hands

[jira] [Updated] (HIVE-15671) RPCServer.registerClient() erroneously uses server/client handshake timeout for connection timeout

2017-01-19 Thread Xuefu Zhang (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15671?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuefu Zhang updated HIVE-15671: --- Attachment: HIVE-15671.patch > RPCServer.registerClient() erroneously uses server/client handshake tim

[jira] [Updated] (HIVE-15580) Eliminate unbounded memory usage for orderBy and groupBy in Hive on Spark

2017-01-19 Thread Xuefu Zhang (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15580?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuefu Zhang updated HIVE-15580: --- Description: Currently, orderBy (sortBy) and groupBy in Hive on Spark uses unbounded memory. For orde

[jira] [Updated] (HIVE-15580) Eliminate unbounded memory usage for orderBy and groupBy in Hive on Spark

2017-01-19 Thread Xuefu Zhang (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15580?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuefu Zhang updated HIVE-15580: --- Description: Currently, orderBy (sortBy) and groupBy in Hive on Spark uses unbounded memory. For order

[jira] [Updated] (HIVE-15580) Eliminate unbounded memory usage for orderBy and groupBy in Hive on Spark

2017-01-19 Thread Xuefu Zhang (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15580?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuefu Zhang updated HIVE-15580: --- Summary: Eliminate unbounded memory usage for orderBy and groupBy in Hive on Spark (was: Replace Spar

[jira] [Commented] (HIVE-9410) ClassNotFoundException occurs during hive query case execution with UDF defined [Spark Branch]

2017-01-19 Thread Xuefu Zhang (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-9410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15830306#comment-15830306 ] Xuefu Zhang commented on HIVE-9410: --- I think what [~vanzin] suggested above is incorporat

[jira] [Updated] (HIVE-15580) Replace Spark's groupByKey operator with something with bounded memory

2017-01-19 Thread Xuefu Zhang (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15580?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuefu Zhang updated HIVE-15580: --- Attachment: HIVE-15580.5.patch > Replace Spark's groupByKey operator with something with bounded memor

[jira] [Commented] (HIVE-15659) StackOverflowError when ClassLoader.loadClass for Spark

2017-01-19 Thread Xuefu Zhang (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15829905#comment-15829905 ] Xuefu Zhang commented on HIVE-15659: [~csun], do you know whether the SOFE exception h

[jira] [Updated] (HIVE-15580) Replace Spark's groupByKey operator with something with bounded memory

2017-01-18 Thread Xuefu Zhang (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15580?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuefu Zhang updated HIVE-15580: --- Attachment: HIVE-15580.4.patch > Replace Spark's groupByKey operator with something with bounded memor

[jira] [Commented] (HIVE-15297) Hive should not split semicolon within quoted string literals

2017-01-18 Thread Xuefu Zhang (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15297?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15829364#comment-15829364 ] Xuefu Zhang commented on HIVE-15297: I noticed that a couple of trailing space/tabs ar

[jira] [Commented] (HIVE-15580) Replace Spark's groupByKey operator with something with bounded memory

2017-01-18 Thread Xuefu Zhang (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15829203#comment-15829203 ] Xuefu Zhang commented on HIVE-15580: [~dapengsun], for the OOM error you get, you can

[jira] [Updated] (HIVE-15580) Replace Spark's groupByKey operator with something with bounded memory

2017-01-18 Thread Xuefu Zhang (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15580?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuefu Zhang updated HIVE-15580: --- Attachment: HIVE-15580.3.patch > Replace Spark's groupByKey operator with something with bounded memor

[jira] [Updated] (HIVE-15580) Replace Spark's groupByKey operator with something with bounded memory

2017-01-18 Thread Xuefu Zhang (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15580?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuefu Zhang updated HIVE-15580: --- Attachment: HIVE-15580.2.patch > Replace Spark's groupByKey operator with something with bounded memor

[jira] [Comment Edited] (HIVE-15580) Replace Spark's groupByKey operator with something with bounded memory

2017-01-18 Thread Xuefu Zhang (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15828155#comment-15828155 ] Xuefu Zhang edited comment on HIVE-15580 at 1/18/17 2:24 PM: -

[jira] [Commented] (HIVE-15580) Replace Spark's groupByKey operator with something with bounded memory

2017-01-18 Thread Xuefu Zhang (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15828155#comment-15828155 ] Xuefu Zhang commented on HIVE-15580: Hi [~lirui], your understanding is correct. And

[jira] [Commented] (HIVE-15580) Replace Spark's groupByKey operator with something with bounded memory

2017-01-17 Thread Xuefu Zhang (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15827470#comment-15827470 ] Xuefu Zhang commented on HIVE-15580: [~Ferd], Functionally, I don't see anything bad b

[jira] [Commented] (HIVE-15527) Memory usage is unbound in SortByShuffler for Spark

2017-01-17 Thread Xuefu Zhang (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15827378#comment-15827378 ] Xuefu Zhang commented on HIVE-15527: Hi [~Ferd], [~dapengsun], We found a better fix,

[jira] [Commented] (HIVE-15544) Support scalar subqueries

2017-01-16 Thread Xuefu Zhang (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15544?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15825491#comment-15825491 ] Xuefu Zhang commented on HIVE-15544: Re: Can you explain what do you mean by semantic

[jira] [Commented] (HIVE-15324) Enable round() function to accept scale argument as non-constants

2017-01-13 Thread Xuefu Zhang (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15822231#comment-15822231 ] Xuefu Zhang commented on HIVE-15324: Result type needs to determined statically. Scale

[jira] [Commented] (HIVE-15431) Round(1234567891.1234567891,50) returns null, result is not consistent with Mysql.

2017-01-13 Thread Xuefu Zhang (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1580#comment-1580 ] Xuefu Zhang commented on HIVE-15431: We cannot be completely be consistent with mysql.

[jira] [Updated] (HIVE-15580) Replace Spark's groupByKey operator with something with bounded memory

2017-01-13 Thread Xuefu Zhang (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15580?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuefu Zhang updated HIVE-15580: --- Attachment: HIVE-15580.2.patch > Replace Spark's groupByKey operator with something with bounded memor

[jira] [Commented] (HIVE-15544) Support scalar subqueries

2017-01-11 Thread Xuefu Zhang (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15544?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15819577#comment-15819577 ] Xuefu Zhang commented on HIVE-15544: Thanks for sharing your findings. However, I don

[jira] [Updated] (HIVE-15580) Replace Spark's groupByKey operator with something with bounded memory

2017-01-11 Thread Xuefu Zhang (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15580?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuefu Zhang updated HIVE-15580: --- Attachment: HIVE-15580.1.patch > Replace Spark's groupByKey operator with something with bounded memor

[jira] [Updated] (HIVE-15580) Replace Spark's groupByKey operator with something with bounded memory

2017-01-11 Thread Xuefu Zhang (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15580?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuefu Zhang updated HIVE-15580: --- Attachment: HIVE-15580.1.patch > Replace Spark's groupByKey operator with something with bounded memor

[jira] [Updated] (HIVE-15580) Replace Spark's groupByKey operator with something with bounded memory

2017-01-11 Thread Xuefu Zhang (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15580?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuefu Zhang updated HIVE-15580: --- Status: Patch Available (was: Open) > Replace Spark's groupByKey operator with something with bounded

[jira] [Updated] (HIVE-15580) Replace Spark's groupByKey operator with something with bounded memory

2017-01-10 Thread Xuefu Zhang (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15580?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuefu Zhang updated HIVE-15580: --- Attachment: HIVE-15580.patch > Replace Spark's groupByKey operator with something with bounded memory

[jira] [Updated] (HIVE-15527) Memory usage is unbound in SortByShuffler for Spark

2017-01-10 Thread Xuefu Zhang (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15527?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuefu Zhang updated HIVE-15527: --- Attachment: HIVE-15527.0.patch > Memory usage is unbound in SortByShuffler for Spark > ---

[jira] [Commented] (HIVE-15299) Yarn-cluster and yarn-client deprecated in Spark 2.0

2017-01-10 Thread Xuefu Zhang (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15815480#comment-15815480 ] Xuefu Zhang commented on HIVE-15299: +1 > Yarn-cluster and yarn-client deprecated in

[jira] [Updated] (HIVE-15527) Memory usage is unbound in SortByShuffler for Spark

2017-01-10 Thread Xuefu Zhang (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15527?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuefu Zhang updated HIVE-15527: --- Attachment: HIVE-15527.0.patch > Memory usage is unbound in SortByShuffler for Spark > ---

[jira] [Updated] (HIVE-15527) Memory usage is unbound in SortByShuffler for Spark

2017-01-10 Thread Xuefu Zhang (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15527?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuefu Zhang updated HIVE-15527: --- Attachment: (was: HIVE-15527.0.patch) > Memory usage is unbound in SortByShuffler for Spark >

[jira] [Updated] (HIVE-15527) Memory usage is unbound in SortByShuffler for Spark

2017-01-10 Thread Xuefu Zhang (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15527?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuefu Zhang updated HIVE-15527: --- Attachment: HIVE-15527.0.patch > Memory usage is unbound in SortByShuffler for Spark > ---

[jira] [Commented] (HIVE-15544) Support scalar subqueries

2017-01-05 Thread Xuefu Zhang (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15544?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15802285#comment-15802285 ] Xuefu Zhang commented on HIVE-15544: It's interesting to know how we know if a subquer

[jira] [Updated] (HIVE-15543) Don't try to get memory/cores to decide parallelism when Spark dynamic allocation is enabled

2017-01-05 Thread Xuefu Zhang (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15543?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuefu Zhang updated HIVE-15543: --- Resolution: Fixed Fix Version/s: 2.2.0 Status: Resolved (was: Patch Available) Comm

[jira] [Updated] (HIVE-15543) Don't try to get memory/cores to decide parallelism when Spark dynamic allocation is enabled

2017-01-04 Thread Xuefu Zhang (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15543?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuefu Zhang updated HIVE-15543: --- Attachment: HIVE-15543.patch cc: [~csun], [~ruili] > Don't try to get memory/cores to decide parallel

[jira] [Updated] (HIVE-15543) Don't try to get memory/cores to decide parallelism when Spark dynamic allocation is enabled

2017-01-04 Thread Xuefu Zhang (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15543?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuefu Zhang updated HIVE-15543: --- Status: Patch Available (was: Open) > Don't try to get memory/cores to decide parallelism when Spark

[jira] [Commented] (HIVE-15526) Some tests need SORT_QUERY_RESULTS

2017-01-03 Thread Xuefu Zhang (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15797135#comment-15797135 ] Xuefu Zhang commented on HIVE-15526: +1. Thanks for the explanation. > Some tests nee

[jira] [Updated] (HIVE-15528) Expose Spark job error in SparkTask

2017-01-03 Thread Xuefu Zhang (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuefu Zhang updated HIVE-15528: --- Resolution: Fixed Fix Version/s: 2.2.0 Status: Resolved (was: Patch Available) Comm

[jira] [Commented] (HIVE-15528) Expose Spark job error in SparkTask

2017-01-03 Thread Xuefu Zhang (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15795697#comment-15795697 ] Xuefu Zhang commented on HIVE-15528: +1 > Expose Spark job error in SparkTask > -

[jira] [Commented] (HIVE-15526) Some tests need SORT_QUERY_RESULTS

2017-01-03 Thread Xuefu Zhang (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15795195#comment-15795195 ] Xuefu Zhang commented on HIVE-15526: [~lirui], thanks for working on this. The patch s

[jira] [Commented] (HIVE-15527) Memory usage is unbound in SortByShuffler for Spark

2017-01-02 Thread Xuefu Zhang (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15794194#comment-15794194 ] Xuefu Zhang commented on HIVE-15527: [~csun], [~lirui], and [~kellyzly], thanks for yo

[jira] [Updated] (HIVE-15527) Memory usage is unbound in SortByShuffler for Spark

2017-01-01 Thread Xuefu Zhang (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15527?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuefu Zhang updated HIVE-15527: --- Attachment: HIVE-15527.3.patch > Memory usage is unbound in SortByShuffler for Spark > ---

[jira] [Updated] (HIVE-15527) Memory usage is unbound in SortByShuffler for Spark

2017-01-01 Thread Xuefu Zhang (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15527?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuefu Zhang updated HIVE-15527: --- Attachment: HIVE-15527.2.patch > Memory usage is unbound in SortByShuffler for Spark > ---

[jira] [Updated] (HIVE-15527) Memory usage is unbound in SortByShuffler for Spark

2017-01-01 Thread Xuefu Zhang (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15527?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuefu Zhang updated HIVE-15527: --- Attachment: HIVE-15527.1.patch > Memory usage is unbound in SortByShuffler for Spark > ---

[jira] [Commented] (HIVE-15324) Enable round() function to accept scale argument as non-constants

2016-12-31 Thread Xuefu Zhang (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15789811#comment-15789811 ] Xuefu Zhang commented on HIVE-15324: It's at lease unclear to me if such a change make

[jira] [Updated] (HIVE-15527) Memory usage is unbound in SortByShuffler for Spark

2016-12-30 Thread Xuefu Zhang (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15527?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuefu Zhang updated HIVE-15527: --- Description: In SortByShuffler.java, an ArrayList is used to back the iterator for values that have t

[jira] [Updated] (HIVE-15527) Memory usage is unbound in SortByShuffler for Spark

2016-12-30 Thread Xuefu Zhang (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15527?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuefu Zhang updated HIVE-15527: --- Status: Patch Available (was: Open) > Memory usage is unbound in SortByShuffler for Spark > -

[jira] [Updated] (HIVE-15527) Memory usage is unbound in SortByShuffler for Spark

2016-12-30 Thread Xuefu Zhang (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15527?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuefu Zhang updated HIVE-15527: --- Attachment: HIVE-15527.patch CC: [~lirui], [~csun] > Memory usage is unbound in SortByShuffler for Sp

[jira] [Commented] (HIVE-15474) Extend limit propagation for chain of RS-GB-RS operators

2016-12-20 Thread Xuefu Zhang (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15474?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15765925#comment-15765925 ] Xuefu Zhang commented on HIVE-15474: [~lirui], could you also take a look at the propo

[jira] [Commented] (HIVE-15474) Extend limit propagation for chain of RS-GB-RS operators

2016-12-20 Thread Xuefu Zhang (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15474?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15765068#comment-15765068 ] Xuefu Zhang commented on HIVE-15474: Hi [~jcamachorodriguez], thanks for the explanati

[jira] [Commented] (HIVE-15474) Extend limit propagation for chain of RS-GB-RS operators

2016-12-20 Thread Xuefu Zhang (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15474?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15764826#comment-15764826 ] Xuefu Zhang commented on HIVE-15474: Didn't read the patch, but I'm trying to understa

[jira] [Commented] (HIVE-13278) Avoid FileNotFoundException when map/reduce.xml is not available

2016-12-15 Thread Xuefu Zhang (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-13278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15751899#comment-15751899 ] Xuefu Zhang commented on HIVE-13278: Yeah, let's try [~lirui]'s idea to cover more cas

[jira] [Commented] (HIVE-15272) "LEFT OUTER JOIN" Is not populating correct records with Hive On Spark

2016-12-15 Thread Xuefu Zhang (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15751333#comment-15751333 ] Xuefu Zhang commented on HIVE-15272: cc: [~lirui] > "LEFT OUTER JOIN" Is not populati

[jira] [Commented] (HIVE-13278) Avoid FileNotFoundException when map/reduce.xml is not available

2016-12-14 Thread Xuefu Zhang (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-13278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15750533#comment-15750533 ] Xuefu Zhang commented on HIVE-13278: [~lirui], the concern is valid and shared, but on

[jira] [Commented] (HIVE-13278) Many redundant 'File not found' messages appeared in container log during query execution with Hive on Spark

2016-12-14 Thread Xuefu Zhang (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-13278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15749533#comment-15749533 ] Xuefu Zhang commented on HIVE-13278: +1 on patch #3. > Many redundant 'File not found

[jira] [Commented] (HIVE-13278) Many redundant 'File not found' messages appeared in container log during query execution with Hive on Spark

2016-12-14 Thread Xuefu Zhang (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-13278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15749097#comment-15749097 ] Xuefu Zhang commented on HIVE-13278: I like the new patch, which takes a simpler, easy

[jira] [Issue Comment Deleted] (HIVE-13278) Many redundant 'File not found' messages appeared in container log during query execution with Hive on Spark

2016-12-14 Thread Xuefu Zhang (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-13278?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuefu Zhang updated HIVE-13278: --- Comment: was deleted (was: [~csun], please feel free to address MR case first if we need more time fo

[jira] [Commented] (HIVE-13278) Many redundant 'File not found' messages appeared in container log during query execution with Hive on Spark

2016-12-13 Thread Xuefu Zhang (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-13278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15747344#comment-15747344 ] Xuefu Zhang commented on HIVE-13278: [~csun], please feel free to address MR case firs

[jira] [Commented] (HIVE-13278) Many redundant 'File not found' messages appeared in container log during query execution with Hive on Spark

2016-12-13 Thread Xuefu Zhang (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-13278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15747342#comment-15747342 ] Xuefu Zhang commented on HIVE-13278: [~csun], please feel free to address MR case firs

[jira] [Commented] (HIVE-13278) Many redundant 'File not found' messages appeared in container log during query execution with Hive on Spark

2016-12-13 Thread Xuefu Zhang (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-13278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15746850#comment-15746850 ] Xuefu Zhang commented on HIVE-13278: [~stakiar], let us know if you like to work on th

[jira] [Commented] (HIVE-13278) Many redundant 'File not found' messages appeared in container log during query execution with Hive on Spark

2016-12-13 Thread Xuefu Zhang (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-13278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15745219#comment-15745219 ] Xuefu Zhang commented on HIVE-13278: [~lirui], thanks for the summary. Following your

[jira] [Comment Edited] (HIVE-13278) Many redundant 'File not found' messages appeared in container log during query execution with Hive on Spark

2016-12-12 Thread Xuefu Zhang (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-13278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15744091#comment-15744091 ] Xuefu Zhang edited comment on HIVE-13278 at 12/13/16 4:15 AM: --

[jira] [Commented] (HIVE-13278) Many redundant 'File not found' messages appeared in container log during query execution with Hive on Spark

2016-12-12 Thread Xuefu Zhang (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-13278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15744091#comment-15744091 ] Xuefu Zhang commented on HIVE-13278: [~lirui], I'm not sure if I understand the conclu

[jira] [Commented] (HIVE-15386) Expose Spark task counts and stage Ids information in SparkTask from SparkJobMonitor

2016-12-12 Thread Xuefu Zhang (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15744066#comment-15744066 ] Xuefu Zhang commented on HIVE-15386: Patch looks good to me as well. +1 > Expose Spar

[jira] [Commented] (HIVE-15386) Expose Spark task counts and stage Ids information in SparkTask from SparkJobMonitor

2016-12-08 Thread Xuefu Zhang (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15733883#comment-15733883 ] Xuefu Zhang commented on HIVE-15386: Yeah, it seems a little invasive with the propose

[jira] [Commented] (HIVE-15361) Dynamic partition INSERT fails with a MoveTask failure

2016-12-05 Thread Xuefu Zhang (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15723096#comment-15723096 ] Xuefu Zhang commented on HIVE-15361: [~spena], thanks for working on this. I'm wonderi

[jira] [Commented] (HIVE-15239) hive on spark combine equivalentwork get wrong result because of tablescan operation compare

2016-12-01 Thread Xuefu Zhang (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15713840#comment-15713840 ] Xuefu Zhang commented on HIVE-15239: +1 > hive on spark combine equivalentwork get wr

[jira] [Commented] (HIVE-15331) Decimal multiplication with high precision/scale often returns NULL

2016-12-01 Thread Xuefu Zhang (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15713110#comment-15713110 ] Xuefu Zhang commented on HIVE-15331: I dont' think this can be made bullet proof. I ca

[jira] [Commented] (HIVE-15331) Decimal multiplication with high precision/scale often returns NULL

2016-12-01 Thread Xuefu Zhang (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15712964#comment-15712964 ] Xuefu Zhang commented on HIVE-15331: There are more than a couple of JIRAs on this top

[jira] [Commented] (HIVE-15239) hive on spark combine equivalentwork get wrong result because of tablescan operation compare

2016-12-01 Thread Xuefu Zhang (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15712934#comment-15712934 ] Xuefu Zhang commented on HIVE-15239: +1 with minor comment on RB. > hive on spark com

[jira] [Commented] (HIVE-15239) hive on spark combine equivalentwork get wrong result because of tablescan operation compare

2016-11-30 Thread Xuefu Zhang (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15710944#comment-15710944 ] Xuefu Zhang commented on HIVE-15239: [~lirui] do mind creating a RB for this? Thanks.

[jira] [Commented] (HIVE-15313) Add export spark.yarn.archive or spark.yarn.jars variable in Hive on Spark document

2016-11-30 Thread Xuefu Zhang (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15710935#comment-15710935 ] Xuefu Zhang commented on HIVE-15313: +1 from me as well. Let's first flush out everyth

[jira] [Commented] (HIVE-15302) Relax the requirement that HoS needs Spark built w/o Hive

2016-11-30 Thread Xuefu Zhang (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15710922#comment-15710922 ] Xuefu Zhang commented on HIVE-15302: I think there are two dependency on Spark from Hi

[jira] [Comment Edited] (HIVE-15239) hive on spark combine equivalentwork get wrong result because of tablescan operation compare

2016-11-30 Thread Xuefu Zhang (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15710435#comment-15710435 ] Xuefu Zhang edited comment on HIVE-15239 at 12/1/16 1:12 AM: -

[jira] [Commented] (HIVE-15239) hive on spark combine equivalentwork get wrong result because of tablescan operation compare

2016-11-30 Thread Xuefu Zhang (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15710435#comment-15710435 ] Xuefu Zhang commented on HIVE-15239: Sorry for the delay. Re: my point #1, I was refe

[jira] [Commented] (HIVE-15301) Expose SparkStatistics information in SparkTask

2016-11-29 Thread Xuefu Zhang (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15705367#comment-15705367 ] Xuefu Zhang commented on HIVE-15301: +1 > Expose SparkStatistics information in Spark

<    1   2   3   4   5   6   7   8   9   10   >