[jira] [Updated] (HIVE-9428) LocalSparkJobStatus may return failed job as successful [Spark Branch]

2015-01-21 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-9428?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-9428: - Attachment: HIVE-9428.2-spark.patch Call future.get() to test is a future is successful. > LocalSparkJobSta

[jira] [Commented] (HIVE-9409) Avoid ser/de loggers as logging framework can be incompatible on driver and workers

2015-01-21 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-9409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14286783#comment-14286783 ] Rui Li commented on HIVE-9409: -- I don't think the failures are related. >

[jira] [Commented] (HIVE-9409) Avoid ser/de loggers as logging framework can be incompatible on driver and workers

2015-01-21 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-9409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14286744#comment-14286744 ] Rui Li commented on HIVE-9409: -- That's good to know. Thank you Xin! > Avoid se

[jira] [Updated] (HIVE-9428) LocalSparkJobStatus may return failed job as successful [Spark Branch]

2015-01-21 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-9428?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-9428: - Status: Open (was: Patch Available) Cancel the patch. It's not sufficient to solve the

[jira] [Updated] (HIVE-9409) Avoid ser/de loggers as logging framework can be incompatible on driver and workers

2015-01-21 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-9409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-9409: - Summary: Avoid ser/de loggers as logging framework can be incompatible on driver and workers (was: Spark branch

[jira] [Updated] (HIVE-9409) Spark branch, ClassNotFoundException: org.apache.commons.logging.impl.SLF4JLocationAwareLog occurs during some hive query case execution [Spark Branch]

2015-01-21 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-9409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-9409: - Status: Patch Available (was: Open) > Spark branch, ClassNotFoundExcept

[jira] [Updated] (HIVE-9409) Spark branch, ClassNotFoundException: org.apache.commons.logging.impl.SLF4JLocationAwareLog occurs during some hive query case execution [Spark Branch]

2015-01-21 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-9409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-9409: - Attachment: HIVE-9409.1.patch The patch is for trunk. [~xhao1] please help to verify if it can solve your problem

[jira] [Commented] (HIVE-9409) Spark branch, ClassNotFoundException: org.apache.commons.logging.impl.SLF4JLocationAwareLog occurs during some hive query case execution [Spark Branch]

2015-01-21 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-9409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14285316#comment-14285316 ] Rui Li commented on HIVE-9409: -- I tried making hive use jcl-over-slf4j. But even that w

[jira] [Updated] (HIVE-9428) LocalSparkJobStatus may return failed job as successful [Spark Branch]

2015-01-20 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-9428?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-9428: - Attachment: HIVE-9428.1-spark.patch > LocalSparkJobStatus may return failed job as successful [Spark Bra

[jira] [Updated] (HIVE-9428) LocalSparkJobStatus may return failed job as successful [Spark Branch]

2015-01-20 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-9428?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-9428: - Priority: Minor (was: Major) > LocalSparkJobStatus may return failed job as successful [Spark Bra

[jira] [Updated] (HIVE-9428) LocalSparkJobStatus may return failed job as successful [Spark Branch]

2015-01-20 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-9428?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-9428: - Status: Patch Available (was: Open) > LocalSparkJobStatus may return failed job as successful [Spark Bra

[jira] [Updated] (HIVE-9428) LocalSparkJobStatus may return failed job as successful [Spark Branch]

2015-01-20 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-9428?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-9428: - Summary: LocalSparkJobStatus may return failed job as successful [Spark Branch] (was: LocalSparkJobStatus may

[jira] [Created] (HIVE-9428) LocalSparkJobStatus may return failed job as successful

2015-01-20 Thread Rui Li (JIRA)
Rui Li created HIVE-9428: Summary: LocalSparkJobStatus may return failed job as successful Key: HIVE-9428 URL: https://issues.apache.org/jira/browse/HIVE-9428 Project: Hive Issue Type: Sub-task

[jira] [Commented] (HIVE-9425) External Function Jar files are not available for Driver when running with yarn-cluster mode [Spark Branch]

2015-01-20 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-9425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14285297#comment-14285297 ] Rui Li commented on HIVE-9425: -- Does this duplicate HIVE-9410? cc [~chengxian

[jira] [Commented] (HIVE-9339) Optimize split grouping for CombineHiveInputFormat [Spark Branch]

2015-01-20 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-9339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14285010#comment-14285010 ] Rui Li commented on HIVE-9339: -- FYI SPARK-4857 enhanced listeners to listen for executor

[jira] [Commented] (HIVE-9409) Spark branch, ClassNotFoundException: org.apache.commons.logging.impl.SLF4JLocationAwareLog occurs during some hive query case execution [Spark Branch]

2015-01-20 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-9409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14284995#comment-14284995 ] Rui Li commented on HIVE-9409: -- Yeah I noticed we depend on several different log

[jira] [Assigned] (HIVE-9409) Spark branch, ClassNotFoundException: org.apache.commons.logging.impl.SLF4JLocationAwareLog occurs during some hive query case execution [Spark Branch]

2015-01-19 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-9409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li reassigned HIVE-9409: Assignee: Rui Li > Spark branch, ClassNotFoundExcept

[jira] [Commented] (HIVE-9409) Spark branch, ClassNotFoundException: org.apache.commons.logging.impl.SLF4JLocationAwareLog occurs during some hive query case execution [Spark Branch]

2015-01-19 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-9409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14283528#comment-14283528 ] Rui Li commented on HIVE-9409: -- Spark uses {{jcl-over-slf4j}} to bridge calls to com

[jira] [Commented] (HIVE-9409) Spark branch, ClassNotFoundException: org.apache.commons.logging.impl.SLF4JLocationAwareLog occurs during some hive query case execution [Spark Branch]

2015-01-19 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-9409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14283326#comment-14283326 ] Rui Li commented on HIVE-9409: -- Right. Let's fix it together with this issue. Tha

[jira] [Commented] (HIVE-9395) Make WAIT_SUBMISSION_TIMEOUT configuable and check timeout in SparkJobMonitor level.[Spark Branch]

2015-01-19 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-9395?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14283301#comment-14283301 ] Rui Li commented on HIVE-9395: -- I see, thanks for the explanation! In that case, shal

[jira] [Commented] (HIVE-9409) Spark branch, ClassNotFoundException: org.apache.commons.logging.impl.SLF4JLocationAwareLog occurs during some hive query case execution [Spark Branch]

2015-01-19 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-9409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14283264#comment-14283264 ] Rui Li commented on HIVE-9409: -- Hi [~brocknoland], do you think that's related to

[jira] [Commented] (HIVE-9395) Make WAIT_SUBMISSION_TIMEOUT configuable and check timeout in SparkJobMonitor level.[Spark Branch]

2015-01-19 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-9395?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14282594#comment-14282594 ] Rui Li commented on HIVE-9395: -- I think jobId available basically means the submission

[jira] [Commented] (HIVE-9395) Make WAIT_SUBMISSION_TIMEOUT configuable and check timeout in SparkJobMonitor level.[Spark Branch]

2015-01-19 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-9395?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14282484#comment-14282484 ] Rui Li commented on HIVE-9395: -- I think HIVE-9370 shows we do need some extra informatio

[jira] [Commented] (HIVE-9409) Spark branch, ClassNotFoundException: org.apache.commons.logging.impl.SLF4JLocationAwareLog occurs during some hive query case execution [Spark Branch]

2015-01-19 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-9409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14282461#comment-14282461 ] Rui Li commented on HIVE-9409: -- I verified with [~xhao1] that removing spark-assembly

[jira] [Commented] (HIVE-9395) Make WAIT_SUBMISSION_TIMEOUT configuable and check timeout in SparkJobMonitor level.[Spark Branch]

2015-01-19 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-9395?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14282303#comment-14282303 ] Rui Li commented on HIVE-9395: -- Any reason why we want to move the timeout to SparkJobMon

[jira] [Commented] (HIVE-9110) Performance of SELECT COUNT(*) FROM store_sales WHERE ss_item_sk IS NOT NULL [Spark Branch]

2015-01-18 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-9110?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14282181#comment-14282181 ] Rui Li commented on HIVE-9110: -- Hi [~brocknoland], With some recent fixes, shall we run

[jira] [Updated] (HIVE-9372) Parallel checking non-combinable paths in CombineHiveInputFormat

2015-01-18 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-9372?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-9372: - Attachment: HIVE-9372.2.patch None of the failures can be reproduced locally. Try again. > Parallel checking

[jira] [Commented] (HIVE-9370) Enable Hive on Spark for BigBench and run Query 8, the test failed [Spark Branch]

2015-01-15 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-9370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14279858#comment-14279858 ] Rui Li commented on HIVE-9370: -- If the additional jobs cannot be avoided, maybe we can a

[jira] [Commented] (HIVE-9370) Enable Hive on Spark for BigBench and run Query 8, the test failed [Spark Branch]

2015-01-15 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-9370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14279839#comment-14279839 ] Rui Li commented on HIVE-9370: -- I see. Thanks for the clarification. > Enable Hive o

[jira] [Commented] (HIVE-9370) Enable Hive on Spark for BigBench and run Query 8, the test failed [Spark Branch]

2015-01-15 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-9370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14279816#comment-14279816 ] Rui Li commented on HIVE-9370: -- Currently we only use sortByKey for global sorting. I t

[jira] [Commented] (HIVE-9370) Enable Hive on Spark for BigBench and run Query 8, the test failed [Spark Branch]

2015-01-15 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-9370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14279813#comment-14279813 ] Rui Li commented on HIVE-9370: -- Yes sortByKey will launch additional jobs if the numbe

[jira] [Commented] (HIVE-9370) Enable Hive on Spark for BigBench and run Query 8, the test failed [Spark Branch]

2015-01-15 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-9370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14279807#comment-14279807 ] Rui Li commented on HIVE-9370: -- The problem seems to be sortByKey may launch extra spark

[jira] [Commented] (HIVE-9370) Enable Hive on Spark for BigBench and run Query 8, the test failed [Spark Branch]

2015-01-15 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-9370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14279806#comment-14279806 ] Rui Li commented on HIVE-9370: -- The problem seems to be sortByKey may launch extra spark

[jira] [Updated] (HIVE-9372) Parallel checking non-combinable paths in CombineHiveInputFormat

2015-01-15 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-9372?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-9372: - Attachment: HIVE-9372.3-spark.patch Run test again > Parallel checking non-combinable paths

[jira] [Commented] (HIVE-9372) Parallel checking non-combinable paths in CombineHiveInputFormat

2015-01-15 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-9372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14279770#comment-14279770 ] Rui Li commented on HIVE-9372: -- Checked the log of failed tests. mapjoin_memcheck failed

[jira] [Updated] (HIVE-9387) Shutting down cli takes quite some time [Spark Branch]

2015-01-15 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-9387?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-9387: - Attachment: HIVE-9387.1-spark.patch Seems we shouldn't wait for a reply from an EndSession as the chann

[jira] [Updated] (HIVE-9387) Shutting down cli takes quite some time [Spark Branch]

2015-01-15 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-9387?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-9387: - Status: Patch Available (was: Open) > Shutting down cli takes quite some time [Spark Bra

[jira] [Assigned] (HIVE-9387) Shutting down cli takes quite some time [Spark Branch]

2015-01-15 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-9387?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li reassigned HIVE-9387: Assignee: Rui Li > Shutting down cli takes quite some time [Spark Bra

[jira] [Commented] (HIVE-9370) Enable Hive on Spark for BigBench and run Query 8, the test failed [Spark Branch]

2015-01-15 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-9370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14278557#comment-14278557 ] Rui Li commented on HIVE-9370: -- At client side we wait 30s for a job to be submitted be

[jira] [Created] (HIVE-9387) Shutting down cli takes quite some time [Spark Branch]

2015-01-15 Thread Rui Li (JIRA)
Rui Li created HIVE-9387: Summary: Shutting down cli takes quite some time [Spark Branch] Key: HIVE-9387 URL: https://issues.apache.org/jira/browse/HIVE-9387 Project: Hive Issue Type: Sub-task

[jira] [Updated] (HIVE-9372) Parallel checking non-combinable paths in CombineHiveInputFormat

2015-01-15 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-9372?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-9372: - Attachment: HIVE-9372.3-spark.patch Upload a spark version > Parallel checking non-combinable paths

[jira] [Commented] (HIVE-9372) Parallel checking non-combinable paths in CombineHiveInputFormat

2015-01-14 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-9372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14278213#comment-14278213 ] Rui Li commented on HIVE-9372: -- [~xuefuz] - right... I just didn't realize it could

[jira] [Updated] (HIVE-9372) Parallel checking non-combinable paths in CombineHiveInputFormat

2015-01-14 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-9372?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-9372: - Attachment: HIVE-9372.2.patch Address Xuefu's comment. I also make the perf log to cover the

[jira] [Commented] (HIVE-9372) Parallel checking non-combinable paths in CombineHiveInputFormat

2015-01-14 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-9372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14278032#comment-14278032 ] Rui Li commented on HIVE-9372: -- Yeah will do that. I wonder if the test hasn't star

[jira] [Updated] (HIVE-9372) Parallel checking non-combinable paths in CombineHiveInputFormat

2015-01-14 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-9372?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-9372: - Status: Patch Available (was: Open) > Parallel checking non-combinable paths in CombineHiveInputFor

[jira] [Updated] (HIVE-9372) Parallel checking non-combinable paths in CombineHiveInputFormat

2015-01-14 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-9372?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-9372: - Attachment: HIVE-9372.1.patch > Parallel checking non-combinable paths in CombineHiveInputFor

[jira] [Updated] (HIVE-9372) Parallel checking non-combinable paths in CombineHiveInputFormat

2015-01-14 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-9372?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-9372: - Description: Checking if an input path is combinable is expensive. So we should make it parallel. > Paral

[jira] [Created] (HIVE-9372) Parallel checking non-combinable paths in CombineHiveInputFormat

2015-01-14 Thread Rui Li (JIRA)
Rui Li created HIVE-9372: Summary: Parallel checking non-combinable paths in CombineHiveInputFormat Key: HIVE-9372 URL: https://issues.apache.org/jira/browse/HIVE-9372 Project: Hive Issue Type

[jira] [Commented] (HIVE-9367) CombineFileInputFormatShim#getDirIndices is expensive

2015-01-13 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-9367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14276527#comment-14276527 ] Rui Li commented on HIVE-9367: -- I just verified the patch here can reduce the getSplits

[jira] [Commented] (HIVE-9367) CombineFileInputFormatShim#getDirIndices is expensive

2015-01-13 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-9367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14276444#comment-14276444 ] Rui Li commented on HIVE-9367: -- I see. Thanks [~jxiang] for the explana

[jira] [Commented] (HIVE-9367) CombineFileInputFormatShim#getDirIndices is expensive

2015-01-13 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-9367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14276414#comment-14276414 ] Rui Li commented on HIVE-9367: -- Hi [~jxiang], could you elaborate a little how this

[jira] [Commented] (HIVE-9110) Performance of SELECT COUNT(*) FROM store_sales WHERE ss_item_sk IS NOT NULL [Spark Branch]

2015-01-12 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-9110?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14274792#comment-14274792 ] Rui Li commented on HIVE-9110: -- In spark standalone mode, the executor won't ha

[jira] [Commented] (HIVE-9339) Optimize split grouping for CombineHiveInputFormat [Spark Branch]

2015-01-12 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-9339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14274751#comment-14274751 ] Rui Li commented on HIVE-9339: -- Just some thoughts, since we can't combine the spl

[jira] [Updated] (HIVE-9343) Fix windowing.q for Spark on trunk

2015-01-11 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-9343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-9343: - Attachment: HIVE-9343.1.patch > Fix windowing.q for Spark on tr

[jira] [Updated] (HIVE-9343) Fix windowing.q for Spark on trunk

2015-01-11 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-9343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-9343: - Assignee: Rui Li Status: Patch Available (was: Open) > Fix windowing.q for Spark on tr

[jira] [Commented] (HIVE-9343) Fix windowing.q for Spark on trunk

2015-01-11 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-9343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14273261#comment-14273261 ] Rui Li commented on HIVE-9343: -- OK I'll take a look > Fix windowing.q for Sp

[jira] [Commented] (HIVE-9339) Optimize split grouping for CombineHiveInputFormat [Spark Branch]

2015-01-11 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-9339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14273191#comment-14273191 ] Rui Li commented on HIVE-9339: -- Using listener is fine. We currently use listeners to col

[jira] [Updated] (HIVE-9251) SetSparkReducerParallelism is likely to set too small number of reducers [Spark Branch]

2015-01-09 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-9251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-9251: - Attachment: HIVE-9251.6-spark.patch Rebase the patch and include more update. > SetSparkReducerParallelism

[jira] [Updated] (HIVE-9251) SetSparkReducerParallelism is likely to set too small number of reducers [Spark Branch]

2015-01-09 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-9251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-9251: - Attachment: HIVE-9251.5-spark.patch I missed some update to optimize_nullscan.q Update patch

[jira] [Commented] (HIVE-9290) Make some test results deterministic

2015-01-09 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-9290?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14270742#comment-14270742 ] Rui Li commented on HIVE-9290: -- Hi [~szehon], sorry I didn't provide a patch for spa

[jira] [Updated] (HIVE-9251) SetSparkReducerParallelism is likely to set too small number of reducers [Spark Branch]

2015-01-08 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-9251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-9251: - Attachment: HIVE-9251.4-spark.patch Update more golden files. > SetSparkReducerParallelism is likely to set

[jira] [Commented] (HIVE-9290) Make some test results deterministic

2015-01-08 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-9290?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14270456#comment-14270456 ] Rui Li commented on HIVE-9290: -- The failed test is not related to the patch here. &g

[jira] [Updated] (HIVE-9290) Make some test results deterministic

2015-01-08 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-9290?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-9290: - Attachment: HIVE-9290.1.patch Reload patch to trigger test > Make some test results determinis

[jira] [Commented] (HIVE-9290) Make some test results deterministic

2015-01-07 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-9290?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14268843#comment-14268843 ] Rui Li commented on HIVE-9290: -- Thanks [~xuefuz] for the explanation! > Make so

[jira] [Updated] (HIVE-9290) Make some test results deterministic

2015-01-07 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-9290?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-9290: - Status: Patch Available (was: Open) > Make some test results determinis

[jira] [Updated] (HIVE-9290) Make some test results deterministic

2015-01-07 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-9290?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-9290: - Attachment: HIVE-9290.1.patch Not sure if it's correct to make limit_pushdown.q deterministic. cc [~x

[jira] [Commented] (HIVE-9251) SetSparkReducerParallelism is likely to set too small number of reducers [Spark Branch]

2015-01-07 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-9251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14268728#comment-14268728 ] Rui Li commented on HIVE-9251: -- [~xuefuz] - you're right. I think we should fix

[jira] [Updated] (HIVE-9290) Make some test results deterministic

2015-01-07 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-9290?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-9290: - Description: {noformat} limit_pushdown.q optimize_nullscan.q ppd_gby_join.q vector_string_concat.q {noformat

[jira] [Updated] (HIVE-9251) SetSparkReducerParallelism is likely to set too small number of reducers [Spark Branch]

2015-01-07 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-9251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-9251: - Attachment: HIVE-9251.3-spark.patch Addressed RB comments and updated golden files. Some notes about the reducer

[jira] [Created] (HIVE-9290) Make some test results deterministic

2015-01-07 Thread Rui Li (JIRA)
Rui Li created HIVE-9290: Summary: Make some test results deterministic Key: HIVE-9290 URL: https://issues.apache.org/jira/browse/HIVE-9290 Project: Hive Issue Type: Test Reporter: Rui

[jira] [Commented] (HIVE-9251) SetSparkReducerParallelism is likely to set too small number of reducers [Spark Branch]

2015-01-06 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-9251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14267276#comment-14267276 ] Rui Li commented on HIVE-9251: -- That basically means cluster info is not available. So

[jira] [Commented] (HIVE-9251) SetSparkReducerParallelism is likely to set too small number of reducers [Spark Branch]

2015-01-06 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-9251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14267229#comment-14267229 ] Rui Li commented on HIVE-9251: -- Hi [~xuefuz], yeah I'll update the golden files if

[jira] [Commented] (HIVE-9251) SetSparkReducerParallelism is likely to set too small number of reducers [Spark Branch]

2015-01-06 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-9251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14267213#comment-14267213 ] Rui Li commented on HIVE-9251: -- I quickly checked the failed tests. Most of them are in q

[jira] [Updated] (HIVE-9251) SetSparkReducerParallelism is likely to set too small number of reducers [Spark Branch]

2015-01-06 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-9251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-9251: - Status: Patch Available (was: Open) > SetSparkReducerParallelism is likely to set too small number of reduc

[jira] [Updated] (HIVE-9251) SetSparkReducerParallelism is likely to set too small number of reducers [Spark Branch]

2015-01-06 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-9251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-9251: - Attachment: HIVE-9251.2-spark.patch Thanks [~jxiang] and [~xuefuz]. Upload another patch. I didn't remov

[jira] [Commented] (HIVE-9251) SetSparkReducerParallelism is likely to set too small number of reducers [Spark Branch]

2015-01-04 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-9251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14264135#comment-14264135 ] Rui Li commented on HIVE-9251: -- Hi [~xuefuz], I think {{hive.exec.reducers.bytes.per.red

[jira] [Updated] (HIVE-9251) SetSparkReducerParallelism is likely to set too small number of reducers [Spark Branch]

2015-01-04 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-9251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-9251: - Attachment: HIVE-9251.1-spark.patch Submit a patch for review. BTW, maybe we don't need memory per ta

[jira] [Assigned] (HIVE-9251) SetSparkReducerParallelism is likely to set too small number of reducers [Spark Branch]

2015-01-04 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-9251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li reassigned HIVE-9251: Assignee: Rui Li > SetSparkReducerParallelism is likely to set too small number of reducers >

[jira] [Updated] (HIVE-9251) SetSparkReducerParallelism is likely to set too small number of reducers [Spark Branch]

2015-01-04 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-9251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-9251: - Description: This may hurt performance or even lead to task failures. For example, spark's netty-based sh

[jira] [Commented] (HIVE-9251) SetSparkReducerParallelism is likely to set too small number of reducers [Spark Branch]

2015-01-04 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-9251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14263821#comment-14263821 ] Rui Li commented on HIVE-9251: -- Basically the following problems will lead to small numbe

[jira] [Created] (HIVE-9251) SetSparkReducerParallelism is likely to set too small number of reducers [Spark Branch]

2015-01-04 Thread Rui Li (JIRA)
Rui Li created HIVE-9251: Summary: SetSparkReducerParallelism is likely to set too small number of reducers [Spark Branch] Key: HIVE-9251 URL: https://issues.apache.org/jira/browse/HIVE-9251 Project: Hive

[jira] [Created] (HIVE-9227) Make HiveInputSplit support InputSplitWithLocationInfo

2014-12-30 Thread Rui Li (JIRA)
Rui Li created HIVE-9227: Summary: Make HiveInputSplit support InputSplitWithLocationInfo Key: HIVE-9227 URL: https://issues.apache.org/jira/browse/HIVE-9227 Project: Hive Issue Type: Improvement

[jira] [Commented] (HIVE-8956) Hive hangs while some error/exception happens beyond job execution [Spark Branch]

2014-12-29 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-8956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14260652#comment-14260652 ] Rui Li commented on HIVE-8956: -- Hi [~chirag.aggarwal], which query triggered this error?

[jira] [Commented] (HIVE-9153) Evaluate CombineHiveInputFormat versus HiveInputFormat

2014-12-26 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-9153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14259261#comment-14259261 ] Rui Li commented on HIVE-9153: -- Hi [~xuefuz], I don't think it's related

[jira] [Updated] (HIVE-9216) Avoid redundant clone of JobConf [Spark Branch]

2014-12-25 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-9216?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-9216: - Status: Patch Available (was: Open) > Avoid redundant clone of JobConf [Spark Bra

[jira] [Updated] (HIVE-9216) Avoid redundant clone of JobConf [Spark Branch]

2014-12-25 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-9216?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-9216: - Attachment: HIVE-9216.1-spark.patch > Avoid redundant clone of JobConf [Spark Bra

[jira] [Updated] (HIVE-9216) Avoid redundant clone of JobConf [Spark Branch]

2014-12-25 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-9216?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-9216: - Priority: Minor (was: Major) > Avoid redundant clone of JobConf [Spark Bra

[jira] [Updated] (HIVE-9216) Avoid redundant clone of JobConf [Spark Branch]

2014-12-25 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-9216?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-9216: - Summary: Avoid redundant clone of JobConf [Spark Branch] (was: Avoid redundant clone of JobConf) > Av

[jira] [Created] (HIVE-9216) Avoid redundant clone of JobConf

2014-12-25 Thread Rui Li (JIRA)
Rui Li created HIVE-9216: Summary: Avoid redundant clone of JobConf Key: HIVE-9216 URL: https://issues.apache.org/jira/browse/HIVE-9216 Project: Hive Issue Type: Sub-task Components: Spark

[jira] [Commented] (HIVE-9153) Evaluate CombineHiveInputFormat versus HiveInputFormat [Spark Branch]

2014-12-25 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-9153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14258909#comment-14258909 ] Rui Li commented on HIVE-9153: -- Strange thing is that {{Utilities}} is different in trunk

[jira] [Updated] (HIVE-9153) Evaluate CombineHiveInputFormat versus HiveInputFormat [Spark Branch]

2014-12-25 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-9153?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-9153: - Attachment: HIVE-9153.3.patch Seems the redundant code in {{Utilities.getBasework}} has been taken care of in

[jira] [Updated] (HIVE-9153) Evaluate CombineHiveInputFormat versus HiveInputFormat [Spark Branch]

2014-12-25 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-9153?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-9153: - Attachment: HIVE-9153.2.patch Upload trunk patch > Evaluate CombineHiveInputFormat versus HiveInputFormat [Sp

[jira] [Commented] (HIVE-9153) Evaluate CombineHiveInputFormat versus HiveInputFormat [Spark Branch]

2014-12-25 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-9153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14258885#comment-14258885 ] Rui Li commented on HIVE-9153: -- Hi [~brocknoland] and [~xuefuz], Sorry maybe I was b

[jira] [Commented] (HIVE-9135) Cache Map and Reduce works in RSC [Spark Branch]

2014-12-25 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-9135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14258741#comment-14258741 ] Rui Li commented on HIVE-9135: -- I'm not sure if this is correct: we clone J

[jira] [Updated] (HIVE-9153) Evaluate CombineHiveInputFormat versus HiveInputFormat [Spark Branch]

2014-12-25 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-9153?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-9153: - Status: Patch Available (was: Open) > Evaluate CombineHiveInputFormat versus HiveInputFormat [Spark Bra

[jira] [Updated] (HIVE-9153) Evaluate CombineHiveInputFormat versus HiveInputFormat [Spark Branch]

2014-12-25 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-9153?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-9153: - Attachment: HIVE-9153.1-spark.patch This patch should further improve spark performance by avoid retrieving

[jira] [Commented] (HIVE-9191) TimeOutException when using RSC with beeline [Spark Branch]

2014-12-23 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-9191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14256825#comment-14256825 ] Rui Li commented on HIVE-9191: -- I also run into this using Cli. > TimeOutExcepti

[jira] [Commented] (HIVE-8722) Enhance InputSplitShims to extend InputSplitWithLocationInfo [Spark Branch]

2014-12-22 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-8722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14255668#comment-14255668 ] Rui Li commented on HIVE-8722: -- Hi [~brocknoland], I suppose this will be a little tricky

[jira] [Commented] (HIVE-9153) Evaluate CombineHiveInputFormat versus HiveInputFormat [Spark Branch]

2014-12-22 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-9153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14255650#comment-14255650 ] Rui Li commented on HIVE-9153: -- [~xuefuz] - I was wrong about turning off delay sche

[jira] [Commented] (HIVE-9153) Evaluate CombineHiveInputFormat versus HiveInputFormat [Spark Branch]

2014-12-19 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-9153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14253164#comment-14253164 ] Rui Li commented on HIVE-9153: -- Investigated a bit about

<    1   2   3   4   5   6   7   8   >