[jira] [Created] (HIVE-12888) TestSparkNegativeCliDriver does not run in Spark mode[Spark Branch]

2016-01-19 Thread Chengxiang Li (JIRA)
Chengxiang Li created HIVE-12888: Summary: TestSparkNegativeCliDriver does not run in Spark mode[Spark Branch] Key: HIVE-12888 URL: https://issues.apache.org/jira/browse/HIVE-12888 Project: Hive

[jira] [Created] (HIVE-12515) Clean the SparkCounters related code after remove counter based stats collection[Spark Branch]

2015-11-24 Thread Chengxiang Li (JIRA)
Chengxiang Li created HIVE-12515: Summary: Clean the SparkCounters related code after remove counter based stats collection[Spark Branch] Key: HIVE-12515 URL: https://issues.apache.org/jira/browse/HIVE-12515

Re: Review Request 36475: HIVE-11082 Support multi edge between nodes in SparkPlan[Spark Branch]

2015-07-15 Thread chengxiang li
/clientpositive/spark/union9.q.out d420ef1 Diff: https://reviews.apache.org/r/36475/diff/ Testing --- Thanks, chengxiang li

Re: Review Request 36475: HIVE-11082 Support multi edge between nodes in SparkPlan[Spark Branch]

2015-07-15 Thread chengxiang li
just return null here. - chengxiang --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/36475/#review91736 --- On July 15

[jira] [Created] (HIVE-11267) Combine equavilent leaf works in SparkWork[Spark Branch]

2015-07-15 Thread Chengxiang Li (JIRA)
Chengxiang Li created HIVE-11267: Summary: Combine equavilent leaf works in SparkWork[Spark Branch] Key: HIVE-11267 URL: https://issues.apache.org/jira/browse/HIVE-11267 Project: Hive Issue

Re: Review Request 36475: HIVE-11082 Support multi edge between nodes in SparkPlan[Spark Branch]

2015-07-15 Thread chengxiang li
/clientpositive/spark/union9.q.out d420ef1 Diff: https://reviews.apache.org/r/36475/diff/ Testing --- Thanks, chengxiang li

Review Request 36475: HIVE-11082 Support multi edge between nodes in SparkPlan[Spark Branch]

2015-07-13 Thread chengxiang li
/src/test/queries/clientpositive/dynamic_rdd_cache.q a380b15 ql/src/test/results/clientpositive/dynamic_rdd_cache.q.out bc716a0 ql/src/test/results/clientpositive/spark/dynamic_rdd_cache.q.out 505cc59 Diff: https://reviews.apache.org/r/36475/diff/ Testing --- Thanks, chengxiang li

Re: Review Request 34666: HIVE-9152 - Dynamic Partition Pruning [Spark Branch]

2015-07-12 Thread chengxiang li
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/34666/#review91427 --- Ship it! Ship It! - chengxiang li On 七月 8, 2015, 6:04 p.m

[jira] [Created] (HIVE-11204) Research on recent failed qtests[Spark Branch]

2015-07-08 Thread Chengxiang Li (JIRA)
Chengxiang Li created HIVE-11204: Summary: Research on recent failed qtests[Spark Branch] Key: HIVE-11204 URL: https://issues.apache.org/jira/browse/HIVE-11204 Project: Hive Issue Type: Sub

Re: Review Request 36156: HIVE-11053: Add more tests for HIVE-10844[Spark Branch]

2015-07-07 Thread chengxiang li
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/36156/#review90859 --- Ship it! Ship It! - chengxiang li On 七月 8, 2015, 3:05 a.m., lun

Re: Review Request 36156: HIVE-11053: Add more tests for HIVE-10844[Spark Branch]

2015-07-06 Thread chengxiang li
) <https://reviews.apache.org/r/36156/#comment143770> Are the temp table X/Y/Z be created actually? - chengxiang li On 七月 6, 2015, 6:35 a.m., lun gao wrote: > > --- > This is an automatically generated e-mail. To reply,

Re: Review Request 34666: HIVE-9152 - Dynamic Partition Pruning [Spark Branch]

2015-07-06 Thread chengxiang li
> On 七月 2, 2015, 6:36 a.m., chengxiang li wrote: > > ql/src/java/org/apache/hadoop/hive/ql/optimizer/SparkRemoveDynamicPruningBySize.java, > > line 59 > > <https://reviews.apache.org/r/34666/diff/1/?file=971706#file971706line59> > > > > The statist

Re: Review Request 36156: HIVE-11053: Add more tests for HIVE-10844[Spark Branch]

2015-07-03 Thread chengxiang li
) <https://reviews.apache.org/r/36156/#comment143335> this query is quite same as the previous one, we shoud just need one of thoese. - chengxiang li On 七月 3, 2015, 7:34 a.m., lun gao wrote: > > --- > This is an automati

Re: Review Request 36156: HIVE-11053: Add more tests for HIVE-10844[Spark Branch]

2015-07-03 Thread chengxiang li
) <https://reviews.apache.org/r/36156/#comment143334> drop temp table at the end. - chengxiang li On 七月 3, 2015, 7:34 a.m., lun gao wrote: > > --- > This is an automatically generated e-mail. To reply,

Re: Review Request 34666: HIVE-9152 - Dynamic Partition Pruning [Spark Branch]

2015-07-01 Thread chengxiang li
verwhelm its capability, DataOutputBuffer expand its byte array size by create a new byte array with 2x size and copy old one to new one. A estimated initial byte array size should be able to reduce most array copy. - chengxiang li On 五月 26, 2015, 4:

Re: Review Request 34666: HIVE-9152 - Dynamic Partition Pruning [Spark Branch]

2015-07-01 Thread chengxiang li
Log in error level should means some error happens,the process would be interrupted, if we really expect single field here, should we throw an exception while it has more? otherwise, we should downgrade the log level to WARN with more precise information. - chengxiang li On 五月 26, 2015, 4:28 p.

Re: Review Request 34757: HIVE-10844: Combine equivalent Works for HoS[Spark Branch]

2015-06-23 Thread chengxiang li
sure, introduced by latest merge from trunk. - chengxiang --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/34757/#review88966 --- On 六月 2

Re: Review Request 34757: HIVE-10844: Combine equivalent Works for HoS[Spark Branch]

2015-06-23 Thread chengxiang li
leave works output should be read by further SparkWork/FetchWork, we does not able to update work reference across SparkWork, so combine leave works may lead to error. - chengxiang --- This is an automatically generated e-mail. To reply, vi

Re: Review Request 34757: HIVE-10844: Combine equivalent Works for HoS[Spark Branch]

2015-06-23 Thread chengxiang li
spark/union_top_level.q.out dede1ef Diff: https://reviews.apache.org/r/34757/diff/ Testing --- Thanks, chengxiang li

Re: Review Request 34757: HIVE-10844: Combine equivalent Works for HoS[Spark Branch]

2015-06-22 Thread chengxiang li
o trans. > > If there are two links between a parent to a child, the input will be self > > unioned and the result is the input to the child. > > chengxiang li wrote: > Take self-join for example, there would be 2 MapWork connect to same > ReduceWork. if we combine t

[jira] [Created] (HIVE-11082) Support multi edge between nodes in SparkPlan[Spark Branch]

2015-06-22 Thread Chengxiang Li (JIRA)
Chengxiang Li created HIVE-11082: Summary: Support multi edge between nodes in SparkPlan[Spark Branch] Key: HIVE-11082 URL: https://issues.apache.org/jira/browse/HIVE-11082 Project: Hive

Re: Review Request 34757: HIVE-10844: Combine equivalent Works for HoS[Spark Branch]

2015-06-19 Thread chengxiang li
spark/union_top_level.q.out dede1ef Diff: https://reviews.apache.org/r/34757/diff/ Testing --- Thanks, chengxiang li

Re: Review Request 34757: HIVE-10844: Combine equivalent Works for HoS[Spark Branch]

2015-06-19 Thread chengxiang li
-- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/34757/#review88484 ----------- On June 17, 2015, 8:59 a.m., chengxiang li wrote: > > --

Re: Review Request 34757: HIVE-10844: Combine equivalent Works for HoS[Spark Branch]

2015-06-19 Thread chengxiang li
s into 1, SparkPlan::connect would throw exception during SparkPlan generation. - chengxiang --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/34757/#review88484 -------

[jira] [Created] (HIVE-11053) Add more tests for HIVE-10844[Spark Branch]

2015-06-18 Thread Chengxiang Li (JIRA)
Chengxiang Li created HIVE-11053: Summary: Add more tests for HIVE-10844[Spark Branch] Key: HIVE-11053 URL: https://issues.apache.org/jira/browse/HIVE-11053 Project: Hive Issue Type: Sub

Re: Review Request 34757: HIVE-10844: Combine equivalent Works for HoS[Spark Branch]

2015-06-17 Thread chengxiang li
/clientpositive/spark/union_top_level.q.out dede1ef Diff: https://reviews.apache.org/r/34757/diff/ Testing --- Thanks, chengxiang li

Review Request 34757: HIVE-10844: Combine equivalent Works for HoS[Spark Branch]

2015-05-27 Thread chengxiang li
--- Thanks, chengxiang li

[jira] [Created] (HIVE-10844) Combine equivalent Works for HoS[Spark Branch]

2015-05-27 Thread Chengxiang Li (JIRA)
Chengxiang Li created HIVE-10844: Summary: Combine equivalent Works for HoS[Spark Branch] Key: HIVE-10844 URL: https://issues.apache.org/jira/browse/HIVE-10844 Project: Hive Issue Type: Sub

Re: Review Request 34455: HIVE-10550 Dynamic RDD caching optimization for HoS.[Spark Branch]

2015-05-27 Thread chengxiang li
/java/org/apache/hadoop/hive/ql/exec/spark/SparkPlanGenerator.java 3f240f5 ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkUtilities.java e6c845c Diff: https://reviews.apache.org/r/34455/diff/ Testing --- Thanks, chengxiang li

Re: Review Request 34455: HIVE-10550 Dynamic RDD caching optimization for HoS.[Spark Branch]

2015-05-27 Thread chengxiang li
is later on. At this moment, I don't feel > > confident to make the call. > > chengxiang li wrote: > persistent to MEM + DISK may hurt the performance in certain cases, i > think at least we should have a switch to open/close this optimization, > > Xuefu Zhang wrote: >

Re: Review Request 34455: HIVE-10550 Dynamic RDD caching optimization for HoS.[Spark Branch]

2015-05-27 Thread chengxiang li
is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/34455/#review85451 ------- On 五月 27, 2015, 1:50 a.m., chengxiang li wrote: > > --- > Thi

Re: Review Request 34455: HIVE-10550 Dynamic RDD caching optimization for HoS.[Spark Branch]

2015-05-26 Thread chengxiang li
/hadoop/hive/ql/parse/spark/SparkCompiler.java 19aae70 ql/src/java/org/apache/hadoop/hive/ql/plan/SparkWork.java bb5dd79 Diff: https://reviews.apache.org/r/34455/diff/ Testing --- Thanks, chengxiang li

Re: Review Request 34455: HIVE-10550 Dynamic RDD caching optimization for HoS.[Spark Branch]

2015-05-21 Thread chengxiang li
/src/java/org/apache/hadoop/hive/ql/parse/spark/SparkCompiler.java 19aae70 ql/src/java/org/apache/hadoop/hive/ql/plan/SparkWork.java bb5dd79 Diff: https://reviews.apache.org/r/34455/diff/ Testing --- Thanks, chengxiang li

Re: Review Request 34455: HIVE-10550 Dynamic RDD caching optimization for HoS.[Spark Branch]

2015-05-21 Thread chengxiang li
e for share cached RDD cross Spark job. - chengxiang --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/34455/#review84572 --- On 五月 20, 2015, 2:37 a.

Review Request 34455: HIVE-10550 Dynamic RDD caching optimization for HoS.[Spark Branch]

2015-05-19 Thread chengxiang li
://reviews.apache.org/r/34455/diff/ Testing --- Thanks, chengxiang li

Re: Review Request 34293: HIVE-10721 SparkSessionManagerImpl leaks SparkSessions [Spark Branch]

2015-05-17 Thread chengxiang li
> On 五月 18, 2015, 2:37 a.m., chengxiang li wrote: > > ql/src/java/org/apache/hadoop/hive/ql/exec/spark/session/SparkSessionManagerImpl.java, > > line 88 > > <https://reviews.apache.org/r/34293/diff/1/?file=961679#file961679line88> > > > > Just curio

Re: Review Request 34293: HIVE-10721 SparkSessionManagerImpl leaks SparkSessions [Spark Branch]

2015-05-17 Thread chengxiang li
> On 五月 18, 2015, 2:26 a.m., chengxiang li wrote: > > ql/src/java/org/apache/hadoop/hive/ql/exec/spark/session/SparkSessionManagerImpl.java, > > line 96 > > <https://reviews.apache.org/r/34293/diff/1/?file=961679#file961679line96> > > > > SparkClientF

Re: Review Request 34293: HIVE-10721 SparkSessionManagerImpl leaks SparkSessions [Spark Branch]

2015-05-17 Thread chengxiang li
/SparkSessionManagerImpl.java <https://reviews.apache.org/r/34293/#comment135204> Just curious, it looks to me that AtomaticBoolean works here either, is that possible 2 threads executed this block both? - chengxiang li On 五月 15, 2015, 9:53 p.m., Jimmy Xiang

Re: Review Request 34293: HIVE-10721 SparkSessionManagerImpl leaks SparkSessions [Spark Branch]

2015-05-17 Thread chengxiang li
. This should be another issue, i just list here as it's found while read the code. - chengxiang li On 五月 15, 2015, 9:53 p.m., Jimmy Xiang wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https

[jira] [Created] (HIVE-10550) Dynamic RDD caching optimization for HoS.[Spark Branch]

2015-04-30 Thread Chengxiang Li (JIRA)
Chengxiang Li created HIVE-10550: Summary: Dynamic RDD caching optimization for HoS.[Spark Branch] Key: HIVE-10550 URL: https://issues.apache.org/jira/browse/HIVE-10550 Project: Hive Issue

Review Request 33119: HIVE-10235: Loop optimization for SIMD in ColumnDivideColumn.txt

2015-04-12 Thread chengxiang li
- trunk/itests/hive-jmh/src/main/java/org/apache/hive/benchmark/vectorization/VectorizationBench.java 1673092 trunk/ql/src/gen/vectorization/ExpressionTemplates/ColumnDivideColumn.txt 1673092 Diff: https://reviews.apache.org/r/33119/diff/ Testing --- Thanks, chengxiang li

Re: Review Request 32920: HIVE-10189: Create a micro benchmark tool for vectorization to evaluate the performance gain after SIMD optimization

2015-04-08 Thread chengxiang li
> On 四月 8, 2015, 9:22 a.m., chengxiang li wrote: > > itests/hive-jmh/src/main/java/org/apache/hive/benchmark/vectorization/VectorizationBench.java, > > line 73 > > <https://reviews.apache.org/r/32920/diff/3/?file=920776#file920776line73> > > > >

Re: Review Request 32920: HIVE-10189: Create a micro benchmark tool for vectorization to evaluate the performance gain after SIMD optimization

2015-04-08 Thread chengxiang li
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/32920/#review79341 --- Ship it! Ship It! - chengxiang li On 四月 8, 2015, 8:42 a.m

Re: Review Request 32920: HIVE-10189: Create a micro benchmark tool for vectorization to evaluate the performance gain after SIMD optimization

2015-04-08 Thread chengxiang li
/vectorization/VectorizationBench.java <https://reviews.apache.org/r/32920/#comment128590> This static variables is specified for expressions of 2 paramater operator, i think we can move it to each setup() method. - chengxiang li On 四月 8, 2015, 8:42 a.m., cheng xu

[jira] [Created] (HIVE-10238) Loop optimization for SIMD in IfExprColumnColumn.txt

2015-04-07 Thread Chengxiang Li (JIRA)
Chengxiang Li created HIVE-10238: Summary: Loop optimization for SIMD in IfExprColumnColumn.txt Key: HIVE-10238 URL: https://issues.apache.org/jira/browse/HIVE-10238 Project: Hive Issue Type

Re: Review Request 32918: HIVE-10180 Loop optimization for SIMD in ColumnArithmeticColumn.txt

2015-04-07 Thread chengxiang li
trunk/ql/src/gen/vectorization/ExpressionTemplates/ColumnArithmeticColumn.txt 1671736 Diff: https://reviews.apache.org/r/32918/diff/ Testing --- Thanks, chengxiang li

Re: Review Request 32920: HIVE-10189: Create a micro benchmark tool for vectorization to evaluate the performance gain after SIMD optimization

2015-04-07 Thread chengxiang li
/vectorization/VectorizationBench.java <https://reviews.apache.org/r/32920/#comment128267> The benchmark look good, my only concern is that how could we expand this benchmark to other expressions? - chengxiang li On April 7, 2015, 6:06 a.m., cheng xu

[jira] [Created] (HIVE-10235) Loop optimization for SIMD in ColumnDivideColumn.txt

2015-04-06 Thread Chengxiang Li (JIRA)
Chengxiang Li created HIVE-10235: Summary: Loop optimization for SIMD in ColumnDivideColumn.txt Key: HIVE-10235 URL: https://issues.apache.org/jira/browse/HIVE-10235 Project: Hive Issue Type

Review Request 32918: HIVE-10180 Loop optimization for SIMD in ColumnArithmeticColumn.txt

2015-04-06 Thread chengxiang li
671736 Diff: https://reviews.apache.org/r/32918/diff/ Testing --- Thanks, chengxiang li

[jira] [Created] (HIVE-10180) Loop optimization in ColumnArithmeticColumn.txt

2015-04-01 Thread Chengxiang Li (JIRA)
Chengxiang Li created HIVE-10180: Summary: Loop optimization in ColumnArithmeticColumn.txt Key: HIVE-10180 URL: https://issues.apache.org/jira/browse/HIVE-10180 Project: Hive Issue Type: Sub

[jira] [Created] (HIVE-10179) Optimization for SIMD instructions in Hive

2015-04-01 Thread Chengxiang Li (JIRA)
Chengxiang Li created HIVE-10179: Summary: Optimization for SIMD instructions in Hive Key: HIVE-10179 URL: https://issues.apache.org/jira/browse/HIVE-10179 Project: Hive Issue Type

[jira] [Created] (HIVE-10052) HiveInputFormat implementations getsplits may lead to memory leak.[Spark Branch]

2015-03-22 Thread Chengxiang Li (JIRA)
Chengxiang Li created HIVE-10052: Summary: HiveInputFormat implementations getsplits may lead to memory leak.[Spark Branch] Key: HIVE-10052 URL: https://issues.apache.org/jira/browse/HIVE-10052

Review Request 32288: HIVE-10006 RSC has memory leak while execute multi queries

2015-03-19 Thread chengxiang li
he/hadoop/hive/serde2/typeinfo/TypeInfoFactory.java 1667894 branches/spark/serde/src/java/org/apache/hadoop/hive/serde2/typeinfo/TypeInfoUtils.java 1667894 Diff: https://reviews.apache.org/r/32288/diff/ Testing --- Thanks, chengxiang li

[jira] [Created] (HIVE-10006) RSC has memory leak while execute multi queries.[Spark Branch]

2015-03-18 Thread Chengxiang Li (JIRA)
Chengxiang Li created HIVE-10006: Summary: RSC has memory leak while execute multi queries.[Spark Branch] Key: HIVE-10006 URL: https://issues.apache.org/jira/browse/HIVE-10006 Project: Hive

[jira] [Updated] (HIVE-9425) External Function Jar files are not available for Driver when running with yarn-cluster mode [Spark Branch]

2015-02-04 Thread Chengxiang Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-9425?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chengxiang Li updated HIVE-9425: Assignee: Rui Li (was: Chengxiang Li) > External Function Jar files are not available for Dri

[jira] [Commented] (HIVE-9410) ClassNotFoundException occurs during hive query case execution with UDF defined [Spark Branch]

2015-02-03 Thread Chengxiang Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-9410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14304382#comment-14304382 ] Chengxiang Li commented on HIVE-9410: - Not actually, as you can see from the patc

[jira] [Updated] (HIVE-9211) Research on build mini HoS cluster on YARN for unit test[Spark Branch]

2015-02-01 Thread Chengxiang Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-9211?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chengxiang Li updated HIVE-9211: Attachment: HIVE-9211.7-spark.patch TestSparkCliDriver launch local spark cluster with \[2,2,1024

[jira] [Updated] (HIVE-9542) SparkSessionImpl calcualte wrong cores number in TestSparkCliDriver [Spark Branch]

2015-02-01 Thread Chengxiang Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-9542?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chengxiang Li updated HIVE-9542: Summary: SparkSessionImpl calcualte wrong cores number in TestSparkCliDriver [Spark Branch] (was

[jira] [Created] (HIVE-9542) SparkSessionImpl calcualte wrong number of cores number in TestSparkCliDriver [Spark Branch]

2015-02-01 Thread Chengxiang Li (JIRA)
Chengxiang Li created HIVE-9542: --- Summary: SparkSessionImpl calcualte wrong number of cores number in TestSparkCliDriver [Spark Branch] Key: HIVE-9542 URL: https://issues.apache.org/jira/browse/HIVE-9542

[jira] [Updated] (HIVE-9211) Research on build mini HoS cluster on YARN for unit test[Spark Branch]

2015-02-01 Thread Chengxiang Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-9211?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chengxiang Li updated HIVE-9211: Attachment: HIVE-9211.6-spark.patch [~xuefuz], the output of infer_bucket_sort_dyn_part.q changes

[jira] [Created] (HIVE-9540) Enable infer_bucket_sort_dyn_part.q for TestMiniSparkOnYarnCliDriver test. [Spark Branch]

2015-02-01 Thread Chengxiang Li (JIRA)
Chengxiang Li created HIVE-9540: --- Summary: Enable infer_bucket_sort_dyn_part.q for TestMiniSparkOnYarnCliDriver test. [Spark Branch] Key: HIVE-9540 URL: https://issues.apache.org/jira/browse/HIVE-9540

[jira] [Commented] (HIVE-9211) Research on build mini HoS cluster on YARN for unit test[Spark Branch]

2015-01-30 Thread Chengxiang Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-9211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14298642#comment-14298642 ] Chengxiang Li commented on HIVE-9211: - I build spark v1.2.0 with -Dhadoop.ver

[jira] [Updated] (HIVE-9211) Research on build mini HoS cluster on YARN for unit test[Spark Branch]

2015-01-30 Thread Chengxiang Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-9211?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chengxiang Li updated HIVE-9211: Attachment: HIVE-9211.5-spark.patch > Research on build mini HoS cluster on YARN for unit test[Sp

[jira] [Commented] (HIVE-9211) Research on build mini HoS cluster on YARN for unit test[Spark Branch]

2015-01-30 Thread Chengxiang Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-9211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14298426#comment-14298426 ] Chengxiang Li commented on HIVE-9211: - Hi, [~brocknoland], the missed class is

[jira] [Updated] (HIVE-9449) Push YARN configuration to Spark while deply Spark on YARN[Spark Branch]

2015-01-30 Thread Chengxiang Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-9449?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chengxiang Li updated HIVE-9449: Attachment: HiveonSparkconfiguration.pdf > Push YARN configuration to Spark while deply Spark

[jira] [Updated] (HIVE-9211) Research on build mini HoS cluster on YARN for unit test[Spark Branch]

2015-01-29 Thread Chengxiang Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-9211?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chengxiang Li updated HIVE-9211: Attachment: HIVE-9211.4-spark.patch [~brocknoland], what code base is our current Spark installation

[jira] [Commented] (HIVE-9487) Make Remote Spark Context secure [Spark Branch]

2015-01-28 Thread Chengxiang Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-9487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14296395#comment-14296395 ] Chengxiang Li commented on HIVE-9487: - +1, the patch looks good to me. > Make

[jira] [Commented] (HIVE-9211) Research on build mini HoS cluster on YARN for unit test[Spark Branch]

2015-01-27 Thread Chengxiang Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-9211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14293613#comment-14293613 ] Chengxiang Li commented on HIVE-9211: - No log files found in the container

[jira] [Updated] (HIVE-9211) Research on build mini HoS cluster on YARN for unit test[Spark Branch]

2015-01-27 Thread Chengxiang Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-9211?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chengxiang Li updated HIVE-9211: Attachment: HIVE-9211.3-spark.patch > Research on build mini HoS cluster on YARN for unit test[Sp

[jira] [Commented] (HIVE-9211) Research on build mini HoS cluster on YARN for unit test[Spark Branch]

2015-01-26 Thread Chengxiang Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-9211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14293070#comment-14293070 ] Chengxiang Li commented on HIVE-9211: - Great, thanks, [~brocknoland]. > Rese

[jira] [Commented] (HIVE-9211) Research on build mini HoS cluster on YARN for unit test[Spark Branch]

2015-01-26 Thread Chengxiang Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-9211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14293057#comment-14293057 ] Chengxiang Li commented on HIVE-9211: - I work on Linux. > Research on build m

[jira] [Commented] (HIVE-9211) Research on build mini HoS cluster on YARN for unit test[Spark Branch]

2015-01-26 Thread Chengxiang Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-9211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14293022#comment-14293022 ] Chengxiang Li commented on HIVE-9211: - >From hive.log, seems like some error

[jira] [Assigned] (HIVE-9425) External Function Jar files are not available for Driver when running with yarn-cluster mode [Spark Branch]

2015-01-26 Thread Chengxiang Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-9425?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chengxiang Li reassigned HIVE-9425: --- Assignee: Chengxiang Li > External Function Jar files are not available for Driver w

[jira] [Updated] (HIVE-9211) Research on build mini HoS cluster on YARN for unit test[Spark Branch]

2015-01-26 Thread Chengxiang Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-9211?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chengxiang Li updated HIVE-9211: Attachment: (was: HIVE-9211.2-spark.patch) > Research on build mini HoS cluster on YARN for u

[jira] [Updated] (HIVE-9211) Research on build mini HoS cluster on YARN for unit test[Spark Branch]

2015-01-26 Thread Chengxiang Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-9211?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chengxiang Li updated HIVE-9211: Attachment: HIVE-9211.2-spark.patch > Research on build mini HoS cluster on YARN for unit test[Sp

Re: Review Request 30264: HIVE-9221 enable unit test for mini Spark on YARN cluster[Spark Branch]

2015-01-26 Thread chengxiang li
: https://reviews.apache.org/r/30264/diff/ Testing --- Thanks, chengxiang li

[jira] [Updated] (HIVE-9211) Research on build mini HoS cluster on YARN for unit test[Spark Branch]

2015-01-26 Thread Chengxiang Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-9211?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chengxiang Li updated HIVE-9211: Status: Patch Available (was: Open) > Research on build mini HoS cluster on YARN for unit t

[jira] [Updated] (HIVE-9211) Research on build mini HoS cluster on YARN for unit test[Spark Branch]

2015-01-26 Thread Chengxiang Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-9211?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chengxiang Li updated HIVE-9211: Attachment: HIVE-9211.2-spark.patch > Research on build mini HoS cluster on YARN for unit test[Sp

Re: Review Request 30264: HIVE-9221 enable unit test for mini Spark on YARN cluster[Spark Branch]

2015-01-26 Thread chengxiang li
ke sense. On 一月 26, 2015, 10:30 p.m., chengxiang li wrote: > > I'm wondering why we have a new set of .out files? Every Test*CliDriver has its own output directory, i didn't think much about this previously. With your remind, i think, yes, we could share the golden files with

Re: Review Request 30264: HIVE-9221 enable unit test for mini Spark on YARN cluster[Spark Branch]

2015-01-25 Thread chengxiang li
/ Testing --- Thanks, chengxiang li

Review Request 30264: HIVE-9221 enable unit test for mini Spark on YARN cluster[Spark Branch]

2015-01-25 Thread chengxiang li
/MiniSparkOnYARNCluster.java PRE-CREATION shims/common/src/main/java/org/apache/hadoop/hive/shims/HadoopShims.java 064304c spark-client/src/main/java/org/apache/hive/spark/client/SparkClientImpl.java aea90db Diff: https://reviews.apache.org/r/30264/diff/ Testing --- Thanks, chengxiang

[jira] [Updated] (HIVE-9211) Research on build mini HoS cluster on YARN for unit test[Spark Branch]

2015-01-25 Thread Chengxiang Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-9211?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chengxiang Li updated HIVE-9211: Attachment: HIVE-9211.1-spark.patch > Research on build mini HoS cluster on YARN for unit test[Sp

Re: Review Request 30208: HIVE-9449 Push YARN configuration to Spark while deply Spark on YARN[Spark Branch]

2015-01-25 Thread chengxiang li
. Diffs (updated) - common/src/java/org/apache/hadoop/hive/conf/HiveConf.java d4d98d7 ql/src/java/org/apache/hadoop/hive/ql/exec/spark/HiveSparkClientFactory.java 9dc6c47 Diff: https://reviews.apache.org/r/30208/diff/ Testing --- Thanks, chengxiang li

[jira] [Updated] (HIVE-9449) Push YARN configuration to Spark while deply Spark on YARN[Spark Branch]

2015-01-25 Thread Chengxiang Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-9449?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chengxiang Li updated HIVE-9449: Attachment: HIVE-9449.2-spark.patch > Push YARN configuration to Spark while deply Spark on Y

Review Request 30208: HIVE-9449 Push YARN configuration to Spark while deply Spark on YARN[Spark Branch]

2015-01-22 Thread chengxiang li
ql/src/java/org/apache/hadoop/hive/ql/exec/spark/HiveSparkClientFactory.java 9dc6c47 Diff: https://reviews.apache.org/r/30208/diff/ Testing --- Thanks, chengxiang li

[jira] [Updated] (HIVE-9449) Push YARN configuration to Spark while deply Spark on YARN[Spark Branch]

2015-01-22 Thread Chengxiang Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-9449?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chengxiang Li updated HIVE-9449: Attachment: HIVE-9449.1-spark.patch > Push YARN configuration to Spark while deply Spark on Y

[jira] [Updated] (HIVE-9449) Push YARN configuration to Spark while deply Spark on YARN[Spark Branch]

2015-01-22 Thread Chengxiang Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-9449?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chengxiang Li updated HIVE-9449: Status: Patch Available (was: Open) > Push YARN configuration to Spark while deply Spark on Y

[jira] [Created] (HIVE-9449) Push YARN configuration to Spark while deply Spark on YARN[Spark Branch]

2015-01-22 Thread Chengxiang Li (JIRA)
Chengxiang Li created HIVE-9449: --- Summary: Push YARN configuration to Spark while deply Spark on YARN[Spark Branch] Key: HIVE-9449 URL: https://issues.apache.org/jira/browse/HIVE-9449 Project: Hive

Re: Review Request 30107: HIVE-9410, ClassNotFoundException occurs during hive query case execution with UDF defined[Spark Branch]

2015-01-22 Thread chengxiang li
tImpl.java 1eb3ff2 spark-client/src/main/java/org/apache/hive/spark/client/SparkClientImpl.java 5f9be65 spark-client/src/main/java/org/apache/hive/spark/client/SparkClientUtilities.java PRE-CREATION Diff: https://reviews.apache.org/r/30107/diff/ Testing --- Thanks, chengxiang li

[jira] [Updated] (HIVE-9410) ClassNotFoundException occurs during hive query case execution with UDF defined [Spark Branch]

2015-01-22 Thread Chengxiang Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-9410?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chengxiang Li updated HIVE-9410: Attachment: (was: HIVE-9410.4-spark.patch) > ClassNotFoundException occurs during hive qu

[jira] [Updated] (HIVE-9410) ClassNotFoundException occurs during hive query case execution with UDF defined [Spark Branch]

2015-01-22 Thread Chengxiang Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-9410?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chengxiang Li updated HIVE-9410: Attachment: HIVE-9410.4-spark.patch > ClassNotFoundException occurs during hive query case execut

Re: Review Request 30107: HIVE-9410, ClassNotFoundException occurs during hive query case execution with UDF defined[Spark Branch]

2015-01-22 Thread chengxiang li
apache.org/r/30107/diff/ Testing --- Thanks, chengxiang li

[jira] [Updated] (HIVE-9410) ClassNotFoundException occurs during hive query case execution with UDF defined [Spark Branch]

2015-01-22 Thread Chengxiang Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-9410?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chengxiang Li updated HIVE-9410: Attachment: HIVE-9410.4-spark.patch > ClassNotFoundException occurs during hive query case execut

[jira] [Commented] (HIVE-9410) ClassNotFoundException occurs during hive query case execution with UDF defined [Spark Branch]

2015-01-22 Thread Chengxiang Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-9410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14288848#comment-14288848 ] Chengxiang Li commented on HIVE-9410: - As ser/deser between Hive driver and re

[jira] [Commented] (HIVE-9410) ClassNotFoundException occurs during hive query case execution with UDF defined [Spark Branch]

2015-01-22 Thread Chengxiang Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-9410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14288797#comment-14288797 ] Chengxiang Li commented on HIVE-9410: - Yes, Spark would address this issue

Re: Review Request 30107: HIVE-9410, ClassNotFoundException occurs during hive query case execution with UDF defined[Spark Branch]

2015-01-22 Thread chengxiang li
> On 一月 23, 2015, 3:02 a.m., chengxiang li wrote: > > ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java, line 371 > > <https://reviews.apache.org/r/30107/diff/4/?file=829688#file829688line371> > > > > #3 this would be executed in akka thread, get extr

Re: Review Request 30107: HIVE-9410, ClassNotFoundException occurs during hive query case execution with UDF defined[Spark Branch]

2015-01-22 Thread chengxiang li
that adds the jars to the classpath of the remote > > driver? > > > > I'm wondering why these jars are necessary in order to deserailize > > SparkWork. > > chengxiang li wrote: > Same as previous comments, SparkWork contains MapWork/ReduceWo

[jira] [Commented] (HIVE-9370) SparkJobMonitor timeout as sortByKey would launch extra Spark job before original job get submitted [Spark Branch]

2015-01-22 Thread Chengxiang Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-9370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14288684#comment-14288684 ] Chengxiang Li commented on HIVE-9370: - RSC have timeout in netty level, so if re

Re: Review Request 30107: HIVE-9410, ClassNotFoundException occurs during hive query case execution with UDF defined[Spark Branch]

2015-01-22 Thread chengxiang li
/#comment114012> #1 add extra jar path to JobContext, this job is executed in netty connection thread. - chengxiang li On 一月 22, 2015, 9:23 a.m., chengxiang li wrote: > > --- > This is an automatically generated e-mai

Re: Review Request 30107: HIVE-9410, ClassNotFoundException occurs during hive query case execution with UDF defined[Spark Branch]

2015-01-22 Thread chengxiang li
ame as previous comments, SparkWork contains MapWork/ReduceWork which contains operator tree, UTFFOperator need to load added jar class. - chengxiang --- This is an automatically generated e-mail. To reply, visit: http

  1   2   3   4   5   6   7   8   >