Re: Precommit unstable?

2015-12-21 Thread Thejas Nair 
Forgot to get back on this one. That delay was actually caused by a
hanging test, it was addressed here -
https://issues.apache.org/jira/browse/HIVE-12560


On Wed, Dec 2, 2015 at 10:08 AM, Sergio Pena  wrote:
> I see the last tests run in average of 1 hour. Should I still restart
> Jenkins?
>
> On Tue, Dec 1, 2015 at 12:11 PM, Thejas Nair  wrote:
>>
>> I killed the current 10+hr test run to see if next test would run
>> faster. (Discussed with Prasanth, the author of patch being tested).
>>
>> However,  I doubt if the next one is going to perform any better. If
>> we could try restarting the ptest2 system, I think that might be
>> better thing to do (instead of waiting several hours to realize that
>> the issue has not gone away).
>>
>>
>> On Tue, Dec 1, 2015 at 10:00 AM, Thejas Nair 
>> wrote:
>> > Would restart of jenkins or the hosts help with this ?
>> >
>> >
>> > On Tue, Dec 1, 2015 at 9:59 AM, Thejas Nair 
>> > wrote:
>> >> The build running right now has been running for almost 10 hours.
>> >> I am wondering if there is something wrong with the hosts being used
>> >> right now. Some commands seem to be taking longer and I get some
>> >> intermittent errors from the jenkins UI.
>> >>
>> >>
>> >> http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6187/
>> >> This rsync command seems to have taken a long time to finish -
>> >>
>> >> 2015-12-01 11:42:41,394  INFO LocalCommand.:35 Starting timeout
>> >> 1h rsync -vaPe "ssh -i /home/hiveptest/.ssh/hive-ptest-user-key"
>> >> --timeout 600
>> >> hiveptest@54.159.202.234:/home/hiveptest/54.159.202.234-hiveptest-0/logs/
>> >>
>> >> /data/hive-ptest/logs/PreCommit-HIVE-TRUNK-Build-6187/succeeded/TestHBaseMinimrCliDriver
>> >> 2015-12-01 12:54:52,522  INFO LocalCommand.:35 Starting timeout
>> >> 1h rsync -vaPe "ssh -i /home/hiveptest/.ssh/hive-ptest-user-key"
>> >> --timeout 600
>> >> hiveptest@54.157.22.221:/home/hiveptest/54.157.22.221-hiveptest-0/logs/
>> >>
>> >> /data/hive-ptest/logs/PreCommit-HIVE-TRUNK-Build-6187/failed/TestMiniTezCliDriver-script_pipe.q-tez_smb_empty.q-mapjoin_decimal.q-and-12-more
>> >> 2015-12-01 12:54:54,240  INFO HostExecutor.executeTests:176 Starting
>> >> isolated execution on 54.157.22.221
>> >>
>> >>
>> >>
>> >>
>> >>
>> >> On Tue, Dec 1, 2015 at 8:57 AM, Sergio Pena 
>> >> wrote:
>> >>> I see HIVE-12541 files were attached 4 times on JIRA while 'Patch
>> >>> Available' status was set. I does not say at what time, but Jenkins
>> >>> runs a
>> >>> job every 5 min to detect which files must be tested for all "Patch
>> >>> Available' JIRAS. That's why you probably saw HIVE-12541 in the queue
>> >>> many
>> >>> times.
>> >>>
>> >>> I would recommend to set the JIRA to 'Patch Available' when the patch
>> >>> is
>> >>> reviewed to avoid many patches to be added to the queue.
>> >>>
>> >>> On Mon, Nov 30, 2015 at 11:09 PM, Prasanth J 
>> >>> wrote:
>> >>>
>>  Hi All
>> 
>>  I am noticing something weird in precommit queue. I can see
>>  HIVE-12541currenly running, but is also waiting next in the queue.
>>  The same
>>  jira ran few more times recently. I cannot see any other patches in
>>  the
>>  queue. Sometime I see bunch of patches in the queue but it will soon
>>  be
>>  removed. But I am seeing HIVE-12541 running quite often. Has anyone
>>  seen
>>  this issue?
>> 
>>  Thanks
>>  Prasanth
>
>

[jira] [Created] (HIVE-12722) Create abstract subclass for HiveAuthorizer to shield implementations from interface changes

2015-12-21 Thread Thejas M Nair (JIRA) 
Thejas M Nair created HIVE-12722:


 Summary: Create abstract subclass for HiveAuthorizer to shield 
implementations from interface changes
 Key: HIVE-12722
 URL: https://issues.apache.org/jira/browse/HIVE-12722
 Project: Hive
  Issue Type: Bug
  Components: Authorization
Reporter: Thejas M Nair
Assignee: Thejas M Nair


A Abstract class that extends HiveAuthorizer will help to shield Hive 
authorization implementations from some of the changes to HiveAuthorizer 
interface by providing default implementation of new methods in HiveAuthorizer 
when possible.




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Precommit result failures snowballing ?

2015-12-21 Thread Thejas Nair 
Some recent failures -

org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join_stats2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_index_auto_partitioned
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_filemetadata
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver_encryption_insert_partition_dynamic
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_stats_filemetadata
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_uri_import
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_columnstats_partlvl_multiple_part_clause
org.apache.hadoop.hive.ql.exec.spark.session.TestSparkSessionManagerImpl.testMultiSessionMultipleUse
org.apache.hadoop.hive.ql.exec.spark.session.TestSparkSessionManagerImpl.testSingleSessionMultipleUse
org.apache.hadoop.hive.ql.security.authorization.plugin.TestHiveOperationType.checkHiveOperationTypeMatch
org.apache.hive.jdbc.TestSSL.testSSLVersion
org.apache.hive.spark.client.TestSparkClient.testAddJarsAndFiles
org.apache.hive.spark.client.TestSparkClient.testCounters
org.apache.hive.spark.client.TestSparkClient.testErrorJob
org.apache.hive.spark.client.TestSparkClient.testJobSubmission
org.apache.hive.spark.client.TestSparkClient.testMetricsCollection
org.apache.hive.spark.client.TestSparkClient.testRemoteClient
org.apache.hive.spark.client.TestSparkClient.testSimpleSparkJob
org.apache.hive.spark.client.TestSparkClient.testSyncRpc


On Mon, Dec 21, 2015 at 10:44 AM, Thejas Nair  wrote:
> The number of tests that fail in precommit seems to be on the rise.
> The larger the number of seemingly unrelated failures to a patch, the
> higher are the chances of missing out on actual failures caused by a
> the patch, resulting in further increase in failures.
>
> I think we need to give this some attention soon. Should we create an
> umbrella jira to track the current set ?
>
> Are there any bugs already open to track them ? I didn't see any when
> i searched by component test/tests.
>
> Thanks,
> Thejas

Re: Review Request 41377: HIVE-12528 don't start HS2 Tez sessions in a single thread

2015-12-21 Thread Sergey Shelukhin 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/41377/#review111539
---



ql/src/java/org/apache/hadoop/hive/ql/exec/tez/DagUtils.java (line 974)


Added a field comment



ql/src/java/org/apache/hadoop/hive/ql/exec/tez/DagUtils.java (line 987)


The old path would just re-check the file immediately before sleep. This 
actually adds an extra wait :) 
We need to check HDFS anyway, so I think it's faster to go into IOEx path 
than to wait in cases where it won't be helpful.



ql/src/java/org/apache/hadoop/hive/ql/exec/tez/DagUtils.java (line 1025)


Where? In case of IOEx path above it's checked before every sleep, and 
after the last sleep. In case of in-process collision it's checked once before 
and after sleep.



ql/src/java/org/apache/hadoop/hive/ql/exec/tez/DagUtils.java (line 1032)


moved the loop into the method


- Sergey Shelukhin


On Dec. 18, 2015, 10:10 p.m., Sergey Shelukhin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/41377/
> ---
> 
> (Updated Dec. 18, 2015, 10:10 p.m.)
> 
> 
> Review request for hive, Gunther Hagleitner and Siddharth Seth.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> see JIRA
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 36e281a 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/DagUtils.java 6e196e6 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionPoolManager.java 
> 0d84340 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionState.java e1a8041 
>   ql/src/test/org/apache/hadoop/hive/ql/exec/tez/TestTezSessionPool.java 
> 11c0325 
> 
> Diff: https://reviews.apache.org/r/41377/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Sergey Shelukhin
> 
>

[jira] [Created] (HIVE-12721) Add UUID built in function

2015-12-21 Thread Jeremy Beard (JIRA) 
Jeremy Beard created HIVE-12721:
---

 Summary: Add UUID built in function
 Key: HIVE-12721
 URL: https://issues.apache.org/jira/browse/HIVE-12721
 Project: Hive
  Issue Type: Improvement
  Components: UDF
Reporter: Jeremy Beard
Assignee: Jeremy Beard


A UUID function would be very useful for ETL jobs that need to generate 
surrogate keys.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Precommit result failures snowballing ?

2015-12-21 Thread Thejas Nair 
The number of tests that fail in precommit seems to be on the rise.
The larger the number of seemingly unrelated failures to a patch, the
higher are the chances of missing out on actual failures caused by a
the patch, resulting in further increase in failures.

I think we need to give this some attention soon. Should we create an
umbrella jira to track the current set ?

Are there any bugs already open to track them ? I didn't see any when
i searched by component test/tests.

Thanks,
Thejas

[GitHub] hive pull request: Hive 12722

2015-12-21 Thread thejasmn 
GitHub user thejasmn opened a pull request:

https://github.com/apache/hive/pull/59

Hive 12722



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/thejasmn/hive HIVE-12722

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/hive/pull/59.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #59


commit d86c776df6194bbae33ccd6ec8c62615bf5354db
Author: Thejas Nair 
Date:   2015-12-21T20:17:06Z

HIVE-12722 - adding abstract class for HiveAuthorizer

commit 04abbf53b2c2506df75bf8b7150cad6df50cd548
Author: Thejas Nair 
Date:   2015-12-21T20:26:08Z

HIVE-12722 - remove the function from subclasses




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (HIVE-12723) stats_filemetadata test was added to the wrong driver

2015-12-21 Thread Sergey Shelukhin (JIRA) 
Sergey Shelukhin created HIVE-12723:
---

 Summary: stats_filemetadata test was added to the wrong driver
 Key: HIVE-12723
 URL: https://issues.apache.org/jira/browse/HIVE-12723
 Project: Hive
  Issue Type: Bug
Reporter: Sergey Shelukhin
Assignee: Sergey Shelukhin


HBase metastore is only used in MiniTez, but the test somehow got added to 
MiniLlap



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 40865: Implement/Enable constant related optimization rules in Calcite: enable HiveReduceExpressionsRule to fold constants

2015-12-21 Thread John Pullokkaran 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40865/#review111537
---

Ship it!


Ship It!

- John Pullokkaran


On Dec. 21, 2015, 5:44 p.m., pengcheng xiong wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40865/
> ---
> 
> (Updated Dec. 21, 2015, 5:44 p.m.)
> 
> 
> Review request for hive and John Pullokkaran.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-11927
> 
> 
> Diffs
> -
> 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/CalciteSemanticException.java
>  336745b 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/HiveRexExecutorImpl.java
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveReduceExpressionsRule.java
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/ASTBuilder.java
>  425514d 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/RexNodeConverter.java
>  631a4ca 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java 87b18b7 
>   ql/src/test/queries/clientpositive/cbo_const.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/constantfolding.q PRE-CREATION 
>   ql/src/test/results/clientpositive/annotate_stats_select.q.out c4d59c8 
>   ql/src/test/results/clientpositive/bucketizedhiveinputformat.q.out 277b0f7 
>   ql/src/test/results/clientpositive/cast1.q.out 0bdecba 
>   ql/src/test/results/clientpositive/cbo_const.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/cbo_rp_cross_product_check_2.q.out 
> 13d8a50 
>   ql/src/test/results/clientpositive/cbo_rp_lineage2.q.out 79f76bf 
>   ql/src/test/results/clientpositive/constantfolding.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/cross_product_check_1.q.out d9143c8 
>   ql/src/test/results/clientpositive/cross_product_check_2.q.out 4e2b93e 
>   ql/src/test/results/clientpositive/dynamic_rdd_cache.q.out 3153c7e 
>   ql/src/test/results/clientpositive/dynpart_sort_optimization2.q.out 24ac550 
>   ql/src/test/results/clientpositive/groupby_ppd.q.out 6164a26 
>   ql/src/test/results/clientpositive/groupby_sort_1_23.q.out 7333677 
>   ql/src/test/results/clientpositive/groupby_sort_skew_1_23.q.out e19d1de 
>   ql/src/test/results/clientpositive/input_part1.q.out d6f4d3e 
>   ql/src/test/results/clientpositive/input_part5.q.out f2d7335 
>   ql/src/test/results/clientpositive/input_part6.q.out fa51cdf 
>   ql/src/test/results/clientpositive/lineage2.q.out ec8b76b 
>   ql/src/test/results/clientpositive/lineage3.q.out 747dc9a 
>   ql/src/test/results/clientpositive/list_bucket_query_oneskew_2.q.out 
> be77ba8 
>   ql/src/test/results/clientpositive/llap/bucket_map_join_tez1.q.out 79348f3 
>   ql/src/test/results/clientpositive/orc_predicate_pushdown.q.out f6d8388 
>   ql/src/test/results/clientpositive/parquet_predicate_pushdown.q.out b322ef1 
>   ql/src/test/results/clientpositive/partition_multilevels.q.out c1c8778 
>   ql/src/test/results/clientpositive/perf/query31.q.out 52cbdf4 
>   ql/src/test/results/clientpositive/perf/query39.q.out a18cdaf 
>   ql/src/test/results/clientpositive/perf/query42.q.out 3954829 
>   ql/src/test/results/clientpositive/perf/query52.q.out 63f0bf6 
>   ql/src/test/results/clientpositive/perf/query64.q.out a63 
>   ql/src/test/results/clientpositive/perf/query66.q.out 22eaf61 
>   ql/src/test/results/clientpositive/perf/query75.q.out d54000b 
>   ql/src/test/results/clientpositive/pointlookup2.q.out a442425 
>   ql/src/test/results/clientpositive/quotedid_basic.q.out 519f647 
>   ql/src/test/results/clientpositive/quotedid_partition.q.out d34a005 
>   ql/src/test/results/clientpositive/spark/bucket_map_join_tez1.q.out b5e7846 
>   ql/src/test/results/clientpositive/spark/cross_product_check_1.q.out 
> 0656cd5 
>   ql/src/test/results/clientpositive/spark/cross_product_check_2.q.out 
> 2d6eb38 
>   ql/src/test/results/clientpositive/spark/dynamic_rdd_cache.q.out 9043fb1 
>   ql/src/test/results/clientpositive/spark/groupby_sort_1_23.q.out 239e803 
>   ql/src/test/results/clientpositive/spark/groupby_sort_skew_1_23.q.out 
> 8370bbe 
>   ql/src/test/results/clientpositive/spark/union_remove_25.q.out b771fe9 
>   ql/src/test/results/clientpositive/spark/union_view.q.out cce7710 
>   ql/src/test/results/clientpositive/subquery_notin.q.out ed86079 
>   ql/src/test/results/clientpositive/subquery_notin_having.q.java1.7.out 
> d6c6edc 
>   ql/src/test/results/clientpositive/subquery_unqualcolumnrefs.q.out ad5f72b 
>   ql/src/test/results/clientpositive/tez/bucket_map_join_tez1.q.out 9582334 
>   ql/src/test/results/clientpositive/tez/cross_product_check_1.q.out ca44973 
>

Re: Review Request 40865: Implement/Enable constant related optimization rules in Calcite: enable HiveReduceExpressionsRule to fold constants

2015-12-21 Thread John Pullokkaran 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40865/#review111536
---



ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveReduceExpressionsRule.java
 (line 320)


Remove this code; seems like not used.


- John Pullokkaran


On Dec. 21, 2015, 5:44 p.m., pengcheng xiong wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40865/
> ---
> 
> (Updated Dec. 21, 2015, 5:44 p.m.)
> 
> 
> Review request for hive and John Pullokkaran.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-11927
> 
> 
> Diffs
> -
> 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/CalciteSemanticException.java
>  336745b 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/HiveRexExecutorImpl.java
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveReduceExpressionsRule.java
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/ASTBuilder.java
>  425514d 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/RexNodeConverter.java
>  631a4ca 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java 87b18b7 
>   ql/src/test/queries/clientpositive/cbo_const.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/constantfolding.q PRE-CREATION 
>   ql/src/test/results/clientpositive/annotate_stats_select.q.out c4d59c8 
>   ql/src/test/results/clientpositive/bucketizedhiveinputformat.q.out 277b0f7 
>   ql/src/test/results/clientpositive/cast1.q.out 0bdecba 
>   ql/src/test/results/clientpositive/cbo_const.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/cbo_rp_cross_product_check_2.q.out 
> 13d8a50 
>   ql/src/test/results/clientpositive/cbo_rp_lineage2.q.out 79f76bf 
>   ql/src/test/results/clientpositive/constantfolding.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/cross_product_check_1.q.out d9143c8 
>   ql/src/test/results/clientpositive/cross_product_check_2.q.out 4e2b93e 
>   ql/src/test/results/clientpositive/dynamic_rdd_cache.q.out 3153c7e 
>   ql/src/test/results/clientpositive/dynpart_sort_optimization2.q.out 24ac550 
>   ql/src/test/results/clientpositive/groupby_ppd.q.out 6164a26 
>   ql/src/test/results/clientpositive/groupby_sort_1_23.q.out 7333677 
>   ql/src/test/results/clientpositive/groupby_sort_skew_1_23.q.out e19d1de 
>   ql/src/test/results/clientpositive/input_part1.q.out d6f4d3e 
>   ql/src/test/results/clientpositive/input_part5.q.out f2d7335 
>   ql/src/test/results/clientpositive/input_part6.q.out fa51cdf 
>   ql/src/test/results/clientpositive/lineage2.q.out ec8b76b 
>   ql/src/test/results/clientpositive/lineage3.q.out 747dc9a 
>   ql/src/test/results/clientpositive/list_bucket_query_oneskew_2.q.out 
> be77ba8 
>   ql/src/test/results/clientpositive/llap/bucket_map_join_tez1.q.out 79348f3 
>   ql/src/test/results/clientpositive/orc_predicate_pushdown.q.out f6d8388 
>   ql/src/test/results/clientpositive/parquet_predicate_pushdown.q.out b322ef1 
>   ql/src/test/results/clientpositive/partition_multilevels.q.out c1c8778 
>   ql/src/test/results/clientpositive/perf/query31.q.out 52cbdf4 
>   ql/src/test/results/clientpositive/perf/query39.q.out a18cdaf 
>   ql/src/test/results/clientpositive/perf/query42.q.out 3954829 
>   ql/src/test/results/clientpositive/perf/query52.q.out 63f0bf6 
>   ql/src/test/results/clientpositive/perf/query64.q.out a63 
>   ql/src/test/results/clientpositive/perf/query66.q.out 22eaf61 
>   ql/src/test/results/clientpositive/perf/query75.q.out d54000b 
>   ql/src/test/results/clientpositive/pointlookup2.q.out a442425 
>   ql/src/test/results/clientpositive/quotedid_basic.q.out 519f647 
>   ql/src/test/results/clientpositive/quotedid_partition.q.out d34a005 
>   ql/src/test/results/clientpositive/spark/bucket_map_join_tez1.q.out b5e7846 
>   ql/src/test/results/clientpositive/spark/cross_product_check_1.q.out 
> 0656cd5 
>   ql/src/test/results/clientpositive/spark/cross_product_check_2.q.out 
> 2d6eb38 
>   ql/src/test/results/clientpositive/spark/dynamic_rdd_cache.q.out 9043fb1 
>   ql/src/test/results/clientpositive/spark/groupby_sort_1_23.q.out 239e803 
>   ql/src/test/results/clientpositive/spark/groupby_sort_skew_1_23.q.out 
> 8370bbe 
>   ql/src/test/results/clientpositive/spark/union_remove_25.q.out b771fe9 
>   ql/src/test/results/clientpositive/spark/union_view.q.out cce7710 
>   ql/src/test/results/clientpositive/subquery_notin.q.out ed86079 
>   ql/src/test/results/clientpositive/subquery_notin_having.q.java1.7.out 
> d6c6edc 
>   ql/src/test/results/clientpositive/subquery_unqualcolumnrefs.q.out ad5f72b 
>

Re: Review Request 41377: HIVE-12528 don't start HS2 Tez sessions in a single thread

2015-12-21 Thread Sergey Shelukhin 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/41377/
---

(Updated Dec. 21, 2015, 9:47 p.m.)


Review request for hive, Gunther Hagleitner and Siddharth Seth.


Repository: hive-git


Description
---

see JIRA


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 0251a6f 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/DagUtils.java 6e196e6 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionPoolManager.java 
0d84340 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionState.java e5df2ec 
  ql/src/test/org/apache/hadoop/hive/ql/exec/tez/TestTezSessionPool.java 
11c0325 

Diff: https://reviews.apache.org/r/41377/diff/


Testing
---


Thanks,

Sergey Shelukhin

[jira] [Created] (HIVE-12725) CompactionTxnHandler.findNextToCompact() may produce "Operation not allowed after ResultSet closed"

2015-12-21 Thread Eugene Koifman (JIRA) 
Eugene Koifman created HIVE-12725:
-

 Summary: CompactionTxnHandler.findNextToCompact() may produce 
"Operation not allowed after ResultSet closed"
 Key: HIVE-12725
 URL: https://issues.apache.org/jira/browse/HIVE-12725
 Project: Hive
  Issue Type: Bug
  Components: Transactions
Affects Versions: 1.3.0, 2.0.0
Reporter: Eugene Koifman
Assignee: Eugene Koifman


{noformat}
2015-12-21 22:34:55,427 ERROR txn.CompactionTxnHandler 
(CompactionTxnHandler.java:findNextToCompact(210)) - 
[pre-prod-poc-4.novalocal-41] Unable to select next element for \
compaction, Operation not allowed after ResultSet closed
2015-12-21 22:34:55,431 ERROR compactor.Worker (Worker.java:run(181)) - 
[pre-prod-poc-4.novalocal-41] Caught an exception in the main loop of compactor 
worker pre-prod-poc-\
4.novalocal-41, MetaException(message:Unable to connect to transaction database 
java.sql.SQLException: Operation not allowed after ResultSet closed
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1073)
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:987)
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:982)
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:927)
at com.mysql.jdbc.ResultSetImpl.checkClosed(ResultSetImpl.java:794)
at com.mysql.jdbc.ResultSetImpl.next(ResultSetImpl.java:7139)
at 
org.apache.hadoop.hive.metastore.txn.CompactionTxnHandler.findNextToCompact(CompactionTxnHandler.java:206)
at org.apache.hadoop.hive.ql.txn.compactor.Worker.run(Worker.java:81)
)
at 
org.apache.hadoop.hive.metastore.txn.CompactionTxnHandler.findNextToCompact(CompactionTxnHandler.java:214)
at org.apache.hadoop.hive.ql.txn.compactor.Worker.run(Worker.java:81)
{noformat}

https://docs.oracle.com/javase/7/docs/api/java/sql/Statement.html

{noformat}
int updCount = stmt.executeUpdate(s);
  if(updCount == 1) {
dbConn.commit();
return info;
  }
  if(updCount == 0) {
LOG.debug("Another Worker picked up " + info);
continue;
  }
{noformat}
executeUpdate() closes the RS and if competing Worker "steals" the item then 
updCount == 0 is true...



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 41102: HIVE-12597 LLAP - allow using elevator without cache

2015-12-21 Thread j . prasanth . j 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/41102/#review111559
---



common/src/java/org/apache/hadoop/hive/conf/HiveConf.java (line 2332)


Can you also fix description accordingly to make to generic (remove ORC 
specific parts)?



llap-server/src/java/org/apache/hadoop/hive/llap/cache/BufferUsageManager.java 
(line 26)


Interface audience + javadoc?



llap-server/src/java/org/apache/hadoop/hive/llap/cache/SimpleAllocator.java 
(line 47)


How frequently is this called? If it is called frequently, then cost of 
allocateDirect() will be very high as direct allocation will sleep for 100ms 
when reserving direct memory.



llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/LlapIoImpl.java 
(line 76)


ignoreCase to be safer



llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/LlapIoImpl.java 
(line 120)


Shouldn't this be other way? If useAllocOnly is true then we use 
SimpleAllocator without cache right?



llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/OrcEncodedDataReader.java
 (line 620)


Whis is this outside if condition? Can you put this all inside a single if 
not null condition?


- Prasanth_J


On Dec. 8, 2015, 9:20 p.m., Sergey Shelukhin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/41102/
> ---
> 
> (Updated Dec. 8, 2015, 9:20 p.m.)
> 
> 
> Review request for hive and Prasanth_J.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> see jira
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java d52f994 
>   data/conf/hive-site.xml 2ebb1c4 
>   data/conf/llap/hive-site.xml 9440611 
>   llap-server/src/java/org/apache/hadoop/hive/llap/cache/BuddyAllocator.java 
> 0c96efa 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/cache/BufferUsageManager.java
>  PRE-CREATION 
>   llap-server/src/java/org/apache/hadoop/hive/llap/cache/LowLevelCache.java 
> b17edb5 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/cache/LowLevelCacheImpl.java 
> c2a130a 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/cache/LowLevelCacheMemoryManager.java
>  8788e15 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/cache/LowLevelLrfuCachePolicy.java
>  40cb92d 
>   llap-server/src/java/org/apache/hadoop/hive/llap/cache/SimpleAllocator.java 
> PRE-CREATION 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/cache/SimpleBufferManager.java
>  PRE-CREATION 
>   llap-server/src/java/org/apache/hadoop/hive/llap/cli/LlapServiceDriver.java 
> 08d573b 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemon.java 
> dbdf571 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/LlapIoImpl.java 
> b38f472 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/io/decode/OrcColumnVectorProducer.java
>  259c483 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/OrcEncodedDataReader.java
>  885acc0 
>   
> llap-server/src/test/org/apache/hadoop/hive/llap/cache/TestBuddyAllocator.java
>  6375996 
>   
> llap-server/src/test/org/apache/hadoop/hive/llap/cache/TestLowLevelCacheImpl.java
>  8324b21 
>   
> llap-server/src/test/org/apache/hadoop/hive/llap/cache/TestLowLevelLrfuCachePolicy.java
>  d0abfa3 
> 
> Diff: https://reviews.apache.org/r/41102/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Sergey Shelukhin
> 
>

[jira] [Created] (HIVE-12724) ACID: Major compaction fails to include the original bucket files into MR job

2015-12-21 Thread Wei Zheng (JIRA) 
Wei Zheng created HIVE-12724:


 Summary: ACID: Major compaction fails to include the original 
bucket files into MR job
 Key: HIVE-12724
 URL: https://issues.apache.org/jira/browse/HIVE-12724
 Project: Hive
  Issue Type: Bug
  Components: Hive
Affects Versions: 2.0.0, 2.1.0
Reporter: Wei Zheng
Assignee: Wei Zheng


How the problem happens:
* Create a non-ACID table
* Before non-ACID to ACID table conversion, we inserted row one
* After non-ACID to ACID table conversion, we inserted row two
* Both rows can be retrieved before MAJOR compaction
* After MAJOR compaction, row one is lost
{code}
hive> USE acidtest;
OK
Time taken: 0.77 seconds
hive> CREATE TABLE t1 (nationkey INT, name STRING, regionkey INT, comment 
STRING)
> CLUSTERED BY (regionkey) INTO 2 BUCKETS
> STORED AS ORC;
OK
Time taken: 0.179 seconds
hive> DESC FORMATTED t1;
OK
# col_name  data_type   comment

nationkey   int
namestring
regionkey   int
comment string

# Detailed Table Information
Database:   acidtest
Owner:  wzheng
CreateTime: Mon Dec 14 15:50:40 PST 2015
LastAccessTime: UNKNOWN
Retention:  0
Location:   file:/Users/wzheng/hivetmp/warehouse/acidtest.db/t1
Table Type: MANAGED_TABLE
Table Parameters:
transient_lastDdlTime   1450137040

# Storage Information
SerDe Library:  org.apache.hadoop.hive.ql.io.orc.OrcSerde
InputFormat:org.apache.hadoop.hive.ql.io.orc.OrcInputFormat
OutputFormat:   org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat
Compressed: No
Num Buckets:2
Bucket Columns: [regionkey]
Sort Columns:   []
Storage Desc Params:
serialization.format1
Time taken: 0.198 seconds, Fetched: 28 row(s)
hive> dfs -ls /Users/wzheng/hivetmp/warehouse/acidtest.db;
Found 1 items
drwxr-xr-x   - wzheng staff 68 2015-12-14 15:50 
/Users/wzheng/hivetmp/warehouse/acidtest.db/t1
hive> dfs -ls /Users/wzheng/hivetmp/warehouse/acidtest.db/t1;
hive> INSERT INTO TABLE t1 VALUES (1, 'USA', 1, 'united states');
WARNING: Hive-on-MR is deprecated in Hive 2 and may not be available in the 
future versions. Consider using a different execution engine (i.e. tez, spark) 
or using Hive 1.X releases.
Query ID = wzheng_20151214155028_630098c6-605f-4e7e-a797-6b49fb48360d
Total jobs = 1
Launching Job 1 out of 1
Number of reduce tasks determined at compile time: 2
In order to change the average load for a reducer (in bytes):
  set hive.exec.reducers.bytes.per.reducer=
In order to limit the maximum number of reducers:
  set hive.exec.reducers.max=
In order to set a constant number of reducers:
  set mapreduce.job.reduces=
Job running in-process (local Hadoop)
2015-12-14 15:51:58,070 Stage-1 map = 100%,  reduce = 100%
Ended Job = job_local73977356_0001
Loading data to table acidtest.t1
MapReduce Jobs Launched:
Stage-Stage-1:  HDFS Read: 0 HDFS Write: 0 SUCCESS
Total MapReduce CPU Time Spent: 0 msec
OK
Time taken: 2.825 seconds
hive> dfs -ls /Users/wzheng/hivetmp/warehouse/acidtest.db/t1;
Found 2 items
-rwxr-xr-x   1 wzheng staff112 2015-12-14 15:51 
/Users/wzheng/hivetmp/warehouse/acidtest.db/t1/00_0
-rwxr-xr-x   1 wzheng staff472 2015-12-14 15:51 
/Users/wzheng/hivetmp/warehouse/acidtest.db/t1/01_0
hive> SELECT * FROM t1;
OK
1   USA 1   united states
Time taken: 0.434 seconds, Fetched: 1 row(s)
hive> ALTER TABLE t1 SET TBLPROPERTIES ('transactional' = 'true');
OK
Time taken: 0.071 seconds
hive> DESC FORMATTED t1;
OK
# col_name  data_type   comment

nationkey   int
namestring
regionkey   int
comment string

# Detailed Table Information
Database:   acidtest
Owner:  wzheng
CreateTime: Mon Dec 14 15:50:40 PST 2015
LastAccessTime: UNKNOWN
Retention:  0
Location:   file:/Users/wzheng/hivetmp/warehouse/acidtest.db/t1
Table Type: MANAGED_TABLE
Table Parameters:
COLUMN_STATS_ACCURATE   false
last_modified_bywzheng
last_modified_time  1450137141
numFiles2
numRows -1
rawDataSize -1
totalSize   584
transactional   true
transient_lastDdlTime   1450137141

# Storage Information
SerDe Library:  org.apache.hadoop.hive.ql.io.orc.OrcSerde
InputFormat:org.apache.hadoop.hive.ql.io.orc.OrcInputFormat
OutputFormat:   org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat
Compressed: No
Num Buckets:2
Bucket Columns: [regionkey]
Sort Columns:   []
Storage Desc Params:

Re: Review Request 41377: HIVE-12528 don't start HS2 Tez sessions in a single thread

2015-12-21 Thread Sergey Shelukhin 


> On Dec. 19, 2015, 2 a.m., Siddharth Seth wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionPoolManager.java, 
> > line 89
> > 
> >
> > Why is this required ?
> > 
> > and why not in the single thread case.

some part of Tez init in Hive accesses the session. The single thread has it 
set; we just propagate it to other threads.


> On Dec. 19, 2015, 2 a.m., Siddharth Seth wrote:
> > ql/src/test/org/apache/hadoop/hive/ql/exec/tez/TestTezSessionPool.java, 
> > line 124
> > 
> >
> > Is there any part of the test which verifies that threads are being 
> > used / sessions are being setup in parallel. Not sure if there's a good way 
> > to verify that though - without setting up explicit properties for tests 
> > only.

No. Rather, I saw the order change in the other test, that's how I know it 
works :)


- Sergey


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/41377/#review111342
---


On Dec. 18, 2015, 10:10 p.m., Sergey Shelukhin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/41377/
> ---
> 
> (Updated Dec. 18, 2015, 10:10 p.m.)
> 
> 
> Review request for hive, Gunther Hagleitner and Siddharth Seth.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> see JIRA
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 36e281a 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/DagUtils.java 6e196e6 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionPoolManager.java 
> 0d84340 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionState.java e1a8041 
>   ql/src/test/org/apache/hadoop/hive/ql/exec/tez/TestTezSessionPool.java 
> 11c0325 
> 
> Diff: https://reviews.apache.org/r/41377/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Sergey Shelukhin
> 
>

[jira] [Created] (HIVE-12719) As a hive user, I am facing issues using permanent UDAF's.

2015-12-21 Thread Surbhit (JIRA) 
Surbhit created HIVE-12719:
--

 Summary: As a hive user, I am facing issues using permanent UDAF's.
 Key: HIVE-12719
 URL: https://issues.apache.org/jira/browse/HIVE-12719
 Project: Hive
  Issue Type: Bug
  Components: Hive
Affects Versions: 1.2.1
Reporter: Surbhit






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 41102: HIVE-12597 LLAP - allow using elevator without cache

2015-12-21 Thread Sergey Shelukhin 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/41102/
---

(Updated Dec. 22, 2015, 2:13 a.m.)


Review request for hive and Prasanth_J.


Repository: hive-git


Description
---

see jira


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java d2dd9c6 
  data/conf/hive-site.xml 2a5f891 
  data/conf/llap/hive-site.xml e3425c8 
  llap-server/src/java/org/apache/hadoop/hive/llap/cache/BuddyAllocator.java 
0c96efa 
  
llap-server/src/java/org/apache/hadoop/hive/llap/cache/BufferUsageManager.java 
PRE-CREATION 
  llap-server/src/java/org/apache/hadoop/hive/llap/cache/LowLevelCache.java 
b17edb5 
  llap-server/src/java/org/apache/hadoop/hive/llap/cache/LowLevelCacheImpl.java 
c2a130a 
  
llap-server/src/java/org/apache/hadoop/hive/llap/cache/LowLevelCacheMemoryManager.java
 8788e15 
  
llap-server/src/java/org/apache/hadoop/hive/llap/cache/LowLevelLrfuCachePolicy.java
 40cb92d 
  llap-server/src/java/org/apache/hadoop/hive/llap/cache/SimpleAllocator.java 
PRE-CREATION 
  
llap-server/src/java/org/apache/hadoop/hive/llap/cache/SimpleBufferManager.java 
PRE-CREATION 
  llap-server/src/java/org/apache/hadoop/hive/llap/cli/LlapServiceDriver.java 
8e5377f 
  llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemon.java 
467ab71 
  llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/LlapIoImpl.java 
b38f472 
  
llap-server/src/java/org/apache/hadoop/hive/llap/io/decode/OrcColumnVectorProducer.java
 e156eaa 
  
llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/OrcEncodedDataReader.java
 58d2ac8 
  
llap-server/src/test/org/apache/hadoop/hive/llap/cache/TestBuddyAllocator.java 
6375996 
  
llap-server/src/test/org/apache/hadoop/hive/llap/cache/TestLowLevelCacheImpl.java
 8324b21 
  
llap-server/src/test/org/apache/hadoop/hive/llap/cache/TestLowLevelLrfuCachePolicy.java
 d0abfa3 

Diff: https://reviews.apache.org/r/41102/diff/


Testing
---


Thanks,

Sergey Shelukhin

[jira] [Created] (HIVE-12726) exclude hadoop jars from webhcat hcatalog/share/webhcat/svr/lib

2015-12-21 Thread Eugene Koifman (JIRA) 
Eugene Koifman created HIVE-12726:
-

 Summary: exclude hadoop jars from webhcat 
hcatalog/share/webhcat/svr/lib
 Key: HIVE-12726
 URL: https://issues.apache.org/jira/browse/HIVE-12726
 Project: Hive
  Issue Type: Bug
  Components: WebHCat
Affects Versions: 2.0.0
Reporter: Eugene Koifman
Assignee: Eugene Koifman






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-12727) allow full table queries in strict mode

2015-12-21 Thread Sergey Shelukhin (JIRA) 
Sergey Shelukhin created HIVE-12727:
---

 Summary: allow full table queries in strict mode
 Key: HIVE-12727
 URL: https://issues.apache.org/jira/browse/HIVE-12727
 Project: Hive
  Issue Type: Bug
Reporter: Sergey Shelukhin
Priority: Blocker


Making strict mode the default recently appears to have broken many normal 
queries, such as some TPCDS benchmark queries, e.g. Q85:
Response message: org.apache.hive.service.cli.HiveSQLException: Error while 
compiling statement: FAILED: SemanticException [Error 10041]: No partition 
predicate found for Alias "web_sales" Table "web_returns"

We should remove this restriction from strict mode, or change the default back 
to non-strict. Perhaps make a 3-value parameter, nonstrict, semistrict, and 
strict, for backward compat for people who are relying on strict already.





--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 41431: HIVE-12674 HS2 Tez session lifetime

2015-12-21 Thread Sergey Shelukhin 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/41431/
---

(Updated Dec. 22, 2015, 3:18 a.m.)


Review request for hive, Siddharth Seth and Vikram Dixit Kumaraswamy.


Repository: hive-git


Description
---

see JIRA


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java d2dd9c6 
  ql/src/java/org/apache/hadoop/hive/ql/exec/mr/ExecDriver.java 971dac9 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezJobMonitor.java f6bc19c 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionPoolManager.java 
0d84340 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionState.java e5df2ec 
  ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java c066c7a 
  ql/src/test/org/apache/hadoop/hive/ql/exec/tez/SampleTezSessionState.java 
d55c9fe 
  ql/src/test/org/apache/hadoop/hive/ql/exec/tez/TestTezSessionPool.java 
11c0325 

Diff: https://reviews.apache.org/r/41431/diff/


Testing
---


Thanks,

Sergey Shelukhin

Re: Review Request 41102: HIVE-12597 LLAP - allow using elevator without cache

2015-12-21 Thread Sergey Shelukhin 


> On Dec. 22, 2015, 12:45 a.m., Prasanth_J wrote:
> > llap-server/src/java/org/apache/hadoop/hive/llap/cache/BufferUsageManager.java,
> >  line 26
> > 
> >
> > Interface audience + javadoc?

interface audiences are not used in LLAP IO so far... added the javadoc


> On Dec. 22, 2015, 12:45 a.m., Prasanth_J wrote:
> > llap-server/src/java/org/apache/hadoop/hive/llap/cache/SimpleAllocator.java,
> >  line 47
> > 
> >
> > How frequently is this called? If it is called frequently, then cost of 
> > allocateDirect() will be very high as direct allocation will sleep for 
> > 100ms when reserving direct memory.

Such as life if you don't use the recommended settings :P


> On Dec. 22, 2015, 12:45 a.m., Prasanth_J wrote:
> > llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/LlapIoImpl.java,
> >  line 124
> > 
> >
> > Shouldn't this be other way? If useAllocOnly is true then we use 
> > SimpleAllocator without cache right?

No; updated the HiveConf description to be more clear


> On Dec. 22, 2015, 12:45 a.m., Prasanth_J wrote:
> > llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/OrcEncodedDataReader.java,
> >  line 620
> > 
> >
> > Whis is this outside if condition? Can you put this all inside a single 
> > if not null condition?

eh? We don't count cache hits or misses if there's no cache or the item is not 
cacheable. Added an else for clarity, this relied on return before


- Sergey


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/41102/#review111559
---


On Dec. 8, 2015, 9:20 p.m., Sergey Shelukhin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/41102/
> ---
> 
> (Updated Dec. 8, 2015, 9:20 p.m.)
> 
> 
> Review request for hive and Prasanth_J.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> see jira
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java d52f994 
>   data/conf/hive-site.xml 2ebb1c4 
>   data/conf/llap/hive-site.xml 9440611 
>   llap-server/src/java/org/apache/hadoop/hive/llap/cache/BuddyAllocator.java 
> 0c96efa 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/cache/BufferUsageManager.java
>  PRE-CREATION 
>   llap-server/src/java/org/apache/hadoop/hive/llap/cache/LowLevelCache.java 
> b17edb5 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/cache/LowLevelCacheImpl.java 
> c2a130a 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/cache/LowLevelCacheMemoryManager.java
>  8788e15 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/cache/LowLevelLrfuCachePolicy.java
>  40cb92d 
>   llap-server/src/java/org/apache/hadoop/hive/llap/cache/SimpleAllocator.java 
> PRE-CREATION 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/cache/SimpleBufferManager.java
>  PRE-CREATION 
>   llap-server/src/java/org/apache/hadoop/hive/llap/cli/LlapServiceDriver.java 
> 08d573b 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemon.java 
> dbdf571 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/LlapIoImpl.java 
> b38f472 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/io/decode/OrcColumnVectorProducer.java
>  259c483 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/OrcEncodedDataReader.java
>  885acc0 
>   
> llap-server/src/test/org/apache/hadoop/hive/llap/cache/TestBuddyAllocator.java
>  6375996 
>   
> llap-server/src/test/org/apache/hadoop/hive/llap/cache/TestLowLevelCacheImpl.java
>  8324b21 
>   
> llap-server/src/test/org/apache/hadoop/hive/llap/cache/TestLowLevelLrfuCachePolicy.java
>  d0abfa3 
> 
> Diff: https://reviews.apache.org/r/41102/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Sergey Shelukhin
> 
>

[jira] [Created] (HIVE-12728) Apply DDL restrictions for ORC schema evolution

2015-12-21 Thread Prasanth Jayachandran (JIRA) 
Prasanth Jayachandran created HIVE-12728:


 Summary: Apply DDL restrictions for ORC schema evolution
 Key: HIVE-12728
 URL: https://issues.apache.org/jira/browse/HIVE-12728
 Project: Hive
  Issue Type: Bug
Affects Versions: 2.0.0, 2.1.0
Reporter: Prasanth Jayachandran
Assignee: Prasanth Jayachandran
Priority: Critical


HIVE-11981 added schema evolution for ORC. However, it does not enforce any 
restrictions in DDL that can break schema evolution. Following changes have to 
be enforced in DDL to support the assumptions in schema evolution (that columns 
will only be added).

1) Restrict changing the file format of the table
2) Restrict changing the serde of the table
3) Restrict replacing columns
4) Restrict reordering columns
5) Restrict unsupported type promotions



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 41102: HIVE-12597 LLAP - allow using elevator without cache

2015-12-21 Thread j . prasanth . j 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/41102/#review111571
---



llap-server/src/java/org/apache/hadoop/hive/llap/cache/SimpleAllocator.java 
(line 54)


Are you sure this will be GC'ed properly for direct memory? I remember 
something about using clean() for direct BB. If not cleaned properly this can 
trigger full GC to clear off unused memory.


- Prasanth_J


On Dec. 22, 2015, 2:13 a.m., Sergey Shelukhin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/41102/
> ---
> 
> (Updated Dec. 22, 2015, 2:13 a.m.)
> 
> 
> Review request for hive and Prasanth_J.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> see jira
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java d2dd9c6 
>   data/conf/hive-site.xml 2a5f891 
>   data/conf/llap/hive-site.xml e3425c8 
>   llap-server/src/java/org/apache/hadoop/hive/llap/cache/BuddyAllocator.java 
> 0c96efa 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/cache/BufferUsageManager.java
>  PRE-CREATION 
>   llap-server/src/java/org/apache/hadoop/hive/llap/cache/LowLevelCache.java 
> b17edb5 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/cache/LowLevelCacheImpl.java 
> c2a130a 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/cache/LowLevelCacheMemoryManager.java
>  8788e15 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/cache/LowLevelLrfuCachePolicy.java
>  40cb92d 
>   llap-server/src/java/org/apache/hadoop/hive/llap/cache/SimpleAllocator.java 
> PRE-CREATION 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/cache/SimpleBufferManager.java
>  PRE-CREATION 
>   llap-server/src/java/org/apache/hadoop/hive/llap/cli/LlapServiceDriver.java 
> 8e5377f 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemon.java 
> 467ab71 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/LlapIoImpl.java 
> b38f472 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/io/decode/OrcColumnVectorProducer.java
>  e156eaa 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/OrcEncodedDataReader.java
>  58d2ac8 
>   
> llap-server/src/test/org/apache/hadoop/hive/llap/cache/TestBuddyAllocator.java
>  6375996 
>   
> llap-server/src/test/org/apache/hadoop/hive/llap/cache/TestLowLevelCacheImpl.java
>  8324b21 
>   
> llap-server/src/test/org/apache/hadoop/hive/llap/cache/TestLowLevelLrfuCachePolicy.java
>  d0abfa3 
> 
> Diff: https://reviews.apache.org/r/41102/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Sergey Shelukhin
> 
>

Review Request 41628: HIVE-12664 - Bug in reduce deduplication optimization causing ArrayOutOfBoundException

2015-12-21 Thread Johan Gustavsson 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/41628/
---

Review request for hive.


Repository: hive-git


Description
---

See jira [HIVE-12664](https://issues.apache.org/jira/browse/HIVE-12664)


Diffs
-

  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/correlation/ReduceSinkDeDuplication.java
 7b5f9b2 

Diff: https://reviews.apache.org/r/41628/diff/


Testing
---

See query in jira [HIVE-12664](https://issues.apache.org/jira/browse/HIVE-12664)


Thanks,

Johan Gustavsson

[jira] [Created] (HIVE-12718) skip.footer.line.count misbehaves on larger text files

2015-12-21 Thread Gergely Nagy (JIRA) 
Gergely Nagy created HIVE-12718:
---

 Summary: skip.footer.line.count misbehaves on larger text files
 Key: HIVE-12718
 URL: https://issues.apache.org/jira/browse/HIVE-12718
 Project: Hive
  Issue Type: Bug
Affects Versions: 1.1.0
 Environment: The bug was discovered and reproduced on a Cloudera 
Hadoop 5.4 distribution running on CentOS 6.4.
Reporter: Gergely Nagy
Priority: Minor


We noticed that when working on a table backed by a larger (large enough to 
require splitting) text file, the {{skip.footer.line.count}} property of the 
table misbehaves: the footer is not being ignored.

To reproduce, follow these steps:

1) Create a large file: {{for i in $(seq 1 100); do cat /usr/share/dict/words; 
done >large.txt}}
2) Upload it to HDFS (eg, as {{/tmp/words}})
3) Create an external table with {{skip.footer.line.count}} set: 

{quote}
CREATE EXTERNAL TABLE ext_words (word STRING)
  ROW FORMAT DELIMITED FIELDS TERMINATED BY '|'
  LINES TERMINATED BY '\n'
  STORED AS TEXTFILE LOCATION '/tmp/words'
  tblproperties("skip.header.line.count"="1", "skip.footer.line.count"="1");
{quote}
4) Count the number of times the last line (in this example, I assume that to 
be {{ZZZ}}) appears: {{SELECT COUNT( * ) FROM ext_words WHERE word = 'ZZZ';}}
5) Observe that it returns 100 instead of 99.

Investigation showed that this happens when there are more than one mappers 
used for the job. If we increase the split size, to force using one mapper 
only, the problem did not occur.

There may be other related issues as well, like the wrong line being skipped -- 
but we did not reproduce those yet.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 41582: HIVE-12713: Miscellaneous improvements in driver compile and execute logging

2015-12-21 Thread Xuefu Zhang 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/41582/#review111472
---



ql/src/java/org/apache/hadoop/hive/ql/Driver.java (line 407)


To clarify, the above log redaction is needed because of the addition of 
this line, right?



ql/src/java/org/apache/hadoop/hive/ql/parse/ParseDriver.java (line 185)


It might be better if we check if debug log is enabled. Same below


- Xuefu Zhang


On Dec. 20, 2015, 1:05 a.m., Chaoyu Tang wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/41582/
> ---
> 
> (Updated Dec. 20, 2015, 1:05 a.m.)
> 
> 
> Review request for hive.
> 
> 
> Bugs: HIVE-12713
> https://issues.apache.org/jira/browse/HIVE-12713
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Patch improves the driver compile and execute logging in following:
> 1. ensuring that only the redacted query to be logged out
> 2. removing redundant variable substitution in HS2 SQLOperation
> 3. logging out the query and its compilation time without having to enable 
> PerfLogger debug, to help identify badly written queries which take a lot of 
> time to compile and probably cause other good queries to be queued 
> (HIVE-12516)
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/ql/log/PerfLogger.java 98ebd50 
>   
> itests/hive-unit/src/test/java/org/apache/hive/beeline/TestBeeLineWithArgs.java
>  7cc0acf 
>   itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestJdbcDriver2.java 
> e9206b9 
>   
> itests/hive-unit/src/test/java/org/apache/hive/jdbc/cbo_rp_TestJdbcDriver2.java
>  c66f166 
>   
> itests/hive-unit/src/test/java/org/apache/hive/service/cli/operation/TestOperationLoggingAPIWithMr.java
>  d21571e 
>   
> itests/hive-unit/src/test/java/org/apache/hive/service/cli/operation/TestOperationLoggingAPIWithTez.java
>  8b21647 
>   ql/src/java/org/apache/hadoop/hive/ql/Driver.java 3d5f3b5 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/ParseDriver.java c33bb66 
>   service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java 
> d90dd0d 
> 
> Diff: https://reviews.apache.org/r/41582/diff/
> 
> 
> Testing
> ---
> 
> 1. Manual tests
> 2. Precommit tests
> 
> 
> Thanks,
> 
> Chaoyu Tang
> 
>

Re: Review Request 41582: HIVE-12713: Miscellaneous improvements in driver compile and execute logging

2015-12-21 Thread Chaoyu Tang 


> On Dec. 21, 2015, 3:13 p.m., Xuefu Zhang wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/Driver.java, line 407
> > 
> >
> > To clarify, the above log redaction is needed because of the addition 
> > of this line, right?

Yes, the redaction logic, which was already there, is moved to the beginning of 
the method and also before this logging is called.


> On Dec. 21, 2015, 3:13 p.m., Xuefu Zhang wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/parse/ParseDriver.java, line 185
> > 
> >
> > It might be better if we check if debug log is enabled. Same below

Yeah, it is recommended to call isLogEnabled for any debug logging message 
which needs to be concatenated. Fixed. Thanks


- Chaoyu


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/41582/#review111472
---


On Dec. 20, 2015, 1:05 a.m., Chaoyu Tang wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/41582/
> ---
> 
> (Updated Dec. 20, 2015, 1:05 a.m.)
> 
> 
> Review request for hive.
> 
> 
> Bugs: HIVE-12713
> https://issues.apache.org/jira/browse/HIVE-12713
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Patch improves the driver compile and execute logging in following:
> 1. ensuring that only the redacted query to be logged out
> 2. removing redundant variable substitution in HS2 SQLOperation
> 3. logging out the query and its compilation time without having to enable 
> PerfLogger debug, to help identify badly written queries which take a lot of 
> time to compile and probably cause other good queries to be queued 
> (HIVE-12516)
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/ql/log/PerfLogger.java 98ebd50 
>   
> itests/hive-unit/src/test/java/org/apache/hive/beeline/TestBeeLineWithArgs.java
>  7cc0acf 
>   itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestJdbcDriver2.java 
> e9206b9 
>   
> itests/hive-unit/src/test/java/org/apache/hive/jdbc/cbo_rp_TestJdbcDriver2.java
>  c66f166 
>   
> itests/hive-unit/src/test/java/org/apache/hive/service/cli/operation/TestOperationLoggingAPIWithMr.java
>  d21571e 
>   
> itests/hive-unit/src/test/java/org/apache/hive/service/cli/operation/TestOperationLoggingAPIWithTez.java
>  8b21647 
>   ql/src/java/org/apache/hadoop/hive/ql/Driver.java 3d5f3b5 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/ParseDriver.java c33bb66 
>   service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java 
> d90dd0d 
> 
> Diff: https://reviews.apache.org/r/41582/diff/
> 
> 
> Testing
> ---
> 
> 1. Manual tests
> 2. Precommit tests
> 
> 
> Thanks,
> 
> Chaoyu Tang
> 
>

[jira] [Created] (HIVE-12720) Partition pruning not happening on left join with CTE

2015-12-21 Thread Clemens Valiente (JIRA) 
Clemens Valiente created HIVE-12720:
---

 Summary: Partition pruning not happening on left join with CTE
 Key: HIVE-12720
 URL: https://issues.apache.org/jira/browse/HIVE-12720
 Project: Hive
  Issue Type: Bug
Affects Versions: 1.1.0
Reporter: Clemens Valiente


In certain cases hive is not pruning partitions as well as it could be, and 
instead scanning the whole table.
Minimal use case:
{code:sql}
create table tbl1 (
id1 int,
ext_id2 int
)
PARTITIONED BY (ymd int);

create table tbl2 (
id2 int,
prop string
)
PARTITIONED BY (ymd int);

INSERT INTO tbl1 PARTITION(ymd=20151201) VALUES (1,1);
INSERT INTO tbl1 PARTITION(ymd=20151202) VALUES (1,2);
INSERT INTO tbl1 PARTITION(ymd=20151203) VALUES (1,3);

INSERT INTO tbl2 PARTITION(ymd=20151201) VALUES (1,'testa');
INSERT INTO tbl2 PARTITION(ymd=20151202) VALUES (2,'testb');
INSERT INTO tbl2 PARTITION(ymd=20151203) VALUES (3,'testc');


-- query 1
EXPLAIN
select * from
tbl1
LEFT OUTER JOIN
tbl2
ON
(tbl1.ext_id2 = tbl2.id2
 and tbl2.ymd = tbl1.ymd)
WHERE
tbl1.ymd = 20151203


-- query2
EXPLAIN
with cte as (select * from tbl1
WHERE
tbl1.ymd = 20151203)

select * from
cte
LEFT OUTER JOIN
tbl2
ON
(cte.ext_id2 = tbl2.id2
 and tbl2.ymd = cte.ymd)
WHERE
tbl2.ymd = 20151203
{code}

Query1 prunes correctly:
alias: tbl2
Statistics: Num rows: 1 Data size: 7 Basic stats: COMPLETE Column stats: NONE
Query2 does a full table scan on tbl2:
 Statistics: Num rows: 4 Data size: 28 Basic stats: COMPLETE Column stats: NONE
I know both queries are not equivalent and the second query is not really a 
LEFT JOIN in its current state, but I think the optimiser could do a better job 
here.

This example was (re)produced with a cdh 5.5 docker image.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 41582: HIVE-12713: Miscellaneous improvements in driver compile and execute logging

2015-12-21 Thread Chaoyu Tang 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/41582/
---

(Updated Dec. 21, 2015, 3:46 p.m.)


Review request for hive.


Changes
---

Revised the patch based on the review


Bugs: HIVE-12713
https://issues.apache.org/jira/browse/HIVE-12713


Repository: hive-git


Description
---

Patch improves the driver compile and execute logging in following:
1. ensuring that only the redacted query to be logged out
2. removing redundant variable substitution in HS2 SQLOperation
3. logging out the query and its compilation time without having to enable 
PerfLogger debug, to help identify badly written queries which take a lot of 
time to compile and probably cause other good queries to be queued (HIVE-12516)


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/ql/log/PerfLogger.java 98ebd50 
  
itests/hive-unit/src/test/java/org/apache/hive/beeline/TestBeeLineWithArgs.java 
7cc0acf 
  itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestJdbcDriver2.java 
e9206b9 
  
itests/hive-unit/src/test/java/org/apache/hive/jdbc/cbo_rp_TestJdbcDriver2.java 
c66f166 
  
itests/hive-unit/src/test/java/org/apache/hive/service/cli/operation/TestOperationLoggingAPIWithMr.java
 d21571e 
  
itests/hive-unit/src/test/java/org/apache/hive/service/cli/operation/TestOperationLoggingAPIWithTez.java
 8b21647 
  ql/src/java/org/apache/hadoop/hive/ql/Driver.java 3d5f3b5 
  ql/src/java/org/apache/hadoop/hive/ql/parse/ParseDriver.java c33bb66 
  service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java 
d90dd0d 

Diff: https://reviews.apache.org/r/41582/diff/


Testing
---

1. Manual tests
2. Precommit tests


Thanks,

Chaoyu Tang

Re: Review Request 40865: Implement/Enable constant related optimization rules in Calcite: enable HiveReduceExpressionsRule to fold constants

2015-12-21 Thread pengcheng xiong 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40865/
---

(Updated Dec. 21, 2015, 5:44 p.m.)


Review request for hive and John Pullokkaran.


Changes
---

address John's comments


Repository: hive-git


Description
---

HIVE-11927


Diffs (updated)
-

  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/CalciteSemanticException.java
 336745b 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/HiveRexExecutorImpl.java
 PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveReduceExpressionsRule.java
 PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/ASTBuilder.java
 425514d 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/RexNodeConverter.java
 631a4ca 
  ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java 87b18b7 
  ql/src/test/queries/clientpositive/cbo_const.q PRE-CREATION 
  ql/src/test/queries/clientpositive/constantfolding.q PRE-CREATION 
  ql/src/test/results/clientpositive/annotate_stats_select.q.out c4d59c8 
  ql/src/test/results/clientpositive/bucketizedhiveinputformat.q.out 277b0f7 
  ql/src/test/results/clientpositive/cast1.q.out 0bdecba 
  ql/src/test/results/clientpositive/cbo_const.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/cbo_rp_cross_product_check_2.q.out 13d8a50 
  ql/src/test/results/clientpositive/cbo_rp_lineage2.q.out 79f76bf 
  ql/src/test/results/clientpositive/constantfolding.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/cross_product_check_1.q.out d9143c8 
  ql/src/test/results/clientpositive/cross_product_check_2.q.out 4e2b93e 
  ql/src/test/results/clientpositive/dynamic_rdd_cache.q.out 3153c7e 
  ql/src/test/results/clientpositive/dynpart_sort_optimization2.q.out 24ac550 
  ql/src/test/results/clientpositive/groupby_ppd.q.out 6164a26 
  ql/src/test/results/clientpositive/groupby_sort_1_23.q.out 7333677 
  ql/src/test/results/clientpositive/groupby_sort_skew_1_23.q.out e19d1de 
  ql/src/test/results/clientpositive/input_part1.q.out d6f4d3e 
  ql/src/test/results/clientpositive/input_part5.q.out f2d7335 
  ql/src/test/results/clientpositive/input_part6.q.out fa51cdf 
  ql/src/test/results/clientpositive/lineage2.q.out ec8b76b 
  ql/src/test/results/clientpositive/lineage3.q.out 747dc9a 
  ql/src/test/results/clientpositive/list_bucket_query_oneskew_2.q.out be77ba8 
  ql/src/test/results/clientpositive/llap/bucket_map_join_tez1.q.out 79348f3 
  ql/src/test/results/clientpositive/orc_predicate_pushdown.q.out f6d8388 
  ql/src/test/results/clientpositive/parquet_predicate_pushdown.q.out b322ef1 
  ql/src/test/results/clientpositive/partition_multilevels.q.out c1c8778 
  ql/src/test/results/clientpositive/perf/query31.q.out 52cbdf4 
  ql/src/test/results/clientpositive/perf/query39.q.out a18cdaf 
  ql/src/test/results/clientpositive/perf/query42.q.out 3954829 
  ql/src/test/results/clientpositive/perf/query52.q.out 63f0bf6 
  ql/src/test/results/clientpositive/perf/query64.q.out a63 
  ql/src/test/results/clientpositive/perf/query66.q.out 22eaf61 
  ql/src/test/results/clientpositive/perf/query75.q.out d54000b 
  ql/src/test/results/clientpositive/pointlookup2.q.out a442425 
  ql/src/test/results/clientpositive/quotedid_basic.q.out 519f647 
  ql/src/test/results/clientpositive/quotedid_partition.q.out d34a005 
  ql/src/test/results/clientpositive/spark/bucket_map_join_tez1.q.out b5e7846 
  ql/src/test/results/clientpositive/spark/cross_product_check_1.q.out 0656cd5 
  ql/src/test/results/clientpositive/spark/cross_product_check_2.q.out 2d6eb38 
  ql/src/test/results/clientpositive/spark/dynamic_rdd_cache.q.out 9043fb1 
  ql/src/test/results/clientpositive/spark/groupby_sort_1_23.q.out 239e803 
  ql/src/test/results/clientpositive/spark/groupby_sort_skew_1_23.q.out 8370bbe 
  ql/src/test/results/clientpositive/spark/union_remove_25.q.out b771fe9 
  ql/src/test/results/clientpositive/spark/union_view.q.out cce7710 
  ql/src/test/results/clientpositive/subquery_notin.q.out ed86079 
  ql/src/test/results/clientpositive/subquery_notin_having.q.java1.7.out 
d6c6edc 
  ql/src/test/results/clientpositive/subquery_unqualcolumnrefs.q.out ad5f72b 
  ql/src/test/results/clientpositive/tez/bucket_map_join_tez1.q.out 9582334 
  ql/src/test/results/clientpositive/tez/cross_product_check_1.q.out ca44973 
  ql/src/test/results/clientpositive/tez/cross_product_check_2.q.out ecd7f6b 
  ql/src/test/results/clientpositive/tez/dynpart_sort_optimization2.q.out 
346e52f 
  ql/src/test/results/clientpositive/tez/explainuser_1.q.out 78c68c3 
  ql/src/test/results/clientpositive/tez/vector_decimal_round.q.out 9a5d047 
  ql/src/test/results/clientpositive/udf1.q.out dffbccf 
  ql/src/test/results/clientpositive/udf_10_trims.q.out 2f79723 
  ql/src/test/results/clientpositive/udf_folder_constants.q.out

Re: Review Request 40865: Implement/Enable constant related optimization rules in Calcite: enable HiveReduceExpressionsRule to fold constants

2015-12-21 Thread pengcheng xiong 


> On Dec. 19, 2015, 2:44 a.m., John Pullokkaran wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveReduceExpressionsRule.java,
> >  line 320
> > 
> >
> > Rename class?

CalcReduceExpression is not used in Hive right now. We can remove it or just 
keep it as a place holder.


- pengcheng


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40865/#review111351
---


On Dec. 21, 2015, 5:44 p.m., pengcheng xiong wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40865/
> ---
> 
> (Updated Dec. 21, 2015, 5:44 p.m.)
> 
> 
> Review request for hive and John Pullokkaran.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-11927
> 
> 
> Diffs
> -
> 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/CalciteSemanticException.java
>  336745b 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/HiveRexExecutorImpl.java
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveReduceExpressionsRule.java
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/ASTBuilder.java
>  425514d 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/RexNodeConverter.java
>  631a4ca 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java 87b18b7 
>   ql/src/test/queries/clientpositive/cbo_const.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/constantfolding.q PRE-CREATION 
>   ql/src/test/results/clientpositive/annotate_stats_select.q.out c4d59c8 
>   ql/src/test/results/clientpositive/bucketizedhiveinputformat.q.out 277b0f7 
>   ql/src/test/results/clientpositive/cast1.q.out 0bdecba 
>   ql/src/test/results/clientpositive/cbo_const.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/cbo_rp_cross_product_check_2.q.out 
> 13d8a50 
>   ql/src/test/results/clientpositive/cbo_rp_lineage2.q.out 79f76bf 
>   ql/src/test/results/clientpositive/constantfolding.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/cross_product_check_1.q.out d9143c8 
>   ql/src/test/results/clientpositive/cross_product_check_2.q.out 4e2b93e 
>   ql/src/test/results/clientpositive/dynamic_rdd_cache.q.out 3153c7e 
>   ql/src/test/results/clientpositive/dynpart_sort_optimization2.q.out 24ac550 
>   ql/src/test/results/clientpositive/groupby_ppd.q.out 6164a26 
>   ql/src/test/results/clientpositive/groupby_sort_1_23.q.out 7333677 
>   ql/src/test/results/clientpositive/groupby_sort_skew_1_23.q.out e19d1de 
>   ql/src/test/results/clientpositive/input_part1.q.out d6f4d3e 
>   ql/src/test/results/clientpositive/input_part5.q.out f2d7335 
>   ql/src/test/results/clientpositive/input_part6.q.out fa51cdf 
>   ql/src/test/results/clientpositive/lineage2.q.out ec8b76b 
>   ql/src/test/results/clientpositive/lineage3.q.out 747dc9a 
>   ql/src/test/results/clientpositive/list_bucket_query_oneskew_2.q.out 
> be77ba8 
>   ql/src/test/results/clientpositive/llap/bucket_map_join_tez1.q.out 79348f3 
>   ql/src/test/results/clientpositive/orc_predicate_pushdown.q.out f6d8388 
>   ql/src/test/results/clientpositive/parquet_predicate_pushdown.q.out b322ef1 
>   ql/src/test/results/clientpositive/partition_multilevels.q.out c1c8778 
>   ql/src/test/results/clientpositive/perf/query31.q.out 52cbdf4 
>   ql/src/test/results/clientpositive/perf/query39.q.out a18cdaf 
>   ql/src/test/results/clientpositive/perf/query42.q.out 3954829 
>   ql/src/test/results/clientpositive/perf/query52.q.out 63f0bf6 
>   ql/src/test/results/clientpositive/perf/query64.q.out a63 
>   ql/src/test/results/clientpositive/perf/query66.q.out 22eaf61 
>   ql/src/test/results/clientpositive/perf/query75.q.out d54000b 
>   ql/src/test/results/clientpositive/pointlookup2.q.out a442425 
>   ql/src/test/results/clientpositive/quotedid_basic.q.out 519f647 
>   ql/src/test/results/clientpositive/quotedid_partition.q.out d34a005 
>   ql/src/test/results/clientpositive/spark/bucket_map_join_tez1.q.out b5e7846 
>   ql/src/test/results/clientpositive/spark/cross_product_check_1.q.out 
> 0656cd5 
>   ql/src/test/results/clientpositive/spark/cross_product_check_2.q.out 
> 2d6eb38 
>   ql/src/test/results/clientpositive/spark/dynamic_rdd_cache.q.out 9043fb1 
>   ql/src/test/results/clientpositive/spark/groupby_sort_1_23.q.out 239e803 
>   ql/src/test/results/clientpositive/spark/groupby_sort_skew_1_23.q.out 
> 8370bbe 
>   ql/src/test/results/clientpositive/spark/union_remove_25.q.out b771fe9 
>   ql/src/test/results/clientpositive/spark/union_view.q.out cce7710 
>   ql/src/test/results/clientpositive/subquery_notin.q.out ed86079

Re: Review Request 41582: HIVE-12713: Miscellaneous improvements in driver compile and execute logging

2015-12-21 Thread Xuefu Zhang 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/41582/#review111501
---

Ship it!


Ship It!

- Xuefu Zhang


On Dec. 21, 2015, 3:46 p.m., Chaoyu Tang wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/41582/
> ---
> 
> (Updated Dec. 21, 2015, 3:46 p.m.)
> 
> 
> Review request for hive.
> 
> 
> Bugs: HIVE-12713
> https://issues.apache.org/jira/browse/HIVE-12713
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Patch improves the driver compile and execute logging in following:
> 1. ensuring that only the redacted query to be logged out
> 2. removing redundant variable substitution in HS2 SQLOperation
> 3. logging out the query and its compilation time without having to enable 
> PerfLogger debug, to help identify badly written queries which take a lot of 
> time to compile and probably cause other good queries to be queued 
> (HIVE-12516)
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/ql/log/PerfLogger.java 98ebd50 
>   
> itests/hive-unit/src/test/java/org/apache/hive/beeline/TestBeeLineWithArgs.java
>  7cc0acf 
>   itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestJdbcDriver2.java 
> e9206b9 
>   
> itests/hive-unit/src/test/java/org/apache/hive/jdbc/cbo_rp_TestJdbcDriver2.java
>  c66f166 
>   
> itests/hive-unit/src/test/java/org/apache/hive/service/cli/operation/TestOperationLoggingAPIWithMr.java
>  d21571e 
>   
> itests/hive-unit/src/test/java/org/apache/hive/service/cli/operation/TestOperationLoggingAPIWithTez.java
>  8b21647 
>   ql/src/java/org/apache/hadoop/hive/ql/Driver.java 3d5f3b5 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/ParseDriver.java c33bb66 
>   service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java 
> d90dd0d 
> 
> Diff: https://reviews.apache.org/r/41582/diff/
> 
> 
> Testing
> ---
> 
> 1. Manual tests
> 2. Precommit tests
> 
> 
> Thanks,
> 
> Chaoyu Tang
> 
>