[jira] [Commented] (HIVE-13928) Hive2: float value need to be single quoted inside where clause to return rows when it doesn't have to be

[Takuma Wakamori (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-13928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15328965#comment-15328965
 ] 

Takuma Wakamori commented on HIVE-13928:


Hi.
Does anyone working on this issue?
If not, I would like to fix it. Thanks!

> Hive2: float value need to be single quoted inside where clause to return 
> rows when it doesn't have to be
> -
>
> Key: HIVE-13928
> URL: https://issues.apache.org/jira/browse/HIVE-13928
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 2.1.0
>Reporter: Takahiko Saito
>Priority: Critical
>
> The below select where with float value does not return any row:
> {noformat}
> 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4> drop table test;
> No rows affected (0.212 seconds)
> 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4> create table test (f float);
> No rows affected (1.131 seconds)
> 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4> insert into table test values 
> (-35664.76),(29497.34);
> No rows affected (2.482 seconds)
> 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4> select * from test;
> ++--+
> |   test.f   |
> ++--+
> | -35664.76  |
> | 29497.34   |
> ++--+
> 2 rows selected (0.142 seconds)
> 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4> select * from test where f = 
> -35664.76;
> +-+--+
> | test.f  |
> +-+--+
> +-+--+
> {noformat}
> The workaround is to single quote float value:
> {noformat}
> 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4> select * from test where f = 
> '-35664.76';
> ++--+
> |   test.f   |
> ++--+
> | -35664.76  |
> ++--+
> 1 row selected (0.163 seconds)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-14008) Duplicate line in LLAP SecretManager

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-14008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15328942#comment-15328942
 ] 

Hive QA commented on HIVE-14008:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12810162/HIVE-14008.patch

{color:red}ERROR:{color} -1 due to no test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 10230 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_acid_globallimit
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_multiinsert
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3
{noformat}

Test results: 
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/119/testReport
Console output: 
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/119/console
Test logs: 
http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-119/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 3 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12810162 - PreCommit-HIVE-MASTER-Build

> Duplicate line in LLAP SecretManager
> 
>
> Key: HIVE-14008
> URL: https://issues.apache.org/jira/browse/HIVE-14008
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Trivial
> Attachments: HIVE-14008.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-13901) Hivemetastore add partitions can be slow depending on filesystems

[Rajesh Balamohan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-13901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15328900#comment-15328900
 ] 

Rajesh Balamohan commented on HIVE-13901:
-

These tests had multiple creations of HSMHandler creating threadpools often 
than needed. "hive.metastore.fshandler.threads" can be a static daemon thread 
pool which can be shared in the same JVM. Will make the changes along with 
rename and post it. 

> Hivemetastore add partitions can be slow depending on filesystems
> -
>
> Key: HIVE-13901
> URL: https://issues.apache.org/jira/browse/HIVE-13901
> Project: Hive
>  Issue Type: Sub-task
>  Components: Metastore
>Reporter: Rajesh Balamohan
>Assignee: Rajesh Balamohan
>Priority: Minor
> Attachments: HIVE-13901.1.patch, HIVE-13901.2.patch
>
>
> Depending on FS, creating external tables & adding partitions can be 
> expensive (e.g msck which adds all partitions).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-14009) Acid DB creation error in HiveQA

[Eugene Koifman (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-14009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15328890#comment-15328890
 ] 

Eugene Koifman commented on HIVE-14009:
---

why would there be parallel threads creating the same DB?
I thought UTs are supposed to run sequentially (on a given machine)

> Acid DB creation error in HiveQA
> 
>
> Key: HIVE-14009
> URL: https://issues.apache.org/jira/browse/HIVE-14009
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>
> Seen when running TestEncryptedHDFSCliDriver, at least with Hadoop 2.7.2 
> (HIVE-13930). 
> Looks like such issues are usually caused by concurrent db creation from 
> multiple threads.
> {noformat}
> java.lang.RuntimeException: Unable to set up transaction database for 
> testing: Exception during creation of file 
> /home/hiveptest/54.219.24.101-hiveptest-0/apache-github-source-source/itests/qtest/target/tmp/junit_metastore_db/seg0/cc60.dat
>  for container
>   at 
> org.apache.hadoop.hive.metastore.txn.TxnHandler.checkQFileTestHack(TxnHandler.java:2172)
>  ~[hive-metastore-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
>   at 
> org.apache.hadoop.hive.metastore.txn.TxnHandler.setConf(TxnHandler.java:228) 
> ~[hive-metastore-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
>   at 
> org.apache.hadoop.hive.metastore.txn.TxnUtils.getTxnStore(TxnUtils.java:96) 
> [hive-metastore-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
>   at 
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getTxnHandler(HiveMetaStore.java:557)
>  [hive-metastore-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
>   at 
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.heartbeat(HiveMetaStore.java:5902)
>  [hive-metastore-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 
> ~[?:1.8.0_25]
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) 
> ~[?:1.8.0_25]
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>  ~[?:1.8.0_25]
>   at java.lang.reflect.Method.invoke(Method.java:483) ~[?:1.8.0_25]
>   at 
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:140)
>  [hive-metastore-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
>   at 
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:99)
>  [hive-metastore-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
>   at com.sun.proxy.$Proxy111.heartbeat(Unknown Source) [?:?]
>   at 
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.heartbeat(HiveMetaStoreClient.java:2140)
>  [hive-metastore-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 
> ~[?:1.8.0_25]
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) 
> ~[?:1.8.0_25]
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>  ~[?:1.8.0_25]
>   at java.lang.reflect.Method.invoke(Method.java:483) ~[?:1.8.0_25]
>   at 
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:154)
>  [hive-metastore-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
>   at com.sun.proxy.$Proxy112.heartbeat(Unknown Source) [?:?]
>   at 
> org.apache.hadoop.hive.ql.lockmgr.DbTxnManager$SynchronizedMetaStoreClient.heartbeat(DbTxnManager.java:663)
>  [hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
>   at 
> org.apache.hadoop.hive.ql.lockmgr.DbTxnManager.heartbeat(DbTxnManager.java:423)
>  [hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
>   at 
> org.apache.hadoop.hive.ql.lockmgr.DbTxnManager$Heartbeater.run(DbTxnManager.java:633)
>  [hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) 
> [?:1.8.0_25]
>   at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) 
> [?:1.8.0_25]
>   at 
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
>  [?:1.8.0_25]
>   at 
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
>  [?:1.8.0_25]
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>  [?:1.8.0_25]
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>  [?:1.8.0_25]
>   at java.lang.Thread.run(Thread.java:745) [?:1.8.0_25]
> Caused by: java.sql.SQLException: Exception during creation of file 
> /home/hiveptest/54.219.24.101-hiveptest-0/apache-github-source-source/itests/qtest/target/tmp/junit_metastore_db/seg0/cc60.dat
>  for container
>   at 
> org.apache.derby.impl.jdbc.SQLExceptionFactory40.getSQLException(Unknown 
> Source) 

[jira] [Commented] (HIVE-13986) LLAP: kill Tez AM on token errors from plugin

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-13986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15328859#comment-15328859
 ] 

Hive QA commented on HIVE-13986:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12810091/HIVE-13986.01.patch

{color:red}ERROR:{color} -1 due to no test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 6 failed/errored test(s), 10228 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_acid_globallimit
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_13
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_list_bucket
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_multiinsert
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3
org.apache.hadoop.hive.ql.TestTxnCommands.testSimpleAcidInsert
{noformat}

Test results: 
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/118/testReport
Console output: 
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/118/console
Test logs: 
http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-118/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 6 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12810091 - PreCommit-HIVE-MASTER-Build

> LLAP: kill Tez AM on token errors from plugin
> -
>
> Key: HIVE-13986
> URL: https://issues.apache.org/jira/browse/HIVE-13986
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-13986.01.patch, HIVE-13986.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-13771) LLAPIF: generate app ID

[Sergey Shelukhin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-13771?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HIVE-13771:

Attachment: HIVE-13771.02.patch

Fixing the build issue.

> LLAPIF: generate app ID
> ---
>
> Key: HIVE-13771
> URL: https://issues.apache.org/jira/browse/HIVE-13771
> Project: Hive
>  Issue Type: Sub-task
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-13771.01.patch, HIVE-13771.02.patch, 
> HIVE-13771.patch
>
>
> See comments in the HIVE-13675 patch. The uniqueness needs to be ensured; the 
> user may be allowed to supply a prefix (e.g. his YARN app Id, if any) for 
> ease of tracking



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-13827) LLAPIF: authentication on the output channel

[Sergey Shelukhin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-13827?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HIVE-13827:

   Resolution: Fixed
Fix Version/s: 2.2.0
   Status: Resolved  (was: Patch Available)

Committed to master

> LLAPIF: authentication on the output channel
> 
>
> Key: HIVE-13827
> URL: https://issues.apache.org/jira/browse/HIVE-13827
> Project: Hive
>  Issue Type: Sub-task
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Fix For: 2.2.0
>
> Attachments: HIVE-13827.01.patch, HIVE-13827.patch
>
>
> The current thinking is that we'd send the token. There's no protocol on the 
> channel right now.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-13957) vectorized IN is inconsistent with non-vectorized (at least for decimal in (string))

[Sergey Shelukhin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-13957?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HIVE-13957:

  Resolution: Fixed
   Fix Version/s: 2.0.2
  2.2.0
  1.3.0
Target Version/s: 2.1.1
  Status: Resolved  (was: Patch Available)

Committed to some branches. [~jcamachorodriguez] what is the state of 2.1? Can 
I commit there now? I put 2.1.1 as target version for now

> vectorized IN is inconsistent with non-vectorized (at least for decimal in 
> (string))
> 
>
> Key: HIVE-13957
> URL: https://issues.apache.org/jira/browse/HIVE-13957
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Fix For: 1.3.0, 2.2.0, 2.0.2
>
> Attachments: HIVE-13957.01.patch, HIVE-13957.02.patch, 
> HIVE-13957.03.patch, HIVE-13957.patch, HIVE-13957.patch
>
>
> The cast is applied to the column in regular IN, but vectorized IN applies it 
> to the IN() list.
> This can cause queries to produce incorrect results.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-13884) Disallow queries fetching more than a configured number of partitions in PartitionPruner

[Mohit Sabharwal (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-13884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15328764#comment-15328764
 ] 

Mohit Sabharwal commented on HIVE-13884:


Since we are moving the functionality from driver to HMS, should we deprecate 
{{hive.limit.query.max.table.partition}} and introduce a new config called 
{{hive.metastore.retrieve.max.partitions}} ?

All metastore configs have "hive.metastore" prefix. 

Otherwise:
1) The change is backward incompatible for existing users that
are setting this config at HS2 level and are now expected to set it
at HMS level to get the same functionality.
2) Name would be confusing.

We could do the following:
1) Mark {{hive.limit.query.max.table.partition}} as deprecated in HiveConf and 
suggest that user move to {{hive.metastore.retrieve.max.partitions}} at HMS
level.
2) Do not remove current functionality associated with 
{{hive.limit.query.max.table.partition}} in PartitionPruner.
It does do what the description promises - i.e. fail the query before execution 
stage if number of 
partitions associated with any scan operator exceed configured value.
3) Add new config {{hive.metastore.retrieve.max.partitions}} to configure 
functionality in this patch.

Makes sense ?

> Disallow queries fetching more than a configured number of partitions in 
> PartitionPruner
> 
>
> Key: HIVE-13884
> URL: https://issues.apache.org/jira/browse/HIVE-13884
> Project: Hive
>  Issue Type: Improvement
>Reporter: Mohit Sabharwal
>Assignee: Sergio Peña
> Attachments: HIVE-13884.1.patch, HIVE-13884.2.patch, 
> HIVE-13884.3.patch
>
>
> Currently the PartitionPruner requests either all partitions or partitions 
> based on filter expression. In either scenarios, if the number of partitions 
> accessed is large there can be significant memory pressure at the HMS server 
> end.
> We already have a config {{hive.limit.query.max.table.partition}} that 
> enforces limits on number of partitions that may be scanned per operator. But 
> this check happens after the PartitionPruner has already fetched all 
> partitions.
> We should add an option at PartitionPruner level to disallow queries that 
> attempt to access number of partitions beyond a configurable limit.
> Note that {{hive.mapred.mode=strict}} disallow queries without a partition 
> filter in PartitionPruner, but this check accepts any query with a pruning 
> condition, even if partitions fetched are large. In multi-tenant 
> environments, admins could use more control w.r.t. number of partitions 
> allowed based on HMS memory capacity.
> One option is to have PartitionPruner first fetch the partition names 
> (instead of partition specs) and throw an exception if number of partitions 
> exceeds the configured value. Otherwise, fetch the partition specs.
> Looks like the existing {{listPartitionNames}} call could be used if extended 
> to take partition filter expressions like {{getPartitionsByExpr}} call does.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-13930) upgrade Hive to latest Hadoop version

[Sergey Shelukhin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-13930?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HIVE-13930:

Attachment: HIVE-13930.02.patch

> upgrade Hive to latest Hadoop version
> -
>
> Key: HIVE-13930
> URL: https://issues.apache.org/jira/browse/HIVE-13930
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-13930.01.patch, HIVE-13930.02.patch, 
> HIVE-13930.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-13930) upgrade Hive to latest Hadoop version

[Sergey Shelukhin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-13930?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HIVE-13930:

Attachment: (was: HIVE-13930.02.patch)

> upgrade Hive to latest Hadoop version
> -
>
> Key: HIVE-13930
> URL: https://issues.apache.org/jira/browse/HIVE-13930
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-13930.01.patch, HIVE-13930.02.patch, 
> HIVE-13930.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Issue Comment Deleted] (HIVE-13930) upgrade Hive to latest Hadoop version

[Sergey Shelukhin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-13930?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HIVE-13930:

Comment: was deleted

(was: Looks like all spark tests fail with this:
{noformat}
2016-06-11T15:51:36,748 INFO  [stderr-redir-1[]]: client.SparkClientImpl 
(SparkClientImpl.java:run(616)) - java.lang.NoClassDefFoundError: 
org/apache/hadoop/crypto/key/KeyProvider
2016-06-11T15:51:36,748 INFO  [stderr-redir-1[]]: client.SparkClientImpl 
(SparkClientImpl.java:run(616)) -  at java.lang.Class.forName0(Native 
Method)
2016-06-11T15:51:36,748 INFO  [stderr-redir-1[]]: client.SparkClientImpl 
(SparkClientImpl.java:run(616)) -  at 
java.lang.Class.forName(Class.java:260)
2016-06-11T15:51:36,748 INFO  [stderr-redir-1[]]: client.SparkClientImpl 
(SparkClientImpl.java:run(616)) -  at 
org.apache.hadoop.hive.shims.ShimLoader.createShim(ShimLoader.java:143)
2016-06-11T15:51:36,748 INFO  [stderr-redir-1[]]: client.SparkClientImpl 
(SparkClientImpl.java:run(616)) -  at 
org.apache.hadoop.hive.shims.ShimLoader.loadShims(ShimLoader.java:138)
2016-06-11T15:51:36,748 INFO  [stderr-redir-1[]]: client.SparkClientImpl 
(SparkClientImpl.java:run(616)) -  at 
org.apache.hadoop.hive.shims.ShimLoader.getHadoopShims(ShimLoader.java:95)
2016-06-11T15:51:36,748 INFO  [stderr-redir-1[]]: client.SparkClientImpl 
(SparkClientImpl.java:run(616)) -  at 
org.apache.hadoop.hive.ql.io.AcidUtils.(AcidUtils.java:115)
2016-06-11T15:51:36,748 INFO  [stderr-redir-1[]]: client.SparkClientImpl 
(SparkClientImpl.java:run(616)) -  at 
org.apache.hadoop.hive.ql.plan.TableScanDesc.(TableScanDesc.java:129)
2016-06-11T15:51:36,749 INFO  [stderr-redir-1[]]: client.SparkClientImpl 
(SparkClientImpl.java:run(616)) -  at 
org.apache.hadoop.hive.ql.plan.TableScanDesc.(TableScanDesc.java:122)
2016-06-11T15:51:36,749 INFO  [stderr-redir-1[]]: client.SparkClientImpl 
(SparkClientImpl.java:run(616)) -  at 
org.apache.hadoop.hive.ql.plan.TableScanDesc.(TableScanDesc.java:113)
2016-06-11T15:51:36,749 INFO  [stderr-redir-1[]]: client.SparkClientImpl 
(SparkClientImpl.java:run(616)) -  at 
org.apache.hadoop.hive.ql.plan.TableScanDescConstructorAccess.newInstance(Unknown
 Source)
2016-06-11T15:51:36,749 INFO  [stderr-redir-1[]]: client.SparkClientImpl 
(SparkClientImpl.java:run(616)) -  at 
org.apache.hive.com.esotericsoftware.kryo.Kryo$DefaultInstantiatorStrategy$1.newInstance(Kryo.java:1237)
2016-06-11T15:51:36,749 INFO  [stderr-redir-1[]]: client.SparkClientImpl 
(SparkClientImpl.java:run(616)) -  at 
org.apache.hive.com.esotericsoftware.kryo.Kryo.newInstance(Kryo.java:1090)
2016-06-11T15:51:36,749 INFO  [stderr-redir-1[]]: client.SparkClientImpl 
(SparkClientImpl.java:run(616)) -  at 
org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.create(FieldSerializer.java:570)
2016-06-11T15:51:36,749 INFO  [stderr-redir-1[]]: client.SparkClientImpl 
(SparkClientImpl.java:run(616)) -  at 
org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:546)
2016-06-11T15:51:36,749 INFO  [stderr-redir-1[]]: client.SparkClientImpl 
(SparkClientImpl.java:run(616)) -  at 
org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:708)
2016-06-11T15:51:36,749 INFO  [stderr-redir-1[]]: client.SparkClientImpl 
(SparkClientImpl.java:run(616)) -  at 
org.apache.hadoop.hive.ql.exec.SerializationUtilities$KryoWithHooks.readObject(SerializationUtilities.java:213)
2016-06-11T15:51:36,749 INFO  [stderr-redir-1[]]: client.SparkClientImpl 
(SparkClientImpl.java:run(616)) -  at 
org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:125)
2016-06-11T15:51:36,749 INFO  [stderr-redir-1[]]: client.SparkClientImpl 
(SparkClientImpl.java:run(616)) -  at 
org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:551)
2016-06-11T15:51:36,749 INFO  [stderr-redir-1[]]: client.SparkClientImpl 
(SparkClientImpl.java:run(616)) -  at 
org.apache.hive.com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:790)
2016-06-11T15:51:36,749 INFO  [stderr-redir-1[]]: client.SparkClientImpl 
(SparkClientImpl.java:run(616)) -  at 
org.apache.hadoop.hive.ql.exec.SerializationUtilities$KryoWithHooks.readClassAndObject(SerializationUtilities.java:175)
2016-06-11T15:51:36,749 INFO  [stderr-redir-1[]]: client.SparkClientImpl 
(SparkClientImpl.java:run(616)) -  at 
org.apache.hive.com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:161)
2016-06-11T15:51:36,749 INFO  [stderr-redir-1[]]: client.SparkClientImpl 
(SparkClientImpl.java:run(616)) -  at 
org.apache.hive.com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:39)
2016-06-11T15:51:36,749 INFO  [stderr-redir-1[]]: client.SparkClientImpl 
(SparkClientImpl.java:run(616)) -  at 

[jira] [Updated] (HIVE-13930) upgrade Hive to latest Hadoop version

[Sergey Shelukhin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-13930?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HIVE-13930:

Attachment: (was: HIVE-13930.02.patch)

> upgrade Hive to latest Hadoop version
> -
>
> Key: HIVE-13930
> URL: https://issues.apache.org/jira/browse/HIVE-13930
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-13930.01.patch, HIVE-13930.02.patch, 
> HIVE-13930.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-13930) upgrade Hive to latest Hadoop version

[Sergey Shelukhin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-13930?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HIVE-13930:

Attachment: HIVE-13930.02.patch

Some more pom updates, and better error from shim loading.

> upgrade Hive to latest Hadoop version
> -
>
> Key: HIVE-13930
> URL: https://issues.apache.org/jira/browse/HIVE-13930
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-13930.01.patch, HIVE-13930.02.patch, 
> HIVE-13930.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-13987) Clarify current error shown when HS2 is down

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-13987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15328740#comment-15328740
 ] 

Hive QA commented on HIVE-13987:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12810085/HIVE-13987.02.patch

{color:red}ERROR:{color} -1 due to no test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 9 failed/errored test(s), 10226 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_acid_globallimit
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_12
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_13
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_list_bucket
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_multiinsert
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3
org.apache.hadoop.hive.llap.tezplugins.TestLlapTaskSchedulerService.testDelayedLocalityNodeCommErrorImmediateAllocation
org.apache.hadoop.hive.ql.TestTxnCommands.testSimpleAcidInsert
org.apache.hive.hcatalog.hbase.TestPigHBaseStorageHandler.org.apache.hive.hcatalog.hbase.TestPigHBaseStorageHandler
{noformat}

Test results: 
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/117/testReport
Console output: 
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/117/console
Test logs: 
http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-117/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 9 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12810085 - PreCommit-HIVE-MASTER-Build

> Clarify current error shown when HS2 is down
> 
>
> Key: HIVE-13987
> URL: https://issues.apache.org/jira/browse/HIVE-13987
> Project: Hive
>  Issue Type: Bug
>  Components: Beeline
>Affects Versions: 2.0.1
>Reporter: Abdullah Yousufi
>Assignee: Abdullah Yousufi
>Priority: Minor
> Fix For: 2.2.0
>
> Attachments: HIVE-13987.01.patch, HIVE-13987.02.patch
>
>
> When HS2 is down and a query is run, the following error is shown in beeline:
> {code}
> 0: jdbc:hive2://localhost:1> show tables;
> Error: org.apache.thrift.transport.TTransportException (state=08S01,code=0)
> {code}
> It may be more helpful to also indicate that the reason for this is that HS2 
> is down, such as:
> {code}
> 0: jdbc:hive2://localhost:1> show tables;
> HS2 may be unavailable, check server status
> Error: org.apache.thrift.transport.TTransportException (state=08S01,code=0)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-13930) upgrade Hive to latest Hadoop version

[Sergey Shelukhin (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-13930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15328738#comment-15328738
 ] 

Sergey Shelukhin commented on HIVE-13930:
-

Actually nm the nm, the real error for Spark tests is
{noformat}
2016-06-13T17:22:47,157 INFO  [stderr-redir-1[]]: client.SparkClientImpl 
(SparkClientImpl.java:run(616)) - 16/06/13 17:22:47 INFO RemoteDriver: Failed 
to run job 27c59fd3-9610-4ae4-b126-f548a64dfdf3
2016-06-13T17:22:47,157 INFO  [stderr-redir-1[]]: client.SparkClientImpl 
(SparkClientImpl.java:run(616)) - java.lang.NoClassDefFoundError: Could not 
initialize class org.apache.hadoop.hive.ql.io.AcidUtils
2016-06-13T17:22:47,157 INFO  [stderr-redir-1[]]: client.SparkClientImpl 
(SparkClientImpl.java:run(616)) -  at 
org.apache.hadoop.hive.ql.plan.TableScanDesc.(TableScanDesc.java:129)
2016-06-13T17:22:47,157 INFO  [stderr-redir-1[]]: client.SparkClientImpl 
(SparkClientImpl.java:run(616)) -  at 
org.apache.hadoop.hive.ql.plan.TableScanDesc.(TableScanDesc.java:122)
2016-06-13T17:22:47,157 INFO  [stderr-redir-1[]]: client.SparkClientImpl 
(SparkClientImpl.java:run(616)) -  at 
org.apache.hadoop.hive.ql.plan.TableScanDesc.(TableScanDesc.java:113)
2016-06-13T17:22:47,157 INFO  [stderr-redir-1[]]: client.SparkClientImpl 
(SparkClientImpl.java:run(616)) -  at 
org.apache.hadoop.hive.ql.plan.TableScanDescConstructorAccess.newInstance(Unknown
 Source)
2016-06-13T17:22:47,157 INFO  [stderr-redir-1[]]: client.SparkClientImpl 
(SparkClientImpl.java:run(616)) -  at 
org.apache.hive.com.esotericsoftware.kryo.Kryo$DefaultInstantiatorStrategy$1.newInstance(Kryo.java:1237)
{noformat}
It looks like AcidUtils fails to initialize due to static init error from 
shims. When logged, the shims init has the same error - KeyProvider class is 
not found.
{noformat}
2016-06-13T17:45:01,037 INFO  [stderr-redir-1[]]: client.SparkClientImpl 
(SparkClientImpl.java:run(616)) - java.lang.NoClassDefFoundError: 
org/apache/hadoop/crypto/key/KeyProvider
2016-06-13T17:45:01,037 INFO  [stderr-redir-1[]]: client.SparkClientImpl 
(SparkClientImpl.java:run(616)) -  at java.lang.Class.forName0(Native 
Method)
2016-06-13T17:45:01,037 INFO  [stderr-redir-1[]]: client.SparkClientImpl 
(SparkClientImpl.java:run(616)) -  at 
java.lang.Class.forName(Class.java:264)
2016-06-13T17:45:01,037 INFO  [stderr-redir-1[]]: client.SparkClientImpl 
(SparkClientImpl.java:run(616)) -  at 
org.apache.hadoop.hive.shims.ShimLoader.createShim(ShimLoader.java:143)
2016-06-13T17:45:01,037 INFO  [stderr-redir-1[]]: client.SparkClientImpl 
(SparkClientImpl.java:run(616)) -  at 
org.apache.hadoop.hive.shims.ShimLoader.loadShims(ShimLoader.java:138)
2016-06-13T17:45:01,037 INFO  [stderr-redir-1[]]: client.SparkClientImpl 
(SparkClientImpl.java:run(616)) -  at 
org.apache.hadoop.hive.shims.ShimLoader.getHadoopShims(ShimLoader.java:95)
2016-06-13T17:45:01,037 INFO  [stderr-redir-1[]]: client.SparkClientImpl 
(SparkClientImpl.java:run(616)) -  at 
org.apache.hadoop.hive.ql.io.AcidUtils.(AcidUtils.java:119)
{noformat}
As far as I see, the packaging for KeyProvider didn't change between Hadoop 2.6 
and 2.7.2.

I wonder if it's caused by the fact that HoS tests use prepackaged spark that 
includes Hadoop 2.4. KeyProvider class didn't exist in that version of Hadoop, 
so it might be causing the issue. Not sure why it worked on 2.6

[~xuefuz] can you please comment? As I have mentioned before this dependency 
scheme is really problematic, but whereas previously it was just inconvenient, 
in this case it's likely causing a real issue where we would be blocked from 
upgrading Hadoop unless we disable  Spark mini-cluster tests, or there's some 
quick way to the the tgz repackaged with a newer hadoop version.
Or otherwise, I wonder if there's a different error here that isn't in hive.log




> upgrade Hive to latest Hadoop version
> -
>
> Key: HIVE-13930
> URL: https://issues.apache.org/jira/browse/HIVE-13930
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-13930.01.patch, HIVE-13930.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-13930) upgrade Hive to latest Hadoop version

[Sergey Shelukhin (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-13930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15328706#comment-15328706
 ] 

Sergey Shelukhin commented on HIVE-13930:
-

Looks like all spark tests fail with this:
{noformat}
2016-06-11T15:51:36,748 INFO  [stderr-redir-1[]]: client.SparkClientImpl 
(SparkClientImpl.java:run(616)) - java.lang.NoClassDefFoundError: 
org/apache/hadoop/crypto/key/KeyProvider
2016-06-11T15:51:36,748 INFO  [stderr-redir-1[]]: client.SparkClientImpl 
(SparkClientImpl.java:run(616)) -  at java.lang.Class.forName0(Native 
Method)
2016-06-11T15:51:36,748 INFO  [stderr-redir-1[]]: client.SparkClientImpl 
(SparkClientImpl.java:run(616)) -  at 
java.lang.Class.forName(Class.java:260)
2016-06-11T15:51:36,748 INFO  [stderr-redir-1[]]: client.SparkClientImpl 
(SparkClientImpl.java:run(616)) -  at 
org.apache.hadoop.hive.shims.ShimLoader.createShim(ShimLoader.java:143)
2016-06-11T15:51:36,748 INFO  [stderr-redir-1[]]: client.SparkClientImpl 
(SparkClientImpl.java:run(616)) -  at 
org.apache.hadoop.hive.shims.ShimLoader.loadShims(ShimLoader.java:138)
2016-06-11T15:51:36,748 INFO  [stderr-redir-1[]]: client.SparkClientImpl 
(SparkClientImpl.java:run(616)) -  at 
org.apache.hadoop.hive.shims.ShimLoader.getHadoopShims(ShimLoader.java:95)
2016-06-11T15:51:36,748 INFO  [stderr-redir-1[]]: client.SparkClientImpl 
(SparkClientImpl.java:run(616)) -  at 
org.apache.hadoop.hive.ql.io.AcidUtils.(AcidUtils.java:115)
2016-06-11T15:51:36,748 INFO  [stderr-redir-1[]]: client.SparkClientImpl 
(SparkClientImpl.java:run(616)) -  at 
org.apache.hadoop.hive.ql.plan.TableScanDesc.(TableScanDesc.java:129)
2016-06-11T15:51:36,749 INFO  [stderr-redir-1[]]: client.SparkClientImpl 
(SparkClientImpl.java:run(616)) -  at 
org.apache.hadoop.hive.ql.plan.TableScanDesc.(TableScanDesc.java:122)
2016-06-11T15:51:36,749 INFO  [stderr-redir-1[]]: client.SparkClientImpl 
(SparkClientImpl.java:run(616)) -  at 
org.apache.hadoop.hive.ql.plan.TableScanDesc.(TableScanDesc.java:113)
2016-06-11T15:51:36,749 INFO  [stderr-redir-1[]]: client.SparkClientImpl 
(SparkClientImpl.java:run(616)) -  at 
org.apache.hadoop.hive.ql.plan.TableScanDescConstructorAccess.newInstance(Unknown
 Source)
2016-06-11T15:51:36,749 INFO  [stderr-redir-1[]]: client.SparkClientImpl 
(SparkClientImpl.java:run(616)) -  at 
org.apache.hive.com.esotericsoftware.kryo.Kryo$DefaultInstantiatorStrategy$1.newInstance(Kryo.java:1237)
2016-06-11T15:51:36,749 INFO  [stderr-redir-1[]]: client.SparkClientImpl 
(SparkClientImpl.java:run(616)) -  at 
org.apache.hive.com.esotericsoftware.kryo.Kryo.newInstance(Kryo.java:1090)
2016-06-11T15:51:36,749 INFO  [stderr-redir-1[]]: client.SparkClientImpl 
(SparkClientImpl.java:run(616)) -  at 
org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.create(FieldSerializer.java:570)
2016-06-11T15:51:36,749 INFO  [stderr-redir-1[]]: client.SparkClientImpl 
(SparkClientImpl.java:run(616)) -  at 
org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:546)
2016-06-11T15:51:36,749 INFO  [stderr-redir-1[]]: client.SparkClientImpl 
(SparkClientImpl.java:run(616)) -  at 
org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:708)
2016-06-11T15:51:36,749 INFO  [stderr-redir-1[]]: client.SparkClientImpl 
(SparkClientImpl.java:run(616)) -  at 
org.apache.hadoop.hive.ql.exec.SerializationUtilities$KryoWithHooks.readObject(SerializationUtilities.java:213)
2016-06-11T15:51:36,749 INFO  [stderr-redir-1[]]: client.SparkClientImpl 
(SparkClientImpl.java:run(616)) -  at 
org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:125)
2016-06-11T15:51:36,749 INFO  [stderr-redir-1[]]: client.SparkClientImpl 
(SparkClientImpl.java:run(616)) -  at 
org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:551)
2016-06-11T15:51:36,749 INFO  [stderr-redir-1[]]: client.SparkClientImpl 
(SparkClientImpl.java:run(616)) -  at 
org.apache.hive.com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:790)
2016-06-11T15:51:36,749 INFO  [stderr-redir-1[]]: client.SparkClientImpl 
(SparkClientImpl.java:run(616)) -  at 
org.apache.hadoop.hive.ql.exec.SerializationUtilities$KryoWithHooks.readClassAndObject(SerializationUtilities.java:175)
2016-06-11T15:51:36,749 INFO  [stderr-redir-1[]]: client.SparkClientImpl 
(SparkClientImpl.java:run(616)) -  at 
org.apache.hive.com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:161)
2016-06-11T15:51:36,749 INFO  [stderr-redir-1[]]: client.SparkClientImpl 
(SparkClientImpl.java:run(616)) -  at 
org.apache.hive.com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:39)
2016-06-11T15:51:36,749 INFO  [stderr-redir-1[]]: client.SparkClientImpl 
(SparkClientImpl.java:run(616)) -  at 

[jira] [Commented] (HIVE-14009) Acid DB creation error in HiveQA

[Sergey Shelukhin (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-14009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15328700#comment-15328700
 ] 

Sergey Shelukhin commented on HIVE-14009:
-

[~ekoifman] fyi

> Acid DB creation error in HiveQA
> 
>
> Key: HIVE-14009
> URL: https://issues.apache.org/jira/browse/HIVE-14009
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>
> Seen when running TestEncryptedHDFSCliDriver, at least with Hadoop 2.7.2 
> (HIVE-13930). 
> Looks like such issues are usually caused by concurrent db creation from 
> multiple threads.
> {noformat}
> java.lang.RuntimeException: Unable to set up transaction database for 
> testing: Exception during creation of file 
> /home/hiveptest/54.219.24.101-hiveptest-0/apache-github-source-source/itests/qtest/target/tmp/junit_metastore_db/seg0/cc60.dat
>  for container
>   at 
> org.apache.hadoop.hive.metastore.txn.TxnHandler.checkQFileTestHack(TxnHandler.java:2172)
>  ~[hive-metastore-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
>   at 
> org.apache.hadoop.hive.metastore.txn.TxnHandler.setConf(TxnHandler.java:228) 
> ~[hive-metastore-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
>   at 
> org.apache.hadoop.hive.metastore.txn.TxnUtils.getTxnStore(TxnUtils.java:96) 
> [hive-metastore-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
>   at 
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getTxnHandler(HiveMetaStore.java:557)
>  [hive-metastore-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
>   at 
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.heartbeat(HiveMetaStore.java:5902)
>  [hive-metastore-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 
> ~[?:1.8.0_25]
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) 
> ~[?:1.8.0_25]
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>  ~[?:1.8.0_25]
>   at java.lang.reflect.Method.invoke(Method.java:483) ~[?:1.8.0_25]
>   at 
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:140)
>  [hive-metastore-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
>   at 
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:99)
>  [hive-metastore-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
>   at com.sun.proxy.$Proxy111.heartbeat(Unknown Source) [?:?]
>   at 
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.heartbeat(HiveMetaStoreClient.java:2140)
>  [hive-metastore-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 
> ~[?:1.8.0_25]
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) 
> ~[?:1.8.0_25]
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>  ~[?:1.8.0_25]
>   at java.lang.reflect.Method.invoke(Method.java:483) ~[?:1.8.0_25]
>   at 
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:154)
>  [hive-metastore-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
>   at com.sun.proxy.$Proxy112.heartbeat(Unknown Source) [?:?]
>   at 
> org.apache.hadoop.hive.ql.lockmgr.DbTxnManager$SynchronizedMetaStoreClient.heartbeat(DbTxnManager.java:663)
>  [hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
>   at 
> org.apache.hadoop.hive.ql.lockmgr.DbTxnManager.heartbeat(DbTxnManager.java:423)
>  [hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
>   at 
> org.apache.hadoop.hive.ql.lockmgr.DbTxnManager$Heartbeater.run(DbTxnManager.java:633)
>  [hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) 
> [?:1.8.0_25]
>   at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) 
> [?:1.8.0_25]
>   at 
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
>  [?:1.8.0_25]
>   at 
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
>  [?:1.8.0_25]
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>  [?:1.8.0_25]
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>  [?:1.8.0_25]
>   at java.lang.Thread.run(Thread.java:745) [?:1.8.0_25]
> Caused by: java.sql.SQLException: Exception during creation of file 
> /home/hiveptest/54.219.24.101-hiveptest-0/apache-github-source-source/itests/qtest/target/tmp/junit_metastore_db/seg0/cc60.dat
>  for container
>   at 
> org.apache.derby.impl.jdbc.SQLExceptionFactory40.getSQLException(Unknown 
> Source) ~[derby-10.10.2.0.jar:?]
>   at org.apache.derby.impl.jdbc.Util.newEmbedSQLException(Unknown Source) 
> 

[jira] [Commented] (HIVE-13930) upgrade Hive to latest Hadoop version

[Sergey Shelukhin (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-13930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15328687#comment-15328687
 ] 

Sergey Shelukhin commented on HIVE-13930:
-

Looks like all spark tests fail with this:
{noformat}
2016-06-11T15:51:36,748 INFO  [stderr-redir-1[]]: client.SparkClientImpl 
(SparkClientImpl.java:run(616)) - java.lang.NoClassDefFoundError: 
org/apache/hadoop/crypto/key/KeyProvider
2016-06-11T15:51:36,748 INFO  [stderr-redir-1[]]: client.SparkClientImpl 
(SparkClientImpl.java:run(616)) -  at java.lang.Class.forName0(Native 
Method)
2016-06-11T15:51:36,748 INFO  [stderr-redir-1[]]: client.SparkClientImpl 
(SparkClientImpl.java:run(616)) -  at 
java.lang.Class.forName(Class.java:260)
2016-06-11T15:51:36,748 INFO  [stderr-redir-1[]]: client.SparkClientImpl 
(SparkClientImpl.java:run(616)) -  at 
org.apache.hadoop.hive.shims.ShimLoader.createShim(ShimLoader.java:143)
2016-06-11T15:51:36,748 INFO  [stderr-redir-1[]]: client.SparkClientImpl 
(SparkClientImpl.java:run(616)) -  at 
org.apache.hadoop.hive.shims.ShimLoader.loadShims(ShimLoader.java:138)
2016-06-11T15:51:36,748 INFO  [stderr-redir-1[]]: client.SparkClientImpl 
(SparkClientImpl.java:run(616)) -  at 
org.apache.hadoop.hive.shims.ShimLoader.getHadoopShims(ShimLoader.java:95)
2016-06-11T15:51:36,748 INFO  [stderr-redir-1[]]: client.SparkClientImpl 
(SparkClientImpl.java:run(616)) -  at 
org.apache.hadoop.hive.ql.io.AcidUtils.(AcidUtils.java:115)
2016-06-11T15:51:36,748 INFO  [stderr-redir-1[]]: client.SparkClientImpl 
(SparkClientImpl.java:run(616)) -  at 
org.apache.hadoop.hive.ql.plan.TableScanDesc.(TableScanDesc.java:129)
2016-06-11T15:51:36,749 INFO  [stderr-redir-1[]]: client.SparkClientImpl 
(SparkClientImpl.java:run(616)) -  at 
org.apache.hadoop.hive.ql.plan.TableScanDesc.(TableScanDesc.java:122)
2016-06-11T15:51:36,749 INFO  [stderr-redir-1[]]: client.SparkClientImpl 
(SparkClientImpl.java:run(616)) -  at 
org.apache.hadoop.hive.ql.plan.TableScanDesc.(TableScanDesc.java:113)
2016-06-11T15:51:36,749 INFO  [stderr-redir-1[]]: client.SparkClientImpl 
(SparkClientImpl.java:run(616)) -  at 
org.apache.hadoop.hive.ql.plan.TableScanDescConstructorAccess.newInstance(Unknown
 Source)
2016-06-11T15:51:36,749 INFO  [stderr-redir-1[]]: client.SparkClientImpl 
(SparkClientImpl.java:run(616)) -  at 
org.apache.hive.com.esotericsoftware.kryo.Kryo$DefaultInstantiatorStrategy$1.newInstance(Kryo.java:1237)
2016-06-11T15:51:36,749 INFO  [stderr-redir-1[]]: client.SparkClientImpl 
(SparkClientImpl.java:run(616)) -  at 
org.apache.hive.com.esotericsoftware.kryo.Kryo.newInstance(Kryo.java:1090)
2016-06-11T15:51:36,749 INFO  [stderr-redir-1[]]: client.SparkClientImpl 
(SparkClientImpl.java:run(616)) -  at 
org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.create(FieldSerializer.java:570)
2016-06-11T15:51:36,749 INFO  [stderr-redir-1[]]: client.SparkClientImpl 
(SparkClientImpl.java:run(616)) -  at 
org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:546)
2016-06-11T15:51:36,749 INFO  [stderr-redir-1[]]: client.SparkClientImpl 
(SparkClientImpl.java:run(616)) -  at 
org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:708)
2016-06-11T15:51:36,749 INFO  [stderr-redir-1[]]: client.SparkClientImpl 
(SparkClientImpl.java:run(616)) -  at 
org.apache.hadoop.hive.ql.exec.SerializationUtilities$KryoWithHooks.readObject(SerializationUtilities.java:213)
2016-06-11T15:51:36,749 INFO  [stderr-redir-1[]]: client.SparkClientImpl 
(SparkClientImpl.java:run(616)) -  at 
org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:125)
2016-06-11T15:51:36,749 INFO  [stderr-redir-1[]]: client.SparkClientImpl 
(SparkClientImpl.java:run(616)) -  at 
org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:551)
2016-06-11T15:51:36,749 INFO  [stderr-redir-1[]]: client.SparkClientImpl 
(SparkClientImpl.java:run(616)) -  at 
org.apache.hive.com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:790)
2016-06-11T15:51:36,749 INFO  [stderr-redir-1[]]: client.SparkClientImpl 
(SparkClientImpl.java:run(616)) -  at 
org.apache.hadoop.hive.ql.exec.SerializationUtilities$KryoWithHooks.readClassAndObject(SerializationUtilities.java:175)
2016-06-11T15:51:36,749 INFO  [stderr-redir-1[]]: client.SparkClientImpl 
(SparkClientImpl.java:run(616)) -  at 
org.apache.hive.com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:161)
2016-06-11T15:51:36,749 INFO  [stderr-redir-1[]]: client.SparkClientImpl 
(SparkClientImpl.java:run(616)) -  at 
org.apache.hive.com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:39)
2016-06-11T15:51:36,749 INFO  [stderr-redir-1[]]: client.SparkClientImpl 
(SparkClientImpl.java:run(616)) -  at 

[jira] [Updated] (HIVE-13964) Add a parameter to beeline to allow a properties file to be passed in

[Abdullah Yousufi (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-13964?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Abdullah Yousufi updated HIVE-13964:

Attachment: HIVE-13964.04.patch

> Add a parameter to beeline to allow a properties file to be passed in
> -
>
> Key: HIVE-13964
> URL: https://issues.apache.org/jira/browse/HIVE-13964
> Project: Hive
>  Issue Type: New Feature
>  Components: Beeline
>Affects Versions: 2.0.1
>Reporter: Abdullah Yousufi
>Assignee: Abdullah Yousufi
>Priority: Minor
> Fix For: 2.2.0
>
> Attachments: HIVE-13964.01.patch, HIVE-13964.02.patch, 
> HIVE-13964.03.patch, HIVE-13964.04.patch
>
>
> HIVE-6652 removed the ability to pass in a properties file as a beeline 
> parameter. It may be a useful feature to be able to pass the file in is a 
> parameter, such as --property-file.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-13965) Empty resultset run into Exception when using Thrift Binary Serde

[Ziyang Zhao (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-13965?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ziyang Zhao updated HIVE-13965:
---
Attachment: HIVE-13965.1.patch.txt

this patch check if the resultset is empty, if yes then skip the writing 
resultset step.

> Empty resultset run into Exception when using Thrift Binary Serde
> -
>
> Key: HIVE-13965
> URL: https://issues.apache.org/jira/browse/HIVE-13965
> Project: Hive
>  Issue Type: Sub-task
>  Components: HiveServer2
>Affects Versions: 2.1.0
>Reporter: Ziyang Zhao
>Assignee: Ziyang Zhao
> Attachments: HIVE-13965.1.patch.txt
>
>
> This error can be reproduced by enabling thrift binary serde, using beeline 
> connect to hiveserver2 and executing the following commands:
> >create table test3(num1 int);
> >create table test4(num1 int);
> >insert into test3 values(1);
> >insert into test4 values(2);
> >select * from test3 join test4 on test3.num1=test4.num1;
> The result should be empty, but it gives an exception:
> Diagnostic Messages for this Task:
> Error: java.lang.RuntimeException: Hive Runtime Error while closing operators
> at 
> org.apache.hadoop.hive.ql.exec.mr.ExecMapper.close(ExecMapper.java:206)
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:61)
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
> Caused by: java.lang.NullPointerException
> at 
> org.apache.hadoop.hive.ql.exec.FileSinkOperator.closeOp(FileSinkOperator.java:1029)
> at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:641)
> at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:655)
> at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:655)
> at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:655)
> at 
> org.apache.hadoop.hive.ql.exec.mr.ExecMapper.close(ExecMapper.java:195)
> ... 8 more
> This error is caused in FileSinkOperator.java. 
> If the resultset is empty, function process() will not be called, so variable 
> "fpaths" will not be set. When run into CloseOp(), 
> if (conf.isHiveServerQuery() && HiveConf.getBoolVar(hconf,
>  HiveConf.ConfVars.HIVE_SERVER2_THRIFT_RESULTSET_SERIALIZE_IN_TASKS) 
> &&
>  
> serializer.getClass().getName().equalsIgnoreCase(ThriftJDBCBinarySerDe.class.getName()))
>  {
>  try {
>recordValue = serializer.serialize(null, inputObjInspectors[0]);
>rowOutWriters = fpaths.outWriters;
>rowOutWriters[0].write(recordValue);
>  } catch (SerDeException | IOException e) {
>throw new HiveException(e);
>  }
>  }
> Here fpaths is null.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-14008) Duplicate line in LLAP SecretManager

[Sergey Shelukhin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-14008?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HIVE-14008:

Status: Patch Available  (was: Open)

> Duplicate line in LLAP SecretManager
> 
>
> Key: HIVE-14008
> URL: https://issues.apache.org/jira/browse/HIVE-14008
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Trivial
> Attachments: HIVE-14008.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-14008) Duplicate line in LLAP SecretManager

[Sergey Shelukhin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-14008?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HIVE-14008:

Attachment: HIVE-14008.patch

[~sseth] trivial patch, can you take a look?

> Duplicate line in LLAP SecretManager
> 
>
> Key: HIVE-14008
> URL: https://issues.apache.org/jira/browse/HIVE-14008
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Trivial
> Attachments: HIVE-14008.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-13866) flatten callstack for directSQL errors

[Sergey Shelukhin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-13866?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HIVE-13866:

   Resolution: Fixed
Fix Version/s: 2.2.0
   Status: Resolved  (was: Patch Available)

Committed to master. Thanks for the review!

> flatten callstack for directSQL errors
> --
>
> Key: HIVE-13866
> URL: https://issues.apache.org/jira/browse/HIVE-13866
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Fix For: 2.2.0
>
> Attachments: HIVE-13866.01.patch, HIVE-13866.patch
>
>
> These errors look like final errors and confuse people. The callstack may be 
> useful if it's some datanucleus/db issue, but it needs to be flattened and 
> logged with a warning that this is not a final query error and that there's a 
> fallback



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-13988) zero length file is being created for empty bucket in tez mode (I)

[Pengcheng Xiong (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-13988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pengcheng Xiong updated HIVE-13988:
---
Summary: zero length file is being created for empty bucket in tez mode (I) 
 (was: zero length file is being created for empty bucket in tez mode)

> zero length file is being created for empty bucket in tez mode (I)
> --
>
> Key: HIVE-13988
> URL: https://issues.apache.org/jira/browse/HIVE-13988
> Project: Hive
>  Issue Type: Bug
>Reporter: Pengcheng Xiong
>Assignee: Pengcheng Xiong
> Attachments: HIVE-13988.01.patch
>
>
> Even though bucket is empty, zero length file is being created in tez mode. 
> steps to reproduce the issue:
> {noformat}
> hive> set hive.execution.engine;
> hive.execution.engine=tez
> hive> drop table if exists emptybucket_orc;
> OK
> Time taken: 5.416 seconds
> hive> create table emptybucket_orc(age int) clustered by (age) sorted by 
> (age) into 99 buckets stored as orc;
> OK
> Time taken: 0.493 seconds
> hive> insert into table emptybucket_orc select distinct(age) from 
> studenttab10k limit 0;
> Query ID = hrt_qa_20160523231955_8b981be7-68c4-4416-8a48-5f8c7ff551c3
> Total jobs = 1
> Launching Job 1 out of 1
> Status: Running (Executing on YARN cluster with App id 
> application_1464045121842_0002)
> --
> VERTICES  MODESTATUS  TOTAL  COMPLETED  RUNNING  PENDING  
> FAILED  KILLED  
> --
> Map 1 ..  llap SUCCEEDED  1  100  
>  0   0  
> Reducer 2 ..  llap SUCCEEDED  1  100  
>  0   0  
> Reducer 3 ..  llap SUCCEEDED  1  100  
>  0   0  
> Reducer 4 ..  llap SUCCEEDED 99 9900  
>  0   0  
> --
> VERTICES: 04/04  [==>>] 100%  ELAPSED TIME: 11.00 s   
>  
> --
> Loading data to table default.emptybucket_orc
> OK
> Time taken: 16.907 seconds
> hive> dfs -ls /apps/hive/warehouse/emptybucket_orc;
> Found 99 items
> -rwxrwxrwx   3 hrt_qa hdfs  0 2016-05-23 23:20 
> /apps/hive/warehouse/emptybucket_orc/00_0
> -rwxrwxrwx   3 hrt_qa hdfs  0 2016-05-23 23:20 
> /apps/hive/warehouse/emptybucket_orc/01_0
> ..
> {noformat}
> Expected behavior:
> In tez mode, zero length file shouldn't get created on hdfs if bucket is empty



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-13986) LLAP: kill Tez AM on token errors from plugin

[Sergey Shelukhin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-13986?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HIVE-13986:

Attachment: HIVE-13986.01.patch

> LLAP: kill Tez AM on token errors from plugin
> -
>
> Key: HIVE-13986
> URL: https://issues.apache.org/jira/browse/HIVE-13986
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-13986.01.patch, HIVE-13986.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-13987) Clarify current error shown when HS2 is down

[Abdullah Yousufi (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-13987?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Abdullah Yousufi updated HIVE-13987:

Attachment: HIVE-13987.02.patch

> Clarify current error shown when HS2 is down
> 
>
> Key: HIVE-13987
> URL: https://issues.apache.org/jira/browse/HIVE-13987
> Project: Hive
>  Issue Type: Bug
>  Components: Beeline
>Affects Versions: 2.0.1
>Reporter: Abdullah Yousufi
>Assignee: Abdullah Yousufi
>Priority: Minor
> Fix For: 2.2.0
>
> Attachments: HIVE-13987.01.patch, HIVE-13987.02.patch
>
>
> When HS2 is down and a query is run, the following error is shown in beeline:
> {code}
> 0: jdbc:hive2://localhost:1> show tables;
> Error: org.apache.thrift.transport.TTransportException (state=08S01,code=0)
> {code}
> It may be more helpful to also indicate that the reason for this is that HS2 
> is down, such as:
> {code}
> 0: jdbc:hive2://localhost:1> show tables;
> HS2 may be unavailable, check server status
> Error: org.apache.thrift.transport.TTransportException (state=08S01,code=0)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-13987) Clarify current error shown when HS2 is down

[JIRA]

[ 
https://issues.apache.org/jira/browse/HIVE-13987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15328414#comment-15328414
 ] 

Sergio Peña commented on HIVE-13987:


Ah cool, it is good then
+1

> Clarify current error shown when HS2 is down
> 
>
> Key: HIVE-13987
> URL: https://issues.apache.org/jira/browse/HIVE-13987
> Project: Hive
>  Issue Type: Bug
>  Components: Beeline
>Affects Versions: 2.0.1
>Reporter: Abdullah Yousufi
>Assignee: Abdullah Yousufi
>Priority: Minor
> Fix For: 2.2.0
>
> Attachments: HIVE-13987.01.patch
>
>
> When HS2 is down and a query is run, the following error is shown in beeline:
> {code}
> 0: jdbc:hive2://localhost:1> show tables;
> Error: org.apache.thrift.transport.TTransportException (state=08S01,code=0)
> {code}
> It may be more helpful to also indicate that the reason for this is that HS2 
> is down, such as:
> {code}
> 0: jdbc:hive2://localhost:1> show tables;
> HS2 may be unavailable, check server status
> Error: org.apache.thrift.transport.TTransportException (state=08S01,code=0)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-14007) Replace ORC module with ORC release

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-14007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15328382#comment-15328382
 ] 

ASF GitHub Bot commented on HIVE-14007:
---

GitHub user omalley opened a pull request:

https://github.com/apache/hive/pull/81

HIVE-14007. Replace ORC module with ORC 1.1.1 release

This patch replaces the inline ORC module with the ORC 1.1.1 release.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/omalley/hive hive-14007

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/hive/pull/81.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #81


commit 4fde122f06709a750dcbbe867be754f04698e9d9
Author: Owen O'Malley 
Date:   2016-06-13T21:53:56Z

HIVE-14007. Replace ORC module with ORC 1.1.1 release




> Replace ORC module with ORC release
> ---
>
> Key: HIVE-14007
> URL: https://issues.apache.org/jira/browse/HIVE-14007
> Project: Hive
>  Issue Type: Bug
>  Components: ORC
>Affects Versions: 2.2.0
>Reporter: Owen O'Malley
>Assignee: Owen O'Malley
> Fix For: 2.2.0
>
>
> This completes moving the core ORC reader & writer to the ORC project.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-13884) Disallow queries fetching more than a configured number of partitions in PartitionPruner

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-13884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15328370#comment-15328370
 ] 

Hive QA commented on HIVE-13884:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12809979/HIVE-13884.3.patch

{color:red}ERROR:{color} -1 due to no test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 361 failed/errored test(s), 10226 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_acid_globallimit
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_alter_partition_coltype
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_analyze_tbl_part
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_autoColumnStats_1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_autoColumnStats_2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_autoColumnStats_8
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin10
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin11
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketsortoptimize_insert_2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cbo_const
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_column_access_stats
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_columnstats_part_coltype
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_columnstats_partlvl
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_columnstats_partlvl_dp
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_combine2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cp_sel
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_explain_dependency
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_explain_dependency2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_explain_logical
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_global_limit
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_index_auto_unused
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_index_bitmap
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_index_bitmap_rc
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_index_compact
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_index_compact_2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_input_part3
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_input_part4
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_input_part6
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join28
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join32
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join32_lessSize
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join33
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_merge_multi_expressions
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_limit_partition_metadataonly
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_12
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_load_dyn_part1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_load_dyn_part10
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_load_dyn_part11
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_load_dyn_part12
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_load_dyn_part13
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_load_dyn_part2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_load_dyn_part3
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_load_dyn_part4
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_load_dyn_part6
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_load_dyn_part7
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_load_dyn_part8
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_load_dyn_part9
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_mapjoin1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_mapjoin_subquery
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_merge3
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_merge_dynamic_partition
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_metadataonly1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_null_column
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_orc_int_type_promotion
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_partition_date
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_partition_decode_name
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_partition_special_char
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_partition_timestamp
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_partition_timestamp2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_partition_type_check
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_partition_type_in_plan
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_partition_varchar1

[jira] [Commented] (HIVE-13964) Add a parameter to beeline to allow a properties file to be passed in

[Abdullah Yousufi (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-13964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15328369#comment-15328369
 ] 

Abdullah Yousufi commented on HIVE-13964:
-

So let's hold off on committing this until I resolve the NullPointerException, 
which occurs when the username and password are not provided in the property 
file.


> Add a parameter to beeline to allow a properties file to be passed in
> -
>
> Key: HIVE-13964
> URL: https://issues.apache.org/jira/browse/HIVE-13964
> Project: Hive
>  Issue Type: New Feature
>  Components: Beeline
>Affects Versions: 2.0.1
>Reporter: Abdullah Yousufi
>Assignee: Abdullah Yousufi
>Priority: Minor
> Fix For: 2.2.0
>
> Attachments: HIVE-13964.01.patch, HIVE-13964.02.patch, 
> HIVE-13964.03.patch
>
>
> HIVE-6652 removed the ability to pass in a properties file as a beeline 
> parameter. It may be a useful feature to be able to pass the file in is a 
> parameter, such as --property-file.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (HIVE-13725) ACID: Streaming API should synchronize calls when multiple threads use the same endpoint

[Eugene Koifman (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-13725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15328300#comment-15328300
 ] 

Eugene Koifman edited comment on HIVE-13725 at 6/13/16 9:24 PM:


[~vgumashta] is it feasible to add a test here?
More importantly, this is not a generally available thread safe metastore 
client.  Using proxy is probably a perf overhead which makes sense if you are 
creating a general purpose client

If this is specifically for use by HiveEndPoint it seems like making a custom 
client that will do simple delegation (with sychronized wrapper) is more 
appropriate since this only uses 4-5 methods.  Something like 
DbTxnManager.SynchronizedMetaStoreClient


was (Author: ekoifman):
[~vgumashta] is it feasible to add a test here?

> ACID: Streaming API should synchronize calls when multiple threads use the 
> same endpoint
> 
>
> Key: HIVE-13725
> URL: https://issues.apache.org/jira/browse/HIVE-13725
> Project: Hive
>  Issue Type: Bug
>  Components: HCatalog, Metastore, Transactions
>Affects Versions: 1.2.1, 2.0.0
>Reporter: Vaibhav Gumashta
>Assignee: Vaibhav Gumashta
>Priority: Critical
>  Labels: ACID, Streaming
> Attachments: HIVE-13725.1.patch
>
>
> Currently, the streaming endpoint creates a metastore client which gets used 
> for RPC. The client itself is not internally thread safe. Therefore, the API 
> methods should provide the relevant synchronization so that the methods can 
> be called from different threads. A sample use case is as follows:
> 1. Thread 1 creates a streaming endpoint and opens a txn batch.
> 2. Thread 2 heartbeats the txn batch.
> With the current impl, this can result in an "out of sequence response", 
> since the response of the calls in thread1 might end up going to thread2 and 
> vice-versa.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-13725) ACID: Streaming API should synchronize calls when multiple threads use the same endpoint

[Eugene Koifman (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-13725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15328300#comment-15328300
 ] 

Eugene Koifman commented on HIVE-13725:
---

[~vgumashta] is it feasible to add a test here?

> ACID: Streaming API should synchronize calls when multiple threads use the 
> same endpoint
> 
>
> Key: HIVE-13725
> URL: https://issues.apache.org/jira/browse/HIVE-13725
> Project: Hive
>  Issue Type: Bug
>  Components: HCatalog, Metastore, Transactions
>Affects Versions: 1.2.1, 2.0.0
>Reporter: Vaibhav Gumashta
>Assignee: Vaibhav Gumashta
>Priority: Critical
>  Labels: ACID, Streaming
> Attachments: HIVE-13725.1.patch
>
>
> Currently, the streaming endpoint creates a metastore client which gets used 
> for RPC. The client itself is not internally thread safe. Therefore, the API 
> methods should provide the relevant synchronization so that the methods can 
> be called from different threads. A sample use case is as follows:
> 1. Thread 1 creates a streaming endpoint and opens a txn batch.
> 2. Thread 2 heartbeats the txn batch.
> With the current impl, this can result in an "out of sequence response", 
> since the response of the calls in thread1 might end up going to thread2 and 
> vice-versa.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-13987) Clarify current error shown when HS2 is down

[Abdullah Yousufi (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-13987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15328218#comment-15328218
 ] 

Abdullah Yousufi commented on HIVE-13987:
-

Unless I'm not understanding, the TTransportException error is also included 
with the fix. For example, once HS2 is killed:
{code}
0: jdbc:hive2://localhost:1> show tables;
HS2 may be unavailable, check server status
Error: org.apache.thrift.transport.TTransportException (state=08S01,code=0)
{code}

> Clarify current error shown when HS2 is down
> 
>
> Key: HIVE-13987
> URL: https://issues.apache.org/jira/browse/HIVE-13987
> Project: Hive
>  Issue Type: Bug
>  Components: Beeline
>Affects Versions: 2.0.1
>Reporter: Abdullah Yousufi
>Assignee: Abdullah Yousufi
>Priority: Minor
> Fix For: 2.2.0
>
> Attachments: HIVE-13987.01.patch
>
>
> When HS2 is down and a query is run, the following error is shown in beeline:
> {code}
> 0: jdbc:hive2://localhost:1> show tables;
> Error: org.apache.thrift.transport.TTransportException (state=08S01,code=0)
> {code}
> It may be more helpful to also indicate that the reason for this is that HS2 
> is down, such as:
> {code}
> 0: jdbc:hive2://localhost:1> show tables;
> HS2 may be unavailable, check server status
> Error: org.apache.thrift.transport.TTransportException (state=08S01,code=0)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-13957) vectorized IN is inconsistent with non-vectorized (at least for decimal in (string))

[Matt McCline (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-13957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15328211#comment-15328211
 ] 

Matt McCline commented on HIVE-13957:
-

(Patch #3) LGTM +1

> vectorized IN is inconsistent with non-vectorized (at least for decimal in 
> (string))
> 
>
> Key: HIVE-13957
> URL: https://issues.apache.org/jira/browse/HIVE-13957
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-13957.01.patch, HIVE-13957.02.patch, 
> HIVE-13957.03.patch, HIVE-13957.patch, HIVE-13957.patch
>
>
> The cast is applied to the column in regular IN, but vectorized IN applies it 
> to the IN() list.
> This can cause queries to produce incorrect results.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-13987) Clarify current error shown when HS2 is down

[JIRA]

[ 
https://issues.apache.org/jira/browse/HIVE-13987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15328206#comment-15328206
 ] 

Sergio Peña commented on HIVE-13987:


I'm thinking we should append the TTransportException error message to the 
error displayed. We don't know exactly why the TTransportException is thrown, 
and there may be different reasons.

Is there a way to append such reason to the error message? I see this code as 
an example in the same method you modified:
{noformat}
error(loc(e instanceof SQLWarning ? "Warning" : "Error",
new Object[] {
e.getMessage() == null ? "" : e.getMessage().trim(),
e.getSQLState() == null ? "" : e.getSQLState().trim(),
new Integer(e.getErrorCode())}));
{noformat}

The above code is pre-appending a Warning or Error, and then adding some other 
messages into the Object[] array. 

Also, once you have the patch ready, could you copy  & paste an example of how 
it runs?

> Clarify current error shown when HS2 is down
> 
>
> Key: HIVE-13987
> URL: https://issues.apache.org/jira/browse/HIVE-13987
> Project: Hive
>  Issue Type: Bug
>  Components: Beeline
>Affects Versions: 2.0.1
>Reporter: Abdullah Yousufi
>Assignee: Abdullah Yousufi
>Priority: Minor
> Fix For: 2.2.0
>
> Attachments: HIVE-13987.01.patch
>
>
> When HS2 is down and a query is run, the following error is shown in beeline:
> {code}
> 0: jdbc:hive2://localhost:1> show tables;
> Error: org.apache.thrift.transport.TTransportException (state=08S01,code=0)
> {code}
> It may be more helpful to also indicate that the reason for this is that HS2 
> is down, such as:
> {code}
> 0: jdbc:hive2://localhost:1> show tables;
> HS2 may be unavailable, check server status
> Error: org.apache.thrift.transport.TTransportException (state=08S01,code=0)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-13957) vectorized IN is inconsistent with non-vectorized (at least for decimal in (string))

[Sergey Shelukhin (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-13957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15328079#comment-15328079
 ] 

Sergey Shelukhin commented on HIVE-13957:
-

[~mmccline] [~gopalv] ping?

> vectorized IN is inconsistent with non-vectorized (at least for decimal in 
> (string))
> 
>
> Key: HIVE-13957
> URL: https://issues.apache.org/jira/browse/HIVE-13957
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-13957.01.patch, HIVE-13957.02.patch, 
> HIVE-13957.03.patch, HIVE-13957.patch, HIVE-13957.patch
>
>
> The cast is applied to the column in regular IN, but vectorized IN applies it 
> to the IN() list.
> This can cause queries to produce incorrect results.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-13725) ACID: Streaming API should synchronize calls when multiple threads use the same endpoint

[Vaibhav Gumashta (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-13725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15328070#comment-15328070
 ] 

Vaibhav Gumashta commented on HIVE-13725:
-

Test failures look unrelated. [~ekoifman] / [~wzheng] can you review?

> ACID: Streaming API should synchronize calls when multiple threads use the 
> same endpoint
> 
>
> Key: HIVE-13725
> URL: https://issues.apache.org/jira/browse/HIVE-13725
> Project: Hive
>  Issue Type: Bug
>  Components: HCatalog, Metastore, Transactions
>Affects Versions: 1.2.1, 2.0.0
>Reporter: Vaibhav Gumashta
>Assignee: Vaibhav Gumashta
>Priority: Critical
>  Labels: ACID, Streaming
> Attachments: HIVE-13725.1.patch
>
>
> Currently, the streaming endpoint creates a metastore client which gets used 
> for RPC. The client itself is not internally thread safe. Therefore, the API 
> methods should provide the relevant synchronization so that the methods can 
> be called from different threads. A sample use case is as follows:
> 1. Thread 1 creates a streaming endpoint and opens a txn batch.
> 2. Thread 2 heartbeats the txn batch.
> With the current impl, this can result in an "out of sequence response", 
> since the response of the calls in thread1 might end up going to thread2 and 
> vice-versa.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-14003) queries running against llap hang at times - preemption issues

[Sergey Shelukhin (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-14003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15328067#comment-15328067
 ] 

Sergey Shelukhin commented on HIVE-14003:
-

I think  TODO: Reviewer:  things need to be investigated (and other TODOs may 
need to be fixed). I don't have exact answers to most of them.
Dummy ops are used for mapjoin. Prasanth may know more about merge. The global 
map does not need to be cleaned up explicitly; you can check the existing 
cleanup, as far as I can tell this patch wouldn't interfere with it.

Overall, I think we should be able to interrupt the execution with interrupt 
exception, so I am not sure why some comments say that having a blocking op is 
a problem - shouldn't an interrupt there abort properly? Abort flag would only 
be an optimization then. The only problem are other library calls that can 
swallow interrupt exceptions... as long as abort flag is checked after those it 
should be alright. Also we should file JIRAs to fix the code if these are 
Hadoop libraries/components. At the very least they should restore the 
interrupt flag. There should be not retries on interrupts, etc.

> queries running against llap hang at times - preemption issues
> --
>
> Key: HIVE-14003
> URL: https://issues.apache.org/jira/browse/HIVE-14003
> Project: Hive
>  Issue Type: Bug
>  Components: llap
>Affects Versions: 2.1.0
>Reporter: Takahiko Saito
>Assignee: Siddharth Seth
> Attachments: HIVE-14003.01.patch
>
>
> The preemption logic in the Hive processor needs some more work. There are 
> definitely windows where the abort flag is completely dropped within the Hive 
> processor.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-13884) Disallow queries fetching more than a configured number of partitions in PartitionPruner

[JIRA]

 [ 
https://issues.apache.org/jira/browse/HIVE-13884?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergio Peña updated HIVE-13884:
---
Attachment: HIVE-13884.3.patch

> Disallow queries fetching more than a configured number of partitions in 
> PartitionPruner
> 
>
> Key: HIVE-13884
> URL: https://issues.apache.org/jira/browse/HIVE-13884
> Project: Hive
>  Issue Type: Improvement
>Reporter: Mohit Sabharwal
>Assignee: Sergio Peña
> Attachments: HIVE-13884.1.patch, HIVE-13884.2.patch, 
> HIVE-13884.3.patch
>
>
> Currently the PartitionPruner requests either all partitions or partitions 
> based on filter expression. In either scenarios, if the number of partitions 
> accessed is large there can be significant memory pressure at the HMS server 
> end.
> We already have a config {{hive.limit.query.max.table.partition}} that 
> enforces limits on number of partitions that may be scanned per operator. But 
> this check happens after the PartitionPruner has already fetched all 
> partitions.
> We should add an option at PartitionPruner level to disallow queries that 
> attempt to access number of partitions beyond a configurable limit.
> Note that {{hive.mapred.mode=strict}} disallow queries without a partition 
> filter in PartitionPruner, but this check accepts any query with a pruning 
> condition, even if partitions fetched are large. In multi-tenant 
> environments, admins could use more control w.r.t. number of partitions 
> allowed based on HMS memory capacity.
> One option is to have PartitionPruner first fetch the partition names 
> (instead of partition specs) and throw an exception if number of partitions 
> exceeds the configured value. Otherwise, fetch the partition specs.
> Looks like the existing {{listPartitionNames}} call could be used if extended 
> to take partition filter expressions like {{getPartitionsByExpr}} call does.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-13884) Disallow queries fetching more than a configured number of partitions in PartitionPruner

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-13884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15327997#comment-15327997
 ] 

Hive QA commented on HIVE-13884:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12809966/HIVE-13884.2.patch

{color:red}ERROR:{color} -1 due to build exiting with an error

Test results: 
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/115/testReport
Console output: 
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/115/console
Test logs: 
http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-115/

Messages:
{noformat}
 This message was trimmed, see log for full details 
[INFO] 
[INFO] --- maven-antrun-plugin:1.7:run (setup-test-dirs) @ hive-service-rpc ---
[INFO] Executing tasks

main:
[mkdir] Created dir: 
/data/hive-ptest/working/apache-github-source-source/service-rpc/target/tmp
[mkdir] Created dir: 
/data/hive-ptest/working/apache-github-source-source/service-rpc/target/warehouse
[mkdir] Created dir: 
/data/hive-ptest/working/apache-github-source-source/service-rpc/target/tmp/conf
 [copy] Copying 15 files to 
/data/hive-ptest/working/apache-github-source-source/service-rpc/target/tmp/conf
[INFO] Executed tasks
[INFO] 
[INFO] --- maven-compiler-plugin:3.1:testCompile (default-testCompile) @ 
hive-service-rpc ---
[INFO] No sources to compile
[INFO] 
[INFO] --- maven-surefire-plugin:2.16:test (default-test) @ hive-service-rpc ---
[INFO] Tests are skipped.
[INFO] 
[INFO] --- maven-jar-plugin:2.2:jar (default-jar) @ hive-service-rpc ---
[INFO] Building jar: 
/data/hive-ptest/working/apache-github-source-source/service-rpc/target/hive-service-rpc-2.2.0-SNAPSHOT.jar
[INFO] 
[INFO] --- maven-site-plugin:3.3:attach-descriptor (attach-descriptor) @ 
hive-service-rpc ---
[INFO] 
[INFO] --- maven-jar-plugin:2.2:test-jar (default) @ hive-service-rpc ---
[INFO] Building jar: 
/data/hive-ptest/working/apache-github-source-source/service-rpc/target/hive-service-rpc-2.2.0-SNAPSHOT-tests.jar
[INFO] 
[INFO] --- maven-install-plugin:2.4:install (default-install) @ 
hive-service-rpc ---
[INFO] Installing 
/data/hive-ptest/working/apache-github-source-source/service-rpc/target/hive-service-rpc-2.2.0-SNAPSHOT.jar
 to 
/data/hive-ptest/working/maven/org/apache/hive/hive-service-rpc/2.2.0-SNAPSHOT/hive-service-rpc-2.2.0-SNAPSHOT.jar
[INFO] Installing 
/data/hive-ptest/working/apache-github-source-source/service-rpc/pom.xml to 
/data/hive-ptest/working/maven/org/apache/hive/hive-service-rpc/2.2.0-SNAPSHOT/hive-service-rpc-2.2.0-SNAPSHOT.pom
[INFO] Installing 
/data/hive-ptest/working/apache-github-source-source/service-rpc/target/hive-service-rpc-2.2.0-SNAPSHOT-tests.jar
 to 
/data/hive-ptest/working/maven/org/apache/hive/hive-service-rpc/2.2.0-SNAPSHOT/hive-service-rpc-2.2.0-SNAPSHOT-tests.jar
[INFO] 
[INFO] 
[INFO] Building Hive Serde 2.2.0-SNAPSHOT
[INFO] 
[INFO] 
[INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ hive-serde ---
[INFO] Deleting 
/data/hive-ptest/working/apache-github-source-source/serde/target
[INFO] Deleting /data/hive-ptest/working/apache-github-source-source/serde 
(includes = [datanucleus.log, derby.log], excludes = [])
[INFO] 
[INFO] --- maven-enforcer-plugin:1.3.1:enforce (enforce-no-snapshots) @ 
hive-serde ---
[INFO] 
[INFO] --- build-helper-maven-plugin:1.8:add-source (add-source) @ hive-serde 
---
[INFO] Source directory: 
/data/hive-ptest/working/apache-github-source-source/serde/src/gen/protobuf/gen-java
 added.
[INFO] Source directory: 
/data/hive-ptest/working/apache-github-source-source/serde/src/gen/thrift/gen-javabean
 added.
[INFO] 
[INFO] --- maven-remote-resources-plugin:1.5:process (default) @ hive-serde ---
[INFO] 
[INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ 
hive-serde ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory 
/data/hive-ptest/working/apache-github-source-source/serde/src/main/resources
[INFO] Copying 3 resources
[INFO] 
[INFO] --- maven-antrun-plugin:1.7:run (define-classpath) @ hive-serde ---
[INFO] Executing tasks

main:
[INFO] Executed tasks
[INFO] 
[INFO] --- maven-compiler-plugin:3.1:compile (default-compile) @ hive-serde ---
[INFO] Compiling 414 source files to 
/data/hive-ptest/working/apache-github-source-source/serde/target/classes
[WARNING] 
/data/hive-ptest/working/apache-github-source-source/serde/src/java/org/apache/hadoop/hive/serde2/AbstractSerDe.java:
 Some input files use or override a deprecated API.
[WARNING] 
/data/hive-ptest/working/apache-github-source-source/serde/src/java/org/apache/hadoop/hive/serde2/AbstractSerDe.java:
 Recompile 

[jira] [Updated] (HIVE-13392) disable speculative execution for ACID Compactor

[Eugene Koifman (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-13392?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eugene Koifman updated HIVE-13392:
--
Fix Version/s: 2.2.0
   1.3.0

> disable speculative execution for ACID Compactor
> 
>
> Key: HIVE-13392
> URL: https://issues.apache.org/jira/browse/HIVE-13392
> Project: Hive
>  Issue Type: Bug
>  Components: Transactions
>Affects Versions: 1.0.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
> Fix For: 1.3.0, 2.2.0
>
> Attachments: HIVE-13392.2.patch, HIVE-13392.3.patch, 
> HIVE-13392.4.patch, HIVE-13392.patch
>
>
> https://developer.yahoo.com/hadoop/tutorial/module4.html
> Speculative execution is enabled by default. You can disable speculative 
> execution for the mappers and reducers by setting the 
> mapred.map.tasks.speculative.execution and 
> mapred.reduce.tasks.speculative.execution JobConf options to false, 
> respectively.
> CompactorMR is currently not set up to handle speculative execution and may 
> lead to something like
> {code}
> 2016-02-08 22:56:38,256 WARN [main] org.apache.hadoop.mapred.YarnChild: 
> Exception running child : 
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.protocol.AlreadyBeingCreatedException):
>  Failed to CREATE_FILE 
> /apps/hive/warehouse/service_logs_v2/ds=2016-01-20/_tmp_6cf08b9f-c2e2-4182-bc81-e032801b147f/base_13858600/bucket_4
>  for DFSClient_attempt_1454628390210_27756_m_01_1_131224698_1 on 
> 172.18.129.12 because this file lease is currently owned by 
> DFSClient_attempt_1454628390210_27756_m_01_0_-2027182532_1 on 
> 172.18.129.18
> at 
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.recoverLeaseInternal(FSNamesystem.java:2937)
> at 
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInternal(FSNamesystem.java:2562)
> at 
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInt(FSNamesystem.java:2451)
> at 
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFile(FSNamesystem.java:2335)
> at 
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.create(NameNodeRpcServer.java:688)
> at 
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.create(ClientNamenodeProtocolServerSideTranslatorPB.java:397)
> at 
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
> at 
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:969)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2151)
> {code}
> Short term: disable speculative execution for this job
> Longer term perhaps make each task write to dir with UUID...



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-13392) disable speculative execution for ACID Compactor

[Eugene Koifman (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-13392?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eugene Koifman updated HIVE-13392:
--
Resolution: Fixed
Status: Resolved  (was: Patch Available)

committed to branch-1 and master
thank Wei and Alan for the review

> disable speculative execution for ACID Compactor
> 
>
> Key: HIVE-13392
> URL: https://issues.apache.org/jira/browse/HIVE-13392
> Project: Hive
>  Issue Type: Bug
>  Components: Transactions
>Affects Versions: 1.0.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
> Attachments: HIVE-13392.2.patch, HIVE-13392.3.patch, 
> HIVE-13392.4.patch, HIVE-13392.patch
>
>
> https://developer.yahoo.com/hadoop/tutorial/module4.html
> Speculative execution is enabled by default. You can disable speculative 
> execution for the mappers and reducers by setting the 
> mapred.map.tasks.speculative.execution and 
> mapred.reduce.tasks.speculative.execution JobConf options to false, 
> respectively.
> CompactorMR is currently not set up to handle speculative execution and may 
> lead to something like
> {code}
> 2016-02-08 22:56:38,256 WARN [main] org.apache.hadoop.mapred.YarnChild: 
> Exception running child : 
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.protocol.AlreadyBeingCreatedException):
>  Failed to CREATE_FILE 
> /apps/hive/warehouse/service_logs_v2/ds=2016-01-20/_tmp_6cf08b9f-c2e2-4182-bc81-e032801b147f/base_13858600/bucket_4
>  for DFSClient_attempt_1454628390210_27756_m_01_1_131224698_1 on 
> 172.18.129.12 because this file lease is currently owned by 
> DFSClient_attempt_1454628390210_27756_m_01_0_-2027182532_1 on 
> 172.18.129.18
> at 
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.recoverLeaseInternal(FSNamesystem.java:2937)
> at 
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInternal(FSNamesystem.java:2562)
> at 
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInt(FSNamesystem.java:2451)
> at 
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFile(FSNamesystem.java:2335)
> at 
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.create(NameNodeRpcServer.java:688)
> at 
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.create(ClientNamenodeProtocolServerSideTranslatorPB.java:397)
> at 
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
> at 
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:969)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2151)
> {code}
> Short term: disable speculative execution for this job
> Longer term perhaps make each task write to dir with UUID...



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-13884) Disallow queries fetching more than a configured number of partitions in PartitionPruner

[JIRA]

 [ 
https://issues.apache.org/jira/browse/HIVE-13884?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergio Peña updated HIVE-13884:
---
Attachment: HIVE-13884.2.patch

Attach 2nd patch.

> Disallow queries fetching more than a configured number of partitions in 
> PartitionPruner
> 
>
> Key: HIVE-13884
> URL: https://issues.apache.org/jira/browse/HIVE-13884
> Project: Hive
>  Issue Type: Improvement
>Reporter: Mohit Sabharwal
>Assignee: Sergio Peña
> Attachments: HIVE-13884.1.patch, HIVE-13884.2.patch
>
>
> Currently the PartitionPruner requests either all partitions or partitions 
> based on filter expression. In either scenarios, if the number of partitions 
> accessed is large there can be significant memory pressure at the HMS server 
> end.
> We already have a config {{hive.limit.query.max.table.partition}} that 
> enforces limits on number of partitions that may be scanned per operator. But 
> this check happens after the PartitionPruner has already fetched all 
> partitions.
> We should add an option at PartitionPruner level to disallow queries that 
> attempt to access number of partitions beyond a configurable limit.
> Note that {{hive.mapred.mode=strict}} disallow queries without a partition 
> filter in PartitionPruner, but this check accepts any query with a pruning 
> condition, even if partitions fetched are large. In multi-tenant 
> environments, admins could use more control w.r.t. number of partitions 
> allowed based on HMS memory capacity.
> One option is to have PartitionPruner first fetch the partition names 
> (instead of partition specs) and throw an exception if number of partitions 
> exceeds the configured value. Otherwise, fetch the partition specs.
> Looks like the existing {{listPartitionNames}} call could be used if extended 
> to take partition filter expressions like {{getPartitionsByExpr}} call does.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-13964) Add a parameter to beeline to allow a properties file to be passed in

[Abdullah Yousufi (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-13964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15327937#comment-15327937
 ] 

Abdullah Yousufi commented on HIVE-13964:
-

With regards to case #3, you need pass in your login credentials as well. For 
example:

{code}
ConnectionURL=jdbc:hive2://localhost:1
ConnectionUserName=hive
ConnectionPassword=
{code}

With case #1, the property-file requires a url because that is how the 
!properties command works: if you run beeline and execute
{code}
!properties 
{code}
you’ll see the ‘Property “url” is required error.

Therefore, I don’t know if it really makes sense to combine command line 
options, such as -u, with the property file, as you do in case #2. What happens 
there is that the shell initially connects to the url specified by -u, but when 
the properties command is run on props, it fails and the shell exits.

> Add a parameter to beeline to allow a properties file to be passed in
> -
>
> Key: HIVE-13964
> URL: https://issues.apache.org/jira/browse/HIVE-13964
> Project: Hive
>  Issue Type: New Feature
>  Components: Beeline
>Affects Versions: 2.0.1
>Reporter: Abdullah Yousufi
>Assignee: Abdullah Yousufi
>Priority: Minor
> Fix For: 2.2.0
>
> Attachments: HIVE-13964.01.patch, HIVE-13964.02.patch, 
> HIVE-13964.03.patch
>
>
> HIVE-6652 removed the ability to pass in a properties file as a beeline 
> parameter. It may be a useful feature to be able to pass the file in is a 
> parameter, such as --property-file.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Issue Comment Deleted] (HIVE-13964) Add a parameter to beeline to allow a properties file to be passed in

[Abdullah Yousufi (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-13964?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Abdullah Yousufi updated HIVE-13964:

Comment: was deleted

(was: With regards to case #3, you need pass in your login credentials as well. 
For example:

{code}
ConnectionURL=jdbc:hive2://localhost:1
ConnectionUserName=hive
ConnectionPassword=
{code}

With case #1, the property-file requires a url because that is how the 
!properties command works: if you run beeline and execute
{code}
!properties 
{code}
you’ll see the ‘Property “url” is required error.

Therefore, I don’t know if it really makes sense to combine command line 
options, such as -u, with the property file, as you do in case #2. What happens 
there is that the shell initially connects to the url specified by -u, but when 
the properties command is run on props, it fails and the shell exits.)

> Add a parameter to beeline to allow a properties file to be passed in
> -
>
> Key: HIVE-13964
> URL: https://issues.apache.org/jira/browse/HIVE-13964
> Project: Hive
>  Issue Type: New Feature
>  Components: Beeline
>Affects Versions: 2.0.1
>Reporter: Abdullah Yousufi
>Assignee: Abdullah Yousufi
>Priority: Minor
> Fix For: 2.2.0
>
> Attachments: HIVE-13964.01.patch, HIVE-13964.02.patch, 
> HIVE-13964.03.patch
>
>
> HIVE-6652 removed the ability to pass in a properties file as a beeline 
> parameter. It may be a useful feature to be able to pass the file in is a 
> parameter, such as --property-file.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-13964) Add a parameter to beeline to allow a properties file to be passed in

[Abdullah Yousufi (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-13964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15327938#comment-15327938
 ] 

Abdullah Yousufi commented on HIVE-13964:
-

With regards to case #3, you need pass in your login credentials as well. For 
example:

{code}
ConnectionURL=jdbc:hive2://localhost:1
ConnectionUserName=hive
ConnectionPassword=
{code}

With case #1, the property-file requires a url because that is how the 
!properties command works: if you run beeline and execute
{code}
!properties 
{code}
you’ll see the ‘Property “url” is required error.

Therefore, I don’t know if it really makes sense to combine command line 
options, such as -u, with the property file, as you do in case #2. What happens 
there is that the shell initially connects to the url specified by -u, but when 
the properties command is run on props, it fails and the shell exits.

> Add a parameter to beeline to allow a properties file to be passed in
> -
>
> Key: HIVE-13964
> URL: https://issues.apache.org/jira/browse/HIVE-13964
> Project: Hive
>  Issue Type: New Feature
>  Components: Beeline
>Affects Versions: 2.0.1
>Reporter: Abdullah Yousufi
>Assignee: Abdullah Yousufi
>Priority: Minor
> Fix For: 2.2.0
>
> Attachments: HIVE-13964.01.patch, HIVE-13964.02.patch, 
> HIVE-13964.03.patch
>
>
> HIVE-6652 removed the ability to pass in a properties file as a beeline 
> parameter. It may be a useful feature to be able to pass the file in is a 
> parameter, such as --property-file.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-13900) HiveStatement.executeAsync() may not work properly when hive.server2.async.exec.async.compile is turned on

[Aihua Xu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-13900?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aihua Xu updated HIVE-13900:

   Resolution: Fixed
Fix Version/s: 2.2.0
   Status: Resolved  (was: Patch Available)

Pushed to master. Thanks Jimmy for reviewing.

> HiveStatement.executeAsync() may not work properly when 
> hive.server2.async.exec.async.compile is turned on
> --
>
> Key: HIVE-13900
> URL: https://issues.apache.org/jira/browse/HIVE-13900
> Project: Hive
>  Issue Type: Bug
>  Components: JDBC
>Affects Versions: 2.2.0
>Reporter: Aihua Xu
>Assignee: Aihua Xu
> Fix For: 2.2.0
>
> Attachments: HIVE-13900.1.patch
>
>
> HIVE-13882 handles HiveStatement.executeQuery() when 
> hive.server2.async.exec.async.compile is turned on. Notice we may also have 
> similar issue when executeAsync() is called. Investigate what would be the 
> good approach for it.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-13723) Executing join query on type Float using Thrift Serde will result in Float cast to Double error

[Ziyang Zhao (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-13723?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ziyang Zhao updated HIVE-13723:
---
Status: Patch Available  (was: Open)

> Executing join query on type Float using Thrift Serde will result in Float 
> cast to Double error
> ---
>
> Key: HIVE-13723
> URL: https://issues.apache.org/jira/browse/HIVE-13723
> Project: Hive
>  Issue Type: Sub-task
>  Components: HiveServer2, JDBC, Serializers/Deserializers
>Affects Versions: 2.1.0
>Reporter: Ziyang Zhao
>Assignee: Ziyang Zhao
>Priority: Critical
> Attachments: HIVE-13723.1.patch, HIVE-13723.2.patch
>
>
> After enable thrift Serde, execute the following queries in beeline,
> >create table test1 (a int);
> >create table test2 (b float);
> >insert into test1 values (1);
> >insert into test2 values (1);
> >select * from test1 join test2 on test1.a=test2.b;
> this will give the error:
> java.lang.Exception: java.lang.RuntimeException: 
> org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while 
> processing row {"b":1.0}
> at 
> org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:462) 
> ~[hadoop-mapreduce-client-common-2.7.1.2.4.0.0-169.jar:?]
> at 
> org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:522) 
> [hadoop-mapreduce-client-common-2.7.1.2.4.0.0-169.jar:?]
> Caused by: java.lang.RuntimeException: 
> org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while 
> processing row {"b":1.0}
> at 
> org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:168) 
> ~[hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54) 
> ~[hadoop-mapreduce-client-core-2.7.1.2.4.0.0-169.jar:?]
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453) 
> ~[hadoop-mapreduce-client-core-2.7.1.2.4.0.0-169.jar:?]
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343) 
> ~[hadoop-mapreduce-client-core-2.7.1.2.4.0.0-169.jar:?]
> at 
> org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:243)
>  ~[hadoop-mapreduce-client-common-2.7.1.2.4.0.0-169.jar:?]
> at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) 
> ~[?:1.7.0_95]
> at java.util.concurrent.FutureTask.run(FutureTask.java:262) 
> ~[?:1.7.0_95]
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>  ~[?:1.7.0_95]
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>  ~[?:1.7.0_95]
> at java.lang.Thread.run(Thread.java:745) ~[?:1.7.0_95]
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime 
> Error while processing row {"b":1.0}
> at 
> org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:568) 
> ~[hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at 
> org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:159) 
> ~[hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54) 
> ~[hadoop-mapreduce-client-core-2.7.1.2.4.0.0-169.jar:?]
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453) 
> ~[hadoop-mapreduce-client-core-2.7.1.2.4.0.0-169.jar:?]
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343) 
> ~[hadoop-mapreduce-client-core-2.7.1.2.4.0.0-169.jar:?]
> at 
> org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:243)
>  ~[hadoop-mapreduce-client-common-2.7.1.2.4.0.0-169.jar:?]
> at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) 
> ~[?:1.7.0_95]
> at java.util.concurrent.FutureTask.run(FutureTask.java:262) 
> ~[?:1.7.0_95]
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>  ~[?:1.7.0_95]
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>  ~[?:1.7.0_95]
> at java.lang.Thread.run(Thread.java:745) ~[?:1.7.0_95]
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Unexpected 
> exception from MapJoinOperator : 
> org.apache.hadoop.hive.serde2.SerDeException: java.lang.ClassCastException: 
> java.lang.Float cannot be cast to java.lang.Double
> at 
> org.apache.hadoop.hive.ql.exec.MapJoinOperator.process(MapJoinOperator.java:454)
>  ~[hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:837) 
> ~[hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at 
> org.apache.hadoop.hive.ql.exec.FilterOperator.process(FilterOperator.java:126)
>  

[jira] [Commented] (HIVE-13903) getFunctionInfo is downloading jar on every call

[Jesus Camacho Rodriguez (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-13903?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15327883#comment-15327883
 ] 

Jesus Camacho Rodriguez commented on HIVE-13903:


LGTM, +1

> getFunctionInfo is downloading jar on every call
> 
>
> Key: HIVE-13903
> URL: https://issues.apache.org/jira/browse/HIVE-13903
> Project: Hive
>  Issue Type: Bug
>Reporter: Rajat Khandelwal
>Assignee: Rajat Khandelwal
> Attachments: HIVE-13903.01.patch, HIVE-13903.01.patch, 
> HIVE-13903.02.patch
>
>
> on queries using permanent udfs, the jar file of the udf is downloaded 
> multiple times. Each call originating from Registry.getFunctionInfo. This 
> increases time for the query, especially if that query is just an explain 
> query. The jar should be downloaded once, and not downloaded again if the udf 
> class is accessible in the current thread. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-14005) Changing a string column of ORC table to boolean sets values to be 'NULL'

[Takahiko Saito (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-14005?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Takahiko Saito updated HIVE-14005:
--
Description: 
{noformat}
0: jdbc:hive2://os-r7-rcmpc-nat-u-hiveserver2>
0: jdbc:hive2://os-r7-rcmpc-nat-u-hiveserver2> set 
hive.metastore.disallow.incompatible.col.type.changes=false;
No rows affected (0.015 seconds)
0: jdbc:hive2://os-r7-rcmpc-nat-u-hiveserver2>
0: jdbc:hive2://os-r7-rcmpc-nat-u-hiveserver2> set 
hive.exec.dynamic.partition.mode=nonstrict;
No rows affected (0.003 seconds)
0: jdbc:hive2://os-r7-rcmpc-nat-u-hiveserver2>
0: jdbc:hive2://os-r7-rcmpc-nat-u-hiveserver2> drop table test;
No rows affected (0.224 seconds)
0: jdbc:hive2://os-r7-rcmpc-nat-u-hiveserver2>
0: jdbc:hive2://os-r7-rcmpc-nat-u-hiveserver2> create table test (s string) 
stored as orc;
No rows affected (0.098 seconds)
0: jdbc:hive2://os-r7-rcmpc-nat-u-hiveserver2>
0: jdbc:hive2://os-r7-rcmpc-nat-u-hiveserver2> insert into table test values 
('true');
No rows affected (1.091 seconds)
0: jdbc:hive2://os-r7-rcmpc-nat-u-hiveserver2>
0: jdbc:hive2://os-r7-rcmpc-nat-u-hiveserver2> select * from test;
+-+--+
| test.s  |
+-+--+
| true|
+-+--+
1 row selected (0.114 seconds)
0: jdbc:hive2://os-r7-rcmpc-nat-u-hiveserver2>
0: jdbc:hive2://os-r7-rcmpc-nat-u-hiveserver2> alter table test change column s 
s boolean;
No rows affected (0.167 seconds)
0: jdbc:hive2://os-r7-rcmpc-nat-u-hiveserver2>
0: jdbc:hive2://os-r7-rcmpc-nat-u-hiveserver2> select * from test;
+-+--+
| test.s  |
+-+--+
| NULL|
+-+--+
1 row selected (0.098 seconds)
{noformat}

The issue is NOT seen with textfile formatted table:
{noformat}
0: jdbc:hive2://os-r7-rcmpc-nat-u-hiveserver2> set 
hive.metastore.disallow.incompatible.col.type.changes=false;
No rows affected (0.013 seconds)
0: jdbc:hive2://os-r7-rcmpc-nat-u-hiveserver2>
0: jdbc:hive2://os-r7-rcmpc-nat-u-hiveserver2> set 
hive.exec.dynamic.partition.mode=nonstrict;
No rows affected (0.006 seconds)
0: jdbc:hive2://os-r7-rcmpc-nat-u-hiveserver2>
0: jdbc:hive2://os-r7-rcmpc-nat-u-hiveserver2> drop table test;
No rows affected (0.224 seconds)
0: jdbc:hive2://os-r7-rcmpc-nat-u-hiveserver2>
0: jdbc:hive2://os-r7-rcmpc-nat-u-hiveserver2> create table test (s string) 
stored as textfile;
No rows affected (0.116 seconds)
0: jdbc:hive2://os-r7-rcmpc-nat-u-hiveserver2>
0: jdbc:hive2://os-r7-rcmpc-nat-u-hiveserver2> insert into table test values 
('true');
No rows affected (1.799 seconds)
0: jdbc:hive2://os-r7-rcmpc-nat-u-hiveserver2>
0: jdbc:hive2://os-r7-rcmpc-nat-u-hiveserver2> select * from test;
+-+--+
| test.s  |
+-+--+
| true|
+-+--+
1 row selected (0.085 seconds)
0: jdbc:hive2://os-r7-rcmpc-nat-u-hiveserver2>
0: jdbc:hive2://os-r7-rcmpc-nat-u-hiveserver2> alter table test change column s 
s boolean;
No rows affected (0.141 seconds)
0: jdbc:hive2://os-r7-rcmpc-nat-u-hiveserver2>
0: jdbc:hive2://os-r7-rcmpc-nat-u-hiveserver2> select * from test;
+-+--+
| test.s  |
+-+--+
| true|
+-+--+
1 row selected (0.093 seconds)
{noformat}

  was:
{noformat]
0: jdbc:hive2://os-r7-rcmpc-nat-u-hiveserver2>
0: jdbc:hive2://os-r7-rcmpc-nat-u-hiveserver2> set 
hive.metastore.disallow.incompatible.col.type.changes=false;
No rows affected (0.015 seconds)
0: jdbc:hive2://os-r7-rcmpc-nat-u-hiveserver2>
0: jdbc:hive2://os-r7-rcmpc-nat-u-hiveserver2> set 
hive.exec.dynamic.partition.mode=nonstrict;
No rows affected (0.003 seconds)
0: jdbc:hive2://os-r7-rcmpc-nat-u-hiveserver2>
0: jdbc:hive2://os-r7-rcmpc-nat-u-hiveserver2> drop table test;
No rows affected (0.224 seconds)
0: jdbc:hive2://os-r7-rcmpc-nat-u-hiveserver2>
0: jdbc:hive2://os-r7-rcmpc-nat-u-hiveserver2> create table test (s string) 
stored as orc;
No rows affected (0.098 seconds)
0: jdbc:hive2://os-r7-rcmpc-nat-u-hiveserver2>
0: jdbc:hive2://os-r7-rcmpc-nat-u-hiveserver2> insert into table test values 
('true');
No rows affected (1.091 seconds)
0: jdbc:hive2://os-r7-rcmpc-nat-u-hiveserver2>
0: jdbc:hive2://os-r7-rcmpc-nat-u-hiveserver2> select * from test;
+-+--+
| test.s  |
+-+--+
| true|
+-+--+
1 row selected (0.114 seconds)
0: jdbc:hive2://os-r7-rcmpc-nat-u-hiveserver2>
0: jdbc:hive2://os-r7-rcmpc-nat-u-hiveserver2> alter table test change column s 
s boolean;
No rows affected (0.167 seconds)
0: jdbc:hive2://os-r7-rcmpc-nat-u-hiveserver2>
0: jdbc:hive2://os-r7-rcmpc-nat-u-hiveserver2> select * from test;
+-+--+
| test.s  |
+-+--+
| NULL|
+-+--+
1 row selected (0.098 seconds)
{noformat}

The issue is NOT seen with textfile formatted table:
{noformat}
0: jdbc:hive2://os-r7-rcmpc-nat-u-hiveserver2> set 
hive.metastore.disallow.incompatible.col.type.changes=false;
No rows affected (0.013 seconds)
0: jdbc:hive2://os-r7-rcmpc-nat-u-hiveserver2>
0: jdbc:hive2://os-r7-rcmpc-nat-u-hiveserver2> set 

[jira] [Updated] (HIVE-14005) Changing a string column of ORC table to boolean sets values to be 'NULL'

[Takahiko Saito (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-14005?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Takahiko Saito updated HIVE-14005:
--
Summary: Changing a string column of ORC table to boolean sets values to be 
'NULL'  (was: Changing a string column of ORC table to boolean change values to 
be 'NULL')

> Changing a string column of ORC table to boolean sets values to be 'NULL'
> -
>
> Key: HIVE-14005
> URL: https://issues.apache.org/jira/browse/HIVE-14005
> Project: Hive
>  Issue Type: Bug
>  Components: Hive
>Affects Versions: 2.1.1
>Reporter: Takahiko Saito
>
> {noformat]
> 0: jdbc:hive2://os-r7-rcmpc-nat-u-hiveserver2>
> 0: jdbc:hive2://os-r7-rcmpc-nat-u-hiveserver2> set 
> hive.metastore.disallow.incompatible.col.type.changes=false;
> No rows affected (0.015 seconds)
> 0: jdbc:hive2://os-r7-rcmpc-nat-u-hiveserver2>
> 0: jdbc:hive2://os-r7-rcmpc-nat-u-hiveserver2> set 
> hive.exec.dynamic.partition.mode=nonstrict;
> No rows affected (0.003 seconds)
> 0: jdbc:hive2://os-r7-rcmpc-nat-u-hiveserver2>
> 0: jdbc:hive2://os-r7-rcmpc-nat-u-hiveserver2> drop table test;
> No rows affected (0.224 seconds)
> 0: jdbc:hive2://os-r7-rcmpc-nat-u-hiveserver2>
> 0: jdbc:hive2://os-r7-rcmpc-nat-u-hiveserver2> create table test (s string) 
> stored as orc;
> No rows affected (0.098 seconds)
> 0: jdbc:hive2://os-r7-rcmpc-nat-u-hiveserver2>
> 0: jdbc:hive2://os-r7-rcmpc-nat-u-hiveserver2> insert into table test values 
> ('true');
> No rows affected (1.091 seconds)
> 0: jdbc:hive2://os-r7-rcmpc-nat-u-hiveserver2>
> 0: jdbc:hive2://os-r7-rcmpc-nat-u-hiveserver2> select * from test;
> +-+--+
> | test.s  |
> +-+--+
> | true|
> +-+--+
> 1 row selected (0.114 seconds)
> 0: jdbc:hive2://os-r7-rcmpc-nat-u-hiveserver2>
> 0: jdbc:hive2://os-r7-rcmpc-nat-u-hiveserver2> alter table test change column 
> s s boolean;
> No rows affected (0.167 seconds)
> 0: jdbc:hive2://os-r7-rcmpc-nat-u-hiveserver2>
> 0: jdbc:hive2://os-r7-rcmpc-nat-u-hiveserver2> select * from test;
> +-+--+
> | test.s  |
> +-+--+
> | NULL|
> +-+--+
> 1 row selected (0.098 seconds)
> {noformat}
> The issue is NOT seen with textfile formatted table:
> {noformat}
> 0: jdbc:hive2://os-r7-rcmpc-nat-u-hiveserver2> set 
> hive.metastore.disallow.incompatible.col.type.changes=false;
> No rows affected (0.013 seconds)
> 0: jdbc:hive2://os-r7-rcmpc-nat-u-hiveserver2>
> 0: jdbc:hive2://os-r7-rcmpc-nat-u-hiveserver2> set 
> hive.exec.dynamic.partition.mode=nonstrict;
> No rows affected (0.006 seconds)
> 0: jdbc:hive2://os-r7-rcmpc-nat-u-hiveserver2>
> 0: jdbc:hive2://os-r7-rcmpc-nat-u-hiveserver2> drop table test;
> No rows affected (0.224 seconds)
> 0: jdbc:hive2://os-r7-rcmpc-nat-u-hiveserver2>
> 0: jdbc:hive2://os-r7-rcmpc-nat-u-hiveserver2> create table test (s string) 
> stored as textfile;
> No rows affected (0.116 seconds)
> 0: jdbc:hive2://os-r7-rcmpc-nat-u-hiveserver2>
> 0: jdbc:hive2://os-r7-rcmpc-nat-u-hiveserver2> insert into table test values 
> ('true');
> No rows affected (1.799 seconds)
> 0: jdbc:hive2://os-r7-rcmpc-nat-u-hiveserver2>
> 0: jdbc:hive2://os-r7-rcmpc-nat-u-hiveserver2> select * from test;
> +-+--+
> | test.s  |
> +-+--+
> | true|
> +-+--+
> 1 row selected (0.085 seconds)
> 0: jdbc:hive2://os-r7-rcmpc-nat-u-hiveserver2>
> 0: jdbc:hive2://os-r7-rcmpc-nat-u-hiveserver2> alter table test change column 
> s s boolean;
> No rows affected (0.141 seconds)
> 0: jdbc:hive2://os-r7-rcmpc-nat-u-hiveserver2>
> 0: jdbc:hive2://os-r7-rcmpc-nat-u-hiveserver2> select * from test;
> +-+--+
> | test.s  |
> +-+--+
> | true|
> +-+--+
> 1 row selected (0.093 seconds)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (HIVE-13946) Decimal value need to be single-quoted when selecting where clause with that decimal value in order to get row

[Takahiko Saito (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-13946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15327852#comment-15327852
 ] 

Takahiko Saito edited comment on HIVE-13946 at 6/13/16 6:01 PM:


[~sircodesalot] I believe you are testing with Hive2 and hitting this issue:
https://issues.apache.org/jira/browse/HIVE-13945

The issue here is for Hive 1.


was (Author: taksaito):
[~sircodesalot] I believe you are testing with Hive2 and hitting this issue:
https://issues.apache.org/jira/browse/HIVE-13945

> Decimal value need to be single-quoted when selecting where clause with that 
> decimal value in order to get row
> --
>
> Key: HIVE-13946
> URL: https://issues.apache.org/jira/browse/HIVE-13946
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 1.2.1
>Reporter: Takahiko Saito
> Fix For: 1.2.1
>
>
> Create a table withe a column of decimal type(38,18) and insert 
> '4327269606205.029297'. Then select with that value does not return anything.
> {noformat}
> 0: jdbc:hive2://ts-0531-1.openstacklocal:2181> drop table if exists test;
> No rows affected (0.175 seconds)
> 0: jdbc:hive2://ts-0531-1.openstacklocal:2181>
> 0: jdbc:hive2://ts-0531-1.openstacklocal:2181> create table test (dc 
> decimal(38,18));
> No rows affected (0.098 seconds)
> 0: jdbc:hive2://ts-0531-1.openstacklocal:2181>
> 0: jdbc:hive2://ts-0531-1.openstacklocal:2181> insert into table test values 
> (4327269606205.029297);
> INFO  : Session is already open
> INFO  : Dag name: insert into table tes...327269606205.029297)(Stage-1)
> INFO  : Tez session was closed. Reopening...
> INFO  : Session re-established.
> INFO  :
> INFO  : Status: Running (Executing on YARN cluster with App id 
> application_1464727816747_0762)
> INFO  : Map 1: -/-
> INFO  : Map 1: 0/1
> INFO  : Map 1: 0(+1)/1
> INFO  : Map 1: 1/1
> INFO  : Loading data to table default.test from 
> hdfs://ts-0531-5.openstacklocal:8020/apps/hive/warehouse/test/.hive-staging_hive_2016-06-04_00-03-54_302_7708281807413586675-940/-ext-1
> INFO  : Table default.test stats: [numFiles=1, numRows=1, totalSize=21, 
> rawDataSize=20]
> No rows affected (13.821 seconds)
> 0: jdbc:hive2://ts-0531-1.openstacklocal:2181>
> 0: jdbc:hive2://ts-0531-1.openstacklocal:2181> select * from test;
> +---+--+
> |test.dc|
> +---+--+
> | 4327269606205.029297  |
> +---+--+
> 1 row selected (0.078 seconds)
> 0: jdbc:hive2://ts-0531-1.openstacklocal:2181> select * from test where dc = 
> 4327269606205.029297;
> +--+--+
> | test.dc  |
> +--+--+
> +--+--+
> No rows selected (0.224 seconds)
> {noformat}
> If you single quote that decimal value, a row is returned.
> {noformat}
> 0: jdbc:hive2://ts-0531-1.openstacklocal:2181> select * from test where dc = 
> '4327269606205.029297';
> +---+--+
> |test.dc|
> +---+--+
> | 4327269606205.029297  |
> +---+--+
> 1 row selected (0.085 seconds)
> {noformat}
> explain shows:
> {noformat}
> 0: jdbc:hive2://ts-0531-1.openstacklocal:2181> explain select * from test 
> where dc = 4327269606205.029297;
> +--+--+
> |   Explain|
> +--+--+
> | STAGE DEPENDENCIES:  |
> |   Stage-0 is a root stage|
> |  |
> | STAGE PLANS: |
> |   Stage: Stage-0 |
> | Fetch Operator   |
> |   limit: -1  |
> |   Processor Tree:|
> | TableScan|
> |   alias: test|
> |   filterExpr: (dc = 4.3272696062050293E12) (type: boolean)   |
> |   Filter Operator|
> | predicate: (dc = 4.3272696062050293E12) (type: boolean)  |
> | Select Operator  |
> |   expressions: dc (type: decimal(38,18)) |
> |   outputColumnNames: _col0   |
> |   ListSink   |
> |

[jira] [Commented] (HIVE-13946) Decimal value need to be single-quoted when selecting where clause with that decimal value in order to get row

[Takahiko Saito (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-13946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15327852#comment-15327852
 ] 

Takahiko Saito commented on HIVE-13946:
---

[~sircodesalot] I believe you are testing with Hive2 and hitting this issue:
https://issues.apache.org/jira/browse/HIVE-13945

> Decimal value need to be single-quoted when selecting where clause with that 
> decimal value in order to get row
> --
>
> Key: HIVE-13946
> URL: https://issues.apache.org/jira/browse/HIVE-13946
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 1.2.1
>Reporter: Takahiko Saito
> Fix For: 1.2.1
>
>
> Create a table withe a column of decimal type(38,18) and insert 
> '4327269606205.029297'. Then select with that value does not return anything.
> {noformat}
> 0: jdbc:hive2://ts-0531-1.openstacklocal:2181> drop table if exists test;
> No rows affected (0.175 seconds)
> 0: jdbc:hive2://ts-0531-1.openstacklocal:2181>
> 0: jdbc:hive2://ts-0531-1.openstacklocal:2181> create table test (dc 
> decimal(38,18));
> No rows affected (0.098 seconds)
> 0: jdbc:hive2://ts-0531-1.openstacklocal:2181>
> 0: jdbc:hive2://ts-0531-1.openstacklocal:2181> insert into table test values 
> (4327269606205.029297);
> INFO  : Session is already open
> INFO  : Dag name: insert into table tes...327269606205.029297)(Stage-1)
> INFO  : Tez session was closed. Reopening...
> INFO  : Session re-established.
> INFO  :
> INFO  : Status: Running (Executing on YARN cluster with App id 
> application_1464727816747_0762)
> INFO  : Map 1: -/-
> INFO  : Map 1: 0/1
> INFO  : Map 1: 0(+1)/1
> INFO  : Map 1: 1/1
> INFO  : Loading data to table default.test from 
> hdfs://ts-0531-5.openstacklocal:8020/apps/hive/warehouse/test/.hive-staging_hive_2016-06-04_00-03-54_302_7708281807413586675-940/-ext-1
> INFO  : Table default.test stats: [numFiles=1, numRows=1, totalSize=21, 
> rawDataSize=20]
> No rows affected (13.821 seconds)
> 0: jdbc:hive2://ts-0531-1.openstacklocal:2181>
> 0: jdbc:hive2://ts-0531-1.openstacklocal:2181> select * from test;
> +---+--+
> |test.dc|
> +---+--+
> | 4327269606205.029297  |
> +---+--+
> 1 row selected (0.078 seconds)
> 0: jdbc:hive2://ts-0531-1.openstacklocal:2181> select * from test where dc = 
> 4327269606205.029297;
> +--+--+
> | test.dc  |
> +--+--+
> +--+--+
> No rows selected (0.224 seconds)
> {noformat}
> If you single quote that decimal value, a row is returned.
> {noformat}
> 0: jdbc:hive2://ts-0531-1.openstacklocal:2181> select * from test where dc = 
> '4327269606205.029297';
> +---+--+
> |test.dc|
> +---+--+
> | 4327269606205.029297  |
> +---+--+
> 1 row selected (0.085 seconds)
> {noformat}
> explain shows:
> {noformat}
> 0: jdbc:hive2://ts-0531-1.openstacklocal:2181> explain select * from test 
> where dc = 4327269606205.029297;
> +--+--+
> |   Explain|
> +--+--+
> | STAGE DEPENDENCIES:  |
> |   Stage-0 is a root stage|
> |  |
> | STAGE PLANS: |
> |   Stage: Stage-0 |
> | Fetch Operator   |
> |   limit: -1  |
> |   Processor Tree:|
> | TableScan|
> |   alias: test|
> |   filterExpr: (dc = 4.3272696062050293E12) (type: boolean)   |
> |   Filter Operator|
> | predicate: (dc = 4.3272696062050293E12) (type: boolean)  |
> | Select Operator  |
> |   expressions: dc (type: decimal(38,18)) |
> |   outputColumnNames: _col0   |
> |   ListSink   |
> |  |
> +--+--+
> 18 rows selected (0.512 seconds)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-13432) ACID ORC CompactorMR job throws java.lang.ArrayIndexOutOfBoundsException: 7

[Eugene Koifman (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-13432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eugene Koifman updated HIVE-13432:
--
Component/s: Transactions

> ACID ORC CompactorMR job throws java.lang.ArrayIndexOutOfBoundsException: 7
> ---
>
> Key: HIVE-13432
> URL: https://issues.apache.org/jira/browse/HIVE-13432
> Project: Hive
>  Issue Type: Bug
>  Components: ORC, Transactions
>Affects Versions: 1.2.1
> Environment: Hadoop 2.6.2+Hive 1.2.1
>Reporter: Qiuzhuang Lian
>Assignee: Matt McCline
> Attachments: orc_hive-site.xml
>
>
> After initiating HIVE ACID ORC table compaction, the CompactorMR job throws 
> exception:
> Error: java.lang.ArrayIndexOutOfBoundsException: 7
>   at 
> org.apache.hadoop.hive.ql.io.orc.TreeReaderFactory$StructTreeReader.(TreeReaderFactory.java:1968)
>   at 
> org.apache.hadoop.hive.ql.io.orc.TreeReaderFactory.createTreeReader(TreeReaderFactory.java:2368)
>   at 
> org.apache.hadoop.hive.ql.io.orc.TreeReaderFactory$StructTreeReader.(TreeReaderFactory.java:1969)
>   at 
> org.apache.hadoop.hive.ql.io.orc.TreeReaderFactory.createTreeReader(TreeReaderFactory.java:2368)
>   at 
> org.apache.hadoop.hive.ql.io.orc.RecordReaderFactory.createTreeReader(RecordReaderFactory.java:69)
>   at 
> org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.(RecordReaderImpl.java:202)
>   at 
> org.apache.hadoop.hive.ql.io.orc.ReaderImpl.rowsOptions(ReaderImpl.java:539)
>   at 
> org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger$ReaderPair.(OrcRawRecordMerger.java:183)
>   at 
> org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger.(OrcRawRecordMerger.java:466)
>   at 
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getRawReader(OrcInputFormat.java:1308)
>   at 
> org.apache.hadoop.hive.ql.txn.compactor.CompactorMR$CompactorMap.map(CompactorMR.java:512)
>   at 
> org.apache.hadoop.hive.ql.txn.compactor.CompactorMR$CompactorMap.map(CompactorMR.java:491)
>   at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
>   at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:450)
>   at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
>   at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at javax.security.auth.Subject.doAs(Subject.java:415)
>   at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656)
>   at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
> As a result, we see hadoop exception stack,
> 297 failed with state FAILED due to: Task failed 
> task_1458819387386_11297_m_08
> Job failed as tasks failed. failedMaps:1 failedReduces:0
> 2016-04-06 11:30:57,891 INFO  [dn209006-27]: mapreduce.Job 
> (Job.java:monitorAndPrintJob(1392)) - Counters: 14
>   Job Counters 
> Failed map tasks=16
> Killed map tasks=7
> Launched map tasks=23
> Other local map tasks=13
> Data-local map tasks=6
> Rack-local map tasks=4
> Total time spent by all maps in occupied slots (ms)=412592
> Total time spent by all reduces in occupied slots (ms)=0
> Total time spent by all map tasks (ms)=206296
> Total vcore-seconds taken by all map tasks=206296
> Total megabyte-seconds taken by all map tasks=422494208
>   Map-Reduce Framework
> CPU time spent (ms)=0
> Physical memory (bytes) snapshot=0
> Virtual memory (bytes) snapshot=0
> 2016-04-06 11:30:57,891 ERROR [dn209006-27]: compactor.Worker 
> (Worker.java:run(176)) - Caught exception while trying to compact 
> lqz.my_orc_acid_table.  Marking clean to avoid repeated failures, 
> java.io.IOException: Job failed!
>   at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:836)
>   at 
> org.apache.hadoop.hive.ql.txn.compactor.CompactorMR.run(CompactorMR.java:186)
>   at org.apache.hadoop.hive.ql.txn.compactor.Worker.run(Worker.java:162)
> 2016-04-06 11:30:57,894 ERROR [dn209006-27]: txn.CompactionTxnHandler 
> (CompactionTxnHandler.java:markCleaned(327)) - Expected to remove at least 
> one row from completed_txn_components when marking compaction entry as clean!



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-13974) ORC Schema Evolution doesn't support add columns to non-last STRUCT columns

[Eugene Koifman (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-13974?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eugene Koifman updated HIVE-13974:
--
Component/s: Transactions
 ORC

> ORC Schema Evolution doesn't support add columns to non-last STRUCT columns
> ---
>
> Key: HIVE-13974
> URL: https://issues.apache.org/jira/browse/HIVE-13974
> Project: Hive
>  Issue Type: Bug
>  Components: Hive, ORC, Transactions
>Affects Versions: 1.3.0, 2.1.0, 2.2.0
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Attachments: HIVE-13974.01.patch
>
>
> Currently, the included columns are based on the fileSchema and not the 
> readerSchema which doesn't work for adding columns to non-last STRUCT data 
> type columns.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (HIVE-13958) hive.strict.checks.type.safety should apply to decimals, as well as IN... and BETWEEN... ops

[Takuma Wakamori (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-13958?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15327800#comment-15327800
 ] 

Takuma Wakamori edited comment on HIVE-13958 at 6/13/16 5:41 PM:
-

[~sershe] Thanks! I am sorry for my late reseponse.
I attached patches that checks the implicit type-cast is allowed to the 
arguments passed to not only {{WHERE}} but also {{IN}} and {{BETWEEN}} operator.
It includes minor typo fixes.
I would appreciate it if you could review it. Thanks!


was (Author: takuma):
Check the implicit type-cast is allowed to the arguments passed to not only 
{{WHERE}} but also {{IN}} and {{BETWEEN}} operator.
It includes minor typo fixes.

> hive.strict.checks.type.safety should apply to decimals, as well as IN... and 
> BETWEEN... ops
> 
>
> Key: HIVE-13958
> URL: https://issues.apache.org/jira/browse/HIVE-13958
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 2.2.0
>Reporter: Sergey Shelukhin
>Assignee: Takuma Wakamori
>  Labels: patch
> Attachments: HIVE-13958.01.patch, HIVE-13958.02.patch
>
>
> String to decimal auto-casts should be prohibited for compares



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-13988) zero length file is being created for empty bucket in tez mode

[Pengcheng Xiong (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-13988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15327815#comment-15327815
 ] 

Pengcheng Xiong commented on HIVE-13988:


[~ashutoshc], could u take a look? The test case failures are legitimate due to 
golden file updates. I will file a follow-up jira to deal with the case when 
source table is not empty. thanks.

> zero length file is being created for empty bucket in tez mode
> --
>
> Key: HIVE-13988
> URL: https://issues.apache.org/jira/browse/HIVE-13988
> Project: Hive
>  Issue Type: Bug
>Reporter: Pengcheng Xiong
>Assignee: Pengcheng Xiong
> Attachments: HIVE-13988.01.patch
>
>
> Even though bucket is empty, zero length file is being created in tez mode. 
> steps to reproduce the issue:
> {noformat}
> hive> set hive.execution.engine;
> hive.execution.engine=tez
> hive> drop table if exists emptybucket_orc;
> OK
> Time taken: 5.416 seconds
> hive> create table emptybucket_orc(age int) clustered by (age) sorted by 
> (age) into 99 buckets stored as orc;
> OK
> Time taken: 0.493 seconds
> hive> insert into table emptybucket_orc select distinct(age) from 
> studenttab10k limit 0;
> Query ID = hrt_qa_20160523231955_8b981be7-68c4-4416-8a48-5f8c7ff551c3
> Total jobs = 1
> Launching Job 1 out of 1
> Status: Running (Executing on YARN cluster with App id 
> application_1464045121842_0002)
> --
> VERTICES  MODESTATUS  TOTAL  COMPLETED  RUNNING  PENDING  
> FAILED  KILLED  
> --
> Map 1 ..  llap SUCCEEDED  1  100  
>  0   0  
> Reducer 2 ..  llap SUCCEEDED  1  100  
>  0   0  
> Reducer 3 ..  llap SUCCEEDED  1  100  
>  0   0  
> Reducer 4 ..  llap SUCCEEDED 99 9900  
>  0   0  
> --
> VERTICES: 04/04  [==>>] 100%  ELAPSED TIME: 11.00 s   
>  
> --
> Loading data to table default.emptybucket_orc
> OK
> Time taken: 16.907 seconds
> hive> dfs -ls /apps/hive/warehouse/emptybucket_orc;
> Found 99 items
> -rwxrwxrwx   3 hrt_qa hdfs  0 2016-05-23 23:20 
> /apps/hive/warehouse/emptybucket_orc/00_0
> -rwxrwxrwx   3 hrt_qa hdfs  0 2016-05-23 23:20 
> /apps/hive/warehouse/emptybucket_orc/01_0
> ..
> {noformat}
> Expected behavior:
> In tez mode, zero length file shouldn't get created on hdfs if bucket is empty



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-13958) hive.strict.checks.type.safety should apply to decimals, as well as IN... and BETWEEN... ops

[Takuma Wakamori (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-13958?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Takuma Wakamori updated HIVE-13958:
---
Attachment: HIVE-13958.02.patch
HIVE-13958.01.patch

> hive.strict.checks.type.safety should apply to decimals, as well as IN... and 
> BETWEEN... ops
> 
>
> Key: HIVE-13958
> URL: https://issues.apache.org/jira/browse/HIVE-13958
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 2.2.0
>Reporter: Sergey Shelukhin
>Assignee: Takuma Wakamori
>  Labels: patch
> Attachments: HIVE-13958.01.patch, HIVE-13958.02.patch
>
>
> String to decimal auto-casts should be prohibited for compares



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-13958) hive.strict.checks.type.safety should apply to decimals, as well as IN... and BETWEEN... ops

[Takuma Wakamori (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-13958?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Takuma Wakamori updated HIVE-13958:
---
   Labels: patch  (was: )
Affects Version/s: 2.2.0
 Target Version/s: 2.2.0
   Status: Patch Available  (was: Open)

Check the implicit type-cast is allowed to the arguments passed to not only 
{{WHERE}} but also {{IN}} and {{BETWEEN}} operator.
It includes minor typo fixes.

> hive.strict.checks.type.safety should apply to decimals, as well as IN... and 
> BETWEEN... ops
> 
>
> Key: HIVE-13958
> URL: https://issues.apache.org/jira/browse/HIVE-13958
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 2.2.0
>Reporter: Sergey Shelukhin
>Assignee: Takuma Wakamori
>  Labels: patch
>
> String to decimal auto-casts should be prohibited for compares



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-13946) Decimal value need to be single-quoted when selecting where clause with that decimal value in order to get row

[Reuben Kuhnert (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-13946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15327793#comment-15327793
 ] 

Reuben Kuhnert commented on HIVE-13946:
---

Also, I'm noticing in your previous ticket 
([HIVE-13945|https://issues.apache.org/jira/browse/HIVE-13945]) your decimal 
expands with a bunch of additional zeros, but in your example above it doesn't?

> Decimal value need to be single-quoted when selecting where clause with that 
> decimal value in order to get row
> --
>
> Key: HIVE-13946
> URL: https://issues.apache.org/jira/browse/HIVE-13946
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 1.2.1
>Reporter: Takahiko Saito
> Fix For: 1.2.1
>
>
> Create a table withe a column of decimal type(38,18) and insert 
> '4327269606205.029297'. Then select with that value does not return anything.
> {noformat}
> 0: jdbc:hive2://ts-0531-1.openstacklocal:2181> drop table if exists test;
> No rows affected (0.175 seconds)
> 0: jdbc:hive2://ts-0531-1.openstacklocal:2181>
> 0: jdbc:hive2://ts-0531-1.openstacklocal:2181> create table test (dc 
> decimal(38,18));
> No rows affected (0.098 seconds)
> 0: jdbc:hive2://ts-0531-1.openstacklocal:2181>
> 0: jdbc:hive2://ts-0531-1.openstacklocal:2181> insert into table test values 
> (4327269606205.029297);
> INFO  : Session is already open
> INFO  : Dag name: insert into table tes...327269606205.029297)(Stage-1)
> INFO  : Tez session was closed. Reopening...
> INFO  : Session re-established.
> INFO  :
> INFO  : Status: Running (Executing on YARN cluster with App id 
> application_1464727816747_0762)
> INFO  : Map 1: -/-
> INFO  : Map 1: 0/1
> INFO  : Map 1: 0(+1)/1
> INFO  : Map 1: 1/1
> INFO  : Loading data to table default.test from 
> hdfs://ts-0531-5.openstacklocal:8020/apps/hive/warehouse/test/.hive-staging_hive_2016-06-04_00-03-54_302_7708281807413586675-940/-ext-1
> INFO  : Table default.test stats: [numFiles=1, numRows=1, totalSize=21, 
> rawDataSize=20]
> No rows affected (13.821 seconds)
> 0: jdbc:hive2://ts-0531-1.openstacklocal:2181>
> 0: jdbc:hive2://ts-0531-1.openstacklocal:2181> select * from test;
> +---+--+
> |test.dc|
> +---+--+
> | 4327269606205.029297  |
> +---+--+
> 1 row selected (0.078 seconds)
> 0: jdbc:hive2://ts-0531-1.openstacklocal:2181> select * from test where dc = 
> 4327269606205.029297;
> +--+--+
> | test.dc  |
> +--+--+
> +--+--+
> No rows selected (0.224 seconds)
> {noformat}
> If you single quote that decimal value, a row is returned.
> {noformat}
> 0: jdbc:hive2://ts-0531-1.openstacklocal:2181> select * from test where dc = 
> '4327269606205.029297';
> +---+--+
> |test.dc|
> +---+--+
> | 4327269606205.029297  |
> +---+--+
> 1 row selected (0.085 seconds)
> {noformat}
> explain shows:
> {noformat}
> 0: jdbc:hive2://ts-0531-1.openstacklocal:2181> explain select * from test 
> where dc = 4327269606205.029297;
> +--+--+
> |   Explain|
> +--+--+
> | STAGE DEPENDENCIES:  |
> |   Stage-0 is a root stage|
> |  |
> | STAGE PLANS: |
> |   Stage: Stage-0 |
> | Fetch Operator   |
> |   limit: -1  |
> |   Processor Tree:|
> | TableScan|
> |   alias: test|
> |   filterExpr: (dc = 4.3272696062050293E12) (type: boolean)   |
> |   Filter Operator|
> | predicate: (dc = 4.3272696062050293E12) (type: boolean)  |
> | Select Operator  |
> |   expressions: dc (type: decimal(38,18)) |
> |   outputColumnNames: _col0   |
> |   ListSink   |
> |  |
> +--+--+
> 18 rows selected (0.512 seconds)
> {noformat}



--

[jira] [Commented] (HIVE-13946) Decimal value need to be single-quoted when selecting where clause with that decimal value in order to get row

[Reuben Kuhnert (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-13946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15327783#comment-15327783
 ] 

Reuben Kuhnert commented on HIVE-13946:
---

I'm getting different results, am I doing something wrong?

{code}
0: jdbc:hive2://localhost:1> show tables;
show tables;
No rows selected (2.659 seconds)
+---+--+
| tab_name  |
+---+--+
+---+--+
0: jdbc:hive2://localhost:1> create table test (dc decimal(38,18));
18));
0: jdbc:hive2://localhost:1> No rows affected (1.367 seconds)
insert into table test values (4327269606205.029297);

27269606205.029297);
WARNING: Hive-on-MR is deprecated in Hive 2 and may not be available in the 
future versions. Consider using a different execution engine (i.e. tez, spark) 
or using Hive 1.X releases.
No rows affected (20.19 seconds)
0: jdbc:hive2://localhost:1> 
0: jdbc:hive2://localhost:1> select * from test;
select * from test;
1 row selected (0.564 seconds)
+---+--+
|  test.dc  |
+---+--+
| 4327269606205.029297  |
+---+--+
0: jdbc:hive2://localhost:1> select * from test where dc = 
4327269606205.029297
7269606205.029297
. . . . . . . . . . . . . . . .> ;
;
1 row selected (6.726 seconds)
+---+--+
|  test.dc  |
+---+--+
| 4327269606205.029300  |
+---+--+
0: jdbc:hive2://localhost:1> explain select * from test where dc = 
4327269606205.029297
dc = 4327269606205.029297
. . . . . . . . . . . . . . . .> ;
;
+---+--+
|Explain
|
+---+--+
| STAGE DEPENDENCIES:   
|
|   Stage-0 is a root stage 
|
|   
|
| STAGE PLANS:  
|
|   Stage: Stage-0  
|
| Fetch Operator
|
|   limit: -1   
|
|   Processor Tree: 
|
| TableScan 
|
|   alias: test 
|
|   Statistics: Num rows: 1 Data size: 32 Basic stats: COMPLETE Column 
stats: NONE  |
|   Filter Operator 
|
| predicate: (UDFToDouble(dc) = 4.3272696062050293E12) (type: 
boolean)  |
| Statistics: Num rows: 1 Data size: 32 Basic stats: COMPLETE 
Column stats: NONE|
| Select Operator   
|
|   expressions: 4327269606205.0293 (type: decimal(38,18))  
|
|   outputColumnNames: _col0
|
|   Statistics: Num rows: 1 Data size: 32 Basic stats: COMPLETE 
Column stats: NONE  |
|   ListSink
|
|   
|
+---+--+
{code}

> Decimal value need to be single-quoted when selecting where clause with that 
> decimal value in order to get row
> --
>
> Key: HIVE-13946
> URL: https://issues.apache.org/jira/browse/HIVE-13946
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 1.2.1
>Reporter: Takahiko Saito
> Fix For: 1.2.1
>
>
> Create a table withe a column of decimal type(38,18) and insert 
> '4327269606205.029297'. Then select with that value does not return anything.
> {noformat}
> 0: jdbc:hive2://ts-0531-1.openstacklocal:2181> drop table if exists test;
> No rows affected (0.175 seconds)
> 0: jdbc:hive2://ts-0531-1.openstacklocal:2181>
> 0: 

[jira] [Commented] (HIVE-13884) Disallow queries fetching more than a configured number of partitions in PartitionPruner

[JIRA]

[ 
https://issues.apache.org/jira/browse/HIVE-13884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15327768#comment-15327768
 ] 

Sergio Peña commented on HIVE-13884:


[~brocknoland] What type of query would you prefer to test? This patch won't 
allow a query to fetch all 100K partitions  if 
{{HiveConf.ConfVars.HIVELIMITTABLESCANPARTITION}} is set to a number > -1. If 
it is not set, no query is executed to reques the # of partitions, so no 
overhead will be added to it.

> Disallow queries fetching more than a configured number of partitions in 
> PartitionPruner
> 
>
> Key: HIVE-13884
> URL: https://issues.apache.org/jira/browse/HIVE-13884
> Project: Hive
>  Issue Type: Improvement
>Reporter: Mohit Sabharwal
>Assignee: Sergio Peña
> Attachments: HIVE-13884.1.patch
>
>
> Currently the PartitionPruner requests either all partitions or partitions 
> based on filter expression. In either scenarios, if the number of partitions 
> accessed is large there can be significant memory pressure at the HMS server 
> end.
> We already have a config {{hive.limit.query.max.table.partition}} that 
> enforces limits on number of partitions that may be scanned per operator. But 
> this check happens after the PartitionPruner has already fetched all 
> partitions.
> We should add an option at PartitionPruner level to disallow queries that 
> attempt to access number of partitions beyond a configurable limit.
> Note that {{hive.mapred.mode=strict}} disallow queries without a partition 
> filter in PartitionPruner, but this check accepts any query with a pruning 
> condition, even if partitions fetched are large. In multi-tenant 
> environments, admins could use more control w.r.t. number of partitions 
> allowed based on HMS memory capacity.
> One option is to have PartitionPruner first fetch the partition names 
> (instead of partition specs) and throw an exception if number of partitions 
> exceeds the configured value. Otherwise, fetch the partition specs.
> Looks like the existing {{listPartitionNames}} call could be used if extended 
> to take partition filter expressions like {{getPartitionsByExpr}} call does.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-13959) MoveTask should only release its query associated locks

[Yongzhi Chen (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-13959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15327753#comment-15327753
 ] 

Yongzhi Chen commented on HIVE-13959:
-

After talked with [~ctang.ma], getLocks return all the locks related to the 
path. It seems a consistent behavior in EmbeddedLockManager
and ZooKeeperHiveLockManager. 
So the first patch is a good fix.
+1 for the fix HIVE-13959.patch

> MoveTask should only release its query associated locks
> ---
>
> Key: HIVE-13959
> URL: https://issues.apache.org/jira/browse/HIVE-13959
> Project: Hive
>  Issue Type: Bug
>  Components: Locking
>Reporter: Chaoyu Tang
>Assignee: Chaoyu Tang
> Attachments: HIVE-13959.1.patch, HIVE-13959.patch, HIVE-13959.patch
>
>
> releaseLocks in MoveTask releases all locks under a HiveLockObject pathNames. 
> But some of locks under this pathNames might be for other queries and should 
> not be released.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-13984) Use multi-threaded approach to listing files for msck

[Pengcheng Xiong (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-13984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pengcheng Xiong updated HIVE-13984:
---
Status: Open  (was: Patch Available)

> Use multi-threaded approach to listing files for msck
> -
>
> Key: HIVE-13984
> URL: https://issues.apache.org/jira/browse/HIVE-13984
> Project: Hive
>  Issue Type: Sub-task
>Reporter: Pengcheng Xiong
>Assignee: Pengcheng Xiong
> Attachments: HIVE-13984.01.patch, HIVE-13984.02.patch, 
> HIVE-13984.03.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-13984) Use multi-threaded approach to listing files for msck

[Pengcheng Xiong (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-13984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pengcheng Xiong updated HIVE-13984:
---
Attachment: HIVE-13984.03.patch

> Use multi-threaded approach to listing files for msck
> -
>
> Key: HIVE-13984
> URL: https://issues.apache.org/jira/browse/HIVE-13984
> Project: Hive
>  Issue Type: Sub-task
>Reporter: Pengcheng Xiong
>Assignee: Pengcheng Xiong
> Attachments: HIVE-13984.01.patch, HIVE-13984.02.patch, 
> HIVE-13984.03.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-13984) Use multi-threaded approach to listing files for msck

[Pengcheng Xiong (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-13984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pengcheng Xiong updated HIVE-13984:
---
Status: Patch Available  (was: Open)

> Use multi-threaded approach to listing files for msck
> -
>
> Key: HIVE-13984
> URL: https://issues.apache.org/jira/browse/HIVE-13984
> Project: Hive
>  Issue Type: Sub-task
>Reporter: Pengcheng Xiong
>Assignee: Pengcheng Xiong
> Attachments: HIVE-13984.01.patch, HIVE-13984.02.patch, 
> HIVE-13984.03.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-13788) hive msck listpartitions need to make use of directSQL instead of datanucleus

[Ashutosh Chauhan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-13788?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashutosh Chauhan updated HIVE-13788:

   Resolution: Fixed
Fix Version/s: 2.1.1
   2.2.0
   Status: Resolved  (was: Patch Available)

Pushed to master & branch-2.1

> hive msck listpartitions need to make use of directSQL instead of datanucleus
> -
>
> Key: HIVE-13788
> URL: https://issues.apache.org/jira/browse/HIVE-13788
> Project: Hive
>  Issue Type: Improvement
>Reporter: Rajesh Balamohan
>Assignee: Hari Sankar Sivarama Subramaniyan
>Priority: Minor
> Fix For: 2.2.0, 2.1.1
>
> Attachments: HIVE-13788.1.patch, HIVE-13788.2.patch, 
> msck_call_stack_with_fix.png, msck_stack_trace.png
>
>
> Currently, for tables having 1000s of partitions too many DB calls are made 
> via datanucleus.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-13982) Extensions to RS dedup: execute with different column order and sorting direction if possible

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-13982?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15327706#comment-15327706
 ] 

Hive QA commented on HIVE-13982:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12809823/HIVE-13982.3.patch

{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 4 failed/errored test(s), 10225 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_acid_globallimit
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_13
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_multiinsert
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3
{noformat}

Test results: 
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/107/testReport
Console output: 
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/107/console
Test logs: 
http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-107/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 4 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12809823 - PreCommit-HIVE-MASTER-Build

> Extensions to RS dedup: execute with different column order and sorting 
> direction if possible
> -
>
> Key: HIVE-13982
> URL: https://issues.apache.org/jira/browse/HIVE-13982
> Project: Hive
>  Issue Type: Improvement
>  Components: Physical Optimizer
>Affects Versions: 2.2.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
> Attachments: HIVE-13982.2.patch, HIVE-13982.3.patch, HIVE-13982.patch
>
>
> Pointed out by [~gopalv].
> RS dedup should kick in for these cases, avoiding an additional shuffle stage.
> {code}
> select state, city, sum(sales) from table
> group by state, city
> order by state, city
> limit 10;
> {code}
> {code}
> select state, city, sum(sales) from table
> group by city, state
> order by state, city
> limit 10;
> {code}
> {code}
> select state, city, sum(sales) from table
> group by city, state
> order by state desc, city
> limit 10;
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (HIVE-14003) queries running against llap hang at times - preemption issues

[Siddharth Seth (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-14003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15327672#comment-15327672
 ] 

Siddharth Seth edited comment on HIVE-14003 at 6/13/16 4:21 PM:


Patch to fix the issue mentioned below.

I've left some TODOs in the code, which will likely be fixed in subsequent 
jiras. Need information from someone more familiar with the code. (Some of the 
comments read TODO: reviewer - those specifically need more info, ideally on RB)

Any operation which can block - e.g. processorContext.waitTillReady within the 
hive codebase has the potential to lead to problems like this. To be fixed in 
subsequent jiras. Some have been identified in the patch.

cc [~prasanth_j], [~sershe]




was (Author: sseth):
Patch to fix the issue mentioned below.

I've left some TODOs in the code, which will likely be fixed in subsequent 
jiras. Need information from someone more familiar with the code. (Some of the 
comments read TODO: reviewer - those specifically need more info, ideally on RB)

cc [~prasanth_j], [~sershe]



> queries running against llap hang at times - preemption issues
> --
>
> Key: HIVE-14003
> URL: https://issues.apache.org/jira/browse/HIVE-14003
> Project: Hive
>  Issue Type: Bug
>  Components: llap
>Affects Versions: 2.1.0
>Reporter: Takahiko Saito
>Assignee: Siddharth Seth
> Attachments: HIVE-14003.01.patch
>
>
> The preemption logic in the Hive processor needs some more work. There are 
> definitely windows where the abort flag is completely dropped within the Hive 
> processor.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-14003) queries running against llap hang at times - preemption issues

[Siddharth Seth (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-14003?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Siddharth Seth updated HIVE-14003:
--
Status: Patch Available  (was: Open)

> queries running against llap hang at times - preemption issues
> --
>
> Key: HIVE-14003
> URL: https://issues.apache.org/jira/browse/HIVE-14003
> Project: Hive
>  Issue Type: Bug
>  Components: llap
>Affects Versions: 2.1.0
>Reporter: Takahiko Saito
>Assignee: Siddharth Seth
> Attachments: HIVE-14003.01.patch
>
>
> The preemption logic in the Hive processor needs some more work. There are 
> definitely windows where the abort flag is completely dropped within the Hive 
> processor.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-14003) queries running against llap hang at times - likely due to preemption'

[Siddharth Seth (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-14003?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Siddharth Seth updated HIVE-14003:
--
Attachment: HIVE-14003.01.patch

Patch to fix the issue mentioned below.

I've left some TODOs in the code, which will likely be fixed in subsequent 
jiras. Need information from someone more familiar with the code. (Some of the 
comments read TODO: reviewer - those specifically need more info, ideally on RB)

cc [~prasanth_j], [~sershe]



> queries running against llap hang at times - likely due to preemption'
> --
>
> Key: HIVE-14003
> URL: https://issues.apache.org/jira/browse/HIVE-14003
> Project: Hive
>  Issue Type: Bug
>  Components: llap
>Affects Versions: 2.1.0
>Reporter: Takahiko Saito
>Assignee: Siddharth Seth
> Attachments: HIVE-14003.01.patch
>
>
> The preemption logic in the Hive processor needs some more work. There are 
> definitely windows where the abort flag is completely dropped within the Hive 
> processor.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-14003) queries running against llap hang at times - preemption issues

[Siddharth Seth (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-14003?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Siddharth Seth updated HIVE-14003:
--
Summary: queries running against llap hang at times - preemption issues  
(was: queries running against llap hang at times - likely due to preemption')

> queries running against llap hang at times - preemption issues
> --
>
> Key: HIVE-14003
> URL: https://issues.apache.org/jira/browse/HIVE-14003
> Project: Hive
>  Issue Type: Bug
>  Components: llap
>Affects Versions: 2.1.0
>Reporter: Takahiko Saito
>Assignee: Siddharth Seth
> Attachments: HIVE-14003.01.patch
>
>
> The preemption logic in the Hive processor needs some more work. There are 
> definitely windows where the abort flag is completely dropped within the Hive 
> processor.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-14003) queries running against llap hang at times - likely due to preemption'

[Siddharth Seth (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-14003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15327669#comment-15327669
 ] 

Siddharth Seth commented on HIVE-14003:
---

This is a continuation of HIVE-13858. Not all problems related to abort 
processing were fixed there.

There's a window at the moment, while the RecordProcessor is being initialized 
where an abort can be ignored. initialize in progress, mapOp/reduceOp not setup 
yet, abort received - this sends an abort to the RecordProcessor - but it's 
essentially useless since it's not propagated to the operator (which hasn't 
been setup yet)

Uploading a patch shortly to resolve this.

> queries running against llap hang at times - likely due to preemption'
> --
>
> Key: HIVE-14003
> URL: https://issues.apache.org/jira/browse/HIVE-14003
> Project: Hive
>  Issue Type: Bug
>  Components: llap
>Affects Versions: 2.1.0
>Reporter: Takahiko Saito
>Assignee: Siddharth Seth
>
> The preemption logic in the Hive processor needs some more work. There are 
> definitely windows where the abort flag is completely dropped within the Hive 
> processor.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (HIVE-14003) queries running against llap hang at times - likely due to preemption'

[Siddharth Seth (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-14003?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Siddharth Seth reassigned HIVE-14003:
-

Assignee: Siddharth Seth

> queries running against llap hang at times - likely due to preemption'
> --
>
> Key: HIVE-14003
> URL: https://issues.apache.org/jira/browse/HIVE-14003
> Project: Hive
>  Issue Type: Bug
>  Components: llap
>Affects Versions: 2.1.0
>Reporter: Takahiko Saito
>Assignee: Siddharth Seth
>
> The preemption logic in the Hive processor needs some more work. There are 
> definitely windows where the abort flag is completely dropped within the Hive 
> processor.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-13617) LLAP: support non-vectorized execution in IO

[JIRA]

[ 
https://issues.apache.org/jira/browse/HIVE-13617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15327652#comment-15327652
 ] 

Sergio Peña commented on HIVE-13617:


[~sershe] Your change should have worked fine. I see that minillap.query.files 
is excluded from TestCliDriver. 
This is part of {{itests/qtest/pom.xml}}
{noformat}

{noformat}

Have you tried to run the same .q file from TestCliDriver to see if it is 
executed in your environment?

> LLAP: support non-vectorized execution in IO
> 
>
> Key: HIVE-13617
> URL: https://issues.apache.org/jira/browse/HIVE-13617
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-13617-wo-11417.patch, HIVE-13617-wo-11417.patch, 
> HIVE-13617.01.patch, HIVE-13617.03.patch, HIVE-13617.04.patch, 
> HIVE-13617.05.patch, HIVE-13617.06.patch, HIVE-13617.patch, HIVE-13617.patch, 
> HIVE-15396-with-oi.patch
>
>
> Two approaches - a separate decoding path, into rows instead of VRBs; or 
> decoding VRBs into rows on a higher level (the original LlapInputFormat). I 
> think the latter might be better - it's not a hugely important path, and perf 
> in non-vectorized case is not the best anyway, so it's better to make do with 
> much less new code and architectural disruption. 
> Some ORC patches in progress introduce an easy to reuse (or so I hope, 
> anyway) VRB-to-row conversion, so we should just use that.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-13964) Add a parameter to beeline to allow a properties file to be passed in

[JIRA]

[ 
https://issues.apache.org/jira/browse/HIVE-13964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15327637#comment-15327637
 ] 

Sergio Peña commented on HIVE-13964:


Thanks [~ayousufi]. This time it works fine.

I was running some tests, but I don't find the properties needed to make it 
work.

1. Using an empty properties file
{noformat}
$ beeline --property-file /tmp/props
Property "url" is required
$ 
{noformat}

2. Passing URL through command line and a properties file
{noformat}
$ beeline -u jdbc:hive2://localhost:1 --property-file=/tmp/props
Connecting to jdbc:hive2://localhost:1
Connected to: Apache Hive (version 2.2.0-SNAPSHOT)
Driver: Hive JDBC (version 2.2.0-SNAPSHOT)
16/06/13 10:44:01 [main]: WARN jdbc.HiveConnection: Request to set autoCommit 
to false; Hive does not support autoCommit=false.
Transaction isolation: TRANSACTION_REPEATABLE_READ
Property "url" is required
Closing: 0: jdbc:hive2://localhost:1
$
{noformat}

3. Adding 'url' to properties file
{noformat}
$ echo "url=jdbc:hive2://localhost:1" > /tmp/props
$ beeline --property-file /tmp/props
Connecting to jdbc:hive2://localhost:1
java.lang.NullPointerException
$
{noformat}

I'm not sure how this properties file should work on beeline. 
- Is 'url' needed even if I pass '-u' through cmdline? If both are the same, I 
think we should take cmdline to connect first, then use the one in props.
- what is the 'url' value passed to props? why is there a NPE?

I imagine props file is just an optional file that can be used to pass other 
hive properties that users usually set dynamically through beeline command 
line, such as:
{noformat}
$ cat /tmp/props
hive.default.fileformat=ORC
{noformat}

Could you make sure that --property-file works as expected?

> Add a parameter to beeline to allow a properties file to be passed in
> -
>
> Key: HIVE-13964
> URL: https://issues.apache.org/jira/browse/HIVE-13964
> Project: Hive
>  Issue Type: New Feature
>  Components: Beeline
>Affects Versions: 2.0.1
>Reporter: Abdullah Yousufi
>Assignee: Abdullah Yousufi
>Priority: Minor
> Fix For: 2.2.0
>
> Attachments: HIVE-13964.01.patch, HIVE-13964.02.patch, 
> HIVE-13964.03.patch
>
>
> HIVE-6652 removed the ability to pass in a properties file as a beeline 
> parameter. It may be a useful feature to be able to pass the file in is a 
> parameter, such as --property-file.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-13981) Operation.toSQLException eats full exception stack

[Thejas M Nair (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-13981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15327624#comment-15327624
 ] 

Thejas M Nair commented on HIVE-13981:
--

Won't this change the error code that would be printed for many errors ?

It seems like the line "  ex.initCause(response.getException())"  should have 
taken care of the problem this patch is trying to address.



> Operation.toSQLException eats full exception stack
> --
>
> Key: HIVE-13981
> URL: https://issues.apache.org/jira/browse/HIVE-13981
> Project: Hive
>  Issue Type: Bug
>Reporter: Daniel Dai
>Assignee: Daniel Dai
> Attachments: HIVE-13981.1.patch
>
>
> Operation.toSQLException eats half of the exception stack and make debug 
> hard. For example, we saw an exception:
> {code}
> org.apache.hive.service.cli.HiveSQL Exception : Error while compiling 
> statement: FAILED : NullPointer Exception null
> at org.apache.hive.service.cli.operation.Operation.toSQL Exception 
> (Operation.java:336)
> at 
> org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:113)
> at 
> org.apache.hive.service.cli.operation.SQLOperation.runInternal(SQLOperation.java:182)
> at org.apache.hive.service.cli.operation.Operation.run(Operation.java:278)
> at 
> org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:421)
> at 
> org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementAsync(HiveSessionImpl.java:408)
> at 
> org.apache.hive.service.cli.CLIService.executeStatementAsync(CLIService.java:276)
> at 
> org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(ThriftCLIService.java:505)
> at 
> org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1317)
> at 
> org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1302)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> at 
> org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor.process(HadoopThriftAuthBridge.java:562)
> at 
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang. NullPointer Exception
> {code}
> The real stack causing the NPE is lost.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-13864) Beeline ignores the command that follows a semicolon and comment

[Reuben Kuhnert (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-13864?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Reuben Kuhnert updated HIVE-13864:
--
Status: Patch Available  (was: Open)

> Beeline ignores the command that follows a semicolon and comment
> 
>
> Key: HIVE-13864
> URL: https://issues.apache.org/jira/browse/HIVE-13864
> Project: Hive
>  Issue Type: Bug
>Reporter: Muthu Manickam
>Assignee: Reuben Kuhnert
> Attachments: HIVE-13864.01.patch, HIVE-13864.02.patch
>
>
> Beeline ignores the next line/command that follows a command with semicolon 
> and comments.
> Example 1:
> select *
> from table1; -- comments
> select * from table2;
> In this case, only the first command is executed.. second command "select * 
> from table2" is not executed.
> --
> Example 2:
> select *
> from table1; -- comments
> select * from table2;
> select * from table3;
> In this case, first command and third command is executed. second command 
> "select * from table2" is not executed.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-13864) Beeline ignores the command that follows a semicolon and comment

[Reuben Kuhnert (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-13864?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Reuben Kuhnert updated HIVE-13864:
--
Status: Open  (was: Patch Available)

> Beeline ignores the command that follows a semicolon and comment
> 
>
> Key: HIVE-13864
> URL: https://issues.apache.org/jira/browse/HIVE-13864
> Project: Hive
>  Issue Type: Bug
>Reporter: Muthu Manickam
>Assignee: Reuben Kuhnert
> Attachments: HIVE-13864.01.patch, HIVE-13864.02.patch
>
>
> Beeline ignores the next line/command that follows a command with semicolon 
> and comments.
> Example 1:
> select *
> from table1; -- comments
> select * from table2;
> In this case, only the first command is executed.. second command "select * 
> from table2" is not executed.
> --
> Example 2:
> select *
> from table1; -- comments
> select * from table2;
> select * from table3;
> In this case, first command and third command is executed. second command 
> "select * from table2" is not executed.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-13864) Beeline ignores the command that follows a semicolon and comment

[Reuben Kuhnert (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-13864?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Reuben Kuhnert updated HIVE-13864:
--
Attachment: HIVE-13864.02.patch

> Beeline ignores the command that follows a semicolon and comment
> 
>
> Key: HIVE-13864
> URL: https://issues.apache.org/jira/browse/HIVE-13864
> Project: Hive
>  Issue Type: Bug
>Reporter: Muthu Manickam
>Assignee: Reuben Kuhnert
> Attachments: HIVE-13864.01.patch, HIVE-13864.02.patch
>
>
> Beeline ignores the next line/command that follows a command with semicolon 
> and comments.
> Example 1:
> select *
> from table1; -- comments
> select * from table2;
> In this case, only the first command is executed.. second command "select * 
> from table2" is not executed.
> --
> Example 2:
> select *
> from table1; -- comments
> select * from table2;
> select * from table3;
> In this case, first command and third command is executed. second command 
> "select * from table2" is not executed.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-11270) Tez gives different responses when run on Physical tables and logical views

[Chetna Chaudhari (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-11270?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15327560#comment-15327560
 ] 

Chetna Chaudhari commented on HIVE-11270:
-

@kannanpoem1984: This used to happen intermittently on tez engine. To unblock, 
we are using MR as an execution engine. Are you hitting this issue ? 

> Tez gives different responses when run on Physical tables and logical views
> ---
>
> Key: HIVE-11270
> URL: https://issues.apache.org/jira/browse/HIVE-11270
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 1.2.0
> Environment: Hive 1.2.0 and Tez 0.7.0,
>Reporter: Soundararajan Velu
>Priority: Critical
>
> The same query, one on view and other on the physical table yields different 
> results, the query on view returns no or few records output.
> CBO is turned on and following are the flags used,
> set hive.cli.print.current.db=true;
> set hive.cli.print.header=true;
> set hive.execution.engine=tez;
> set mapreduce.job.queuename=admin;
> set tez.queue.name=admin;
> set hive.tez.container.size=5096;
> set tez.task.resource.memory.mb=5096;
> set hive.auto.convert.join=true;
> set hive.auto.convert.sortmerge.join.to.mapjoin=true;
> set hive.auto.convert.sortmerge.join=true;
> set hive.enforce.bucketmapjoin=true;
> set hive.enforce.bucketing=true;
> set hive.enforce.sorting=true;
> set hive.enforce.sortmergebucketmapjoin=true;
> set hive.optimize.bucketmapjoin.sortedmerge=true; 
> set hive.optimize.skewjoin=true;
> set hive.optimize.skewjoin.compiletime=true;
> set hive.groupby.skewindata=true;
> set hive.convert.join.bucket.mapjoin.tez=true;
> set hive.exec.parallel=true;
> set hive.vectorized.execution.enabled=true;
> set hive.vectorized.groupby.maxentries=10240;
> set hive.vectorized.groupby.flush.percent=0.1;
> set hive.tez.auto.reducer.parallelism=true;
> set hive.tez.min.partition.factor=50;
> set hive.tez.max.partition.factor=100;
> set io.sort.mb=400;
> set hive.input.format=org.apache.hadoop.hive.ql.io.BucketizedHiveInputFormat;
> set hive.hashtable.initialCapacity=100;
> set hive.hashtable.key.count.adjustment=1.0;
> set hive.hashtable.loadfactor=0.99;
> set tez.runtime.io.sort.mb=1800;
> set tez.runtime.sort.threads=4;
> set tez.runtime.io.sort.factor=200;
> set tez.runtime.shuffle.memory-to-memory.enable=false;
> set tez.runtime.shuffle.memory-to-memory.segments=4;
> set tez.runtime.pipelined-shuffle.enable=true;
> set tez.runtime.optimize.shared.fetch=true;
> set tez.runtime.shuffle.keep-alive.enabled=true;
> set tez.runtime.optimize.local.fetch=false;
> set hive.exec.reducers.max=300;
> set hive.mapjoin.hybridgrace.hashtable=true;
> set hive.mapjoin.hybridgrace.memcheckfrequency=1024;
> set hive.mapjoin.optimized.hashtable=true;
> set hive.mapjoin.optimized.hashtable.wbsize=88;
> set hive.mapjoin.localtask.max.memory.usage=0.99;
> set hive.optimize.skewjoin.compiletime=false;
> set hive.skewjoin.key=1000;
> set hive.skewjoin.mapjoin.map.tasks=200;
> set hive.skewjoin.mapjoin.min.split=134217728;
> set hive.compute.query.using.stats=true;



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-13959) MoveTask should only release its query associated locks

[Chaoyu Tang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-13959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15327551#comment-15327551
 ] 

Chaoyu Tang commented on HIVE-13959:


The failed tests seem not related to the patch.

> MoveTask should only release its query associated locks
> ---
>
> Key: HIVE-13959
> URL: https://issues.apache.org/jira/browse/HIVE-13959
> Project: Hive
>  Issue Type: Bug
>  Components: Locking
>Reporter: Chaoyu Tang
>Assignee: Chaoyu Tang
> Attachments: HIVE-13959.1.patch, HIVE-13959.patch, HIVE-13959.patch
>
>
> releaseLocks in MoveTask releases all locks under a HiveLockObject pathNames. 
> But some of locks under this pathNames might be for other queries and should 
> not be released.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-14002) Extend limit propagation to subsequent RS operators

[Jesus Camacho Rodriguez (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-14002?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jesus Camacho Rodriguez updated HIVE-14002:
---
Attachment: (was: HIVE-14002.patch)

> Extend limit propagation to subsequent RS operators
> ---
>
> Key: HIVE-14002
> URL: https://issues.apache.org/jira/browse/HIVE-14002
> Project: Hive
>  Issue Type: Improvement
>  Components: Physical Optimizer
>Affects Versions: 2.2.0
>Reporter: Nita Dembla
>Assignee: Jesus Camacho Rodriguez
> Attachments: HIVE-14002.patch
>
>
> In some occasions, for instance when RS dedup does not kick in, it is useful 
> to propagate the limit to subsequent RS operators, as this will reduce 
> intermediary results and impact performance. This issue covers that extension.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-14002) Extend limit propagation to subsequent RS operators

[Jesus Camacho Rodriguez (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-14002?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jesus Camacho Rodriguez updated HIVE-14002:
---
Attachment: HIVE-14002.patch

> Extend limit propagation to subsequent RS operators
> ---
>
> Key: HIVE-14002
> URL: https://issues.apache.org/jira/browse/HIVE-14002
> Project: Hive
>  Issue Type: Improvement
>  Components: Physical Optimizer
>Affects Versions: 2.2.0
>Reporter: Nita Dembla
>Assignee: Jesus Camacho Rodriguez
> Attachments: HIVE-14002.patch
>
>
> In some occasions, for instance when RS dedup does not kick in, it is useful 
> to propagate the limit to subsequent RS operators, as this will reduce 
> intermediary results and impact performance. This issue covers that extension.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-14002) Extend limit propagation to subsequent RS operators

[Jesus Camacho Rodriguez (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-14002?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jesus Camacho Rodriguez updated HIVE-14002:
---
Component/s: (was: Hive)
 Physical Optimizer

> Extend limit propagation to subsequent RS operators
> ---
>
> Key: HIVE-14002
> URL: https://issues.apache.org/jira/browse/HIVE-14002
> Project: Hive
>  Issue Type: Improvement
>  Components: Physical Optimizer
>Affects Versions: 2.2.0
>Reporter: Nita Dembla
>Assignee: Jesus Camacho Rodriguez
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-13959) MoveTask should only release its query associated locks

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-13959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15327506#comment-15327506
 ] 

Hive QA commented on HIVE-13959:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12809813/HIVE-13959.1.patch

{color:red}ERROR:{color} -1 due to no test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 4 failed/errored test(s), 10224 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_acid_globallimit
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_list_bucket
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_multiinsert
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3
{noformat}

Test results: 
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/106/testReport
Console output: 
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/106/console
Test logs: 
http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-106/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 4 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12809813 - PreCommit-HIVE-MASTER-Build

> MoveTask should only release its query associated locks
> ---
>
> Key: HIVE-13959
> URL: https://issues.apache.org/jira/browse/HIVE-13959
> Project: Hive
>  Issue Type: Bug
>  Components: Locking
>Reporter: Chaoyu Tang
>Assignee: Chaoyu Tang
> Attachments: HIVE-13959.1.patch, HIVE-13959.patch, HIVE-13959.patch
>
>
> releaseLocks in MoveTask releases all locks under a HiveLockObject pathNames. 
> But some of locks under this pathNames might be for other queries and should 
> not be released.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-14002) Extend limit propagation to subsequent RS operators

[Jesus Camacho Rodriguez (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-14002?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jesus Camacho Rodriguez updated HIVE-14002:
---
Attachment: HIVE-14002.patch

> Extend limit propagation to subsequent RS operators
> ---
>
> Key: HIVE-14002
> URL: https://issues.apache.org/jira/browse/HIVE-14002
> Project: Hive
>  Issue Type: Improvement
>  Components: Physical Optimizer
>Affects Versions: 2.2.0
>Reporter: Nita Dembla
>Assignee: Jesus Camacho Rodriguez
> Attachments: HIVE-14002.patch
>
>
> In some occasions, for instance when RS dedup does not kick in, it is useful 
> to propagate the limit to subsequent RS operators, as this will reduce 
> intermediary results and impact performance. This issue covers that extension.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-14002) Extend limit propagation to subsequent RS operators

[Jesus Camacho Rodriguez (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-14002?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jesus Camacho Rodriguez updated HIVE-14002:
---
Status: Patch Available  (was: In Progress)

> Extend limit propagation to subsequent RS operators
> ---
>
> Key: HIVE-14002
> URL: https://issues.apache.org/jira/browse/HIVE-14002
> Project: Hive
>  Issue Type: Improvement
>  Components: Physical Optimizer
>Affects Versions: 2.2.0
>Reporter: Nita Dembla
>Assignee: Jesus Camacho Rodriguez
>
> In some occasions, for instance when RS dedup does not kick in, it is useful 
> to propagate the limit to subsequent RS operators, as this will reduce 
> intermediary results and impact performance. This issue covers that extension.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-14002) Extend limit propagation to subsequent RS operators

[Jesus Camacho Rodriguez (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-14002?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jesus Camacho Rodriguez updated HIVE-14002:
---
Description: In some occasions, for instance when RS dedup does not kick 
in, it is useful to propagate the limit to subsequent RS operators, as this 
will reduce intermediary results and impact performance. This issue covers that 
extension.

> Extend limit propagation to subsequent RS operators
> ---
>
> Key: HIVE-14002
> URL: https://issues.apache.org/jira/browse/HIVE-14002
> Project: Hive
>  Issue Type: Improvement
>  Components: Physical Optimizer
>Affects Versions: 2.2.0
>Reporter: Nita Dembla
>Assignee: Jesus Camacho Rodriguez
>
> In some occasions, for instance when RS dedup does not kick in, it is useful 
> to propagate the limit to subsequent RS operators, as this will reduce 
> intermediary results and impact performance. This issue covers that extension.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Work started] (HIVE-14002) Extend limit propagation to subsequent RS operators

[Jesus Camacho Rodriguez (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-14002?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Work on HIVE-14002 started by Jesus Camacho Rodriguez.
--
> Extend limit propagation to subsequent RS operators
> ---
>
> Key: HIVE-14002
> URL: https://issues.apache.org/jira/browse/HIVE-14002
> Project: Hive
>  Issue Type: Improvement
>  Components: Physical Optimizer
>Affects Versions: 2.2.0
>Reporter: Nita Dembla
>Assignee: Jesus Camacho Rodriguez
>
> In some occasions, for instance when RS dedup does not kick in, it is useful 
> to propagate the limit to subsequent RS operators, as this will reduce 
> intermediary results and impact performance. This issue covers that extension.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-13899) Beeline CLI is not showing the db name even after setting the property 'set hive.cli.print.current.db=true'

[Peter Vary (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-13899?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15327504#comment-15327504
 ] 

Peter Vary commented on HIVE-13899:
---

Does anyone working on this issue?

If not, then I might try create a patch.

> Beeline CLI is not showing the db name even after setting the property 'set 
> hive.cli.print.current.db=true'
> ---
>
> Key: HIVE-13899
> URL: https://issues.apache.org/jira/browse/HIVE-13899
> Project: Hive
>  Issue Type: Bug
>  Components: Beeline
>Affects Versions: 0.10.0
>Reporter: Sravan Suryadevara
>Priority: Trivial
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-14002) Extend limit propagation to subsequent RS operators

[Jesus Camacho Rodriguez (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-14002?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jesus Camacho Rodriguez updated HIVE-14002:
---
Issue Type: Improvement  (was: Bug)

> Extend limit propagation to subsequent RS operators
> ---
>
> Key: HIVE-14002
> URL: https://issues.apache.org/jira/browse/HIVE-14002
> Project: Hive
>  Issue Type: Improvement
>  Components: Physical Optimizer
>Affects Versions: 2.2.0
>Reporter: Nita Dembla
>Assignee: Jesus Camacho Rodriguez
>Priority: Blocker
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-14001) beeline doesn't give out an error when takes either "-e" or "-f" in command instead of both

[Peter Vary (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-14001?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Peter Vary updated HIVE-14001:
--
Status: Patch Available  (was: Open)

Please review - every comment is welcome (first patch)

> beeline doesn't give out an error when takes either "-e" or "-f" in command 
> instead of both
> ---
>
> Key: HIVE-14001
> URL: https://issues.apache.org/jira/browse/HIVE-14001
> Project: Hive
>  Issue Type: Improvement
>  Components: Beeline
>Affects Versions: 2.0.1, 0.10.0
>Reporter: Peter Vary
>Priority: Trivial
> Attachments: HIVE-14001.patch
>
>
> When providing both arguments there should be an error message



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (HIVE-14002) Extend limit propagation to subsequent RS operators

[Jesus Camacho Rodriguez (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-14002?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jesus Camacho Rodriguez reassigned HIVE-14002:
--

Assignee: Jesus Camacho Rodriguez

> Extend limit propagation to subsequent RS operators
> ---
>
> Key: HIVE-14002
> URL: https://issues.apache.org/jira/browse/HIVE-14002
> Project: Hive
>  Issue Type: Bug
>  Components: Physical Optimizer
>Affects Versions: 2.2.0
>Reporter: Nita Dembla
>Assignee: Jesus Camacho Rodriguez
>Priority: Blocker
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-14002) Extend limit propagation to subsequent RS operators

[Jesus Camacho Rodriguez (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-14002?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jesus Camacho Rodriguez updated HIVE-14002:
---
Priority: Major  (was: Blocker)

> Extend limit propagation to subsequent RS operators
> ---
>
> Key: HIVE-14002
> URL: https://issues.apache.org/jira/browse/HIVE-14002
> Project: Hive
>  Issue Type: Improvement
>  Components: Physical Optimizer
>Affects Versions: 2.2.0
>Reporter: Nita Dembla
>Assignee: Jesus Camacho Rodriguez
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-14001) beeline doesn't give out an error when takes either "-e" or "-f" in command instead of both

[Peter Vary (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-14001?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Peter Vary updated HIVE-14001:
--
Attachment: HIVE-14001.patch

Checking the command line parameters.
If -e are present, and at least one command is specified, and a file is 
specified with a -f argument, then exiting with error code 1.

Testcase is created too.

> beeline doesn't give out an error when takes either "-e" or "-f" in command 
> instead of both
> ---
>
> Key: HIVE-14001
> URL: https://issues.apache.org/jira/browse/HIVE-14001
> Project: Hive
>  Issue Type: Improvement
>  Components: Beeline
>Affects Versions: 0.10.0, 2.0.1
>Reporter: Peter Vary
>Priority: Trivial
> Attachments: HIVE-14001.patch
>
>
> When providing both arguments there should be an error message



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-13982) Extensions to RS dedup: execute with different column order and sorting direction if possible

[Jesus Camacho Rodriguez (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-13982?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jesus Camacho Rodriguez updated HIVE-13982:
---
Description: 
Pointed out by [~gopalv].

RS dedup should kick in for these cases.

{code}
select state, city, sum(sales) from table
group by state, city
order by state, city
limit 10;
{code}

{code}
select state, city, sum(sales) from table
group by city, state
order by state, city
limit 10;
{code}

{code}
select state, city, sum(sales) from table
group by city, state
order by state desc, city
limit 10;
{code}

  was:
Pointed out by [~gopalv].

{code}
select state, city, sum(sales) from table
group by state, city
order by state, city
limit 10;
{code}

{code}
select state, city, sum(sales) from table
group by city, state
order by state, city
limit 10;
{code}

{code}
select state, city, sum(sales) from table
group by city, state
order by state desc, city
limit 10;
{code}


> Extensions to RS dedup: execute with different column order and sorting 
> direction if possible
> -
>
> Key: HIVE-13982
> URL: https://issues.apache.org/jira/browse/HIVE-13982
> Project: Hive
>  Issue Type: Improvement
>  Components: Physical Optimizer
>Affects Versions: 2.2.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
> Attachments: HIVE-13982.2.patch, HIVE-13982.3.patch, HIVE-13982.patch
>
>
> Pointed out by [~gopalv].
> RS dedup should kick in for these cases.
> {code}
> select state, city, sum(sales) from table
> group by state, city
> order by state, city
> limit 10;
> {code}
> {code}
> select state, city, sum(sales) from table
> group by city, state
> order by state, city
> limit 10;
> {code}
> {code}
> select state, city, sum(sales) from table
> group by city, state
> order by state desc, city
> limit 10;
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-13982) Extensions to RS dedup: execute with different column order and sorting direction if possible

[Jesus Camacho Rodriguez (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-13982?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jesus Camacho Rodriguez updated HIVE-13982:
---
Description: 
Pointed out by [~gopalv].

RS dedup should kick in for these cases, avoiding an additional shuffle stage.

{code}
select state, city, sum(sales) from table
group by state, city
order by state, city
limit 10;
{code}

{code}
select state, city, sum(sales) from table
group by city, state
order by state, city
limit 10;
{code}

{code}
select state, city, sum(sales) from table
group by city, state
order by state desc, city
limit 10;
{code}

  was:
Pointed out by [~gopalv].

RS dedup should kick in for these cases.

{code}
select state, city, sum(sales) from table
group by state, city
order by state, city
limit 10;
{code}

{code}
select state, city, sum(sales) from table
group by city, state
order by state, city
limit 10;
{code}

{code}
select state, city, sum(sales) from table
group by city, state
order by state desc, city
limit 10;
{code}


> Extensions to RS dedup: execute with different column order and sorting 
> direction if possible
> -
>
> Key: HIVE-13982
> URL: https://issues.apache.org/jira/browse/HIVE-13982
> Project: Hive
>  Issue Type: Improvement
>  Components: Physical Optimizer
>Affects Versions: 2.2.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
> Attachments: HIVE-13982.2.patch, HIVE-13982.3.patch, HIVE-13982.patch
>
>
> Pointed out by [~gopalv].
> RS dedup should kick in for these cases, avoiding an additional shuffle stage.
> {code}
> select state, city, sum(sales) from table
> group by state, city
> order by state, city
> limit 10;
> {code}
> {code}
> select state, city, sum(sales) from table
> group by city, state
> order by state, city
> limit 10;
> {code}
> {code}
> select state, city, sum(sales) from table
> group by city, state
> order by state desc, city
> limit 10;
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-13982) Extensions to RS dedup: execute with different column order and sorting direction if possible

[Jesus Camacho Rodriguez (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-13982?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jesus Camacho Rodriguez updated HIVE-13982:
---
Description: 
Pointed out by [~gopalv].

{code}
select state, city, sum(sales) from table
group by state, city
order by state, city
limit 10;
{code}

{code}
select state, city, sum(sales) from table
group by city, state
order by state, city
limit 10;
{code}

{code}
select state, city, sum(sales) from table
group by city, state
order by state desc, city
limit 10;
{code}

  was:
Pointed out by [~gopalv].

Queries which follow the format are not optimal with map-side aggregation, 
because the Map 1 does not have TopN in the reduce sink.

These queries shuffle 100% of the aggregate in cases where the reduce de-dup 
does not kick in. 

{code}
select state, city, sum(sales) from table
group by state, city
order by state, city
limit 10;
{code}

{code}
select state, city, sum(sales) from table
group by city, state
order by state, city
limit 10;
{code}

{code}
select state, city, sum(sales) from table
group by city, state
order by state desc, city
limit 10;
{code}


> Extensions to RS dedup: execute with different column order and sorting 
> direction if possible
> -
>
> Key: HIVE-13982
> URL: https://issues.apache.org/jira/browse/HIVE-13982
> Project: Hive
>  Issue Type: Improvement
>  Components: Physical Optimizer
>Affects Versions: 2.2.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
> Attachments: HIVE-13982.2.patch, HIVE-13982.3.patch, HIVE-13982.patch
>
>
> Pointed out by [~gopalv].
> {code}
> select state, city, sum(sales) from table
> group by state, city
> order by state, city
> limit 10;
> {code}
> {code}
> select state, city, sum(sales) from table
> group by city, state
> order by state, city
> limit 10;
> {code}
> {code}
> select state, city, sum(sales) from table
> group by city, state
> order by state desc, city
> limit 10;
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)