[jira] [Created] (HIVE-17811) LLAP: Use NUMA interleaved allocations for memory cache on POWER cpus

[Gopal V (JIRA)]

Gopal V created HIVE-17811:
--

 Summary: LLAP: Use NUMA interleaved allocations for memory cache 
on POWER cpus
 Key: HIVE-17811
 URL: https://issues.apache.org/jira/browse/HIVE-17811
 Project: Hive
  Issue Type: Bug
Reporter: Gopal V






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: Review Request 62981: HIVE-17672: Upgrade Calcite version to 1.14

[Ashutosh Chauhan]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62981/#review188049
---




ql/src/java/org/apache/hadoop/hive/ql/metadata/HiveMaterializedViewsRegistry.java
Line 317 (original), 318 (patched)


Will be good to add a TODO to remove reference to LocalInterval.



ql/src/java/org/apache/hadoop/hive/ql/optimizer/SortedDynPartitionTimeGranularityOptimizer.java
Lines 285-288 (patched)


Will be good to add a comment for a need to do a cast to timestamp first.
Also, should it not be to ts w/ tz ?



ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/functions/HiveSqlCountAggFunction.java
Lines 89 (patched)


Good to add a reason for need to override.



ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/ASTBuilder.java
Lines 285 (patched)


good to add a comment on why we postfix UTC



ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/ExprNodeConverter.java
Lines 309 (patched)


comment for UTC.



ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java
Line 2402 (original), 2404 (patched)


TODO to remove this reference.



ql/src/java/org/apache/hadoop/hive/ql/parse/IdentifiersParser.g
Lines 371 (patched)


We already support ts w/ tz in syntax. What is this change for?


- Ashutosh Chauhan


On Oct. 13, 2017, 4:52 p.m., Jesús Camacho Rodríguez wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/62981/
> ---
> 
> (Updated Oct. 13, 2017, 4:52 p.m.)
> 
> 
> Review request for hive and Ashutosh Chauhan.
> 
> 
> Bugs: HIVE-17672
> https://issues.apache.org/jira/browse/HIVE-17672
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-17672: Upgrade Calcite version to 1.14
> 
> 
> Diffs
> -
> 
>   
> druid-handler/src/java/org/apache/hadoop/hive/druid/io/DruidOutputFormat.java 
> da7642c61a3c1b82e0d505d110bef63863e7be13 
>   druid-handler/src/java/org/apache/hadoop/hive/druid/serde/DruidSerDe.java 
> a43f62ed8203b74fde64e676e75385b4cd52570b 
>   
> druid-handler/src/java/org/apache/hadoop/hive/druid/serde/DruidSerDeUtils.java
>  86c325ba1fc728e452dea8440ab1ca0face45cc3 
>   
> druid-handler/src/test/org/apache/hadoop/hive/druid/serde/TestDruidSerDe.java 
> 655a96ad7b5a24de91707e35b93fb3eb99a2563a 
>   pom.xml ca088da3bfda76fd3da19d96fb6f7705a8adfe57 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java 
> 1d1d2a3e0bee177cd4b145c731b5ea3c75483c84 
>   ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java 
> 68c94820c9368f504401c801f9b599a14f91ddf6 
>   
> ql/src/java/org/apache/hadoop/hive/ql/metadata/HiveMaterializedViewsRegistry.java
>  154ea6844175387e11802e6167343dac7c306112 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/SortedDynPartitionTimeGranularityOptimizer.java
>  4a248d6fe74234c532d32f11c7926971bb44e8a2 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/HiveCalciteUtil.java 
> f70aa7502793f691ab554e2b378f915b57ec52a1 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/HiveConfPlannerContext.java
>  756b671a83f1be30b993c3b76913e5d4317ed2d5 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/HiveRelFactories.java 
> 823b099efd477a01b4facf670efc093732a8ad88 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/HiveType.java 
> 31a628bba2ca5c83cceee7d77208acafaed45714 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/HiveTypeSystemImpl.java
>  dad783d91dfa90fceefc2bd66e538bed0278971e 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/functions/HiveSqlCountAggFunction.java
>  75b7ad2c8d4dd9da7357e6c811326bb29fcc02d7 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/functions/HiveSqlSumEmptyIsZeroAggFunction.java
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/reloperators/HiveAggregate.java
>  fa51b4e0b24b6fb1fde5e1a742dac5efa91c2246 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/reloperators/HiveGroupingID.java
>  2fb950867bccbdac571a0630b8ab4d8fc395e326 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveAggregateJoinTransposeRule.java
>  21b46cfca7661171e78b7e55cc8d14cf3d3a4573 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveAggregateProjectMergeRule.java
>  7c3b4b06ce5ecd8d53d73e43dc45e5726ba37a10 
>   
> 

Review Request 62995: HIVE-17806 Create directory for metrics file if it doesn't exist

[Alexander Kolbasov]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62995/
---

Review request for hive, Aihua Xu, Andrew Sherman, Janaki Lahorani, Sergio 
Pena, Sahil Takiar, and Vihang Karajgaonkar.


Bugs: HIVE-17806
https://issues.apache.org/jira/browse/HIVE-17806


Repository: hive-git


Description
---

HIVE-17806 Create directory for metrics file if it doesn't exist


Diffs
-

  
common/src/java/org/apache/hadoop/hive/common/metrics/metrics2/JsonFileMetricsReporter.java
 96243cb74a154b9a639ffb080256c4b43bd35a4b 
  
common/src/test/org/apache/hadoop/hive/common/metrics/metrics2/TestCodahaleMetrics.java
 254af7d4310578e3883c0dffa64bed0f823696ea 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/metrics/JsonReporter.java
 f71bb25463b56bc741f989454664397996b6a5cf 


Diff: https://reviews.apache.org/r/62995/diff/1/


Testing
---


Thanks,

Alexander Kolbasov

[jira] [Created] (HIVE-17810) Creating a table through HCatClient without specifying columns throws a NullPointerException on the server

[Stephen Patel (JIRA)]

Stephen Patel created HIVE-17810:


 Summary: Creating a table through HCatClient without specifying 
columns throws a NullPointerException on the server
 Key: HIVE-17810
 URL: https://issues.apache.org/jira/browse/HIVE-17810
 Project: Hive
  Issue Type: Bug
  Components: HCatalog
Reporter: Stephen Patel
Priority: Minor


I've attached a simple test case using the AvroSerde (which generates it's own 
columns) that, when run will throw this error:
{noformat}
2017-10-13T15:49:17,697 ERROR [pool-6-thread-2] metastore.RetryingHMSHandler: 
MetaException(message:java.lang.NullPointerException)
at 
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.newMetaException(HiveMetaStore.java:6560)
at 
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_with_environment_context(HiveMetaStore.java:1635)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:148)
at 
org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:107)
at com.sun.proxy.$Proxy30.create_table_with_environment_context(Unknown 
Source)
at 
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$create_table_with_environment_context.getResult(ThriftHiveMetastore.java:11710)
at 
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$create_table_with_environment_context.getResult(ThriftHiveMetastore.java:11694)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
at 
org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:110)
at 
org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:106)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1807)
at 
org.apache.hadoop.hive.metastore.TUGIBasedProcessor.process(TUGIBasedProcessor.java:118)
at 
org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.NullPointerException
at 
org.apache.hadoop.hive.metastore.MetaStoreUtils.validateTblColumns(MetaStoreUtils.java:621)
at 
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_core(HiveMetaStore.java:1433)
at 
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_core(HiveMetaStore.java:1420)
at 
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_with_environment_context(HiveMetaStore.java:1621)
... 20 more
{noformat}

By default the StorageDescriptor in the HCatTable class has a null column list. 
 When calling hCatTable.cols(emptyList), the hCatTable will determine that the 
list is equal to it's current column list and won't set the empty list on the 
StorageDescriptor, thus leading to the NullPointerException.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (HIVE-17809) Implement per pool trigger validation

[Prasanth Jayachandran (JIRA)]

Prasanth Jayachandran created HIVE-17809:


 Summary: Implement per pool trigger validation
 Key: HIVE-17809
 URL: https://issues.apache.org/jira/browse/HIVE-17809
 Project: Hive
  Issue Type: Sub-task
Affects Versions: 3.0.0
Reporter: Prasanth Jayachandran
Assignee: Prasanth Jayachandran


HIVE-17508 trigger validation is applied for all pools at once. This is follow 
up to implement trigger validation at per pool level. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (HIVE-17808) Change System.currentTimeMillis to System.nanoTime for elapsed time

[Prasanth Jayachandran (JIRA)]

Prasanth Jayachandran created HIVE-17808:


 Summary: Change System.currentTimeMillis to System.nanoTime for 
elapsed time
 Key: HIVE-17808
 URL: https://issues.apache.org/jira/browse/HIVE-17808
 Project: Hive
  Issue Type: Bug
Affects Versions: 3.0.0
Reporter: Prasanth Jayachandran


There are many places in QueryInfo and TezJobMonitor that uses 
System.currentTimeMillis() for finding elapsed time. Since currentTimeMillis 
depends on system clock this can cause issues in ntpd environments. Replace 
System.currentTimeMillis() with System.nanoTime() everywhere. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (HIVE-17807) Execute maven commands in batch mode for ptests

[Ashutosh Chauhan (JIRA)]

Ashutosh Chauhan created HIVE-17807:
---

 Summary: Execute maven commands in batch mode for ptests
 Key: HIVE-17807
 URL: https://issues.apache.org/jira/browse/HIVE-17807
 Project: Hive
  Issue Type: Bug
Reporter: Vijay Kumar
Assignee: Vijay Kumar


No need to run in interactive mode in CI environment.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (HIVE-17806) Create directory for metrics file if it doesn't exist

[Alexander Kolbasov (JIRA)]

Alexander Kolbasov created HIVE-17806:
-

 Summary: Create directory for metrics file if it doesn't exist
 Key: HIVE-17806
 URL: https://issues.apache.org/jira/browse/HIVE-17806
 Project: Hive
  Issue Type: Bug
  Components: Hive
Affects Versions: 3.0.0
Reporter: Alexander Kolbasov
Assignee: Alexander Kolbasov


HIVE-17563 changed metrics code to use local file system operations instead of 
Hadoop local file system operations. There is an unintended side effect - 
hadoop file systems create the directory if it doesn't exist and java nio 
interfaces don't. The purpose of this fix is to revert the behavior to the 
original one to avoid surprises.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (HIVE-17805) SchemaTool validate locations should not return exit 1

[Vihang Karajgaonkar (JIRA)]

Vihang Karajgaonkar created HIVE-17805:
--

 Summary: SchemaTool validate locations should not return exit 1
 Key: HIVE-17805
 URL: https://issues.apache.org/jira/browse/HIVE-17805
 Project: Hive
  Issue Type: Improvement
Reporter: Vihang Karajgaonkar
Assignee: Vihang Karajgaonkar
Priority: Minor


{{HiveSchemaTool}} can be used by other applications to validate the metastore 
schema. One of the validation check looks at the location URLs of tables/DBs 
and returns {{false}} which causes HiveSchemaTool to exit 1 to the calling 
script. Invalid locations although are a problem in some instances, cannot be 
termed as catastrophic errors in the schema which should cause Hive service 
failure/unusable. Ideally we should introduce warning levels and error levels 
in schemaTool validations so the caller can take appropriate action.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (HIVE-17804) Vectorization: Bug erroneously causes match for 1st row in batch (SelectStringColLikeStringScalar)

[Matt McCline (JIRA)]

Matt McCline created HIVE-17804:
---

 Summary: Vectorization: Bug erroneously causes match for 1st row 
in batch (SelectStringColLikeStringScalar)
 Key: HIVE-17804
 URL: https://issues.apache.org/jira/browse/HIVE-17804
 Project: Hive
  Issue Type: Bug
  Components: Hive
Reporter: Matt McCline
Assignee: Matt McCline
Priority: Critical
 Fix For: 3.0.0


Code setting output value to LongColumnVector.NULL_VALUE for null candidate 
sets the 0th entry instead of the i'th.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (HIVE-17803) With Pig multi-query, 2 HCatStorers writing to the same table will trample each other's outputs

[Mithun Radhakrishnan (JIRA)]

Mithun Radhakrishnan created HIVE-17803:
---

 Summary: With Pig multi-query, 2 HCatStorers writing to the same 
table will trample each other's outputs
 Key: HIVE-17803
 URL: https://issues.apache.org/jira/browse/HIVE-17803
 Project: Hive
  Issue Type: Bug
  Components: HCatalog
Affects Versions: 2.2.0, 3.0.0
Reporter: Mithun Radhakrishnan
Assignee: Chris Drome


When Pig scripts use multi-query and {{HCatStorer}} with dynamic-partitioning, 
and use more than one {{HCatStorer}} instance to write to the same table, they 
might trample on each other's outputs. The failure looks as follows:

{noformat}
Caused by: org.apache.hive.hcatalog.common.HCatException : 2006 : Error adding 
partition to metastore. Cause : 
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException):
 No lease on /projects/foo/bar/activity_date=2016022306/_placeholder (inode 
2878224200): File does not exist. [Lease.  Holder: 
DFSClient_NONMAPREDUCE_-1281544466_4952, pendingcreates: 1]
at 
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:3429)
at 
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.completeFileInternal(FSNamesystem.java:3517)
at 
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.completeFile(FSNamesystem.java:3484)
at 
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.complete(NameNodeRpcServer.java:791)
at 
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.complete(ClientNamenodeProtocolServerSideTranslatorPB.java:537)
at 
org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at 
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:608)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:969)
at org.apache.hadoop.ipc.Server.call(Server.java:2267)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:648)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:615)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1679)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2217)

at 
org.apache.hive.hcatalog.mapreduce.FileOutputCommitterContainer.registerPartitions(FileOutputCommitterContainer.java:1022)
at 
org.apache.hive.hcatalog.mapreduce.FileOutputCommitterContainer.commitJob(FileOutputCommitterContainer.java:269)
... 20 more
Caused by: 
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException):
 No lease on /projects/foo/bar/activity_date=2016022306/_placeholder (inode 
2878224200): File does not exist. [Lease.  Holder: 
DFSClient_NONMAPREDUCE_-1281544466_4952, pendingcreates: 1]
at 
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:3429)
at 
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.completeFileInternal(FSNamesystem.java:3517)
at 
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.completeFile(FSNamesystem.java:3484)
at 
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.complete(NameNodeRpcServer.java:791)
at 
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.complete(ClientNamenodeProtocolServerSideTranslatorPB.java:537)
at 
org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at 
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:608)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:969)
at org.apache.hadoop.ipc.Server.call(Server.java:2267)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:648)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:615)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1679)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2217)

at org.apache.hadoop.ipc.Client.call(Client.java:1457)
at org.apache.hadoop.ipc.Client.call(Client.java:1394)
at 
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:230)
at com.sun.proxy.$Proxy11.complete(Unknown Source)
at 
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.complete(ClientNamenodeProtocolTranslatorPB.java:462)
at 

Re: Review Request 62975: HIVE-17798: When replacing the src table names in BeeLine testing, the table names shouldn't be changed to lower case

[Peter Vary via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62975/#review188023
---


Ship it!




Ship It!

- Peter Vary


On Oct. 13, 2017, 2:21 p.m., Marta Kuczora wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/62975/
> ---
> 
> (Updated Oct. 13, 2017, 2:21 p.m.)
> 
> 
> Review request for hive and Peter Vary.
> 
> 
> Bugs: HIVE-17798
> https://issues.apache.org/jira/browse/HIVE-17798
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Change the QFile.replaceTableNames method to keep the upper case letters 
> unchanged.
> 
> 
> Diffs
> -
> 
>   itests/util/src/main/java/org/apache/hive/beeline/QFile.java 21be8b0 
> 
> 
> Diff: https://reviews.apache.org/r/62975/diff/1/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Marta Kuczora
> 
>

Re: Review Request 62952: HIVE-17792

[Deepak Jaiswal]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62952/
---

(Updated Oct. 13, 2017, 7:20 p.m.)


Review request for hive, Gopal V and Jason Dere.


Changes
---

Fixed the failing tests.


Bugs: HIVE-17792
https://issues.apache.org/jira/browse/HIVE-17792


Repository: hive-git


Description
---

Enable Bucket Map Join when there are extra keys other than bucketed columns.
Added couple of test cases.


Diffs (updated)
-

  ql/src/java/org/apache/hadoop/hive/ql/optimizer/ConvertJoinMapJoin.java 
e24760b90c 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/metainfo/annotation/OpTraitsRulesProcFactory.java
 875ee9d842 
  ql/src/test/queries/clientpositive/bucket_map_join_tez1.q cac1d6a3d5 
  ql/src/test/results/clientpositive/llap/bucket_map_join_tez1.q.out 36cb4ac6c7 


Diff: https://reviews.apache.org/r/62952/diff/2/

Changes: https://reviews.apache.org/r/62952/diff/1-2/


Testing
---


Thanks,

Deepak Jaiswal

[jira] [Created] (HIVE-17802) Remove unnecessary calls to FileSystem.setOwner() from FileOutputCommitterContainer

[Mithun Radhakrishnan (JIRA)]

Mithun Radhakrishnan created HIVE-17802:
---

 Summary: Remove unnecessary calls to FileSystem.setOwner() from 
FileOutputCommitterContainer
 Key: HIVE-17802
 URL: https://issues.apache.org/jira/browse/HIVE-17802
 Project: Hive
  Issue Type: Bug
  Components: HCatalog
Affects Versions: 2.2.0, 3.0.0
Reporter: Mithun Radhakrishnan
Assignee: Chris Drome


For large Pig/HCat queries that produce a large number of 
partitions/directories/files, we have seen cases where the HDFS NameNode 
groaned under the weight of {{FileSystem.setOwner()}} calls, originating from 
the commit-step. This was the result of the following code in 
FileOutputCommitterContainer:
{code:java}
private void applyGroupAndPerms(FileSystem fs, Path dir, FsPermission 
permission,
  List acls, String group, boolean recursive)
throws IOException {
...
if (recursive) {
  for (FileStatus fileStatus : fs.listStatus(dir)) {
if (fileStatus.isDir()) {
  applyGroupAndPerms(fs, fileStatus.getPath(), permission, acls, group, 
true);
} else {
  fs.setPermission(fileStatus.getPath(), permission);
  chown(fs, fileStatus.getPath(), group);
}
  }
}
  }

  private void chown(FileSystem fs, Path file, String group) throws IOException 
{
try {
  fs.setOwner(file, null, group);
} catch (AccessControlException ignore) {
  // Some users have wrong table group, ignore it.
  LOG.warn("Failed to change group of partition directories/files: " + 
file, ignore);
}
  }
{code}

One call per file/directory is far too many. We have a patch that reduces the 
namenode pressure.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: Review Request 62693: HIVE-17635: Add unit tests to CompactionTxnHandler and use PreparedStatements for queries

[Andrew Sherman via Review Board]

> On Oct. 12, 2017, 5:34 p.m., Sahil Takiar wrote:
> > metastore/src/java/org/apache/hadoop/hive/metastore/txn/CompactionTxnHandler.java
> > Line 387 (original), 411 (patched)
> > 
> >
> > What is this for?

The 'questions' list is used to create the list containing 'IN list values'. If 
not using PreparedStatements these would be actual values. In the 
PreparedStatement case they are a list of ?. This is arguably ugly but it 
allows us to use common code for PreparedStatements and (unprepared) 
Statements. See below for a more complete explanantion.


> On Oct. 12, 2017, 5:34 p.m., Sahil Takiar wrote:
> > metastore/src/java/org/apache/hadoop/hive/metastore/txn/CompactionTxnHandler.java
> > Line 413 (original), 444 (patched)
> > 
> >
> > Is this necessary?

No, its a mistake, thanks


> On Oct. 12, 2017, 5:34 p.m., Sahil Takiar wrote:
> > metastore/src/java/org/apache/hadoop/hive/metastore/txn/CompactionTxnHandler.java
> > Line 432 (original), 475 (patched)
> > 
> >
> > If we are changing this, should we just use try-with-resources.

I agree try-with-resources is great, I wanted to mimimize my changes


> On Oct. 12, 2017, 5:34 p.m., Sahil Takiar wrote:
> > metastore/src/java/org/apache/hadoop/hive/metastore/txn/TxnUtils.java
> > Lines 158 (patched)
> > 
> >
> > why is a new return value necessary?

We are looking at code that generates IN clauses: 
  select count(*) from TXNS where (TXN_ID in (1,2,3,4,5)) and TXN_STATE = 'o'
There are limits on how many values you can have in an IN clause (like maybe 
1000), and the code knows something about that.
If you ask it to generate code for a lot of values then it returns multiple 
queries:
  select count(*) from TXNS where (TXN_ID in (1,2,3,4,5)) and TXN_STATE = 'o'
  select count(*) from TXNS where (TXN_ID in (1001,1002,1003) and TXN_STATE = 
'o'
My change involves using the same logic to build PreparedStatements. These look 
like:
  select count(*) from TXNS where (TXN_ID in (?,?,?,?,?)) and TXN_STATE = 'o'
  select count(*) from TXNS where (TXN_ID in (?,?,?)) and TXN_STATE = 'o'
The difference is that with PreparedStatements the code must also subsequently 
call 
 pStmt.setLong(paramNum, value)
The right number of times for each query. So the new method 
buildQueryWithINClauseStrings,  
in addition to building the list of queries also returns a corresponding list 
of the number of ? i
n the the generated in clause.


- Andrew


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62693/#review187818
---


On Sept. 29, 2017, 4:51 p.m., Andrew Sherman wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/62693/
> ---
> 
> (Updated Sept. 29, 2017, 4:51 p.m.)
> 
> 
> Review request for hive.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Add a unit test which exercises CompactionTxnHandler.markFailed() and change 
> it to use PreparedStament.
> Add test for checkFailedCompactions() and change it to use PreparedStatement
> Add a unit test which exercises purgeCompactionHistory().
> Add buildQueryWithINClauseStrings() which is suitable for building in clauses 
> for PreparedStatement
> Add test code to TestTxnUtils to tickle code in 
> TxnUtils.buildQueryWithINClauseStrings() so that it produces multiple queries.
> Change markCleaned() to use PreparedStatement
> 
> 
> Diffs
> -
> 
>   beeline/src/java/org/apache/hive/beeline/HiveSchemaTool.java 
> 84963af10ec13979a7b3976be434efbc21cf2382 
>   
> metastore/src/java/org/apache/hadoop/hive/metastore/txn/CompactionTxnHandler.java
>  60839faa352cbf959041a455e9e780dfca0afdc3 
>   metastore/src/java/org/apache/hadoop/hive/metastore/txn/TxnUtils.java 
> 30b155f3b3311fed6cd79e46a5b2abcee9927d91 
>   metastore/src/test/org/apache/hadoop/hive/metastore/txn/TestTxnUtils.java 
> 1497c00e5dc77c02e53767b014a23e5fd8cb5b29 
>   
> ql/src/test/org/apache/hadoop/hive/metastore/txn/TestCompactionTxnHandler.java
>  f8ae86bea3fe78374c0e0487d66c661f4f0d78ff 
> 
> 
> Diff: https://reviews.apache.org/r/62693/diff/1/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Andrew Sherman
> 
>

Review Request 62981: HIVE-17672: Upgrade Calcite version to 1.14

[Jesús Camacho Rodríguez]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62981/
---

Review request for hive and Ashutosh Chauhan.


Bugs: HIVE-17672
https://issues.apache.org/jira/browse/HIVE-17672


Repository: hive-git


Description
---

HIVE-17672: Upgrade Calcite version to 1.14


Diffs
-

  druid-handler/src/java/org/apache/hadoop/hive/druid/io/DruidOutputFormat.java 
da7642c61a3c1b82e0d505d110bef63863e7be13 
  druid-handler/src/java/org/apache/hadoop/hive/druid/serde/DruidSerDe.java 
a43f62ed8203b74fde64e676e75385b4cd52570b 
  
druid-handler/src/java/org/apache/hadoop/hive/druid/serde/DruidSerDeUtils.java 
86c325ba1fc728e452dea8440ab1ca0face45cc3 
  druid-handler/src/test/org/apache/hadoop/hive/druid/serde/TestDruidSerDe.java 
655a96ad7b5a24de91707e35b93fb3eb99a2563a 
  pom.xml ca088da3bfda76fd3da19d96fb6f7705a8adfe57 
  ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java 
1d1d2a3e0bee177cd4b145c731b5ea3c75483c84 
  ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java 
68c94820c9368f504401c801f9b599a14f91ddf6 
  
ql/src/java/org/apache/hadoop/hive/ql/metadata/HiveMaterializedViewsRegistry.java
 154ea6844175387e11802e6167343dac7c306112 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/SortedDynPartitionTimeGranularityOptimizer.java
 4a248d6fe74234c532d32f11c7926971bb44e8a2 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/HiveCalciteUtil.java 
f70aa7502793f691ab554e2b378f915b57ec52a1 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/HiveConfPlannerContext.java
 756b671a83f1be30b993c3b76913e5d4317ed2d5 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/HiveRelFactories.java 
823b099efd477a01b4facf670efc093732a8ad88 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/HiveType.java 
31a628bba2ca5c83cceee7d77208acafaed45714 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/HiveTypeSystemImpl.java 
dad783d91dfa90fceefc2bd66e538bed0278971e 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/functions/HiveSqlCountAggFunction.java
 75b7ad2c8d4dd9da7357e6c811326bb29fcc02d7 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/functions/HiveSqlSumEmptyIsZeroAggFunction.java
 PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/reloperators/HiveAggregate.java
 fa51b4e0b24b6fb1fde5e1a742dac5efa91c2246 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/reloperators/HiveGroupingID.java
 2fb950867bccbdac571a0630b8ab4d8fc395e326 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveAggregateJoinTransposeRule.java
 21b46cfca7661171e78b7e55cc8d14cf3d3a4573 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveAggregateProjectMergeRule.java
 7c3b4b06ce5ecd8d53d73e43dc45e5726ba37a10 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveExceptRewriteRule.java
 b63ea022c9dd4c0d46ba5fc98761d4340c0d6291 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveExpandDistinctAggregatesRule.java
 81bb730fba9baeb3a86454b30210afe9874632f7 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveIntersectRewriteRule.java
 5b0a7d7293653ca87a0f4e4dd697ae20242b53aa 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveRelColumnsAlignment.java
 f35bf2f2a9fd4b7123007085af45853f9412586b 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveRelDecorrelator.java
 b922faac5d522e44c1dc6f314877f22769d058be 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/ASTBuilder.java
 eab8f6f110610374f1ac5ab103302b1ee5ab15c5 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/ASTConverter.java
 d6b978fae3588f83b250ec3956d4f48e64733261 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/ExprNodeConverter.java
 3dcceaba694359e3206594cb409ea441323505cf 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/HiveGBOpConvUtil.java
 0f6c5b5355490f3a9b67a29bd3db9ef77a129f13 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/HiveOpConverter.java
 471675b53bbb151eca5c1007a7f3573b5e3fe9c1 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/RexNodeConverter.java
 f8a959d13cf74946ed5e0df21f837cfdd095f993 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/SqlFunctionConverter.java
 31a088ba5fed51f195517a413d146d944fb777bd 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/TypeConverter.java
 34886f367aba7a9665b44e45334bf51fc05f91d4 
  ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java 
65552693f776da5625526a7acb5a5d32b6a418e9 
  ql/src/java/org/apache/hadoop/hive/ql/parse/HiveParser.g 
020a3008db5febd8070505ccc9c8b63dbd43ad6a 
  ql/src/java/org/apache/hadoop/hive/ql/parse/IdentifiersParser.g 
0591fbf586d130857262c31c978057f169016d25 
  

[jira] [Created] (HIVE-17801) OpenCSVserde should store schema in metastore

[Ashutosh Chauhan (JIRA)]

Ashutosh Chauhan created HIVE-17801:
---

 Summary: OpenCSVserde should store schema in metastore
 Key: HIVE-17801
 URL: https://issues.apache.org/jira/browse/HIVE-17801
 Project: Hive
  Issue Type: Bug
  Components: Metastore, Serializers/Deserializers
Reporter: Ashutosh Chauhan


Just need to add opencsv serde in config.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: Review Request 62935: HIVE-17787: Apply more filters on the BeeLine test output files

[Marta Kuczora via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62935/
---

(Updated Oct. 13, 2017, 2:54 p.m.)


Review request for hive and Peter Vary.


Bugs: HIVE-17787
https://issues.apache.org/jira/browse/HIVE-17787


Repository: hive-git


Description
---

This is a follow-up patch for HIVE-17569.
When running the q tests with BeeLine, some known differences came up which 
should be filtered out if the "test.beeline.compare.portable" parameter is set 
to true.


Diffs (updated)
-

  itests/util/src/main/java/org/apache/hive/beeline/QFile.java 21be8b0 


Diff: https://reviews.apache.org/r/62935/diff/2/

Changes: https://reviews.apache.org/r/62935/diff/1-2/


Testing
---


Thanks,

Marta Kuczora

[jira] [Created] (HIVE-17800) input_part6.q wants to test partition pruning, but tests expression evaluation

[Peter Vary (JIRA)]

Peter Vary created HIVE-17800:
-

 Summary: input_part6.q wants to test partition pruning, but tests 
expression evaluation
 Key: HIVE-17800
 URL: https://issues.apache.org/jira/browse/HIVE-17800
 Project: Hive
  Issue Type: Bug
Reporter: Peter Vary
Assignee: Peter Vary


input_part6.q looks like this:
{code}
EXPLAIN
SELECT x.* FROM SRCPART x WHERE x.ds = 2008-04-08 LIMIT 10;
{code}

The intended test most probably is this:
{code}
EXPLAIN
SELECT x.* FROM SRCPART x WHERE x.ds = "2008-04-08" LIMIT 10;
{code}

Currently we evaluete 2008-4-8 to 1996:
{code}
predicate: (UDFToDouble(ds) = 1996.0) (type: boolean)
{code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (HIVE-17799) Add Ellipsis For Truncated Query In Hive Lock

[BELUGA BEHR (JIRA)]

BELUGA BEHR created HIVE-17799:
--

 Summary: Add Ellipsis For Truncated Query In Hive Lock
 Key: HIVE-17799
 URL: https://issues.apache.org/jira/browse/HIVE-17799
 Project: Hive
  Issue Type: Improvement
  Components: Hive
Affects Versions: 3.0.0
Reporter: BELUGA BEHR
Assignee: BELUGA BEHR
Priority: Trivial


[HIVE-16334] introduced truncation for storing queries in ZK lock nodes.  This 
Jira is to add ellipsis into the query to let the operator know that truncation 
has occurred and therefore they will not find the specific query in their logs, 
only a prefix match will work.

{code:sql}
-- Truncation of query may be confusing to operator

-- Without truncation
SELECT * FROM TABLE WHERE COL=1

-- With truncation (will not find this query in workload)
SELECT * FROM TABLE
{code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Review Request 62975: HIVE-17798: When replacing the src table names in BeeLine testing, the table names shouldn't be changed to lower case

[Marta Kuczora via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62975/
---

Review request for hive and Peter Vary.


Bugs: HIVE-17798
https://issues.apache.org/jira/browse/HIVE-17798


Repository: hive-git


Description
---

Change the QFile.replaceTableNames method to keep the upper case letters 
unchanged.


Diffs
-

  itests/util/src/main/java/org/apache/hive/beeline/QFile.java 21be8b0 


Diff: https://reviews.apache.org/r/62975/diff/1/


Testing
---


Thanks,

Marta Kuczora

[jira] [Created] (HIVE-17798) When replacing the src table names in BeeLine testing, the table names shouldn't be changed to lower case

[Marta Kuczora (JIRA)]

Marta Kuczora created HIVE-17798:


 Summary: When replacing the src table names in BeeLine testing, 
the table names shouldn't be changed to lower case
 Key: HIVE-17798
 URL: https://issues.apache.org/jira/browse/HIVE-17798
 Project: Hive
  Issue Type: Bug
  Components: Testing Infrastructure
Affects Versions: 3.0.0
Reporter: Marta Kuczora
Assignee: Marta Kuczora
Priority: Minor


When running the q tests with BeeLine, the name of the src tables are changed 
in all queries to have the database name as prefix, like src -> default.src, 
srcpart -> default.srcpart
This renaming mechanism changes the table names to lower case. For example the 
query "SELECT * FROM SRC" will be "SELECT * FROM src" after the rewite. This 
will cause failure during the comparison of the out files.
Change the QFile.replaceTableNames method to keep the upper case letters 
unchanged.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (HIVE-17797) History of API changes for Hive Common

[Andrey Ponomarenko (JIRA)]

Andrey Ponomarenko created HIVE-17797:
-

 Summary: History of API changes for Hive Common
 Key: HIVE-17797
 URL: https://issues.apache.org/jira/browse/HIVE-17797
 Project: Hive
  Issue Type: Improvement
Reporter: Andrey Ponomarenko


Hi,

I'd like to share the report on API changes and backward binary compatibility 
for the Hive Common library: 
https://abi-laboratory.pro/java/tracker/timeline/hive-common/

The report is generated by the https://github.com/lvc/japi-tracker tool for 
jars found at http://central.maven.org/maven2/org/apache/hive/hive-common/ 
according to https://wiki.eclipse.org/Evolving_Java-based_APIs_2.

Feel free to request other Hive modules to be included to the tracker if you 
are interested.

Also please let me know if the tool should not check some parts of the API (it 
checks all public API methods and classes by default).

Thank you.

!hive-common-2.png|API symbols timeline!
!hive-common-1.png|API changes review!



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)