date:20180413

Re: Review Request 66567: Migrate to Murmur hash for shuffle and bucketing

2018-04-13 Thread Jason Dere


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66567/#review201133
---




hbase-handler/src/test/results/positive/external_table_ppd.q.out
Lines 59 (patched)


Are there any tests for the old-style bucketing, to make sure that 
previously created bucketed tables still work properly?



hcatalog/streaming/src/java/org/apache/hive/hcatalog/streaming/mutate/worker/BucketIdResolverImpl.java
Lines 25 (patched)


Unnecessary change?



itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/TestAcidOnTez.java
Lines 850 (patched)


missing comment?



ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java
Line 1053 (original), 1051 (patched)


If this occurs every row, I wonder if it would be better to determine the 
bucketing version once during initializeOp() and create some object which knows 
which knows which bucketing hash code method to call here



ql/src/java/org/apache/hadoop/hive/ql/metadata/Table.java
Lines 469 (patched)


should we validate that this is a valid bucketing version that we support?



ql/src/java/org/apache/hadoop/hive/ql/optimizer/ConvertJoinMapJoin.java
Lines 639 (patched)


Do we also need to check the bucketing type in the case that op is not a 
TableScan? If op is a ReduceSink or Join, would that end up being 
bucketingVersion 2?



ql/src/java/org/apache/hadoop/hive/ql/optimizer/metainfo/annotation/AnnotateWithOpTraits.java
Lines 72 (patched)


Was this commented code for testing?



ql/src/java/org/apache/hadoop/hive/ql/optimizer/metainfo/annotation/OpTraitsRulesProcFactory.java
Lines 411 (patched)


It seems to me a lot of the logic will treat -1 as bucketing version 1, 
since there are a lot of (bucketingVersion == 2 ? doVersion2 : doVersion1) 
statements. Where in the code would SMB be disabled because of -1 
bucketingVersion?



ql/src/java/org/apache/hadoop/hive/ql/plan/TableDesc.java
Lines 187 (patched)


Maybe make some common utility to parse/validate bucketing version, that 
both places can use?



ql/src/java/org/apache/hadoop/hive/ql/plan/TableDesc.java
Lines 198 (patched)


Validate bucketing version number?



ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFHash.java
Lines 32 (patched)


Docs for this UDF will probably need to mention that this uses the old 
hashing/bucketing scheme which and that a new one has replaced it.



ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFMurmurHash.java
Lines 1 (patched)


Missing Apache header



serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ObjectInspectorUtils.java
Lines 813 (patched)


For these primitive types, might make sense to pre-allocate fixed size 
ByteBuffers of size 2/4/8 which can be used here rather than having to allocate 
new ones for every value.



serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ObjectInspectorUtils.java
Lines 858 (patched)


Old impl (based on DateWritable.hashCode()) did hashCode based on 
daysSinceEpoc value, will be faster than doing toString()



serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ObjectInspectorUtils.java
Lines 866 (patched)


Faster to do hashcode based on the underlying values (totalMonths) rather 
than toString



serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ObjectInspectorUtils.java
Lines 869 (patched)


Faster to do hashcode based on the underlying values (totalSeconds/nanos) 
rather than toString


- Jason Dere


On April 12, 2018, 6:24 p.m., Deepak Jaiswal wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66567/
> ---
> 
> (Updated April 12, 2018, 6:24 p.m.)
> 
> 
> Review request for hive, Eugene Koifman, Jason Dere, and Matt McCline.
> 
> 
> Bugs: HIVE-18910
> https://issues.apache.org/jira/browse/HIVE-18910
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Hive uses

Re: Review Request 66571: HIVE-19161: Add authorizations to information schema

2018-04-13 Thread Thejas Nair


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66571/#review201144
---




common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
Lines 4203 (patched)


needs rename to "hive." here as well



metastore/scripts/upgrade/hive/hive-schema-3.0.0.hive.sql
Lines 1085 (patched)


I think its easier to reason about the correctness of this one if you put 
another set of paranthesis around the group match condition-

OR ((array_contains(current_groups(), P.`PRINCIPAL_NAME`) OR 
P.`PRINCIPAL_NAME` = 'public') AND P.`PRINCIPAL_TYPE`='GROUP'))

Otherwise, it seems like it might get reduced to something like -

P.`PRINCIPAL_TYPE`='USER' OR (array) AND P.`PRINCIPAL_TYPE`='GROUP'



metastore/scripts/upgrade/hive/hive-schema-3.0.0.hive.sql
Lines 1120 (patched)


similar parenthesis would be useful here



metastore/scripts/upgrade/hive/hive-schema-3.0.0.hive.sql
Lines 1152 (patched)


similar parenthesis would be useful here



metastore/scripts/upgrade/hive/hive-schema-3.0.0.hive.sql
Lines 1309 (patched)


similar parenthesis would be useful here



metastore/scripts/upgrade/hive/hive-schema-3.0.0.hive.sql
Lines 1345 (patched)


similar parenthesis would be useful here



metastore/scripts/upgrade/hive/hive-schema-3.0.0.hive.sql
Lines 1382 (patched)


similar parenthesis would be useful here



service/src/java/org/apache/hive/service/server/HiveServer2.java
Lines 141 (patched)


typo, rename variable to -
zooKeeperClientForPrivilegeSynchronizer

or shorter -
zKClientForPrivSync



standalone-metastore/src/main/thrift/hive_metastore.thrift
Lines 1997 (patched)


Update the description ? 
// Revokes all privileges for the object and adds the newly granted 
privileges for it.


- Thejas Nair


On April 13, 2018, 10:30 p.m., Daniel Dai wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66571/
> ---
> 
> (Updated April 13, 2018, 10:30 p.m.)
> 
> 
> Review request for hive.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> See HIVE-19161
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 0627c35 
>   
> itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/listener/DummyRawStoreFailEvent.java
>  801de7a 
>   itests/hive-unit/pom.xml f473d25 
>   
> itests/hive-unit/src/test/java/org/apache/hive/service/server/TestInformationSchemaWithPrivilege.java
>  PRE-CREATION 
>   
> jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/dao/DatabaseAccessorFactory.java
>  7dc690f 
>   
> jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/dao/GenericJdbcDatabaseAccessor.java
>  178c97d 
>   
> jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/dao/JdbcRecordIterator.java
>  638e2b0 
>   
> jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/dao/MsSqlDatabaseAccessor.java
>  PRE-CREATION 
>   
> jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/dao/OracleDatabaseAccessor.java
>  PRE-CREATION 
>   
> jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/dao/PostgresDatabaseAccessor.java
>  PRE-CREATION 
>   metastore/scripts/upgrade/hive/hive-schema-3.0.0.hive.sql 339 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java d59bf1f 
>   
> ql/src/java/org/apache/hadoop/hive/ql/security/authorization/HiveAuthorizationProvider.java
>  60d9dc1 
>   
> ql/src/java/org/apache/hadoop/hive/ql/security/authorization/PrivilegeSynchonizer.java
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/security/authorization/plugin/HiveResourceACLsImpl.java
>  PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java 6003ced 
>   
> ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFCurrentGroups.java
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFRestrictInformationSchema.java
>  PRE-CREATION 
>   service/src/java/org/apache/hive/service/server/HiveServer2.java 6308c5c 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
>  450da4f 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java
>  ebbf465 
>   
>

Re: Review Request 66614: HIVE-19210: Create separate module for streaming ingest

2018-04-13 Thread j . prasanth . j


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66614/
---

(Updated April 13, 2018, 11:58 p.m.)


Review request for hive, Ashutosh Chauhan and EugeneBK EugeneBK.


Changes
---

Not moving mutate package from hcat.


Bugs: HIVE-19210
https://issues.apache.org/jira/browse/HIVE-19210


Repository: hive-git


Description
---

HIVE-19210: Create separate module for streaming ingest


Diffs (updated)
-

  
hcatalog/streaming/src/java/org/apache/hive/hcatalog/streaming/package-info.java
 PRE-CREATION 
  pom.xml 5802bd3 
  streaming/pom.xml PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/AbstractRecordWriter.java 
PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/ConnectionError.java 
PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/DelimitedInputWriter.java 
PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/HeartBeatFailure.java 
PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/HiveEndPoint.java PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/ImpersonationFailed.java 
PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/InvalidColumn.java PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/InvalidPartition.java 
PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/InvalidTable.java PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/InvalidTrasactionState.java 
PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/PartitionCreationFailed.java 
PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/QueryFailedException.java 
PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/RecordWriter.java PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/SerializationError.java 
PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/StreamingConnection.java 
PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/StreamingException.java 
PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/StreamingIOFailure.java 
PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/StrictJsonWriter.java 
PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/StrictRegexWriter.java 
PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/TransactionBatch.java 
PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/TransactionBatchUnAvailable.java 
PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/TransactionError.java 
PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/package.html PRE-CREATION 
  streaming/src/test/org/apache/hive/streaming/TestDelimitedInputWriter.java 
PRE-CREATION 
  streaming/src/test/org/apache/hive/streaming/TestStreaming.java PRE-CREATION 


Diff: https://reviews.apache.org/r/66614/diff/2/

Changes: https://reviews.apache.org/r/66614/diff/1-2/


Testing
---


Thanks,

Prasanth_J

[jira] [Created] (HIVE-19215) JavaUtils.AnyIdDirFilter ignores base_n directories

2018-04-13 Thread Eugene Koifman (JIRA)

Eugene Koifman created HIVE-19215:
-

 Summary: JavaUtils.AnyIdDirFilter ignores base_n directories
 Key: HIVE-19215
 URL: https://issues.apache.org/jira/browse/HIVE-19215
 Project: Hive
  Issue Type: Bug
  Components: Transactions
Affects Versions: 3.0.0
Reporter: Eugene Koifman


cc [~sershe], [~steveyeom2017]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 66614: HIVE-19210: Create separate module for streaming ingest

2018-04-13 Thread j . prasanth . j


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66614/
---

(Updated April 13, 2018, 10:49 p.m.)


Review request for hive, Ashutosh Chauhan and EugeneBK EugeneBK.


Bugs: HIVE-19210
https://issues.apache.org/jira/browse/HIVE-19210


Repository: hive-git


Description
---

HIVE-19210: Create separate module for streaming ingest


Diffs
-

  
hcatalog/streaming/src/java/org/apache/hive/hcatalog/streaming/package-info.java
 PRE-CREATION 
  pom.xml 5802bd3a354ce6a78fd786e43847c7475a3f965e 
  streaming/pom.xml PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/AbstractRecordWriter.java 
PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/ConnectionError.java 
PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/DelimitedInputWriter.java 
PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/HeartBeatFailure.java 
PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/HiveEndPoint.java PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/ImpersonationFailed.java 
PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/InvalidColumn.java PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/InvalidPartition.java 
PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/InvalidTable.java PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/InvalidTrasactionState.java 
PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/PartitionCreationFailed.java 
PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/QueryFailedException.java 
PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/RecordWriter.java PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/SerializationError.java 
PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/StreamingConnection.java 
PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/StreamingException.java 
PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/StreamingIOFailure.java 
PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/StreamingUtil.java PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/StrictJsonWriter.java 
PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/StrictRegexWriter.java 
PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/TransactionBatch.java 
PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/TransactionBatchUnAvailable.java 
PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/TransactionError.java 
PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/mutate/HiveConfFactory.java 
PRE-CREATION 
  
streaming/src/java/org/apache/hive/streaming/mutate/UgiMetaStoreClientFactory.java
 PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/mutate/client/AcidTable.java 
PRE-CREATION 
  
streaming/src/java/org/apache/hive/streaming/mutate/client/AcidTableSerializer.java
 PRE-CREATION 
  
streaming/src/java/org/apache/hive/streaming/mutate/client/ClientException.java 
PRE-CREATION 
  
streaming/src/java/org/apache/hive/streaming/mutate/client/ConnectionException.java
 PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/mutate/client/MutatorClient.java 
PRE-CREATION 
  
streaming/src/java/org/apache/hive/streaming/mutate/client/MutatorClientBuilder.java
 PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/mutate/client/TableType.java 
PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/mutate/client/Transaction.java 
PRE-CREATION 
  
streaming/src/java/org/apache/hive/streaming/mutate/client/TransactionException.java
 PRE-CREATION 
  
streaming/src/java/org/apache/hive/streaming/mutate/client/lock/HeartbeatFactory.java
 PRE-CREATION 
  
streaming/src/java/org/apache/hive/streaming/mutate/client/lock/HeartbeatTimerTask.java
 PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/mutate/client/lock/Lock.java 
PRE-CREATION 
  
streaming/src/java/org/apache/hive/streaming/mutate/client/lock/LockException.java
 PRE-CREATION 
  
streaming/src/java/org/apache/hive/streaming/mutate/client/lock/LockFailureListener.java
 PRE-CREATION 
  
streaming/src/java/org/apache/hive/streaming/mutate/doc-files/system-overview.dot
 PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/mutate/package.html PRE-CREATION 
  
streaming/src/java/org/apache/hive/streaming/mutate/worker/BucketIdException.java
 PRE-CREATION 
  
streaming/src/java/org/apache/hive/streaming/mutate/worker/BucketIdResolver.java
 PRE-CREATION 
  
streaming/src/java/org/apache/hive/streaming/mutate/worker/BucketIdResolverImpl.java
 PRE-CREATION 
  
streaming/src/java/org/apache/hive/streaming/mutate/worker/GroupRevisitedException.java
 PRE-CREATION 
  
streaming/src/java/org/apache/hive/streaming/mutate/worker/GroupingValidator.java
 PRE-CREATION

Review Request 66614: HIVE-19210: Create separate module for streaming ingest

2018-04-13 Thread j . prasanth . j


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66614/
---

Review request for hive, Ashutosh Chauhan and EugeneBK EugeneBK.


Repository: hive-git


Description
---

HIVE-19210: Create separate module for streaming ingest


Diffs
-

  
hcatalog/streaming/src/java/org/apache/hive/hcatalog/streaming/package-info.java
 PRE-CREATION 
  pom.xml 5802bd3a354ce6a78fd786e43847c7475a3f965e 
  streaming/pom.xml PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/AbstractRecordWriter.java 
PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/ConnectionError.java 
PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/DelimitedInputWriter.java 
PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/HeartBeatFailure.java 
PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/HiveEndPoint.java PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/ImpersonationFailed.java 
PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/InvalidColumn.java PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/InvalidPartition.java 
PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/InvalidTable.java PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/InvalidTrasactionState.java 
PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/PartitionCreationFailed.java 
PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/QueryFailedException.java 
PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/RecordWriter.java PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/SerializationError.java 
PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/StreamingConnection.java 
PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/StreamingException.java 
PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/StreamingIOFailure.java 
PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/StreamingUtil.java PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/StrictJsonWriter.java 
PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/StrictRegexWriter.java 
PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/TransactionBatch.java 
PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/TransactionBatchUnAvailable.java 
PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/TransactionError.java 
PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/mutate/HiveConfFactory.java 
PRE-CREATION 
  
streaming/src/java/org/apache/hive/streaming/mutate/UgiMetaStoreClientFactory.java
 PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/mutate/client/AcidTable.java 
PRE-CREATION 
  
streaming/src/java/org/apache/hive/streaming/mutate/client/AcidTableSerializer.java
 PRE-CREATION 
  
streaming/src/java/org/apache/hive/streaming/mutate/client/ClientException.java 
PRE-CREATION 
  
streaming/src/java/org/apache/hive/streaming/mutate/client/ConnectionException.java
 PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/mutate/client/MutatorClient.java 
PRE-CREATION 
  
streaming/src/java/org/apache/hive/streaming/mutate/client/MutatorClientBuilder.java
 PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/mutate/client/TableType.java 
PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/mutate/client/Transaction.java 
PRE-CREATION 
  
streaming/src/java/org/apache/hive/streaming/mutate/client/TransactionException.java
 PRE-CREATION 
  
streaming/src/java/org/apache/hive/streaming/mutate/client/lock/HeartbeatFactory.java
 PRE-CREATION 
  
streaming/src/java/org/apache/hive/streaming/mutate/client/lock/HeartbeatTimerTask.java
 PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/mutate/client/lock/Lock.java 
PRE-CREATION 
  
streaming/src/java/org/apache/hive/streaming/mutate/client/lock/LockException.java
 PRE-CREATION 
  
streaming/src/java/org/apache/hive/streaming/mutate/client/lock/LockFailureListener.java
 PRE-CREATION 
  
streaming/src/java/org/apache/hive/streaming/mutate/doc-files/system-overview.dot
 PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/mutate/package.html PRE-CREATION 
  
streaming/src/java/org/apache/hive/streaming/mutate/worker/BucketIdException.java
 PRE-CREATION 
  
streaming/src/java/org/apache/hive/streaming/mutate/worker/BucketIdResolver.java
 PRE-CREATION 
  
streaming/src/java/org/apache/hive/streaming/mutate/worker/BucketIdResolverImpl.java
 PRE-CREATION 
  
streaming/src/java/org/apache/hive/streaming/mutate/worker/GroupRevisitedException.java
 PRE-CREATION 
  
streaming/src/java/org/apache/hive/streaming/mutate/worker/GroupingValidator.java
 PRE-CREATION 
  
streaming/src/java/org/apache/hive/streaming/mutate/worker/MetaStorePartitionHelper.java
 PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/mutate/worker/Mutator.java

Re: Review Request 66571: HIVE-19161: Add authorizations to information schema

2018-04-13 Thread Thejas Nair


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66571/#review201142
---




jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/dao/OracleDatabaseAccessor.java
Lines 23 (patched)


Adding a prefix to it like "dummy_rownum_col_rn1938392" might help with 
debugging later!


- Thejas Nair


On April 13, 2018, 10:30 p.m., Daniel Dai wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66571/
> ---
> 
> (Updated April 13, 2018, 10:30 p.m.)
> 
> 
> Review request for hive.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> See HIVE-19161
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 0627c35 
>   
> itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/listener/DummyRawStoreFailEvent.java
>  801de7a 
>   itests/hive-unit/pom.xml f473d25 
>   
> itests/hive-unit/src/test/java/org/apache/hive/service/server/TestInformationSchemaWithPrivilege.java
>  PRE-CREATION 
>   
> jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/dao/DatabaseAccessorFactory.java
>  7dc690f 
>   
> jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/dao/GenericJdbcDatabaseAccessor.java
>  178c97d 
>   
> jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/dao/JdbcRecordIterator.java
>  638e2b0 
>   
> jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/dao/MsSqlDatabaseAccessor.java
>  PRE-CREATION 
>   
> jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/dao/OracleDatabaseAccessor.java
>  PRE-CREATION 
>   
> jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/dao/PostgresDatabaseAccessor.java
>  PRE-CREATION 
>   metastore/scripts/upgrade/hive/hive-schema-3.0.0.hive.sql 339 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java d59bf1f 
>   
> ql/src/java/org/apache/hadoop/hive/ql/security/authorization/HiveAuthorizationProvider.java
>  60d9dc1 
>   
> ql/src/java/org/apache/hadoop/hive/ql/security/authorization/PrivilegeSynchonizer.java
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/security/authorization/plugin/HiveResourceACLsImpl.java
>  PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java 6003ced 
>   
> ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFCurrentGroups.java
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFRestrictInformationSchema.java
>  PRE-CREATION 
>   service/src/java/org/apache/hive/service/server/HiveServer2.java 6308c5c 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
>  450da4f 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java
>  ebbf465 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/IMetaStoreClient.java
>  b2c40c2 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
>  2056930 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/RawStore.java
>  f6c46ee 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java
>  c47856d 
>   standalone-metastore/src/main/thrift/hive_metastore.thrift 7450439 
>   
> standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/DummyRawStoreControlledCommit.java
>  304f567 
>   
> standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/DummyRawStoreForJdoConnection.java
>  85c6727 
>   
> standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClientPreCatalog.java
>  7d37262 
> 
> 
> Diff: https://reviews.apache.org/r/66571/diff/4/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Daniel Dai
> 
>

[jira] [Created] (HIVE-19214) High throughput ingest ORC format

2018-04-13 Thread Prasanth Jayachandran (JIRA)

Prasanth Jayachandran created HIVE-19214:


 Summary: High throughput ingest ORC format
 Key: HIVE-19214
 URL: https://issues.apache.org/jira/browse/HIVE-19214
 Project: Hive
  Issue Type: Sub-task
  Components: Streaming
Affects Versions: 3.0.0, 3.1.0
Reporter: Prasanth Jayachandran
Assignee: Prasanth Jayachandran


Create delta files with all ORC overhead disabled (no index, no compression, no 
dictionary). Compactor will recreate the orc files with index, compression and 
dictionary encoding.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 66571: HIVE-19161: Add authorizations to information schema

2018-04-13 Thread Daniel Dai


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66571/
---

(Updated April 13, 2018, 10:30 p.m.)


Review request for hive.


Repository: hive-git


Description
---

See HIVE-19161


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 0627c35 
  
itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/listener/DummyRawStoreFailEvent.java
 801de7a 
  itests/hive-unit/pom.xml f473d25 
  
itests/hive-unit/src/test/java/org/apache/hive/service/server/TestInformationSchemaWithPrivilege.java
 PRE-CREATION 
  
jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/dao/DatabaseAccessorFactory.java
 7dc690f 
  
jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/dao/GenericJdbcDatabaseAccessor.java
 178c97d 
  
jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/dao/JdbcRecordIterator.java
 638e2b0 
  
jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/dao/MsSqlDatabaseAccessor.java
 PRE-CREATION 
  
jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/dao/OracleDatabaseAccessor.java
 PRE-CREATION 
  
jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/dao/PostgresDatabaseAccessor.java
 PRE-CREATION 
  metastore/scripts/upgrade/hive/hive-schema-3.0.0.hive.sql 339 
  ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java d59bf1f 
  
ql/src/java/org/apache/hadoop/hive/ql/security/authorization/HiveAuthorizationProvider.java
 60d9dc1 
  
ql/src/java/org/apache/hadoop/hive/ql/security/authorization/PrivilegeSynchonizer.java
 PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/security/authorization/plugin/HiveResourceACLsImpl.java
 PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java 6003ced 
  
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFCurrentGroups.java 
PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFRestrictInformationSchema.java
 PRE-CREATION 
  service/src/java/org/apache/hive/service/server/HiveServer2.java 6308c5c 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
 450da4f 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java
 ebbf465 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/IMetaStoreClient.java
 b2c40c2 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
 2056930 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/RawStore.java
 f6c46ee 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java
 c47856d 
  standalone-metastore/src/main/thrift/hive_metastore.thrift 7450439 
  
standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/DummyRawStoreControlledCommit.java
 304f567 
  
standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/DummyRawStoreForJdoConnection.java
 85c6727 
  
standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClientPreCatalog.java
 7d37262 


Diff: https://reviews.apache.org/r/66571/diff/4/

Changes: https://reviews.apache.org/r/66571/diff/3-4/


Testing
---


Thanks,

Daniel Dai

[jira] [Created] (HIVE-19213) QueryCache: Stop opening transactions on _dummy_table

2018-04-13 Thread Gopal V (JIRA)

Gopal V created HIVE-19213:
--

 Summary: QueryCache: Stop opening transactions on _dummy_table
 Key: HIVE-19213
 URL: https://issues.apache.org/jira/browse/HIVE-19213
 Project: Hive
  Issue Type: Bug
Reporter: Gopal V


When the query-cache is hit, the ACID TxnManager opens a txn + lock + 
heartbeats on _dummy_database._dummy_table.

This is blocking the query-cache from jumping through the fast-paths for 
transactions.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 66571: HIVE-19161: Add authorizations to information schema

2018-04-13 Thread Daniel Dai


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66571/
---

(Updated April 13, 2018, 8:49 p.m.)


Review request for hive.


Repository: hive-git


Description
---

See HIVE-19161


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 0627c35 
  
itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/listener/DummyRawStoreFailEvent.java
 801de7a 
  itests/hive-unit/pom.xml f473d25 
  
itests/hive-unit/src/test/java/org/apache/hive/service/server/TestInformationSchemaWithPrivilege.java
 PRE-CREATION 
  
jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/dao/DatabaseAccessorFactory.java
 7dc690f 
  
jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/dao/GenericJdbcDatabaseAccessor.java
 178c97d 
  
jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/dao/JdbcRecordIterator.java
 638e2b0 
  
jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/dao/MsSqlDatabaseAccessor.java
 PRE-CREATION 
  
jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/dao/OracleDatabaseAccessor.java
 PRE-CREATION 
  
jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/dao/PostgresDatabaseAccessor.java
 PRE-CREATION 
  metastore/scripts/upgrade/hive/hive-schema-3.0.0.hive.sql 339 
  ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java d59bf1f 
  
ql/src/java/org/apache/hadoop/hive/ql/security/authorization/HiveAuthorizationProvider.java
 60d9dc1 
  
ql/src/java/org/apache/hadoop/hive/ql/security/authorization/PrivilegeSynchonizer.java
 PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/security/authorization/plugin/HiveResourceACLsImpl.java
 PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java 6003ced 
  
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFCurrentGroups.java 
PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFRestrictInformationSchema.java
 PRE-CREATION 
  service/src/java/org/apache/hive/service/server/HiveServer2.java 6308c5c 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
 450da4f 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java
 ebbf465 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/IMetaStoreClient.java
 b2c40c2 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
 2056930 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/RawStore.java
 f6c46ee 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java
 c47856d 
  standalone-metastore/src/main/thrift/hive_metastore.thrift 7450439 
  
standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/DummyRawStoreControlledCommit.java
 304f567 
  
standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/DummyRawStoreForJdoConnection.java
 85c6727 
  
standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClientPreCatalog.java
 7d37262 


Diff: https://reviews.apache.org/r/66571/diff/3/

Changes: https://reviews.apache.org/r/66571/diff/2-3/


Testing
---


Thanks,

Daniel Dai

Re: Review Request 66571: HIVE-19161: Add authorizations to information schema

2018-04-13 Thread Daniel Dai


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66571/
---

(Updated April 13, 2018, 8:48 p.m.)


Review request for hive.


Repository: hive-git


Description
---

See HIVE-19161


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 0627c35 
  
itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/listener/DummyRawStoreFailEvent.java
 801de7a 
  itests/hive-unit/pom.xml f473d25 
  
itests/hive-unit/src/test/java/org/apache/hive/service/server/TestInformationSchemaWithPrivilege.java
 PRE-CREATION 
  
jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/dao/DatabaseAccessorFactory.java
 7dc690f 
  
jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/dao/GenericJdbcDatabaseAccessor.java
 178c97d 
  
jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/dao/JdbcRecordIterator.java
 638e2b0 
  
jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/dao/MsSqlDatabaseAccessor.java
 PRE-CREATION 
  
jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/dao/OracleDatabaseAccessor.java
 PRE-CREATION 
  
jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/dao/PostgresDatabaseAccessor.java
 PRE-CREATION 
  metastore/scripts/upgrade/hive/hive-schema-3.0.0.hive.sql 339 
  ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java d59bf1f 
  
ql/src/java/org/apache/hadoop/hive/ql/security/authorization/HiveAuthorizationProvider.java
 60d9dc1 
  
ql/src/java/org/apache/hadoop/hive/ql/security/authorization/PrivilegeSynchonizer.java
 PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/security/authorization/plugin/HiveResourceACLsImpl.java
 PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java 6003ced 
  
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFCurrentGroups.java 
PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFRestrictInformationSchema.java
 PRE-CREATION 
  service/src/java/org/apache/hive/service/server/HiveServer2.java 6308c5c 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
 450da4f 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java
 ebbf465 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/IMetaStoreClient.java
 b2c40c2 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
 2056930 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/RawStore.java
 f6c46ee 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java
 c47856d 
  standalone-metastore/src/main/thrift/hive_metastore.thrift 7450439 
  
standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/DummyRawStoreControlledCommit.java
 304f567 
  
standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/DummyRawStoreForJdoConnection.java
 85c6727 
  
standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClientPreCatalog.java
 7d37262 


Diff: https://reviews.apache.org/r/66571/diff/2/

Changes: https://reviews.apache.org/r/66571/diff/1-2/


Testing
---


Thanks,

Daniel Dai

Re: Apache Hive 3.0.0 release preparation

2018-04-13 Thread Vineet Garg

Thanks Vihang!


> On Apr 13, 2018, at 12:36 PM, Vihang Karajgaonkar  wrote:
> 
> Hi Vineet,
> 
> I created a profile on ptest-server so that tests can be run on branch-3.
> It is the same as branch-2 patches. You will need to include branch-3 in
> the patch name. Eg. HIVE-1234.01-branch-3.patch
> 
> -Vihang
> 
> 
> 
> On Mon, Apr 9, 2018 at 4:35 PM, Vineet Garg  wrote:
> 
>> I have created an umbrella jira to investigate and fix test failures for
>> hive 3.0.0. LINK : https://issues.apache.org/jira/browse/HIVE-19142.
>> Please link any other existing jira related to test failure with this
>> umbrella jira.
>> 
>> Also, how do we run tests on branch-3? Is there some setup to be done?
>> 
>> -Vineet
>> 
>> On Apr 9, 2018, at 4:26 AM, Zoltan Haindrich > mailto:zhaindr...@hortonworks.com>> wrote:
>> 
>> Hello
>> 
>> A few weeks earlier I've tried to hunt down this problem...
>> so...to my best knowledge the cause of this seems to be the following:
>> 
>> * in some cases the "cleanup" after a failed query may somehow leave some
>> threads behind...
>> * these threads have reference to the "customized" session classloader -
>> this makes the threads more memory hungry
>> * after a while these threads/classloaders eat up the heap...
>> 
>> I've opened HIVE-18522 for this thread issue
>> 
>> I think this problem is not new ...and it might have been present earlier
>> as well...the only thing what changed is that there were a few more new
>> features which have added new udfs/etc which made the memory cost of a
>> session more heavier..
>> ...and as a sidenote: I'm not convinced that this issue will arise in a
>> proper hs2 setup - as it might be easily connected to the fact that these
>> tests are using the cli driver to execute the tests.
>> 
>> 
>> cheers,
>> Zoltan
>> 
>> On 7 Apr 2018 7:15 p.m., Ashutosh Chauhan  ashut...@apache.org>> wrote:
>> We need to investigate and find out root cause of these failures. If its
>> determined that its a corner case and fix is non-trivial then we may
>> release note it under known issues. But ideally we should fix these
>> failures.
>> Cutting a branch should make it easier since branch is expected to receive
>> lot less commits as compared to master so it should be faster to stabilize
>> branch.
>> 
>> On Fri, Apr 6, 2018 at 10:49 AM, Eugene Koifman > mailto:ekoif...@hortonworks.com>>
>> wrote:
>> 
>> Cutting the branch before the tests are stabilized would mean we have to
>> fix them in 2 places.
>> 
>> On 4/6/18, 10:05 AM, "Thejas Nair"  thejas.n...@gmail.com>> wrote:
>> 
>>   That needs to be cleaned up. There are far too many right now, its
>>   just not handful of flaky tests.
>> 
>> 
>>   On Fri, Apr 6, 2018 at 2:48 AM, Peter Vary  pv...@cloudera.com>> wrote:
>> Hi Team,
>> 
>> I am new to the Hive release process and it is not clear to me how
>> the failing tests are handled. Do we plan to fix the failing tests before
>> release? Or it is accepted to cut a new major release with known test
>> issues.
>> 
>> Thanks,
>> Peter
>> 
>> On Apr 5, 2018, at 8:25 PM, Vineet Garg  vg...@hortonworks.com>>
>> wrote:
>> 
>> Hello,
>> 
>> I plan to cut off branch for Hive 3.0.0 on Monday (9 April) since
>> bunch of folks have big patches pending.
>> 
>> Regards,
>> Vineet G
>> 
>> On Apr 2, 2018, at 3:14 PM, Vineet Garg  vg...@hortonworks.com>>
>> wrote:
>> 
>> Hello,
>> 
>> We have enough votes to prepare a release candidate for Hive
>> 3.0.0. I am going to cutoff a branch in a day or two. I’ll send an email as
>> soon as I have the branch ready.
>> Meanwhile there are approximately 69 JIRAs which are currently
>> opened with fix version 3.0.0. I’ll appreciate if their respective owners
>> would update the JIRA if it is a blocker. Otherwise I’ll update them to
>> defer the fix version to next release.
>> 
>> Regards,
>> Vineet G
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>>

[jira] [Created] (HIVE-19212) Fix findbugs yetus pre-commit checks

2018-04-13 Thread Sahil Takiar (JIRA)

Sahil Takiar created HIVE-19212:
---

 Summary: Fix findbugs yetus pre-commit checks
 Key: HIVE-19212
 URL: https://issues.apache.org/jira/browse/HIVE-19212
 Project: Hive
  Issue Type: Bug
  Components: Testing Infrastructure
Reporter: Sahil Takiar
Assignee: Sahil Takiar


Follow up from HIVE-18883, the committed patch isn't working and Findbugs is 
still not working.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Apache Hive 3.0.0 release preparation

2018-04-13 Thread Vihang Karajgaonkar

Hi Vineet,

I created a profile on ptest-server so that tests can be run on branch-3.
It is the same as branch-2 patches. You will need to include branch-3 in
the patch name. Eg. HIVE-1234.01-branch-3.patch

-Vihang



On Mon, Apr 9, 2018 at 4:35 PM, Vineet Garg  wrote:

> I have created an umbrella jira to investigate and fix test failures for
> hive 3.0.0. LINK : https://issues.apache.org/jira/browse/HIVE-19142.
> Please link any other existing jira related to test failure with this
> umbrella jira.
>
> Also, how do we run tests on branch-3? Is there some setup to be done?
>
> -Vineet
>
> On Apr 9, 2018, at 4:26 AM, Zoltan Haindrich  mailto:zhaindr...@hortonworks.com>> wrote:
>
> Hello
>
> A few weeks earlier I've tried to hunt down this problem...
> so...to my best knowledge the cause of this seems to be the following:
>
> * in some cases the "cleanup" after a failed query may somehow leave some
> threads behind...
> * these threads have reference to the "customized" session classloader -
> this makes the threads more memory hungry
> * after a while these threads/classloaders eat up the heap...
>
> I've opened HIVE-18522 for this thread issue
>
> I think this problem is not new ...and it might have been present earlier
> as well...the only thing what changed is that there were a few more new
> features which have added new udfs/etc which made the memory cost of a
> session more heavier..
> ...and as a sidenote: I'm not convinced that this issue will arise in a
> proper hs2 setup - as it might be easily connected to the fact that these
> tests are using the cli driver to execute the tests.
>
>
> cheers,
> Zoltan
>
> On 7 Apr 2018 7:15 p.m., Ashutosh Chauhan > wrote:
> We need to investigate and find out root cause of these failures. If its
> determined that its a corner case and fix is non-trivial then we may
> release note it under known issues. But ideally we should fix these
> failures.
> Cutting a branch should make it easier since branch is expected to receive
> lot less commits as compared to master so it should be faster to stabilize
> branch.
>
> On Fri, Apr 6, 2018 at 10:49 AM, Eugene Koifman  mailto:ekoif...@hortonworks.com>>
> wrote:
>
> Cutting the branch before the tests are stabilized would mean we have to
> fix them in 2 places.
>
> On 4/6/18, 10:05 AM, "Thejas Nair" > wrote:
>
>That needs to be cleaned up. There are far too many right now, its
>just not handful of flaky tests.
>
>
>On Fri, Apr 6, 2018 at 2:48 AM, Peter Vary > wrote:
> Hi Team,
>
> I am new to the Hive release process and it is not clear to me how
> the failing tests are handled. Do we plan to fix the failing tests before
> release? Or it is accepted to cut a new major release with known test
> issues.
>
> Thanks,
> Peter
>
> On Apr 5, 2018, at 8:25 PM, Vineet Garg >
> wrote:
>
> Hello,
>
> I plan to cut off branch for Hive 3.0.0 on Monday (9 April) since
> bunch of folks have big patches pending.
>
> Regards,
> Vineet G
>
> On Apr 2, 2018, at 3:14 PM, Vineet Garg >
> wrote:
>
> Hello,
>
> We have enough votes to prepare a release candidate for Hive
> 3.0.0. I am going to cutoff a branch in a day or two. I’ll send an email as
> soon as I have the branch ready.
> Meanwhile there are approximately 69 JIRAs which are currently
> opened with fix version 3.0.0. I’ll appreciate if their respective owners
> would update the JIRA if it is a blocker. Otherwise I’ll update them to
> defer the fix version to next release.
>
> Regards,
> Vineet G
>
>
>
>
>
>
>
>
>

[jira] [Created] (HIVE-19211) New streaming ingest API and support for dynamic partitioning

2018-04-13 Thread Prasanth Jayachandran (JIRA)

Prasanth Jayachandran created HIVE-19211:


 Summary: New streaming ingest API and support for dynamic 
partitioning
 Key: HIVE-19211
 URL: https://issues.apache.org/jira/browse/HIVE-19211
 Project: Hive
  Issue Type: Sub-task
  Components: Streaming
Affects Versions: 3.0.0, 3.1.0
Reporter: Prasanth Jayachandran
Assignee: Prasanth Jayachandran


- New streaming API under new hive sub-module
- Dynamic partitioning support
- Delta file optimizations



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-19210) Create separate module for streaming ingest

2018-04-13 Thread Prasanth Jayachandran (JIRA)

Prasanth Jayachandran created HIVE-19210:


 Summary: Create separate module for streaming ingest
 Key: HIVE-19210
 URL: https://issues.apache.org/jira/browse/HIVE-19210
 Project: Hive
  Issue Type: Sub-task
  Components: Streaming
Affects Versions: 3.0.0, 3.1.0
Reporter: Prasanth Jayachandran
Assignee: Prasanth Jayachandran


This will retain the old hcat streaming API for old clients. The new streaming 
ingest API will be separate module under hive. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-19209) Streaming ingest record writers should accept input stream

2018-04-13 Thread Prasanth Jayachandran (JIRA)

Prasanth Jayachandran created HIVE-19209:


 Summary: Streaming ingest record writers should accept input stream
 Key: HIVE-19209
 URL: https://issues.apache.org/jira/browse/HIVE-19209
 Project: Hive
  Issue Type: Sub-task
  Components: Streaming
Affects Versions: 3.0.0, 3.1.0
Reporter: Prasanth Jayachandran
Assignee: Prasanth Jayachandran


Record writers in streaming ingest currently accepts byte[]. Provide an option 
for clients to pass in input stream directly from which byte[] for record can 
be constructed. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-19208) Automatic distribution for non-bucketed tables during streaming ingest

2018-04-13 Thread Prasanth Jayachandran (JIRA)

Prasanth Jayachandran created HIVE-19208:


 Summary: Automatic distribution for non-bucketed tables during 
streaming ingest
 Key: HIVE-19208
 URL: https://issues.apache.org/jira/browse/HIVE-19208
 Project: Hive
  Issue Type: Sub-task
  Components: Streaming
Affects Versions: 3.0.0, 3.1.0
Reporter: Prasanth Jayachandran
Assignee: Prasanth Jayachandran


Currently non-bucketed tables write to a single file in transaction batch. This 
limits the concurrency of compactor job to be 1. Provide an option to 
automatically distribute the incoming records to N files within a transaction 
batch. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-19207) Support avro record writer for streaming ingest

2018-04-13 Thread Prasanth Jayachandran (JIRA)

Prasanth Jayachandran created HIVE-19207:


 Summary: Support avro record writer for streaming ingest
 Key: HIVE-19207
 URL: https://issues.apache.org/jira/browse/HIVE-19207
 Project: Hive
  Issue Type: Sub-task
  Components: Streaming
Affects Versions: 3.0.0, 3.1.0
Reporter: Prasanth Jayachandran
Assignee: Prasanth Jayachandran


Add support for Avro record writer in streaming ingest.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-19206) Automatic memory management for open streaming writers

2018-04-13 Thread Prasanth Jayachandran (JIRA)

Prasanth Jayachandran created HIVE-19206:


 Summary: Automatic memory management for open streaming writers
 Key: HIVE-19206
 URL: https://issues.apache.org/jira/browse/HIVE-19206
 Project: Hive
  Issue Type: Sub-task
  Components: Streaming
Affects Versions: 3.0.0, 3.1.0
 Environment: Problem:
When there are 100s of record updaters open, the amount of memory required by 
orc writers keeps growing because of ORC's internal buffers. This can lead to 
potential high GC or OOM during streaming ingest.

Solution:
The high level idea is for the streaming connection to remember all the open 
record updaters and flush the record updater periodically (at some interval). 
Records written to each record updater can be used as a metric to determine the 
candidate record updaters for flushing. 
If stripe size of orc file is 64MB, the default memory management check happens 
only after every 5000 rows which may which may be too late when there are too 
many concurrent writers in a process. Example case would be 100 writers open 
and each of them have almost full stripe of 64MB buffered data, this would take 
100*64MB ~=6GB of memory. When all of the record writers flush, the memory 
usage drops down to 100*~2MB which is just ~200MB memory usage. 
Reporter: Prasanth Jayachandran
Assignee: Prasanth Jayachandran






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 66567: Migrate to Murmur hash for shuffle and bucketing

2018-04-13 Thread Deepak Jaiswal



> On April 13, 2018, 6:05 p.m., Matt McCline wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/exec/ReduceSinkOperator.java
> > Lines 338 (patched)
> > 
> >
> > Logging per row too expensive to leave in.

Thanks for pointing out. It was put in for dev purpose but I forgot to remove.


> On April 13, 2018, 6:05 p.m., Matt McCline wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/exec/ReduceSinkOperator.java
> > Line 338 (original), 344 (patched)
> > 
> >
> > Unnecessary line.

Sure.


> On April 13, 2018, 6:05 p.m., Matt McCline wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/exec/ReduceSinkOperator.java
> > Lines 453 (patched)
> > 
> >
> > Please add comments as to the significanse of checking the acidOp flag.

Will do.


- Deepak


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66567/#review201121
---


On April 12, 2018, 6:24 p.m., Deepak Jaiswal wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66567/
> ---
> 
> (Updated April 12, 2018, 6:24 p.m.)
> 
> 
> Review request for hive, Eugene Koifman, Jason Dere, and Matt McCline.
> 
> 
> Bugs: HIVE-18910
> https://issues.apache.org/jira/browse/HIVE-18910
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Hive uses JAVA hash which is not as good as murmur for better distribution 
> and efficiency in bucketing a table.
> Migrate to murmur hash but still keep backward compatibility for existing 
> users so that they dont have to reload the existing tables.
> 
> To keep backward compatibility, bucket_version is added as a table property, 
> resulting in high number of result updates.
> 
> 
> Diffs
> -
> 
>   hbase-handler/src/test/results/positive/external_table_ppd.q.out cdc43ee560 
>   hbase-handler/src/test/results/positive/hbase_binary_storage_queries.q.out 
> 153613e6d0 
>   hbase-handler/src/test/results/positive/hbase_ddl.q.out ef3f5f704e 
>   hbase-handler/src/test/results/positive/hbasestats.q.out 5d000d2f4f 
>   
> hcatalog/streaming/src/java/org/apache/hive/hcatalog/streaming/AbstractRecordWriter.java
>  924e233293 
>   
> hcatalog/streaming/src/java/org/apache/hive/hcatalog/streaming/mutate/worker/BucketIdResolver.java
>  5dd0b8ea5b 
>   
> hcatalog/streaming/src/java/org/apache/hive/hcatalog/streaming/mutate/worker/BucketIdResolverImpl.java
>  7c2cadefa7 
>   
> hcatalog/streaming/src/java/org/apache/hive/hcatalog/streaming/mutate/worker/MutatorCoordinator.java
>  ad14c7265f 
>   
> hcatalog/streaming/src/test/org/apache/hive/hcatalog/streaming/TestStreaming.java
>  3733e3d02f 
>   
> hcatalog/streaming/src/test/org/apache/hive/hcatalog/streaming/mutate/worker/TestBucketIdResolverImpl.java
>  03c28a33c8 
>   
> hcatalog/webhcat/java-client/src/main/java/org/apache/hive/hcatalog/api/HCatTable.java
>  996329195c 
>   
> hcatalog/webhcat/java-client/src/test/java/org/apache/hive/hcatalog/api/TestHCatClient.java
>  f9ee9d9a03 
>   
> itests/hive-blobstore/src/test/results/clientpositive/insert_into_dynamic_partitions.q.out
>  caa00292b8 
>   
> itests/hive-blobstore/src/test/results/clientpositive/insert_into_table.q.out 
> ab8ad77074 
>   
> itests/hive-blobstore/src/test/results/clientpositive/insert_overwrite_directory.q.out
>  2b28a6677e 
>   
> itests/hive-blobstore/src/test/results/clientpositive/insert_overwrite_dynamic_partitions.q.out
>  cdb67dd786 
>   
> itests/hive-blobstore/src/test/results/clientpositive/insert_overwrite_table.q.out
>  2c23a7e94f 
>   
> itests/hive-blobstore/src/test/results/clientpositive/write_final_output_blobstore.q.out
>  a1be085ea5 
>   itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/TestAcidOnTez.java 
> 353b890b7c 
>   
> itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/TestCompactor.java
>  5966740f88 
>   itests/src/test/resources/testconfiguration.properties 48d62a8bf9 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java c084fa054c 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java d59bf1fb6e 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/ReduceSinkOperator.java 
> d4363fdf91 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/keyseries/VectorKeySeriesSerializedImpl.java
>  86f466fc4e 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/reducesink/VectorReduceSinkCommonOperator.java
>  4077552a56 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/reducesink/VectorReduceSinkObjectHashOperator.java
>  1bc3fdabac 
>

Re: Review Request 66571: HIVE-19161: Add authorizations to information schema

2018-04-13 Thread Thejas Nair


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66571/#review201119
---




common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
Lines 2916 (patched)


The syncrhonizer could be eventually moved to metastore as well. 
I think its better to name both synchronizer variables as "hive." instead 
of "hive.server2."


- Thejas Nair


On April 11, 2018, 10:33 p.m., Daniel Dai wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66571/
> ---
> 
> (Updated April 11, 2018, 10:33 p.m.)
> 
> 
> Review request for hive.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> See HIVE-19161
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 0627c35 
>   
> itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/listener/DummyRawStoreFailEvent.java
>  801de7a 
>   itests/hive-unit/pom.xml f473d25 
>   
> itests/hive-unit/src/test/java/org/apache/hive/service/server/TestInformationSchemaWithPrivilege.java
>  PRE-CREATION 
>   metastore/scripts/upgrade/hive/hive-schema-3.0.0.hive.sql 339 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java d59bf1f 
>   
> ql/src/java/org/apache/hadoop/hive/ql/security/authorization/PrivilegeSynchonizer.java
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/security/authorization/plugin/HiveResourceACLs.java
>  53e221f 
>   ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java 6003ced 
>   
> ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFCurrentGroups.java
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFRestrictInformationSchema.java
>  PRE-CREATION 
>   service/src/java/org/apache/hive/service/server/HiveServer2.java 6308c5c 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
>  450da4f 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java
>  ebbf465 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/IMetaStoreClient.java
>  b2c40c2 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
>  2056930 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/RawStore.java
>  f6c46ee 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java
>  c47856d 
>   standalone-metastore/src/main/thrift/hive_metastore.thrift 7450439 
>   
> standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/DummyRawStoreControlledCommit.java
>  304f567 
>   
> standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/DummyRawStoreForJdoConnection.java
>  85c6727 
>   
> standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClientPreCatalog.java
>  7d37262 
> 
> 
> Diff: https://reviews.apache.org/r/66571/diff/1/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Daniel Dai
> 
>

Re: Recent change to skip tests

2018-04-13 Thread Deepak Jaiswal

I reopened HIVE-19077 to remove the future instance instead of the current one.

On 4/13/18, 11:10 AM, "Deepak Jaiswal"  wrote:

Hi,

It seems someone committed a patch to modify ptests. I see this on the 
ptest console output,


“Checking PreCommit-HIVE-Build queue...
PreCommit-HIVE-Build has the following jira(s) in queue: [18845, 19161, 
19054, 19161, 19126, 19184, 18652, 19184, 19187, 19175, 18902, 19104, 16041, 
12369, 12192, 18862, 19009, 18739, 19104, 19167, 18910, 16144, 19106, 18816, 
19133, 19162, 18986, 19191, 17645, 19186, 18609, 18469, 17824, 19048, 18252, 
18252, 19001, 18739, 18915, 19154, 19096]
Skipping ptest execution, as HIVE-18910 is scheduled in queue in the future 
too.
“

After waiting for 2 days I get to see this which does not make any sense. 
The queue is so long and then you get pushed back? Worst, I am not even in the 
queue anymore.
I cant find the JIRA which did this. So I request the developer who did 
this to revert it back and rework on it to remove the future instance from the 
queue. If you have time constraint, please assign it to me.

Regards,
Deepak

[jira] [Created] (HIVE-19205) Hive streaming ingest improvements (v2)

2018-04-13 Thread Prasanth Jayachandran (JIRA)

Prasanth Jayachandran created HIVE-19205:


 Summary: Hive streaming ingest improvements (v2)
 Key: HIVE-19205
 URL: https://issues.apache.org/jira/browse/HIVE-19205
 Project: Hive
  Issue Type: Improvement
  Components: Streaming
Affects Versions: 3.0.0, 3.1.0
Reporter: Prasanth Jayachandran
Assignee: Prasanth Jayachandran


This is umbrella jira to track hive streaming ingest improvements. At a high 
level following are the improvements
- Support for dynamic partitioning
- API changes (simple streaming connection builder)
- Hide the transaction batches from clients (client can tune the transaction 
batch but doesn't have to know about the transaction batch size)
- Support auto rollover to next transaction batch (clients don't have to worry 
about closing a transaction batch and opening a new one)
- Record writers will all be strict meaning the schema of the record has to 
match table schema. This is to avoid the multiple serialization/deserialization 
for re-ordering columns if there is schema mismatch
- Automatic distribution for non-bucketed tables so that compactor can have 
more parallelism
- Create delta files with all ORC overhead disabled (no compression, no 
dictionary). Compactor will recreate the orc files with compression and 
dictionary encoding.
- Automatic memory management via auto-flushing (will yield smaller stripes for 
delta files but is more scalable and clients don't have to worry about 
distributing the data across writers)
- Support for more writers (Avro specifically. ORC passthrough format?)
- Support to accept input stream instead of record byte[]
- Removing HCatalog dependency (old streaming API will be in the hcatalog 
package for backward compatibility, new streaming API will be in its own hive 
module)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Recent change to skip tests

2018-04-13 Thread Deepak Jaiswal

Hi,

It seems someone committed a patch to modify ptests. I see this on the ptest 
console output,


“Checking PreCommit-HIVE-Build queue...
PreCommit-HIVE-Build has the following jira(s) in queue: [18845, 19161, 19054, 
19161, 19126, 19184, 18652, 19184, 19187, 19175, 18902, 19104, 16041, 12369, 
12192, 18862, 19009, 18739, 19104, 19167, 18910, 16144, 19106, 18816, 19133, 
19162, 18986, 19191, 17645, 19186, 18609, 18469, 17824, 19048, 18252, 18252, 
19001, 18739, 18915, 19154, 19096]
Skipping ptest execution, as HIVE-18910 is scheduled in queue in the future too.
“

After waiting for 2 days I get to see this which does not make any sense. The 
queue is so long and then you get pushed back? Worst, I am not even in the 
queue anymore.
I cant find the JIRA which did this. So I request the developer who did this to 
revert it back and rework on it to remove the future instance from the queue. 
If you have time constraint, please assign it to me.

Regards,
Deepak

Re: Review Request 66567: Migrate to Murmur hash for shuffle and bucketing

2018-04-13 Thread Matt McCline


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66567/#review201121
---




ql/src/java/org/apache/hadoop/hive/ql/exec/ReduceSinkOperator.java
Lines 338 (patched)


Logging per row too expensive to leave in.



ql/src/java/org/apache/hadoop/hive/ql/exec/ReduceSinkOperator.java
Line 338 (original), 344 (patched)


Unnecessary line.



ql/src/java/org/apache/hadoop/hive/ql/exec/ReduceSinkOperator.java
Lines 453 (patched)


Please add comments as to the significanse of checking the acidOp flag.


- Matt McCline


On April 12, 2018, 6:24 p.m., Deepak Jaiswal wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66567/
> ---
> 
> (Updated April 12, 2018, 6:24 p.m.)
> 
> 
> Review request for hive, Eugene Koifman, Jason Dere, and Matt McCline.
> 
> 
> Bugs: HIVE-18910
> https://issues.apache.org/jira/browse/HIVE-18910
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Hive uses JAVA hash which is not as good as murmur for better distribution 
> and efficiency in bucketing a table.
> Migrate to murmur hash but still keep backward compatibility for existing 
> users so that they dont have to reload the existing tables.
> 
> To keep backward compatibility, bucket_version is added as a table property, 
> resulting in high number of result updates.
> 
> 
> Diffs
> -
> 
>   hbase-handler/src/test/results/positive/external_table_ppd.q.out cdc43ee560 
>   hbase-handler/src/test/results/positive/hbase_binary_storage_queries.q.out 
> 153613e6d0 
>   hbase-handler/src/test/results/positive/hbase_ddl.q.out ef3f5f704e 
>   hbase-handler/src/test/results/positive/hbasestats.q.out 5d000d2f4f 
>   
> hcatalog/streaming/src/java/org/apache/hive/hcatalog/streaming/AbstractRecordWriter.java
>  924e233293 
>   
> hcatalog/streaming/src/java/org/apache/hive/hcatalog/streaming/mutate/worker/BucketIdResolver.java
>  5dd0b8ea5b 
>   
> hcatalog/streaming/src/java/org/apache/hive/hcatalog/streaming/mutate/worker/BucketIdResolverImpl.java
>  7c2cadefa7 
>   
> hcatalog/streaming/src/java/org/apache/hive/hcatalog/streaming/mutate/worker/MutatorCoordinator.java
>  ad14c7265f 
>   
> hcatalog/streaming/src/test/org/apache/hive/hcatalog/streaming/TestStreaming.java
>  3733e3d02f 
>   
> hcatalog/streaming/src/test/org/apache/hive/hcatalog/streaming/mutate/worker/TestBucketIdResolverImpl.java
>  03c28a33c8 
>   
> hcatalog/webhcat/java-client/src/main/java/org/apache/hive/hcatalog/api/HCatTable.java
>  996329195c 
>   
> hcatalog/webhcat/java-client/src/test/java/org/apache/hive/hcatalog/api/TestHCatClient.java
>  f9ee9d9a03 
>   
> itests/hive-blobstore/src/test/results/clientpositive/insert_into_dynamic_partitions.q.out
>  caa00292b8 
>   
> itests/hive-blobstore/src/test/results/clientpositive/insert_into_table.q.out 
> ab8ad77074 
>   
> itests/hive-blobstore/src/test/results/clientpositive/insert_overwrite_directory.q.out
>  2b28a6677e 
>   
> itests/hive-blobstore/src/test/results/clientpositive/insert_overwrite_dynamic_partitions.q.out
>  cdb67dd786 
>   
> itests/hive-blobstore/src/test/results/clientpositive/insert_overwrite_table.q.out
>  2c23a7e94f 
>   
> itests/hive-blobstore/src/test/results/clientpositive/write_final_output_blobstore.q.out
>  a1be085ea5 
>   itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/TestAcidOnTez.java 
> 353b890b7c 
>   
> itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/TestCompactor.java
>  5966740f88 
>   itests/src/test/resources/testconfiguration.properties 48d62a8bf9 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java c084fa054c 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java d59bf1fb6e 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/ReduceSinkOperator.java 
> d4363fdf91 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/keyseries/VectorKeySeriesSerializedImpl.java
>  86f466fc4e 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/reducesink/VectorReduceSinkCommonOperator.java
>  4077552a56 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/reducesink/VectorReduceSinkObjectHashOperator.java
>  1bc3fdabac 
>   ql/src/java/org/apache/hadoop/hive/ql/metadata/Table.java a51fdd322f 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/ConvertJoinMapJoin.java 
> 7121bceb22 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/FixedBucketPruningOptimizer.java
>  5f65f638ca 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/PrunerOperatorFactory.java 
> 2be3c9b9a2 
>   
>

[jira] [Created] (HIVE-19204) Detailed errors from some tasks are not displayed to the client because the tasks don't set exception when they fail

2018-04-13 Thread Aihua Xu (JIRA)

Aihua Xu created HIVE-19204:
---

 Summary: Detailed errors from some tasks are not displayed to the 
client because the tasks don't set exception when they fail
 Key: HIVE-19204
 URL: https://issues.apache.org/jira/browse/HIVE-19204
 Project: Hive
  Issue Type: Improvement
  Components: Query Processor
Affects Versions: 3.0.0
Reporter: Aihua Xu
Assignee: Aihua Xu


In TaskRunner.java, if the tasks have exception set, then the task result will 
have such exception set and Driver.java will get such details and display to 
the client. But some tasks don't set such exceptions so the client won't see 
such details unless you check the HS2 log.
  
{noformat}
  public void runSequential() {
int exitVal = -101;
try {
  exitVal = tsk.executeTask(ss == null ? null : ss.getHiveHistory());
} catch (Throwable t) {
  if (tsk.getException() == null) {
tsk.setException(t);
  }
  LOG.error("Error in executeTask", t);
}
result.setExitVal(exitVal);
if (tsk.getException() != null) {
  result.setTaskError(tsk.getException());
}
  }
 {noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Unable to read hive external table data which is linked to s3 after upgradation from 2.1.1 to 2.3.3

2018-04-13 Thread Anup Tiwari

looping dev in for suggestions.

Regards,
Anup Tiwari

On Thu, Apr 12, 2018 at 8:39 PM, Richard A. Bross 
wrote:

> To be honest, my advice was just based on your post - we only use Hive in
> AWS EMR, so I couldn't tell you.
>
> Glad that at least you're back up though.
>
> - Original Message -
> From: "Anup Tiwari" 
> To: u...@hive.apache.org
> Sent: Thursday, April 12, 2018 10:50:23 AM
> Subject: Re: Unable to read hive external table data which is linked to s3
> after upgradation from 2.1.1 to 2.3.3
>
>
>
>
>
>
>
>
> Hi Richard ,
>
> After looking at hive 2.3.3 logs i found that when we are loading all
> configuration parameters then below message is getting printed :-
>
> Values omitted for security reason if present: [fs.s3n.awsAccessKeyId,
> fs.s3a.access.key, fs.s3.awsAccessKeyId, hive.server2.keystore.password, f
> s.s3a.proxy.password, javax.jdo.option.ConnectionPassword,
> fs.s3.awsSecretAccessKey, fs.s3n.awsSecretAccessKey, fs.s3a.secret.key]
>
>
> while in hive 2.1.1 logs i found below message :-
>
> Values omitted for security reason if present: 
> [hive.server2.keystore.password,
> javax.jdo.option.ConnectionPassword]
>
> Can this be the reason why hive 2.3.3 is not able to read s3 related
> params from hive-site.xml?
>
> I found one of JIRA : https://issues.apache.org/jira/browse/HIVE-14588
>
>
>
> Also i have set below property in hive-site.xml by excluding above s3
> variables(fs.s3a.access.key,fs.s3a.secret.key) from list and it worked.
>
> 
> hive.conf.hidden.list
> javax.jdo.option.ConnectionPassword,hive.
> server2.keystore.password,fs.s3.awsAccessKeyId,fs.s3.
> awsSecretAccessKey,fs.s3n.awsAccessKeyId,fs.s3n.awsSecretAccessKey,fs.s3a.
> proxy.password
> Comma separated list of configuration options which should
> not be read by normal user like passwords.Anup has excluded 2 variable
> which were fs.s3a.access.key,fs.s3a.secret.key
> 
>
> Let me know if there is any other solution because i think if these
> variables are by default part of hidden.list then there will be some other
> proper workaround for this.
>
>
>
>
>
>
>
>
>
> Regards,
> Anup Tiwari
>
>
> On Thu, Apr 12, 2018 at 7:44 PM, Richard A. Bross < r...@oaktreepeak.com >
> wrote:
>
>
> I hear you, but given the exception log, it does seem that it can't
> authenticate you. You can try using the AWS environment variables. If that
> resolves the issue then you'll have some more to go on. According to
> Hortonworks here:
>
> https://hortonworks.github.io/hdp-aws/s3-security/
>
> "AWS CLI supports authentication through environment variables. These same
> environment variables will be used by Hadoop if no configuration properties
> are set."
>
>
> - Original Message -
> From: "Anup Tiwari" < anupsdtiw...@gmail.com >
> To: u...@hive.apache.org
> Sent: Thursday, April 12, 2018 10:06:33 AM
> Subject: Re: Unable to read hive external table data which is linked to s3
> after upgradation from 2.1.1 to 2.3.3
>
>
> We are not using EMR. Also we have set below params for accessing s3
> bucket in hive-site.xml which are same as what we have set in hive 2.1.1.
>
>
> * fs.s3a.access.key
> * fs.s3a.secret.key
> * fs.s3a.connection.maximum
> * fs.s3a.impl
>
>
>
>
>
>
>
>
> Regards,
> Anup Tiwari
>
>
> On Thu, Apr 12, 2018 at 7:19 PM, Richard A. Bross < r...@oaktreepeak.com >
> wrote:
>
>
> Based on the exception, it looks more like an AWS credentials issue than a
> Hive issue. Are you running in AWS EMR, on-prem?
>
> In AWS the resource accessing the S3 bucket would have to have an IAM that
> gave permission. If you are running somewhere else whatever AWS login you
> use would have to have the correct permissions in the IAM.
>
>
>
> - Original Message -
> From: "Anup Tiwari" < anupsdtiw...@gmail.com >
> To: u...@hive.apache.org
> Sent: Thursday, April 12, 2018 9:11:37 AM
> Subject: Unable to read hive external table data which is linked to s3
> after upgradation from 2.1.1 to 2.3.3
>
>
>
> Hi All,
>
> When i am trying to read s3 linked external table in 2.3.3 ; i am getting
> errors.It was working properly in 2.1.1. please find below details and let
> me know if i am missing something :-
>
>
> Hadoop Version :- 2.8.0
>
> Query :-
>
> select log_date,count(1) as cnt from test.tt1 group by log_date;
>
> Error :-
>
> Vertex failed, vertexName=Map 1, vertexId=vertex_1523502631429_0029_3_00,
> diagnostics=[Vertex vertex_1523502631429_0029_3_00 [Map 1] killed/failed
> due to:ROOT_INPUT_INIT_FAILURE, Vertex Input: tt1 initializer failed,
> vertex=vertex_1523502631429_0029_3_00 [Map 1], 
> org.apache.hadoop.fs.s3a.AWSClientIOException:
> doesBucketExist on g24x7.new-analytics: com.amazonaws.AmazonClientException:
> No AWS Credentials provided by BasicAWSCredentialsProvider
> EnvironmentVariableCredentialsProvider 
> SharedInstanceProfileCredentialsProvider
> : com.amazonaws.AmazonClientException: The requested metadata is not
> found at

[jira] [Created] (HIVE-19203) Thread-Safety Issue in HiveMetaStore

2018-04-13 Thread BELUGA BEHR (JIRA)

BELUGA BEHR created HIVE-19203:
--

 Summary: Thread-Safety Issue in HiveMetaStore
 Key: HIVE-19203
 URL: https://issues.apache.org/jira/browse/HIVE-19203
 Project: Hive
  Issue Type: Bug
  Components: Metastore
Affects Versions: 3.0.0
Reporter: BELUGA BEHR


[https://github.com/apache/hive/blob/550d1e1196b7c801c572092db974a459aac6c249/standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java#L345-L351]
{code:java}
private static int nextSerialNum = 0;
private static ThreadLocal threadLocalId = new ThreadLocal() {
  @Override
  protected Integer initialValue() {
return nextSerialNum++;
  }
};{code}
 

{{nextSerialNum}} needs to be an atomic value.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-19202) CBO failed due to NullPointerException in HiveAggregate.isBucketedInput()

2018-04-13 Thread zhuwei (JIRA)

zhuwei created HIVE-19202:
-

 Summary: CBO failed due to NullPointerException in 
HiveAggregate.isBucketedInput()
 Key: HIVE-19202
 URL: https://issues.apache.org/jira/browse/HIVE-19202
 Project: Hive
  Issue Type: Bug
Reporter: zhuwei
Assignee: zhuwei


I ran a query with join and group by with below settings, COB failed due to 
NullPointerException in HiveAggregate.isBucketedInput()

set hive.execution.engine=tez;

set hive.cbo.costmodel.extended=true;

 

In class HiveRelMdDistribution, we implemented below functions:

public RelDistribution distribution(HiveAggregate aggregate, RelMetadataQuery 
mq)

public RelDistribution distribution(HiveJoin join, RelMetadataQuery mq)

 

But in HiveAggregate.isBucketedInput, the argument passed to distribution is 
"this.getInput()"

, obviously it's not right here. The right argument needed is "this"



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Review Request 66604: HIVE-19158: Fix NPE in the HiveMetastore add partition tests

2018-04-13 Thread Marta Kuczora via Review Board


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66604/
---

Review request for hive, Peter Vary and Adam Szita.


Bugs: HIVE-19158
https://issues.apache.org/jira/browse/HIVE-19158


Repository: hive-git


Description
---

The TestAddPartitions and TestAddPartitionsFromPartSpec tests revealed that NPE 
is thrown in some cases. These NPEs could be prevented with a simple null check 
and a MetaException with a proper error message should be thrown instead.


Diffs
-

  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
 565549a 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java
 9a43b2c 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/partition/spec/CompositePartitionSpecProxy.java
 92813b9 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/partition/spec/PartitionListComposingSpecProxy.java
 6bd29d0 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/partition/spec/PartitionSpecProxy.java
 ff2dea1 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/partition/spec/PartitionSpecWithSharedSDProxy.java
 61e00ea 
  
standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/client/TestAddPartitions.java
 f8497c7 
  
standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/client/TestAddPartitionsFromPartSpec.java
 fc0c60f 


Diff: https://reviews.apache.org/r/66604/diff/1/


Testing
---


Thanks,

Marta Kuczora

[GitHub] hive pull request #37: Fix bug when OrcFileStripeMergeRecordReader get nextS...

2018-04-13 Thread FanTn

Github user FanTn closed the pull request at:

https://github.com/apache/hive/pull/37


---

[jira] [Created] (HIVE-19201) Hive doesn't read Druid data correctly

2018-04-13 Thread Tournadre (JIRA)

Tournadre created HIVE-19201:


 Summary: Hive doesn't read Druid data correctly
 Key: HIVE-19201
 URL: https://issues.apache.org/jira/browse/HIVE-19201
 Project: Hive
  Issue Type: Bug
  Components: Druid integration, Hive
Affects Versions: 2.3.0
 Environment: Ubuntu 16.4 TLS Desktop
Druid 0.12.0 (StandAlone - Quickstart conf)
Hive 2.3.0 (StandAlone - Quickstart conf)

Also have Hadoop and Zookeeper running
Reporter: Tournadre


I created an external table on hive pointing at my datasource : wikiticker on 
druid (I already defined the broker adress). However few colums appear as NULL 
without knowing why.

I have also many time *.lck file in the metadatadb on Hive and Druid to delete 
(otherwise, queries don't work)

So the describe statement lists the column and their type correctly, however I 
only get "NULL" when I query some column (string type). I checked in my Druid, 
the data are still here.

 

Help ! :(  

PS : Sorry first issue and JIRA, didn't see my issue resolved yet. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-19200) Vectorization: Disable vectorization for LLAP I/O when a non-VECTORIZED_INPUT_FILE_FORMAT mode is needed (i.e. rows) and data type conversion is needed

2018-04-13 Thread Matt McCline (JIRA)

Matt McCline created HIVE-19200:
---

 Summary: Vectorization: Disable vectorization for LLAP I/O when a 
non-VECTORIZED_INPUT_FILE_FORMAT mode is needed (i.e. rows) and data type 
conversion is needed
 Key: HIVE-19200
 URL: https://issues.apache.org/jira/browse/HIVE-19200
 Project: Hive
  Issue Type: Bug
  Components: Hive
Affects Versions: 3.0.0
Reporter: Matt McCline


Disable vectorization for issue in HIVE-18763 until we can do the harder VRB 
conversion code.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-19199) ACID: DbTxnManager heartbeat-service needs static sync init

2018-04-13 Thread Gopal V (JIRA)

Gopal V created HIVE-19199:
--

 Summary: ACID: DbTxnManager heartbeat-service needs static sync 
init 
 Key: HIVE-19199
 URL: https://issues.apache.org/jira/browse/HIVE-19199
 Project: Hive
  Issue Type: Bug
Reporter: Gopal V


{code}
private synchronized void initHeartbeatExecutorService() {
if (heartbeatExecutorService != null && 
!heartbeatExecutorService.isShutdown()
&& !heartbeatExecutorService.isTerminated()) {
  return;
}
...
{code}

heartbeatExecutorService is a static field, so this needs class sync, not 
object sync.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 66567: Migrate to Murmur hash for shuffle and bucketing

Re: Review Request 66571: HIVE-19161: Add authorizations to information schema

Re: Review Request 66614: HIVE-19210: Create separate module for streaming ingest

[jira] [Created] (HIVE-19215) JavaUtils.AnyIdDirFilter ignores base_n directories

Re: Review Request 66614: HIVE-19210: Create separate module for streaming ingest

Review Request 66614: HIVE-19210: Create separate module for streaming ingest

Re: Review Request 66571: HIVE-19161: Add authorizations to information schema

[jira] [Created] (HIVE-19214) High throughput ingest ORC format

Re: Review Request 66571: HIVE-19161: Add authorizations to information schema

[jira] [Created] (HIVE-19213) QueryCache: Stop opening transactions on _dummy_table

Re: Review Request 66571: HIVE-19161: Add authorizations to information schema

Re: Review Request 66571: HIVE-19161: Add authorizations to information schema

Re: Apache Hive 3.0.0 release preparation

[jira] [Created] (HIVE-19212) Fix findbugs yetus pre-commit checks

Re: Apache Hive 3.0.0 release preparation

[jira] [Created] (HIVE-19211) New streaming ingest API and support for dynamic partitioning

[jira] [Created] (HIVE-19210) Create separate module for streaming ingest

[jira] [Created] (HIVE-19209) Streaming ingest record writers should accept input stream

[jira] [Created] (HIVE-19208) Automatic distribution for non-bucketed tables during streaming ingest

[jira] [Created] (HIVE-19207) Support avro record writer for streaming ingest

[jira] [Created] (HIVE-19206) Automatic memory management for open streaming writers

Re: Review Request 66567: Migrate to Murmur hash for shuffle and bucketing

Re: Review Request 66571: HIVE-19161: Add authorizations to information schema

Re: Recent change to skip tests

[jira] [Created] (HIVE-19205) Hive streaming ingest improvements (v2)

Recent change to skip tests

Re: Review Request 66567: Migrate to Murmur hash for shuffle and bucketing

[jira] [Created] (HIVE-19204) Detailed errors from some tasks are not displayed to the client because the tasks don't set exception when they fail

Re: Unable to read hive external table data which is linked to s3 after upgradation from 2.1.1 to 2.3.3

[jira] [Created] (HIVE-19203) Thread-Safety Issue in HiveMetaStore

[jira] [Created] (HIVE-19202) CBO failed due to NullPointerException in HiveAggregate.isBucketedInput()

Review Request 66604: HIVE-19158: Fix NPE in the HiveMetastore add partition tests

[GitHub] hive pull request #37: Fix bug when OrcFileStripeMergeRecordReader get nextS...

[jira] [Created] (HIVE-19201) Hive doesn't read Druid data correctly

[jira] [Created] (HIVE-19200) Vectorization: Disable vectorization for LLAP I/O when a non-VECTORIZED_INPUT_FILE_FORMAT mode is needed (i.e. rows) and data type conversion is needed

[jira] [Created] (HIVE-19199) ACID: DbTxnManager heartbeat-service needs static sync init

36 matches

Site Navigation

Mail list logo

Footer information