Re: Review Request 66567: Migrate to Murmur hash for shuffle and bucketing

2018-04-27 Thread Deepak Jaiswal 


> On April 14, 2018, 1:13 a.m., Jason Dere wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFHash.java
> > Lines 32 (patched)
> > 
> >
> > Docs for this UDF will probably need to mention that this uses the old 
> > hashing/bucketing scheme which and that a new one has replaced it.
> 
> Deepak Jaiswal wrote:
> Should I open a documentation JIRA to track this?

opened https://issues.apache.org/jira/browse/HIVE-19342


- Deepak


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66567/#review201133
---


On April 27, 2018, 1:14 a.m., Deepak Jaiswal wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66567/
> ---
> 
> (Updated April 27, 2018, 1:14 a.m.)
> 
> 
> Review request for hive, Ashutosh Chauhan, Eugene Koifman, Jason Dere, and 
> Matt McCline.
> 
> 
> Bugs: HIVE-18910
> https://issues.apache.org/jira/browse/HIVE-18910
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Hive uses JAVA hash which is not as good as murmur for better distribution 
> and efficiency in bucketing a table.
> Migrate to murmur hash but still keep backward compatibility for existing 
> users so that they dont have to reload the existing tables.
> 
> To keep backward compatibility, bucket_version is added as a table property, 
> resulting in high number of result updates.
> 
> 
> Diffs
> -
> 
>   hbase-handler/src/test/results/positive/external_table_ppd.q.out cdc43ee560 
>   hbase-handler/src/test/results/positive/hbase_binary_storage_queries.q.out 
> 153613e6d0 
>   hbase-handler/src/test/results/positive/hbase_ddl.q.out ef3f5f704e 
>   hbase-handler/src/test/results/positive/hbasestats.q.out 5d000d2f4f 
>   
> hcatalog/streaming/src/java/org/apache/hive/hcatalog/streaming/AbstractRecordWriter.java
>  924e233293 
>   
> hcatalog/streaming/src/test/org/apache/hive/hcatalog/streaming/TestStreaming.java
>  fe2b1c1f3c 
>   
> hcatalog/streaming/src/test/org/apache/hive/hcatalog/streaming/mutate/worker/TestBucketIdResolverImpl.java
>  03c28a33c8 
>   
> hcatalog/webhcat/java-client/src/main/java/org/apache/hive/hcatalog/api/HCatTable.java
>  996329195c 
>   
> hcatalog/webhcat/java-client/src/test/java/org/apache/hive/hcatalog/api/TestHCatClient.java
>  f9ee9d9a03 
>   
> itests/hive-blobstore/src/test/results/clientpositive/insert_into_dynamic_partitions.q.out
>  caa00292b8 
>   
> itests/hive-blobstore/src/test/results/clientpositive/insert_into_table.q.out 
> ab8ad77074 
>   
> itests/hive-blobstore/src/test/results/clientpositive/insert_overwrite_directory.q.out
>  2b28a6677e 
>   
> itests/hive-blobstore/src/test/results/clientpositive/insert_overwrite_dynamic_partitions.q.out
>  cdb67dd786 
>   
> itests/hive-blobstore/src/test/results/clientpositive/insert_overwrite_table.q.out
>  2c23a7e94f 
>   
> itests/hive-blobstore/src/test/results/clientpositive/write_final_output_blobstore.q.out
>  a1be085ea5 
>   
> itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/TestCompactor.java
>  82ba775286 
>   itests/src/test/resources/testconfiguration.properties 1a346593fd 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java c084fa054c 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java d59bf1fb6e 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/Operator.java c28ef99621 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/OperatorFactory.java 21ca04d78a 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/ReduceSinkOperator.java 
> d4363fdf91 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 25035433c7 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/repl/bootstrap/load/table/LoadPartitions.java
>  a42c299537 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/keyseries/VectorKeySeriesSerializedImpl.java
>  86f466fc4e 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/reducesink/VectorReduceSinkObjectHashOperator.java
>  1bc3fdabac 
>   ql/src/java/org/apache/hadoop/hive/ql/io/HiveFileFormatUtils.java 
> 71498a125c 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java 019682fb10 
>   ql/src/java/org/apache/hadoop/hive/ql/metadata/Table.java a51fdd322f 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/ConvertJoinMapJoin.java 
> 7121bceb22 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/FixedBucketPruningOptimizer.java
>  5f65f638ca 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/PrunerOperatorFactory.java 
> 2be3c9b9a2 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/SortedDynPartitionOptimizer.java
>  1c5656267d 
>   
>

Review Request 66862: HIVE-19258 add originals support to MM tables (and make the conversion a metadata only operation)

2018-04-27 Thread Sergey Shelukhin 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66862/
---

Review request for hive and Thejas Nair.


Repository: hive-git


Description
---

see jira


Diffs
-

  
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/TestCompactor.java
 4ebd096d0d 
  ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 3141a7e981 
  ql/src/java/org/apache/hadoop/hive/ql/exec/FetchOperator.java 969c591917 
  ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java 445e126386 
  ql/src/java/org/apache/hadoop/hive/ql/io/HiveInputFormat.java 611a4c346b 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java 019682fb10 
  ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorMR.java 
b698c84080 
  ql/src/test/queries/clientpositive/mm_conversions.q 55565a9428 


Diff: https://reviews.apache.org/r/66862/diff/1/


Testing
---


Thanks,

Sergey Shelukhin

Re: Review Request 66720: HIVE-17657 export/import for MM tables is broken

2018-04-27 Thread Eugene Koifman 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66720/#review202100
---




ql/src/java/org/apache/hadoop/hive/ql/parse/ExportSemanticAnalyzer.java
Line 21 (original), 21 (patched)


unused imports



ql/src/java/org/apache/hadoop/hive/ql/parse/ExportSemanticAnalyzer.java
Lines 121 (patched)


this seems strange - can 2 table names differ?



ql/src/java/org/apache/hadoop/hive/ql/parse/repl/dump/io/FileOperations.java
Lines 99 (patched)


can this be "export_" or something?



ql/src/java/org/apache/hadoop/hive/ql/parse/repl/dump/io/FileOperations.java
Lines 117 (patched)


this should include getOriginalFiles() check if table was converted to MM 
but not yet compacted (I assume the patch to make this conversion metadata-only 
operation is still somewhere in flight)



ql/src/java/org/apache/hadoop/hive/ql/plan/ExportWork.java
Line 19 (original), 19 (patched)


unused imports


- Eugene Koifman


On April 23, 2018, 2:18 p.m., Sergey Shelukhin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66720/
> ---
> 
> (Updated April 23, 2018, 2:18 p.m.)
> 
> 
> Review request for hive and Eugene Koifman.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> .
> 
> 
> Diffs
> -
> 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/CopyTask.java ce683c8a8d 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/ExportTask.java aba65918f8 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 6395c31ec7 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/repl/ReplDumpTask.java 
> ce0757cba2 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/ExportSemanticAnalyzer.java 
> d3c62a2775 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/ImportSemanticAnalyzer.java 
> b850ddc9d0 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzerFactory.java 
> 820046388a 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/repl/dump/PartitionExport.java 
> 5844f3d97f 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/repl/dump/TableExport.java 
> abb2e8874b 
>   
> ql/src/java/org/apache/hadoop/hive/ql/parse/repl/dump/io/FileOperations.java 
> 866d3513b1 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/CopyWork.java c0e4a43d9c 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/ExportWork.java 72ce79836c 
>   ql/src/test/org/apache/hadoop/hive/ql/TestTxnCommands.java 12d57c6feb 
>   ql/src/test/org/apache/hadoop/hive/ql/TestTxnExIm.java 0e53697be2 
>   ql/src/test/org/apache/hadoop/hive/ql/TxnCommandsBaseForTests.java 
> a2adb966fe 
>   ql/src/test/queries/clientpositive/mm_exim.q c47342bd23 
>   ql/src/test/results/clientpositive/llap/mm_exim.q.out 1f40754373 
> 
> 
> Diff: https://reviews.apache.org/r/66720/diff/3/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Sergey Shelukhin
> 
>

Re: Review Request 66571: HIVE-19161: Add authorizations to information schema

2018-04-27 Thread Daniel Dai 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66571/
---

(Updated April 28, 2018, 1:09 a.m.)


Review request for hive.


Repository: hive-git


Description
---

See HIVE-19161


Diffs (updated)
-

  
accumulo-handler/src/java/org/apache/hadoop/hive/accumulo/HiveAccumuloHelper.java
 9fccb49 
  accumulo-handler/src/java/org/apache/hadoop/hive/accumulo/Utils.java 3a2facf 
  
accumulo-handler/src/java/org/apache/hadoop/hive/accumulo/serde/CompositeAccumuloRowIdFactory.java
 d8b9aa3 
  
accumulo-handler/src/java/org/apache/hadoop/hive/accumulo/serde/DefaultAccumuloRowIdFactory.java
 bae2930 
  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java f40c606 
  
itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/listener/DummyRawStoreFailEvent.java
 8ecbaad 
  itests/hive-unit/pom.xml 3ae7f2f 
  itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestRestrictedList.java 
79fdb68 
  
itests/hive-unit/src/test/java/org/apache/hive/service/server/TestInformationSchemaWithPrivilege.java
 PRE-CREATION 
  
jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/JdbcStorageHandler.java 
df55272 
  
jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/dao/DatabaseAccessorFactory.java
 6d3c8d9 
  
jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/dao/GenericJdbcDatabaseAccessor.java
 772bc5d 
  
jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/dao/JdbcRecordIterator.java
 638e2b0 
  
jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/dao/MsSqlDatabaseAccessor.java
 PRE-CREATION 
  
jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/dao/OracleDatabaseAccessor.java
 PRE-CREATION 
  
jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/dao/PostgresDatabaseAccessor.java
 PRE-CREATION 
  metastore/scripts/upgrade/hive/hive-schema-3.0.0.hive.sql 339 
  ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java d59bf1f 
  ql/src/java/org/apache/hadoop/hive/ql/metadata/JarUtils.java PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/security/authorization/HiveAuthorizationProvider.java
 60d9dc1 
  
ql/src/java/org/apache/hadoop/hive/ql/security/authorization/PrivilegeSynchonizer.java
 PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/security/authorization/plugin/HiveResourceACLsImpl.java
 PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java 60b63d4 
  
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFCurrentGroups.java 
PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFRestrictInformationSchema.java
 PRE-CREATION 
  ql/src/test/results/clientpositive/llap/resourceplan.q.out 9850276 
  ql/src/test/results/clientpositive/show_functions.q.out 4df555b 
  service/src/java/org/apache/hive/service/server/HiveServer2.java e373628 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
 397a081 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java
 1c8d223 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/IMetaStoreClient.java
 aee416d 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
 184ecb6 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/RawStore.java
 2c9f2e5 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java
 92d000b 
  standalone-metastore/src/main/thrift/hive_metastore.thrift c56a4f9 
  
standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/DummyRawStoreControlledCommit.java
 defc68f 
  
standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/DummyRawStoreForJdoConnection.java
 20c5d8a 
  
standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClientPreCatalog.java
 bf87cfc 


Diff: https://reviews.apache.org/r/66571/diff/6/

Changes: https://reviews.apache.org/r/66571/diff/5-6/


Testing
---


Thanks,

Daniel Dai

[jira] [Created] (HIVE-19345) createSources fails on branch-2.3

2018-04-27 Thread Vihang Karajgaonkar (JIRA) 
Vihang Karajgaonkar created HIVE-19345:
--

 Summary: createSources fails on branch-2.3
 Key: HIVE-19345
 URL: https://issues.apache.org/jira/browse/HIVE-19345
 Project: Hive
  Issue Type: Bug
Reporter: Vihang Karajgaonkar
Assignee: Vihang Karajgaonkar


I see the following NPE when the source tables are getting created when I try 
to run a qtest.

{noformat}
java.lang.NullPointerException
at 
org.apache.hadoop.hive.ql.stats.StatsUtils.estimateRowSizeFromSchema(StatsUtils.java:546)
at 
org.apache.hadoop.hive.ql.stats.StatsUtils.getNumRows(StatsUtils.java:183)
at 
org.apache.hadoop.hive.ql.stats.StatsUtils.collectStatistics(StatsUtils.java:207)
at 
org.apache.hadoop.hive.ql.stats.StatsUtils.collectStatistics(StatsUtils.java:157)
at 
org.apache.hadoop.hive.ql.stats.StatsUtils.collectStatistics(StatsUtils.java:145)
at 
org.apache.hadoop.hive.ql.optimizer.stats.annotation.StatsRulesProcFactory$TableScanStatsRule.process(StatsRulesProcFactory.java:130)
at 
org.apache.hadoop.hive.ql.lib.DefaultRuleDispatcher.dispatch(DefaultRuleDispatcher.java:90)
at 
org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatchAndReturn(DefaultGraphWalker.java:105)
at 
org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatch(DefaultGraphWalker.java:89)
at 
org.apache.hadoop.hive.ql.lib.LevelOrderWalker.walk(LevelOrderWalker.java:143)
at 
org.apache.hadoop.hive.ql.lib.LevelOrderWalker.startWalking(LevelOrderWalker.java:122)
at 
org.apache.hadoop.hive.ql.optimizer.stats.annotation.AnnotateWithStatistics.transform(AnnotateWithStatistics.java:78)
at 
org.apache.hadoop.hive.ql.parse.spark.SparkCompiler.runStatsAnnotation(SparkCompiler.java:240)
at 
org.apache.hadoop.hive.ql.parse.spark.SparkCompiler.optimizeOperatorPlan(SparkCompiler.java:119)
at 
org.apache.hadoop.hive.ql.parse.TaskCompiler.compile(TaskCompiler.java:140)
at 
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:11273)
at 
org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:286)
at 
org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:258)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:512)
at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1317)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1457)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1237)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1227)
at 
org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:233)
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:184)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:403)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:336)
at 
org.apache.hadoop.hive.ql.QTestUtil.createSources(QTestUtil.java:1096)
at 
org.apache.hadoop.hive.ql.QTestUtil.createSources(QTestUtil.java:1073)
at 
org.apache.hadoop.hive.cli.control.CoreCliDriver$3.invokeInternal(CoreCliDriver.java:81)
at 
org.apache.hadoop.hive.cli.control.CoreCliDriver$3.invokeInternal(CoreCliDriver.java:78)
at 
org.apache.hadoop.hive.util.ElapsedTimeLoggingWrapper.invoke(ElapsedTimeLoggingWrapper.java:33)
at 
org.apache.hadoop.hive.cli.control.CoreCliDriver.beforeClass(CoreCliDriver.java:84)
{noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-19344) Change default value of msck.repair.batch.size

2018-04-27 Thread Vihang Karajgaonkar (JIRA) 
Vihang Karajgaonkar created HIVE-19344:
--

 Summary: Change default value of msck.repair.batch.size 
 Key: HIVE-19344
 URL: https://issues.apache.org/jira/browse/HIVE-19344
 Project: Hive
  Issue Type: Improvement
Reporter: Vihang Karajgaonkar
Assignee: Vihang Karajgaonkar


{{msck.repair.batch.size}} default to 0 which means msck will try to add all 
the partitions in one API call to HMS. This can potentially add huge memory 
pressure on HMS. The default value should be changed to a reasonable number so 
that in case of large number of partitions we can batch the addition of 
partitions. Same goes for {{msck.repair.batch.max.retries}}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 66290: HIVE-14388 : Add number of rows inserted message after insert command in Beeline

2018-04-27 Thread Bharathkrishna Guruvayoor Murali via Review Board 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66290/
---

(Updated April 27, 2018, 11:05 p.m.)


Review request for hive, Sahil Takiar and Vihang Karajgaonkar.


Bugs: HIVE-14388
https://issues.apache.org/jira/browse/HIVE-14388


Repository: hive-git


Description
---

Currently, when you run insert command on beeline, it returns a message saying 
"No rows affected .."
A better and more intuitive msg would be "xxx rows inserted (26.068 seconds)"

Added the numRows parameter as part of QueryState.
Adding the numRows to the response as well to display in beeline.

Getting the count in FileSinkOperator and setting it in statsMap, when it 
operates only on table specific rows for the particular operation. (so that we 
can get only the insert to table count and avoid counting non-table specific 
file-sink operations happening during query execution).


Diffs (updated)
-

  jdbc/src/java/org/apache/hive/jdbc/HiveStatement.java 
06542cee02e5dc4696f2621bb45cc4f24c67dfda 
  ql/src/java/org/apache/hadoop/hive/ql/Driver.java 
41ad002abf3d2a6969ef0d1d48f7db22e096bb47 
  ql/src/java/org/apache/hadoop/hive/ql/MapRedStats.java 
cf9c2273159c0d779ea90ad029613678fb0967a6 
  ql/src/java/org/apache/hadoop/hive/ql/QueryState.java 
706c9ffa48b9c3b4a6fdaae78bab1d39c3d0efda 
  ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java 
c084fa054cb771bfdb033d244935713e3c7eb874 
  ql/src/java/org/apache/hadoop/hive/ql/exec/mr/HadoopJobExecHelper.java 
fcdc9967f12a454a9d3f31031e2261f264479118 
  service-rpc/if/TCLIService.thrift 30f8af7f3e6e0598b410498782900ac27971aef0 
  service-rpc/src/gen/thrift/gen-cpp/TCLIService_types.h 
4321ad6d3c966d30f7a69552f91804cf2f1ba6c4 
  service-rpc/src/gen/thrift/gen-cpp/TCLIService_types.cpp 
b2b62c71492b844f4439367364c5c81aa62f3908 
  
service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TGetOperationStatusResp.java
 15e8220eb3eb12b72c7b64029410dced33bc0d72 
  service-rpc/src/gen/thrift/gen-php/Types.php 
abb7c1ff3a2c8b72dc97689758266b675880e32b 
  service-rpc/src/gen/thrift/gen-py/TCLIService/ttypes.py 
0f8fd0745be0f4ed9e96b7bbe0f092d03649bcdf 
  service-rpc/src/gen/thrift/gen-rb/t_c_l_i_service_types.rb 
60183dae9e9927bd09a9676e49eeb4aea2401737 
  service/src/java/org/apache/hive/service/cli/CLIService.java 
c9914ba9bf8653cbcbca7d6612e98a64058c0fcc 
  service/src/java/org/apache/hive/service/cli/OperationStatus.java 
52cc3ae4f26b990b3e4edb52d9de85b3cc25f269 
  service/src/java/org/apache/hive/service/cli/operation/Operation.java 
3706c72abc77ac8bd77947cc1c5d084ddf965e9f 
  service/src/java/org/apache/hive/service/cli/thrift/ThriftCLIService.java 
c64c99120ad21ee98af81ec6659a2722e3e1d1c7 


Diff: https://reviews.apache.org/r/66290/diff/5/

Changes: https://reviews.apache.org/r/66290/diff/4-5/


Testing
---


Thanks,

Bharathkrishna Guruvayoor Murali

Re: Review Request 66645: HIVE-19211: New streaming ingest API and support for dynamic partitioning

2018-04-27 Thread j . prasanth . j 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66645/
---

(Updated April 27, 2018, 9:36 p.m.)


Review request for hive, Ashutosh Chauhan and Eugene Koifman.


Changes
---

Rebased patch.


Bugs: HIVE-19211
https://issues.apache.org/jira/browse/HIVE-19211


Repository: hive-git


Description
---

HIVE-19211: New streaming ingest API and support for dynamic partitioning


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 6e35653 
  itests/hive-unit/pom.xml 3ae7f2f 
  
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/TestCompactor.java
 4ebd096 
  metastore/src/java/org/apache/hadoop/hive/metastore/HiveClientCache.java 
PRE-CREATION 
  metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStoreUtils.java 
a66c135 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcRecordUpdater.java 09f8802 
  ql/src/java/org/apache/hadoop/hive/ql/lockmgr/DbTxnManager.java 76569d5 
  ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java 4661881 
  serde/src/java/org/apache/hadoop/hive/serde2/JsonSerDe.java PRE-CREATION 
  streaming/pom.xml b58ec01 
  streaming/src/java/org/apache/hive/streaming/AbstractRecordWriter.java 
25998ae 
  streaming/src/java/org/apache/hive/streaming/ConnectionError.java 668bffb 
  streaming/src/java/org/apache/hive/streaming/ConnectionInfo.java PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/DelimitedInputWriter.java 
898b3f9 
  streaming/src/java/org/apache/hive/streaming/HeartBeatFailure.java b1f9520 
  streaming/src/java/org/apache/hive/streaming/HiveEndPoint.java b04e137 
  streaming/src/java/org/apache/hive/streaming/HiveStreamingConnection.java 
PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/ImpersonationFailed.java 23e17e7 
  streaming/src/java/org/apache/hive/streaming/InvalidColumn.java 0011b14 
  streaming/src/java/org/apache/hive/streaming/InvalidPartition.java f1f9804 
  streaming/src/java/org/apache/hive/streaming/InvalidTable.java ef1c91d 
  streaming/src/java/org/apache/hive/streaming/InvalidTransactionState.java 
PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/InvalidTrasactionState.java 
762f5f8 
  streaming/src/java/org/apache/hive/streaming/PartitionCreationFailed.java 
5f9aca6 
  streaming/src/java/org/apache/hive/streaming/PartitionHandler.java 
PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/PartitionInfo.java PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/QueryFailedException.java 
ccd3ae0 
  streaming/src/java/org/apache/hive/streaming/RecordWriter.java dc6d70e 
  streaming/src/java/org/apache/hive/streaming/SerializationError.java a57ba00 
  streaming/src/java/org/apache/hive/streaming/StreamingConnection.java 2f760ea 
  streaming/src/java/org/apache/hive/streaming/StreamingException.java a7f84c1 
  streaming/src/java/org/apache/hive/streaming/StreamingIOFailure.java 0dfbfa7 
  streaming/src/java/org/apache/hive/streaming/StrictDelimitedInputWriter.java 
PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/StrictJsonWriter.java 0077913 
  streaming/src/java/org/apache/hive/streaming/StrictRegexWriter.java c0b7324 
  streaming/src/java/org/apache/hive/streaming/TransactionBatch.java 2b05771 
  streaming/src/java/org/apache/hive/streaming/TransactionBatchUnAvailable.java 
a8c8cd4 
  streaming/src/java/org/apache/hive/streaming/TransactionError.java a331b20 
  streaming/src/test/org/apache/hive/streaming/TestDelimitedInputWriter.java 
f0843a1 
  streaming/src/test/org/apache/hive/streaming/TestStreaming.java 3343d10 
  
streaming/src/test/org/apache/hive/streaming/TestStreamingDynamicPartitioning.java
 PRE-CREATION 


Diff: https://reviews.apache.org/r/66645/diff/9/

Changes: https://reviews.apache.org/r/66645/diff/8-9/


Testing
---


Thanks,

Prasanth_J

Review Request 66857: Replication: The file uris being dumped should contain information about the uri of the source cluster's cm root

2018-04-27 Thread Vaibhav Gumashta 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66857/
---

Review request for hive and Thejas Nair.


Bugs: HIVE-19343
https://issues.apache.org/jira/browse/HIVE-19343


Repository: hive-git


Description
---

https://issues.apache.org/jira/browse/HIVE-19343


Diffs
-

  
itests/hive-unit/src/test/java/org/apache/hadoop/hive/metastore/TestReplChangeManager.java
 6ade76d0c2 
  ql/src/java/org/apache/hadoop/hive/ql/exec/ReplCopyTask.java de270cfcdb 
  
ql/src/java/org/apache/hadoop/hive/ql/parse/repl/load/message/CreateFunctionHandler.java
 f7c90409b7 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/ReplChangeManager.java
 7c1d5f5cca 


Diff: https://reviews.apache.org/r/66857/diff/1/


Testing
---


Thanks,

Vaibhav Gumashta

[jira] [Created] (HIVE-19343) Replication: The file uris being dumped should contain information about the uri of the source cluster's cm root

2018-04-27 Thread Vaibhav Gumashta (JIRA) 
Vaibhav Gumashta created HIVE-19343:
---

 Summary: Replication: The file uris being dumped should contain 
information about the uri of the source cluster's cm root
 Key: HIVE-19343
 URL: https://issues.apache.org/jira/browse/HIVE-19343
 Project: Hive
  Issue Type: Bug
  Components: repl
Affects Versions: 3.0.0, 3.1.0
Reporter: Vaibhav Gumashta


In replication v2, we use change manager (the location is specified by cmroot: 
{{hive.repl.cmrootdir}}) to archive deleted files from the source cluster so 
that they can later be copied on the target cluster. When files are read from 
the cmroot, the target needs to know the appropriate file system. This patch 
adds the fs information of the cmroot on the source to the filenames that get 
written in the repldump command.  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 66788: HIVE-19282 don't nest delta directories inside LB directories for ACID tables

2018-04-27 Thread Sergey Shelukhin 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66788/
---

(Updated April 27, 2018, 8:50 p.m.)


Review request for hive, Prasanth_J and Seong (Steve) Yeom.


Repository: hive-git


Description
---

see jira


Diffs (updated)
-

  ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java c084fa054c 
  ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 25035433c7 
  ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java 4661881301 
  ql/src/test/queries/clientpositive/mm_all.q 4ffbb6b98a 


Diff: https://reviews.apache.org/r/66788/diff/3/

Changes: https://reviews.apache.org/r/66788/diff/2-3/


Testing
---


Thanks,

Sergey Shelukhin

[jira] [Created] (HIVE-19342) Update Wiki with new murmur hash UDF

2018-04-27 Thread Deepak Jaiswal (JIRA) 
Deepak Jaiswal created HIVE-19342:
-

 Summary: Update Wiki with new murmur hash UDF
 Key: HIVE-19342
 URL: https://issues.apache.org/jira/browse/HIVE-19342
 Project: Hive
  Issue Type: Bug
Reporter: Deepak Jaiswal
Assignee: Deepak Jaiswal






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 66567: Migrate to Murmur hash for shuffle and bucketing

2018-04-27 Thread Jason Dere 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66567/#review202074
---




ql/src/java/org/apache/hadoop/hive/ql/optimizer/metainfo/annotation/OpTraitsRulesProcFactory.java
Line 341 (original), 352 (patched)


Can you just add a comment here describing why it is ok to hardcode 
bucketing version to 2 here?


- Jason Dere


On April 27, 2018, 1:14 a.m., Deepak Jaiswal wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66567/
> ---
> 
> (Updated April 27, 2018, 1:14 a.m.)
> 
> 
> Review request for hive, Ashutosh Chauhan, Eugene Koifman, Jason Dere, and 
> Matt McCline.
> 
> 
> Bugs: HIVE-18910
> https://issues.apache.org/jira/browse/HIVE-18910
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Hive uses JAVA hash which is not as good as murmur for better distribution 
> and efficiency in bucketing a table.
> Migrate to murmur hash but still keep backward compatibility for existing 
> users so that they dont have to reload the existing tables.
> 
> To keep backward compatibility, bucket_version is added as a table property, 
> resulting in high number of result updates.
> 
> 
> Diffs
> -
> 
>   hbase-handler/src/test/results/positive/external_table_ppd.q.out cdc43ee560 
>   hbase-handler/src/test/results/positive/hbase_binary_storage_queries.q.out 
> 153613e6d0 
>   hbase-handler/src/test/results/positive/hbase_ddl.q.out ef3f5f704e 
>   hbase-handler/src/test/results/positive/hbasestats.q.out 5d000d2f4f 
>   
> hcatalog/streaming/src/java/org/apache/hive/hcatalog/streaming/AbstractRecordWriter.java
>  924e233293 
>   
> hcatalog/streaming/src/test/org/apache/hive/hcatalog/streaming/TestStreaming.java
>  fe2b1c1f3c 
>   
> hcatalog/streaming/src/test/org/apache/hive/hcatalog/streaming/mutate/worker/TestBucketIdResolverImpl.java
>  03c28a33c8 
>   
> hcatalog/webhcat/java-client/src/main/java/org/apache/hive/hcatalog/api/HCatTable.java
>  996329195c 
>   
> hcatalog/webhcat/java-client/src/test/java/org/apache/hive/hcatalog/api/TestHCatClient.java
>  f9ee9d9a03 
>   
> itests/hive-blobstore/src/test/results/clientpositive/insert_into_dynamic_partitions.q.out
>  caa00292b8 
>   
> itests/hive-blobstore/src/test/results/clientpositive/insert_into_table.q.out 
> ab8ad77074 
>   
> itests/hive-blobstore/src/test/results/clientpositive/insert_overwrite_directory.q.out
>  2b28a6677e 
>   
> itests/hive-blobstore/src/test/results/clientpositive/insert_overwrite_dynamic_partitions.q.out
>  cdb67dd786 
>   
> itests/hive-blobstore/src/test/results/clientpositive/insert_overwrite_table.q.out
>  2c23a7e94f 
>   
> itests/hive-blobstore/src/test/results/clientpositive/write_final_output_blobstore.q.out
>  a1be085ea5 
>   
> itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/TestCompactor.java
>  82ba775286 
>   itests/src/test/resources/testconfiguration.properties 1a346593fd 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java c084fa054c 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java d59bf1fb6e 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/Operator.java c28ef99621 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/OperatorFactory.java 21ca04d78a 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/ReduceSinkOperator.java 
> d4363fdf91 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 25035433c7 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/repl/bootstrap/load/table/LoadPartitions.java
>  a42c299537 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/keyseries/VectorKeySeriesSerializedImpl.java
>  86f466fc4e 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/reducesink/VectorReduceSinkObjectHashOperator.java
>  1bc3fdabac 
>   ql/src/java/org/apache/hadoop/hive/ql/io/HiveFileFormatUtils.java 
> 71498a125c 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java 019682fb10 
>   ql/src/java/org/apache/hadoop/hive/ql/metadata/Table.java a51fdd322f 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/ConvertJoinMapJoin.java 
> 7121bceb22 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/FixedBucketPruningOptimizer.java
>  5f65f638ca 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/PrunerOperatorFactory.java 
> 2be3c9b9a2 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/SortedDynPartitionOptimizer.java
>  1c5656267d 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/SortedDynPartitionTimeGranularityOptimizer.java
>  0e995d79d2 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/metainfo/annotation/OpTraitsRulesProcFactory.java
>  69d9f3125a 
>

Re: [VOTE] Should we release storage-api 2.6.0 rc0?

2018-04-27 Thread Alan Gates 
+1 Did a build in a clean mvn repo, ran rat, looked over NOTICE and LICENSE
files.

On Fri, Apr 27, 2018 at 8:53 AM, Jesus Camacho Rodriguez <
jcama...@apache.org> wrote:

> +1
> - compiled from src
> - ran unit tests
> - ran rat
>
> -Jesús
>
>
>
> On 4/26/18, 8:30 AM, "Owen O'Malley"  wrote:
>
> All,
>I'd like to make a new release of the storage-api.
>
> Artifacts:
> tag: https://github.com/apache/hive/releases/tag/storage-
> release-2.6.0-rc0
> tar ball: http://home.apache.org/~omalley/storage-2.6.0/
>
> Thanks,
>Owen
>
>
>
>

Re: [VOTE] Should we release storage-api 2.6.0 rc0?

2018-04-27 Thread Jesus Camacho Rodriguez 
+1
- compiled from src
- ran unit tests
- ran rat

-Jesús



On 4/26/18, 8:30 AM, "Owen O'Malley"  wrote:

All,
   I'd like to make a new release of the storage-api.

Artifacts:
tag: https://github.com/apache/hive/releases/tag/storage-release-2.6.0-rc0
tar ball: http://home.apache.org/~omalley/storage-2.6.0/

Thanks,
   Owen

Requesting write permission on Hive wiki page

2018-04-27 Thread Adam Szita 
Hi,

I'd like to request an edit permission for the Hive Confluence page please.
My user ID is: szita

Thanks,
Adam

Re: ptest queue

2018-04-27 Thread Adam Szita 
Thanks to all for the responses.
As I see it, option 3 is the winning one. Next week I'm going start working
on this one then (unless any objections of course).

Adam

On 26 April 2018 at 05:48, Deepak Jaiswal  wrote:

> +1 for option 3. Thanks Adam for taking this up again.
>
> Regards,
> Deepak
>
> On 4/25/18, 4:54 PM, "Thejas Nair"  wrote:
>
> Option 3 seems reasonable. I believe that used to be the state a while
> back (maybe 12 months back or so).
> When 2nd ptest for same jira runs, it checks if the latest patch has
> already been run.
>
>
> On Wed, Apr 25, 2018 at 7:37 AM, Peter Vary 
> wrote:
> > I would vote for version 3. It would solve the big patch problem,
> and removes the unnecessary test runs too.
> >
> > Thanks,
> > Peter
> >
> >> On Apr 25, 2018, at 11:01 AM, Adam Szita 
> wrote:
> >>
> >> Hi all,
> >>
> >> I had a patch (HIVE-19077) committed with the original aim being the
> >> prevention of wasting resources when running ptest on the same patch
> >> multiple times:
> >> It is supposed to manage scenarios where a developer uploads
> >> HIVE-XYZ.1.patch, that gets queued in jenkins, then before execution
> >> HIVE-XYZ.2.patch (for the same jira) is uploaded and that gets
> queued also.
> >> When the first patch starts to execute ptest will see that patch2
> is the
> >> latest patch and will use that. After some time the second queued
> job will
> >> also run on this very same patch.
> >> This is just pointless and causes long queues to progress slowly.
> >>
> >> My idea was to remove these duplicates from the queue where I'd
> only keep
> >> the latest queued element if I see more queued entries for the same
> jira
> >> number. It's like when you go grocery shopping and you're already
> in line
> >> at cashier but you realise you also need e.g. milk. You go grab it
> and join
> >> the END of the queue. So I believe it's a fair punishment for
> losing one's
> >> spot in the queue for making amends on their patch.
> >>
> >> That said Deepak made me realise that for big patches this will be
> very
> >> cumbersome due to the need of constant rebasing to avoid conflicts
> on patch
> >> application.
> >> I have three proposals now:
> >>
> >> 1: Leave this as it currently is (with HIVE-19077 committed) -
> *only the
> >> latest queued job will run of the same jira*
> >> pros: no wasting resources to run the same patches more times,
> 'scheduling'
> >> is fair: if you amend you're patch you may loose your original spot
> in the
> >> queue
> >> cons: big patches that are prone to conflicts will be hard to get
> executed
> >> in ptest, devs will have to wait more time for their ptest results
> if they
> >> amend their patches
> >>
> >> 2: *Add a safety switch* to this queue checking feature (currently
> proposed
> >> in HIVE-19077), deduplication can be switch off on request
> >> pros: same as 1st, + ability to have more control on this mechanism
> i.e.
> >> turn it off for big/urgent patches
> >> cons: big patches that use the swich might still waste resources,
> also devs
> >> might use safety switch inappropriately for their own evil benefit
> :)
> >>
> >> 3: Deduplication the other way around - *only the first queued job
> will run
> >> of the same jira*, ptest server will keep record of patch names and
> won't
> >> execute a patch with a seen name and jira number again
> >> pros: same patches will not be executed more times accidentally, big
> >> patches won't be a problem either, devs will get their ptest result
> back
> >> earlier even if more jobs are triggered for same jira/patch name
> >> cons: scheduling is less fair: devs can reserve their spots in the
> queue
> >>
> >>
> >> (0: restore original: I'm strongly against this, ptest queue is
> already too
> >> big as it is, we have to at least try and decrease its size by
> >> deduplicating jiras in it)
> >>
> >> I'm personally fine with any of the 1,2,3 methods listed above,
> with my
> >> favourites being 2 and 3.
> >> Let me know which one you think is the right path to go down on.
> >>
> >> Thanks,
> >> Adam
> >>
> >> On 20 April 2018 at 20:14, Eugene Koifman 
> wrote:
> >>
> >>> Would it be possible to add patch name validation when it gets
> added to
> >>> the queue?
> >>> Currently I think it fails when the bot gets to the patch if it’s
> not
> >>> named correctly.
> >>> More  common for branch patches
> >>>
> >>> On 4/20/18, 8:20 AM, "Zoltan Haindrich"  wrote:
> >>>
> >>>Hello,
> >>>
> >>>Some time ago the ptest queue worked the following way:
> >>>
>

Review Request 66848: HIVE-19211: New streaming ingest API and support for dynamic partitioning

2018-04-27 Thread j . prasanth . j 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66848/
---

Review request for hive and Gopal V.


Bugs: HIVE-19211
https://issues.apache.org/jira/browse/HIVE-19211


Repository: hive-git


Description
---

HIVE-19206: Automatic memory management for open streaming writers


Diffs
-

  common/src/java/org/apache/hadoop/hive/common/HeapMemoryMonitor.java 
PRE-CREATION 
  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 
f40c60606ca704d3f8c434c83107d63652ad359b 
  itests/hive-unit/pom.xml 3ae7f2feeec2df8156f89bdc204e2c35976485e2 
  
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/TestCompactor.java
 82ba77528678d01517dec58c3dc5d6b07077a965 
  metastore/src/java/org/apache/hadoop/hive/metastore/HiveClientCache.java 
PRE-CREATION 
  metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStoreUtils.java 
a66c13507abef42977dfdb315ff7d69404f67ac3 
  ql/src/java/org/apache/hadoop/hive/ql/io/RecordUpdater.java 
0aed172b6978847cdefe97878d2bc91648e37e0b 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcOutputFormat.java 
e69d1a06e2e001a6d706c3e6d92605c5d6a3d678 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcRecordUpdater.java 
09f880221ce74cbe5477e6d514b4a6915532a491 
  ql/src/java/org/apache/hadoop/hive/ql/lockmgr/DbTxnManager.java 
7b7fd5d19842ec691390880dcd1366539e9a6993 
  ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java 
466188130184516459dbd307e9cd0cf22844b46d 
  ql/src/test/org/apache/hadoop/hive/ql/exec/TestFileSinkOperator.java 
7f6077c460504a9a6bf2404e07905a0f6b0da654 
  serde/src/java/org/apache/hadoop/hive/serde2/JsonSerDe.java PRE-CREATION 
  streaming/pom.xml b58ec015aa69e29aacdc0a165ead9439ea2e4b26 
  streaming/src/java/org/apache/hive/streaming/AbstractRecordWriter.java 
25998ae31a3a829aab45f9e526aa03d94feff5e0 
  streaming/src/java/org/apache/hive/streaming/ConnectionError.java 
668bffb1ab17558dec33d599bddd6e28a06b3c5a 
  streaming/src/java/org/apache/hive/streaming/ConnectionInfo.java PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/ConnectionStats.java 
PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/DelimitedInputWriter.java 
898b3f9bb1d1c483cae8c1dd4f2338fc453d514b 
  streaming/src/java/org/apache/hive/streaming/HeartBeatFailure.java 
b1f9520814d260a3d2df23e6050e72d803874da9 
  streaming/src/java/org/apache/hive/streaming/HiveEndPoint.java 
b04e13784485ca097153bbec86f80d22e15e5cdc 
  streaming/src/java/org/apache/hive/streaming/HiveStreamingConnection.java 
PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/ImpersonationFailed.java 
23e17e76237036d8f9419bef2255f4f82c5b18a1 
  streaming/src/java/org/apache/hive/streaming/InvalidColumn.java 
0011b1454f8815816be931bf67cc13e7e78c9c0d 
  streaming/src/java/org/apache/hive/streaming/InvalidPartition.java 
f1f980430f3aceeb044bb549cc1a37a33c144750 
  streaming/src/java/org/apache/hive/streaming/InvalidTable.java 
ef1c91dbeb84b325b019318122fdd1f45b927414 
  streaming/src/java/org/apache/hive/streaming/InvalidTrasactionState.java 
762f5f86fc0df4a59cb54812a5dc79c1e2bc9489 
  streaming/src/java/org/apache/hive/streaming/PartitionCreationFailed.java 
5f9aca66ea0f2a7b2c3d2f6fb805fa1760b69e44 
  streaming/src/java/org/apache/hive/streaming/PartitionInfo.java PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/QueryFailedException.java 
ccd3ae0c98ea6ced0290f1ab027ad6337453fca2 
  streaming/src/java/org/apache/hive/streaming/RecordWriter.java 
dc6d70e92438e037d764099c82f5f654d5f5d801 
  streaming/src/java/org/apache/hive/streaming/SerializationError.java 
a57ba00ba401283aedd3f685171ef6bd810b11cd 
  streaming/src/java/org/apache/hive/streaming/StreamingConnection.java 
2f760ea86eecbbc96db08509405a369abf7d89d5 
  streaming/src/java/org/apache/hive/streaming/StreamingException.java 
a7f84c14f30f2e4753bd99b3d2d1dcb236b0197b 
  streaming/src/java/org/apache/hive/streaming/StreamingIOFailure.java 
0dfbfa71c50215d8f3e25298c8d11634a3cbedc4 
  streaming/src/java/org/apache/hive/streaming/StrictDelimitedInputWriter.java 
PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/StrictJsonWriter.java 
0077913cd1f0afbafe4608c4378398f61e254424 
  streaming/src/java/org/apache/hive/streaming/StrictRegexWriter.java 
c0b732482d35305ceaba1adfff09659e193ab098 
  streaming/src/java/org/apache/hive/streaming/TransactionBatch.java 
2b057718f58dec6de3e2b329a43bb5a06ce7c9ed 
  streaming/src/java/org/apache/hive/streaming/TransactionBatchUnAvailable.java 
a8c8cd48726421003df186fa1e0c2ecd18bdd5b4 
  streaming/src/java/org/apache/hive/streaming/TransactionError.java 
a331b20463e8328148fb08d85cf3ce77a7463062 
  streaming/src/test/org/apache/hive/streaming/TestDelimitedInputWriter.java 
f0843a1748d956ea99dd4807cf0b4ffbe0ef9cba 
  streaming/src/test/org/apache/hive/streaming/TestStreaming.java

Re: Review Request 66848: HIVE-19211: New streaming ingest API and support for dynamic partitioning

2018-04-27 Thread j . prasanth . j 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66848/
---

(Updated April 27, 2018, 9:43 a.m.)


Review request for hive and Gopal V.


Bugs: HIVE-19206
https://issues.apache.org/jira/browse/HIVE-19206


Repository: hive-git


Description
---

HIVE-19206: Automatic memory management for open streaming writers


Diffs
-

  common/src/java/org/apache/hadoop/hive/common/HeapMemoryMonitor.java 
PRE-CREATION 
  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 
f40c60606ca704d3f8c434c83107d63652ad359b 
  itests/hive-unit/pom.xml 3ae7f2feeec2df8156f89bdc204e2c35976485e2 
  
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/TestCompactor.java
 82ba77528678d01517dec58c3dc5d6b07077a965 
  metastore/src/java/org/apache/hadoop/hive/metastore/HiveClientCache.java 
PRE-CREATION 
  metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStoreUtils.java 
a66c13507abef42977dfdb315ff7d69404f67ac3 
  ql/src/java/org/apache/hadoop/hive/ql/io/RecordUpdater.java 
0aed172b6978847cdefe97878d2bc91648e37e0b 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcOutputFormat.java 
e69d1a06e2e001a6d706c3e6d92605c5d6a3d678 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcRecordUpdater.java 
09f880221ce74cbe5477e6d514b4a6915532a491 
  ql/src/java/org/apache/hadoop/hive/ql/lockmgr/DbTxnManager.java 
7b7fd5d19842ec691390880dcd1366539e9a6993 
  ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java 
466188130184516459dbd307e9cd0cf22844b46d 
  ql/src/test/org/apache/hadoop/hive/ql/exec/TestFileSinkOperator.java 
7f6077c460504a9a6bf2404e07905a0f6b0da654 
  serde/src/java/org/apache/hadoop/hive/serde2/JsonSerDe.java PRE-CREATION 
  streaming/pom.xml b58ec015aa69e29aacdc0a165ead9439ea2e4b26 
  streaming/src/java/org/apache/hive/streaming/AbstractRecordWriter.java 
25998ae31a3a829aab45f9e526aa03d94feff5e0 
  streaming/src/java/org/apache/hive/streaming/ConnectionError.java 
668bffb1ab17558dec33d599bddd6e28a06b3c5a 
  streaming/src/java/org/apache/hive/streaming/ConnectionInfo.java PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/ConnectionStats.java 
PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/DelimitedInputWriter.java 
898b3f9bb1d1c483cae8c1dd4f2338fc453d514b 
  streaming/src/java/org/apache/hive/streaming/HeartBeatFailure.java 
b1f9520814d260a3d2df23e6050e72d803874da9 
  streaming/src/java/org/apache/hive/streaming/HiveEndPoint.java 
b04e13784485ca097153bbec86f80d22e15e5cdc 
  streaming/src/java/org/apache/hive/streaming/HiveStreamingConnection.java 
PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/ImpersonationFailed.java 
23e17e76237036d8f9419bef2255f4f82c5b18a1 
  streaming/src/java/org/apache/hive/streaming/InvalidColumn.java 
0011b1454f8815816be931bf67cc13e7e78c9c0d 
  streaming/src/java/org/apache/hive/streaming/InvalidPartition.java 
f1f980430f3aceeb044bb549cc1a37a33c144750 
  streaming/src/java/org/apache/hive/streaming/InvalidTable.java 
ef1c91dbeb84b325b019318122fdd1f45b927414 
  streaming/src/java/org/apache/hive/streaming/InvalidTrasactionState.java 
762f5f86fc0df4a59cb54812a5dc79c1e2bc9489 
  streaming/src/java/org/apache/hive/streaming/PartitionCreationFailed.java 
5f9aca66ea0f2a7b2c3d2f6fb805fa1760b69e44 
  streaming/src/java/org/apache/hive/streaming/PartitionInfo.java PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/QueryFailedException.java 
ccd3ae0c98ea6ced0290f1ab027ad6337453fca2 
  streaming/src/java/org/apache/hive/streaming/RecordWriter.java 
dc6d70e92438e037d764099c82f5f654d5f5d801 
  streaming/src/java/org/apache/hive/streaming/SerializationError.java 
a57ba00ba401283aedd3f685171ef6bd810b11cd 
  streaming/src/java/org/apache/hive/streaming/StreamingConnection.java 
2f760ea86eecbbc96db08509405a369abf7d89d5 
  streaming/src/java/org/apache/hive/streaming/StreamingException.java 
a7f84c14f30f2e4753bd99b3d2d1dcb236b0197b 
  streaming/src/java/org/apache/hive/streaming/StreamingIOFailure.java 
0dfbfa71c50215d8f3e25298c8d11634a3cbedc4 
  streaming/src/java/org/apache/hive/streaming/StrictDelimitedInputWriter.java 
PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/StrictJsonWriter.java 
0077913cd1f0afbafe4608c4378398f61e254424 
  streaming/src/java/org/apache/hive/streaming/StrictRegexWriter.java 
c0b732482d35305ceaba1adfff09659e193ab098 
  streaming/src/java/org/apache/hive/streaming/TransactionBatch.java 
2b057718f58dec6de3e2b329a43bb5a06ce7c9ed 
  streaming/src/java/org/apache/hive/streaming/TransactionBatchUnAvailable.java 
a8c8cd48726421003df186fa1e0c2ecd18bdd5b4 
  streaming/src/java/org/apache/hive/streaming/TransactionError.java 
a331b20463e8328148fb08d85cf3ce77a7463062 
  streaming/src/test/org/apache/hive/streaming/TestDelimitedInputWriter.java 
f0843a1748d956ea99dd4807cf0b4ffbe0ef9cba

Re: Review Request 66805: HIVE-19311 : Partition and bucketing support for “load data” statement

2018-04-27 Thread Deepak Jaiswal 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66805/
---

(Updated April 27, 2018, 7:44 a.m.)


Review request for hive, Ashutosh Chauhan, Eugene Koifman, Jesús Camacho 
Rodríguez, and Vineet Garg.


Changes
---

fixed a bug causing output entities missed from POSTHOOKs.


Bugs: HIVE-19311
https://issues.apache.org/jira/browse/HIVE-19311


Repository: hive-git


Description
---

Currently, "load data" statement is very limited. It errors out if any of the 
information is missing such as partitioning info if table is partitioned or 
appropriate names when table is bucketed.
It should be able to launch an insert job to load the data instead.


Diffs (updated)
-

  data/files/load_data_job/bucketing.txt PRE-CREATION 
  data/files/load_data_job/partitions/load_data_1_partition.txt PRE-CREATION 
  data/files/load_data_job/partitions/load_data_2_partitions.txt PRE-CREATION 
  itests/src/test/resources/testconfiguration.properties 1a346593fd 
  ql/src/java/org/apache/hadoop/hive/ql/Context.java 70846ac3ce 
  ql/src/java/org/apache/hadoop/hive/ql/ErrorMsg.java 7d33fa3892 
  ql/src/java/org/apache/hadoop/hive/ql/metadata/Table.java a51fdd322f 
  ql/src/java/org/apache/hadoop/hive/ql/parse/LoadSemanticAnalyzer.java 
c07991d434 
  ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 1dccf969ff 
  ql/src/java/org/apache/hadoop/hive/ql/parse/UpdateDeleteSemanticAnalyzer.java 
2f3b07f4af 
  ql/src/test/queries/clientpositive/load_data_using_job.q PRE-CREATION 
  ql/src/test/results/clientpositive/llap/load_data_using_job.q.out 
PRE-CREATION 
  ql/src/test/results/clientpositive/spark/load_data_using_job.q.out 
PRE-CREATION 


Diff: https://reviews.apache.org/r/66805/diff/2/

Changes: https://reviews.apache.org/r/66805/diff/1-2/


Testing
---

Added a unit test.


Thanks,

Deepak Jaiswal

Re: Metastore thrift client to get the table location on hdfs

2018-04-27 Thread 侯宗田 
Thanks for your advice, I have found the functions in ThriftHiveMetastore.h. 
That is precisely what I need!! 
> 在 2018年4月26日，下午11:16，Alan Gates  写道：
> 
> The calls you want are in ThriftHiveMetastore.h.  You can use these to get
> lists of tables and objects representing each table.  One of the fields in
> the Table object is a StorageDescriptor, which has a field called
> 'location'.  If you are working with partitioned tables the location
> information for each partition will be in the associated Partition object
> rather than in the Table object.
> 
> Alan.
> 
> On Thu, Apr 26, 2018 at 5:23 AM, 侯宗田  wrote:
> 
>> Hi,
>> 
>> I am writing a simple thrift HMS client to get table location on hdfs, I
>> generate the cpp code from hive_metastore.thrift, Can someone give me some
>> guide of how to do it next, the info I searched on the internet is too
>> abstract and not to my need precisely.

Re: Metastore thrift client to get the table location on hdfs

2018-04-27 Thread 侯宗田 
Thank you, I have found the functions in ThriftHiveMetastore.h, I should get 
the table, then its storage descriptor, then the location, just like Mr.Gates 
said. Since this function I am writing will be called in C project, C/C++ is 
more convenient.
> 在 2018年4月27日，上午1:42，Alexander Kolbasov  写道：
> 
> Do you need to do this from C/C++? I have a set of tools that you can use
> to talk to HMS via thrift using Java, Python or Go.
> 
> On Thu, Apr 26, 2018 at 5:23 AM, 侯宗田  wrote:
> 
>> Hi,
>> 
>> I am writing a simple thrift HMS client to get table location on hdfs, I
>> generate the cpp code from hive_metastore.thrift, Can someone give me some
>> guide of how to do it next, the info I searched on the internet is too
>> abstract and not to my need precisely.