from:"j . prasanth . j"

Re: Review Request 71555: Incompatible java.util.ArrayList for java 11

2019-10-01 Thread j . prasanth . j


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71555/#review218010
---


Ship it!




Ship It!

- Prasanth_J


On Sept. 30, 2019, 7:23 a.m., Attila Magyar wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71555/
> ---
> 
> (Updated Sept. 30, 2019, 7:23 a.m.)
> 
> 
> Review request for hive, Laszlo Bodor, Ashutosh Chauhan, and Prasanth_J.
> 
> 
> Bugs: HIVE-22097
> https://issues.apache.org/jira/browse/HIVE-22097
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> The following exceptions come when running a query on Java 11:
> 
> java.lang.RuntimeException: java.lang.NoSuchFieldException: parentOffset
>   at 
> org.apache.hadoop.hive.ql.exec.SerializationUtilities$ArrayListSubListSerializer.(SerializationUtilities.java:390)
>   at 
> org.apache.hadoop.hive.ql.exec.SerializationUtilities$1.create(SerializationUtilities.java:235)
>   at 
> org.apache.hive.com.esotericsoftware.kryo.pool.KryoPoolQueueImpl.borrow(KryoPoolQueueImpl.java:48)
>   at 
> org.apache.hadoop.hive.ql.exec.SerializationUtilities.borrowKryo(SerializationUtilities.java:280)
>   at 
> org.apache.hadoop.hive.ql.exec.Utilities.setBaseWork(Utilities.java:595)
>   at 
> org.apache.hadoop.hive.ql.exec.Utilities.setMapWork(Utilities.java:587)
>   at 
> org.apache.hadoop.hive.ql.exec.Utilities.setMapRedWork(Utilities.java:579)
>   at 
> org.apache.hadoop.hive.ql.exec.mr.ExecDriver.execute(ExecDriver.java:357)
>   at 
> org.apache.hadoop.hive.ql.exec.mr.MapRedTask.execute(MapRedTask.java:159)
>   at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:212)
>   at 
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:103)
>   at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:2317)
>   at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1969)
>   at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1636)
>   at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1396)
>   at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1390)
>   at 
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:162)
>   at 
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:223)
>   at 
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:242)
>   at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:189)
>   at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:408)
>   at 
> org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:838)
>   at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:777)
>   at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:696)
>   at 
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.base/java.lang.reflect.Method.invoke(Method.java:566)
>   at org.apache.hadoop.util.RunJar.run(RunJar.java:323)
>   at org.apache.hadoop.util.RunJar.main(RunJar.java:236)
> Caused by: java.lang.NoSuchFieldException: parentOffset
>   at java.base/java.lang.Class.getDeclaredField(Class.java:2412)
>   at 
> org.apache.hadoop.hive.ql.exec.SerializationUtilities$ArrayListSubListSerializer.(SerializationUtilities.java:384)
>   ... 29 more
> 
> The internal structure of ArrayList$SubList changed and our serializer fails. 
> This serialzier comes from kryo-serializers package where they already 
> updated the code. This patch does the some.
> 
> 
> Diffs
> -
> 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/SerializationUtilities.java 
> e4d33e82168 
> 
> 
> Diff: https://reviews.apache.org/r/71555/diff/1/
> 
> 
> Testing
> ---
> 
> Tested on a real cluster with Java 11.
> 
> 
> Thanks,
> 
> Attila Magyar
> 
>

Review Request 70442: HIVE-21597: WM trigger validation should happen at the time of create or alter

2019-04-10 Thread j . prasanth . j


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/70442/
---

Review request for hive and Daniel Dai.


Bugs: HIVE-21597
https://issues.apache.org/jira/browse/HIVE-21597


Repository: hive-git


Description
---

HIVE-21597: WM trigger validation should happen at the time of create or alter


Diffs
-

  ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 
a1d795fb08b163f016673fbf707b347ba63cf818 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TriggerValidatorRunnable.java 
670184b0ac3ce4b47b3c44b392c54e5a28a9cfed 
  ql/src/test/queries/clientpositive/resourceplan.q 
46aae72a1100f9efd568a606f08242013c6fc016 
  ql/src/test/results/clientpositive/llap/resourceplan.q.out 
9ae68f487f040f06d786a1e4d999f1b41acb6457 
  service/src/java/org/apache/hive/service/server/KillQueryImpl.java 
c7f2c9117b9605882a3de849de592e027dcab484 


Diff: https://reviews.apache.org/r/70442/diff/1/


Testing
---


Thanks,

Prasanth_J

Re: Review Request 70372: HIVE-21427: Syslog storage handler

2019-04-05 Thread j . prasanth . j


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/70372/
---

(Updated April 5, 2019, 11:05 p.m.)


Review request for hive, Ashutosh Chauhan and Jason Dere.


Changes
---

The timestamp change to ConvertAstToSearchArg is incorrect. Orc does not handle 
TIMESTAMP and TIMESTAMPLOCALTZ types correctly. ORC-189 is required. The 
timestamps in syslog all assume UTC (TIMESTAMP type), but expression filters in 
syslog_parser_file_pruning.q gets converted to local TZ (TIMESTAMPLOCALTZ) 
giving incorrect results. Also there is no way to force using UTC for the qfile 
tests. If we pass "-Duser.timezone=UTC" while running qfile it gives correct 
results but then precommit servers cannot run qfile test with 
"-Duser.timezone=UTC" option. We probably need a separate CliDriver for qfile 
tests that runs tests in UTC but that is out of scope for this ticket. I will 
disable syslog_parser_file_pruning.q for now as enabling it will give incorrect 
results depending on which TZ ptest server runs. Also reverted changes to 
ConvertAstToSearchArg.java file as ORC-189 will take care it.


Bugs: HIVE-21427
https://issues.apache.org/jira/browse/HIVE-21427


Repository: hive-git


Description
---

HIVE-21427: Syslog storage handler


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java aa27a01174 
  data/files/syslog-hs2-2.log PRE-CREATION 
  data/files/syslog-hs2-exception.log PRE-CREATION 
  data/files/syslog-hs2.log PRE-CREATION 
  itests/src/test/resources/testconfiguration.properties 96dfbc4b56 
  itests/util/src/main/java/org/apache/hadoop/hive/cli/control/CliConfigs.java 
6f04602acb 
  
llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/LlapRecordReader.java
 9ef7af4eb0 
  ql/src/java/org/apache/hadoop/hive/ql/log/syslog/SyslogInputFormat.java 
PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/log/syslog/SyslogParser.java 
PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/log/syslog/SyslogSerDe.java 
PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/log/syslog/SyslogStorageHandler.java 
PRE-CREATION 
  ql/src/test/org/apache/hadoop/hive/ql/log/TestSyslogInputFormat.java 
PRE-CREATION 
  ql/src/test/queries/clientpositive/syslog_parser.q PRE-CREATION 
  ql/src/test/queries/clientpositive/syslog_parser_exception.q PRE-CREATION 
  ql/src/test/queries/clientpositive/syslog_parser_file_pruning.q PRE-CREATION 
  ql/src/test/results/clientpositive/llap/syslog_parser.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/llap/syslog_parser_exception.q.out 
PRE-CREATION 
  ql/src/test/results/clientpositive/llap/syslog_parser_file_pruning.q.out 
PRE-CREATION 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/tools/schematool/MetastoreSchemaTool.java
 eafe0c6d46 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/tools/schematool/SchemaToolCommandLine.java
 6282078411 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/tools/schematool/SchemaToolTaskCreateLogsTable.java
 PRE-CREATION 


Diff: https://reviews.apache.org/r/70372/diff/5/

Changes: https://reviews.apache.org/r/70372/diff/4-5/


Testing
---


Thanks,

Prasanth_J

Re: Review Request 70372: HIVE-21427: Syslog storage handler

2019-04-05 Thread j . prasanth . j


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/70372/
---

(Updated April 5, 2019, 7:39 p.m.)


Review request for hive, Ashutosh Chauhan and Jason Dere.


Changes
---

SyslogInputFormat extends TextInputFormat, textIF considers CR as new line if 
line delimiter is unspecified as a result multi-line exceptions were not 
returned by query results. .4 patch fixes it and adds a test case.


Bugs: HIVE-21427
https://issues.apache.org/jira/browse/HIVE-21427


Repository: hive-git


Description
---

HIVE-21427: Syslog storage handler


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java aa27a01174 
  data/files/syslog-hs2-2.log PRE-CREATION 
  data/files/syslog-hs2-exception.log PRE-CREATION 
  data/files/syslog-hs2.log PRE-CREATION 
  itests/src/test/resources/testconfiguration.properties 96dfbc4b56 
  
llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/LlapRecordReader.java
 9ef7af4eb0 
  ql/src/java/org/apache/hadoop/hive/ql/io/sarg/ConvertAstToSearchArg.java 
27fe828b75 
  ql/src/java/org/apache/hadoop/hive/ql/log/syslog/SyslogInputFormat.java 
PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/log/syslog/SyslogParser.java 
PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/log/syslog/SyslogSerDe.java 
PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/log/syslog/SyslogStorageHandler.java 
PRE-CREATION 
  ql/src/test/org/apache/hadoop/hive/ql/log/TestSyslogInputFormat.java 
PRE-CREATION 
  ql/src/test/queries/clientpositive/syslog_parser.q PRE-CREATION 
  ql/src/test/queries/clientpositive/syslog_parser_exception.q PRE-CREATION 
  ql/src/test/queries/clientpositive/syslog_parser_file_pruning.q PRE-CREATION 
  ql/src/test/results/clientpositive/llap/syslog_parser.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/llap/syslog_parser_exception.q.out 
PRE-CREATION 
  ql/src/test/results/clientpositive/llap/syslog_parser_file_pruning.q.out 
PRE-CREATION 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/tools/schematool/MetastoreSchemaTool.java
 eafe0c6d46 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/tools/schematool/SchemaToolCommandLine.java
 6282078411 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/tools/schematool/SchemaToolTaskCreateLogsTable.java
 PRE-CREATION 


Diff: https://reviews.apache.org/r/70372/diff/4/

Changes: https://reviews.apache.org/r/70372/diff/3-4/


Testing
---


Thanks,

Prasanth_J

Re: Review Request 70372: HIVE-21427: Syslog storage handler

2019-04-04 Thread j . prasanth . j


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/70372/
---

(Updated April 4, 2019, 9:28 p.m.)


Review request for hive, Ashutosh Chauhan and Jason Dere.


Changes
---

fixes a bug where input row can have data from previous row.


Bugs: HIVE-21427
https://issues.apache.org/jira/browse/HIVE-21427


Repository: hive-git


Description
---

HIVE-21427: Syslog storage handler


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java aa27a01174 
  data/files/syslog-hs2-2.log PRE-CREATION 
  data/files/syslog-hs2.log PRE-CREATION 
  itests/src/test/resources/testconfiguration.properties 96dfbc4b56 
  
llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/LlapRecordReader.java
 9ef7af4eb0 
  ql/src/java/org/apache/hadoop/hive/ql/io/sarg/ConvertAstToSearchArg.java 
27fe828b75 
  ql/src/java/org/apache/hadoop/hive/ql/log/syslog/SyslogInputFormat.java 
PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/log/syslog/SyslogParser.java 
PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/log/syslog/SyslogSerDe.java 
PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/log/syslog/SyslogStorageHandler.java 
PRE-CREATION 
  ql/src/test/org/apache/hadoop/hive/ql/log/TestSyslogInputFormat.java 
PRE-CREATION 
  ql/src/test/queries/clientpositive/syslog_parser.q PRE-CREATION 
  ql/src/test/queries/clientpositive/syslog_parser_file_pruning.q PRE-CREATION 
  ql/src/test/results/clientpositive/llap/syslog_parser.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/llap/syslog_parser_file_pruning.q.out 
PRE-CREATION 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/tools/schematool/MetastoreSchemaTool.java
 eafe0c6d46 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/tools/schematool/SchemaToolCommandLine.java
 6282078411 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/tools/schematool/SchemaToolTaskCreateLogsTable.java
 PRE-CREATION 


Diff: https://reviews.apache.org/r/70372/diff/3/

Changes: https://reviews.apache.org/r/70372/diff/2-3/


Testing
---


Thanks,

Prasanth_J

Re: Review Request 70372: HIVE-21427: Syslog storage handler

2019-04-04 Thread j . prasanth . j


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/70372/
---

(Updated April 4, 2019, 6:18 p.m.)


Review request for hive, Ashutosh Chauhan and Jason Dere.


Changes
---

review comment update


Bugs: HIVE-21427
https://issues.apache.org/jira/browse/HIVE-21427


Repository: hive-git


Description
---

HIVE-21427: Syslog storage handler


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java aa27a01174 
  data/files/syslog-hs2-2.log PRE-CREATION 
  data/files/syslog-hs2.log PRE-CREATION 
  itests/src/test/resources/testconfiguration.properties 96dfbc4b56 
  
llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/LlapRecordReader.java
 9ef7af4eb0 
  ql/src/java/org/apache/hadoop/hive/ql/io/sarg/ConvertAstToSearchArg.java 
27fe828b75 
  ql/src/java/org/apache/hadoop/hive/ql/log/syslog/SyslogInputFormat.java 
PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/log/syslog/SyslogParser.java 
PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/log/syslog/SyslogSerDe.java 
PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/log/syslog/SyslogStorageHandler.java 
PRE-CREATION 
  ql/src/test/org/apache/hadoop/hive/ql/log/TestSyslogInputFormat.java 
PRE-CREATION 
  ql/src/test/queries/clientpositive/syslog_parser.q PRE-CREATION 
  ql/src/test/queries/clientpositive/syslog_parser_file_pruning.q PRE-CREATION 
  ql/src/test/results/clientpositive/llap/syslog_parser.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/llap/syslog_parser_file_pruning.q.out 
PRE-CREATION 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/tools/schematool/MetastoreSchemaTool.java
 eafe0c6d46 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/tools/schematool/SchemaToolCommandLine.java
 6282078411 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/tools/schematool/SchemaToolTaskCreateLogsTable.java
 PRE-CREATION 


Diff: https://reviews.apache.org/r/70372/diff/2/

Changes: https://reviews.apache.org/r/70372/diff/1-2/


Testing
---


Thanks,

Prasanth_J

Re: Review Request 70372: HIVE-21427: Syslog storage handler

2019-04-03 Thread j . prasanth . j



> On April 3, 2019, 9:54 p.m., Jason Dere wrote:
> > llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/LlapRecordReader.java
> > Line 615 (original), 615 (patched)
> > 
> >
> > Just curious about this one, was there a difference between 
> > rbCtx.getRowColumnTypeInfos() and rbCtx.getDataColumnCount()? Or just the 
> > fact that rbCtx.getDataColumnCount() directly returns an int value?

Yes. There is place in encoded reader creation that counts just the non-null 
columns in vector batch. But in here we include all columns including the nulls 
(partition columns at the last) so the text llap reader was throwing exception.


> On April 3, 2019, 9:54 p.m., Jason Dere wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/log/syslog/SyslogSerDe.java
> > Lines 57 (patched)
> > 
> >
> > Is the list of columns from SyslogSerDe fixed to  (facility, severity, 
> > version, ts, hostname, app_name, proc_id, msg_id, structured_data, msg, 
> > unmatched)? If so then should the column list/types be hardcoded rather 
> > than set via LIST_COLUMNS/LIST_COLUMNS_TYPES properties?

Yes. It is fixed schema. make sense.. The columns and types doesn't change 
anyway. Will fix it in next patch.


- Prasanth_J


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/70372/#review214337
---


On April 2, 2019, 10:29 p.m., Prasanth_J wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/70372/
> ---
> 
> (Updated April 2, 2019, 10:29 p.m.)
> 
> 
> Review request for hive, Ashutosh Chauhan and Jason Dere.
> 
> 
> Bugs: HIVE-21427
> https://issues.apache.org/jira/browse/HIVE-21427
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-21427: Syslog storage handler
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 
> 777f8b51215523fca8e396ddf77139420666311a 
>   data/files/syslog-hs2-2.log PRE-CREATION 
>   data/files/syslog-hs2.log PRE-CREATION 
>   itests/src/test/resources/testconfiguration.properties 
> 96dfbc4b56b6eb3dff6b8e1e42a2371d090426e7 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/LlapRecordReader.java
>  9ef7af4eb0c9787a33d2aa4c9a4528b8f356106b 
>   ql/src/java/org/apache/hadoop/hive/ql/io/sarg/ConvertAstToSearchArg.java 
> 27fe828b7531584138cd002956a9fcc20f238f71 
>   ql/src/java/org/apache/hadoop/hive/ql/log/syslog/SyslogInputFormat.java 
> PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/log/syslog/SyslogParser.java 
> PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/log/syslog/SyslogSerDe.java 
> PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/log/syslog/SyslogStorageHandler.java 
> PRE-CREATION 
>   ql/src/test/org/apache/hadoop/hive/ql/log/TestSyslogInputFormat.java 
> PRE-CREATION 
>   ql/src/test/queries/clientpositive/syslog_parser.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/syslog_parser_file_pruning.q 
> PRE-CREATION 
>   ql/src/test/results/clientpositive/llap/syslog_parser.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/llap/syslog_parser_file_pruning.q.out 
> PRE-CREATION 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/tools/schematool/MetastoreSchemaTool.java
>  eafe0c6d46d448bce287e61fabac0384b12b9295 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/tools/schematool/SchemaToolCommandLine.java
>  6282078411c4c728beed8e957aa857ed3c02133c 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/tools/schematool/SchemaToolTaskCreateLogsTable.java
>  PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/70372/diff/1/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Prasanth_J
> 
>

Review Request 70372: HIVE-21427: Syslog storage handler

2019-04-02 Thread j . prasanth . j


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/70372/
---

Review request for hive, Ashutosh Chauhan and Jason Dere.


Bugs: HIVE-21427
https://issues.apache.org/jira/browse/HIVE-21427


Repository: hive-git


Description
---

HIVE-21427: Syslog storage handler


Diffs
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 
777f8b51215523fca8e396ddf77139420666311a 
  data/files/syslog-hs2-2.log PRE-CREATION 
  data/files/syslog-hs2.log PRE-CREATION 
  itests/src/test/resources/testconfiguration.properties 
96dfbc4b56b6eb3dff6b8e1e42a2371d090426e7 
  
llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/LlapRecordReader.java
 9ef7af4eb0c9787a33d2aa4c9a4528b8f356106b 
  ql/src/java/org/apache/hadoop/hive/ql/io/sarg/ConvertAstToSearchArg.java 
27fe828b7531584138cd002956a9fcc20f238f71 
  ql/src/java/org/apache/hadoop/hive/ql/log/syslog/SyslogInputFormat.java 
PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/log/syslog/SyslogParser.java 
PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/log/syslog/SyslogSerDe.java 
PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/log/syslog/SyslogStorageHandler.java 
PRE-CREATION 
  ql/src/test/org/apache/hadoop/hive/ql/log/TestSyslogInputFormat.java 
PRE-CREATION 
  ql/src/test/queries/clientpositive/syslog_parser.q PRE-CREATION 
  ql/src/test/queries/clientpositive/syslog_parser_file_pruning.q PRE-CREATION 
  ql/src/test/results/clientpositive/llap/syslog_parser.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/llap/syslog_parser_file_pruning.q.out 
PRE-CREATION 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/tools/schematool/MetastoreSchemaTool.java
 eafe0c6d46d448bce287e61fabac0384b12b9295 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/tools/schematool/SchemaToolCommandLine.java
 6282078411c4c728beed8e957aa857ed3c02133c 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/tools/schematool/SchemaToolTaskCreateLogsTable.java
 PRE-CREATION 


Diff: https://reviews.apache.org/r/70372/diff/1/


Testing
---


Thanks,

Prasanth_J

Review Request 70224: HIVE-21457: Perf optimizations in ORC split-generation

2019-03-15 Thread j . prasanth . j


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/70224/
---

Review request for hive and Gopal V.


Bugs: HIVE-21457
https://issues.apache.org/jira/browse/HIVE-21457


Repository: hive-git


Description
---

HIVE-21457: Perf optimizations in ORC split-generation


Diffs
-

  ql/src/java/org/apache/hadoop/hive/ql/exec/FetchOperator.java 
e6b47de877e4931f30f1fab725ea0e62c98bdf26 
  ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java 
50a233d5de20491e0107af7eeefdc1515f706894 
  ql/src/java/org/apache/hadoop/hive/ql/io/HiveContextAwareRecordReader.java 
11876fbb10ac45772153c357202645fe08ed28a7 
  ql/src/java/org/apache/hadoop/hive/ql/io/HiveInputFormat.java 
6bac285c15ced93cf4215281447c7adafa98bd1c 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java 
9dac185067c68fd94fbec53d5bb5274b878bbb00 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcRawRecordMerger.java 
62a1061dfd9499954ff2ed9432ab235d3b28a819 
  
ql/src/java/org/apache/hadoop/hive/ql/io/orc/VectorizedOrcAcidRowBatchReader.java
 1795bb54570e5b71a19b3a9091c2172c6b284cb4 
  ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorMR.java 
10192859a7326a223ec9d9cce7d284fd83122f86 
  ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Initiator.java 
deabec6f8767c5397a7503fa64d1b03f0cb41ac2 


Diff: https://reviews.apache.org/r/70224/diff/1/


Testing
---


Thanks,

Prasanth_J

Re: Review Request 70224: HIVE-21457: Perf optimizations in ORC split-generation

2019-03-15 Thread j . prasanth . j


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/70224/
---

(Updated March 16, 2019, 12:28 a.m.)


Review request for hive and Gopal V.


Changes
---

Another place for reuse.


Bugs: HIVE-21457
https://issues.apache.org/jira/browse/HIVE-21457


Repository: hive-git


Description
---

HIVE-21457: Perf optimizations in ORC split-generation


Diffs (updated)
-

  ql/src/java/org/apache/hadoop/hive/ql/exec/FetchOperator.java e6b47de877 
  ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java 50a233d5de 
  ql/src/java/org/apache/hadoop/hive/ql/io/HiveContextAwareRecordReader.java 
11876fbb10 
  ql/src/java/org/apache/hadoop/hive/ql/io/HiveInputFormat.java 6bac285c15 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java 9dac185067 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcRawRecordMerger.java 
62a1061dfd 
  
ql/src/java/org/apache/hadoop/hive/ql/io/orc/VectorizedOrcAcidRowBatchReader.java
 1795bb5457 
  ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorMR.java 
10192859a7 
  ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Initiator.java deabec6f87 


Diff: https://reviews.apache.org/r/70224/diff/2/

Changes: https://reviews.apache.org/r/70224/diff/1-2/


Testing
---


Thanks,

Prasanth_J

Re: Review Request 69965: HIVE-21254: Pre-upgrade tool should handle exceptions and skip db/tables

2019-02-13 Thread j . prasanth . j


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69965/
---

(Updated Feb. 14, 2019, 12:17 a.m.)


Review request for hive, Eugene Koifman and Jason Dere.


Bugs: HIVE-21254
https://issues.apache.org/jira/browse/HIVE-21254


Repository: hive-git


Description
---

HIVE-21254: Pre-upgrade tool should handle exceptions and skip db/tables


Diffs (updated)
-

  
upgrade-acid/pre-upgrade/src/main/java/org/apache/hadoop/hive/upgrade/acid/PreUpgradeTool.java
 04782a62ad 
  
upgrade-acid/pre-upgrade/src/test/java/org/apache/hadoop/hive/upgrade/acid/TestPreUpgradeTool.java
 fe4b08b3f3 


Diff: https://reviews.apache.org/r/69965/diff/3/

Changes: https://reviews.apache.org/r/69965/diff/2-3/


Testing
---


Thanks,

Prasanth_J

Re: Review Request 69965: HIVE-21254: Pre-upgrade tool should handle exceptions and skip db/tables

2019-02-12 Thread j . prasanth . j


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69965/
---

(Updated Feb. 13, 2019, 1:15 a.m.)


Review request for hive, Eugene Koifman and Jason Dere.


Changes
---

Addressed review comments.


Bugs: HIVE-21254
https://issues.apache.org/jira/browse/HIVE-21254


Repository: hive-git


Description
---

HIVE-21254: Pre-upgrade tool should handle exceptions and skip db/tables


Diffs (updated)
-

  
upgrade-acid/pre-upgrade/src/main/java/org/apache/hadoop/hive/upgrade/acid/PreUpgradeTool.java
 04782a62ad 
  
upgrade-acid/pre-upgrade/src/test/java/org/apache/hadoop/hive/upgrade/acid/TestPreUpgradeTool.java
 fe4b08b3f3 


Diff: https://reviews.apache.org/r/69965/diff/2/

Changes: https://reviews.apache.org/r/69965/diff/1-2/


Testing
---


Thanks,

Prasanth_J

Review Request 69965: HIVE-21254: Pre-upgrade tool should handle exceptions and skip db/tables

2019-02-12 Thread j . prasanth . j


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69965/
---

Review request for hive, Eugene Koifman and Jason Dere.


Bugs: HIVE-21254
https://issues.apache.org/jira/browse/HIVE-21254


Repository: hive-git


Description
---

HIVE-21254: Pre-upgrade tool should handle exceptions and skip db/tables


Diffs
-

  
upgrade-acid/pre-upgrade/src/main/java/org/apache/hadoop/hive/upgrade/acid/PreUpgradeTool.java
 04782a62ad9babb6f2b24ab76c6210a27bdfdceb 
  
upgrade-acid/pre-upgrade/src/test/java/org/apache/hadoop/hive/upgrade/acid/TestPreUpgradeTool.java
 fe4b08b3f360faba52172fecb97e246cc7acb4a1 


Diff: https://reviews.apache.org/r/69965/diff/1/


Testing
---


Thanks,

Prasanth_J

Re: Review Request 68946: HIVE-20707: Automatic MSCK REPAIR for external tables

2018-10-29 Thread j . prasanth . j


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68946/
---

(Updated Oct. 29, 2018, 9:17 a.m.)


Review request for hive, Ashutosh Chauhan and Jason Dere.


Changes
---

Uploaded wrong patch before. Moved partition management task to remote 
metastore only task. This fixed druid test failures.


Bugs: HIVE-20707
https://issues.apache.org/jira/browse/HIVE-20707


Repository: hive-git


Description
---

HIVE-20707: Automatic partition management


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java e226a1f 
  hbase-handler/src/test/results/positive/external_table_ppd.q.out edcbe7e 
  hbase-handler/src/test/results/positive/hbase_binary_storage_queries.q.out 
1209c88 
  hbase-handler/src/test/results/positive/hbase_ddl.q.out ccd4148 
  hbase-handler/src/test/results/positive/hbase_queries.q.out eeb97f0 
  hbase-handler/src/test/results/positive/hbasestats.q.out 5a4aea9 
  
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/TestCompactor.java
 a9d7468 
  ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 807f159 
  ql/src/java/org/apache/hadoop/hive/ql/exec/ExplainTask.java 4cc5fa8 
  ql/src/java/org/apache/hadoop/hive/ql/metadata/CheckResult.java 0b4240f 
  ql/src/java/org/apache/hadoop/hive/ql/metadata/HiveMetaStoreChecker.java 
598bb2e 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java cff32d3 
  ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java bba7d6c 
  ql/src/java/org/apache/hadoop/hive/ql/plan/CreateTableDesc.java 27f677e 
  
ql/src/test/org/apache/hadoop/hive/ql/exec/TestMsckCreatePartitionsInBatches.java
 ce2b186 
  
ql/src/test/org/apache/hadoop/hive/ql/exec/TestMsckDropPartitionsInBatches.java 
9480d38 
  ql/src/test/org/apache/hadoop/hive/ql/metadata/TestHiveMetaStoreChecker.java 
a2a0583 
  ql/src/test/queries/clientpositive/msck_repair_acid.q PRE-CREATION 
  ql/src/test/queries/clientpositive/partition_discovery.q PRE-CREATION 
  ql/src/test/results/clientpositive/create_like.q.out f4a5ed5 
  ql/src/test/results/clientpositive/create_like_view.q.out 870f280 
  ql/src/test/results/clientpositive/default_file_format.q.out 0adf5ae 
  ql/src/test/results/clientpositive/druid/druidkafkamini_basic.q.out 883994c 
  ql/src/test/results/clientpositive/druid/druidmini_expressions.q.out 9c9af44 
  ql/src/test/results/clientpositive/druid_topn.q.out 179902a 
  ql/src/test/results/clientpositive/explain_locks.q.out ed7f1e8 
  ql/src/test/results/clientpositive/llap/external_table_purge.q.out 24c778e 
  ql/src/test/results/clientpositive/llap/mm_exim.q.out ee6cf06 
  ql/src/test/results/clientpositive/llap/strict_managed_tables2.q.out f3b6152 
  ql/src/test/results/clientpositive/llap/table_nonprintable.q.out 8221b8c 
  ql/src/test/results/clientpositive/llap/whroot_external1.q.out cac158c 
  ql/src/test/results/clientpositive/msck_repair_acid.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/msck_repair_drop.q.out 2456734 
  ql/src/test/results/clientpositive/partition_discovery.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/rename_external_partition_location.q.out 
02cd814 
  ql/src/test/results/clientpositive/repl_2_exim_basic.q.out b2bcd51 
  ql/src/test/results/clientpositive/show_create_table_alter.q.out 2c75c36 
  ql/src/test/results/clientpositive/show_create_table_partitioned.q.out 
e554a18 
  ql/src/test/results/clientpositive/show_create_table_serde.q.out 8b95c9b 
  ql/src/test/results/clientpositive/spark/stats_noscan_2.q.out 2d713a8 
  ql/src/test/results/clientpositive/stats_noscan_2.q.out 182820f 
  ql/src/test/results/clientpositive/temp_table_display_colstats_tbllvl.q.out 
2a442b4 
  
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/CheckResult.java
 PRE-CREATION 
  
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/Warehouse.java
 294dfb7 
  
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/api/MetastoreException.java
 PRE-CREATION 
  
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java
 1d64cce 
  
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/utils/MetaStoreUtils.java
 8fb1fa7 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreChecker.java
 PRE-CREATION 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/Msck.java
 PRE-CREATION 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MsckInfo.java
 PRE-CREATION 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MsckPartitionExpressionProxy.java
 PRE-CREATION

Re: Review Request 69148: HIVE-20793 add RP namespacing to workload management

2018-10-26 Thread j . prasanth . j


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69148/#review210118
---


Ship it!




Ship It!

- Prasanth_J


On Oct. 26, 2018, 11:23 p.m., Sergey Shelukhin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/69148/
> ---
> 
> (Updated Oct. 26, 2018, 11:23 p.m.)
> 
> 
> Review request for hive, Jason Dere and Prasanth_J.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> see jira
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 
> e226a1f82d44550f389308f91d578e7aa4ea170a 
>   
> itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/listener/DummyRawStoreFailEvent.java
>  c3e1e8e88c69d7713e16c7061ce8cf73a0d5e833 
>   metastore/scripts/upgrade/hive/hive-schema-4.0.0.hive.sql 
> a69046f961cdf0fff7989492c489bb62f2a66d72 
>   metastore/scripts/upgrade/hive/upgrade-3.1.0-to-4.0.0.hive.sql 
> 4c770206fe3dcceb8570be1c1ef078b376f5cafd 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 
> 807f159daa98d40e667914adc6c53fb8ecabf998 
>   ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java 
> 012a670064c155a31dab5337dc97ac5bb7b39aec 
>   ql/src/test/org/apache/hadoop/hive/ql/metadata/TestHive.java 
> e57db935d9420508ed6091e12ca6b6cd3382db5d 
>   ql/src/test/queries/clientpositive/resourceplan.q 
> fae9701ebaeaa521904a383f5fb741c13be08d8e 
>   ql/src/test/results/clientpositive/llap/resourceplan.q.out 
> c11daf728cdd5bd6fe36618aff113b3d60579129 
>   
> standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java
>  90b5764a2e24d5ddf71d227f1364a0d09d63c185 
>   
> standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/IMetaStoreClient.java
>  d1c0c4d1f60016f28cea69348b1b30ecb61bf083 
>   standalone-metastore/metastore-common/src/main/thrift/hive_metastore.thrift 
> 4b7b61520a2d55635f474317053a17410f3a4bb7 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
>  8cd46e3f44e7c4e47fbf7f2ce2b6350a5814106f 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
>  9c158040497cd3d2762620ce35e2b46bb6d5fffe 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/RawStore.java
>  c3914b668fac18ead6196a4fc449e909f5af01b1 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java
>  47ac68c667bea8f09f5301a6364c854bc18b3c0d 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/model/MWMResourcePlan.java
>  ac51f2d77145b37da468ce8df2ac5c42f4d6c538 
>   standalone-metastore/metastore-server/src/main/resources/package.jdo 
> fef6a42038bb2aa0cba6dfda8d710fd37cb720e7 
>   
> standalone-metastore/metastore-server/src/main/sql/derby/hive-schema-4.0.0.derby.sql
>  c889bbdf96b887b29be858e41ee854f0731cd5cd 
>   
> standalone-metastore/metastore-server/src/main/sql/derby/upgrade-3.2.0-to-4.0.0.derby.sql
>  aca5227a5bb6192da6c5f070c04d2941d636bad2 
>   
> standalone-metastore/metastore-server/src/main/sql/mssql/hive-schema-4.0.0.mssql.sql
>  91ba134325094e413887a89e1d605efa99218288 
>   
> standalone-metastore/metastore-server/src/main/sql/mssql/upgrade-3.2.0-to-4.0.0.mssql.sql
>  f0d861b3a9bc982c1e24fa49415dcfc6c105cd68 
>   
> standalone-metastore/metastore-server/src/main/sql/mysql/hive-schema-4.0.0.mysql.sql
>  3af2ebb253f82bb85976d229d4ac2225deffdbde 
>   
> standalone-metastore/metastore-server/src/main/sql/mysql/upgrade-3.2.0-to-4.0.0.mysql.sql
>  ee0f691b524a6e822ac14e09d24d3a49ae8565b1 
>   
> standalone-metastore/metastore-server/src/main/sql/oracle/hive-schema-4.0.0.oracle.sql
>  33aa08015a9e17585c42d64d44b364be96e69eaf 
>   
> standalone-metastore/metastore-server/src/main/sql/oracle/upgrade-3.2.0-to-4.0.0.oracle.sql
>  bbb4a39ec4f6f616c9a1a9042a35cafd45cf9796 
>   
> standalone-metastore/metastore-server/src/main/sql/postgres/hive-schema-4.0.0.postgres.sql
>  ea088d77fdaec85834b8fd3f01eacdfac58dd245 
>   
> standalone-metastore/metastore-server/src/main/sql/postgres/upgrade-3.2.0-to-4.0.0.postgres.sql
>  2a2d70ae802eb6f5b9ab7f4f9519a0af30d2c5b4 
>   
> standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/DummyRawStoreControlledCommit.java
>  966979891b71f1cbfe50f56c40c35af8b304c47f 
>   
> standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/DummyRawStoreForJdoConnection.java
>  593d562c3498660861201f58d83c27d59d184046 
>   
> standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClientPreCatalog.java
>

Re: Review Request 69148: HIVE-20793 add RP namespacing to workload management

2018-10-26 Thread j . prasanth . j


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69148/#review210105
---




ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java
Lines 5451 (patched)


nit. Cannot 'create'



ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java
Lines 5536 (patched)


nit: Cannot create trigger



ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java
Lines 5550 (patched)


nit: trigger
similarly for all other exceptions below. To say what specific resource is 
being created/modified/dropped



ql/src/test/org/apache/hadoop/hive/ql/metadata/TestHive.java
Lines 434 (patched)


unused :)



ql/src/test/results/clientpositive/llap/resourceplan.q.out
Line 3918 (original), 3970 (patched)


hmm.. expected?



ql/src/test/results/clientpositive/llap/resourceplan.q.out
Line 4051 (original), 4101 (patched)


Similarly are these support to show up in clientpositive tests?



ql/src/test/results/clientpositive/llap/resourceplan.q.out
Line 4713 (original), 4763 (patched)


can you add a non-default ns test as well?


- Prasanth_J


On Oct. 25, 2018, 9:43 p.m., Sergey Shelukhin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/69148/
> ---
> 
> (Updated Oct. 25, 2018, 9:43 p.m.)
> 
> 
> Review request for hive, Jason Dere and Prasanth_J.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> see jira
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 
> e226a1f82d44550f389308f91d578e7aa4ea170a 
>   
> itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/listener/DummyRawStoreFailEvent.java
>  c3e1e8e88c69d7713e16c7061ce8cf73a0d5e833 
>   metastore/scripts/upgrade/hive/hive-schema-4.0.0.hive.sql 
> a69046f961cdf0fff7989492c489bb62f2a66d72 
>   metastore/scripts/upgrade/hive/upgrade-3.1.0-to-4.0.0.hive.sql 
> 4c770206fe3dcceb8570be1c1ef078b376f5cafd 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 
> 807f159daa98d40e667914adc6c53fb8ecabf998 
>   ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java 
> 4de038913a5c9a2c199f71702b8f70ca84d0856b 
>   ql/src/test/org/apache/hadoop/hive/ql/metadata/TestHive.java 
> e57db935d9420508ed6091e12ca6b6cd3382db5d 
>   ql/src/test/queries/clientpositive/resourceplan.q 
> fae9701ebaeaa521904a383f5fb741c13be08d8e 
>   ql/src/test/results/clientpositive/llap/resourceplan.q.out 
> c11daf728cdd5bd6fe36618aff113b3d60579129 
>   
> standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java
>  aba63f050b5b98a2aeeb0df6ff2de5e6e06761f2 
>   
> standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/IMetaStoreClient.java
>  d1c0c4d1f60016f28cea69348b1b30ecb61bf083 
>   standalone-metastore/metastore-common/src/main/thrift/hive_metastore.thrift 
> 4b7b61520a2d55635f474317053a17410f3a4bb7 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
>  8cd46e3f44e7c4e47fbf7f2ce2b6350a5814106f 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
>  9c158040497cd3d2762620ce35e2b46bb6d5fffe 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/RawStore.java
>  c3914b668fac18ead6196a4fc449e909f5af01b1 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java
>  47ac68c667bea8f09f5301a6364c854bc18b3c0d 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/model/MWMResourcePlan.java
>  ac51f2d77145b37da468ce8df2ac5c42f4d6c538 
>   standalone-metastore/metastore-server/src/main/resources/package.jdo 
> fef6a42038bb2aa0cba6dfda8d710fd37cb720e7 
>   
> standalone-metastore/metastore-server/src/main/sql/derby/hive-schema-4.0.0.derby.sql
>  c889bbdf96b887b29be858e41ee854f0731cd5cd 
>   
> standalone-metastore/metastore-server/src/main/sql/derby/upgrade-3.2.0-to-4.0.0.derby.sql
>  aca5227a5bb6192da6c5f070c04d2941d636bad2 
>   
> standalone-metastore/metastore-server/src/main/sql/mssql/hive-schema-4.0.0.mssql.sql
>  91ba134325094e413887a89e1d605efa99218288 
>   
> standalone-metastore/metastore-server/src/main/sql/mssql/upgrade-3.2.0-to-4.0.0.mssql.sql
>  f0d861b3a9bc982c1e24fa49415dcfc6c105cd68 
>   
>

Re: Review Request 68946: HIVE-20707: Automatic MSCK REPAIR for external tables

2018-10-17 Thread j . prasanth . j


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68946/
---

(Updated Oct. 17, 2018, 7:14 p.m.)


Review request for hive, Ashutosh Chauhan and Jason Dere.


Changes
---

Added sorting to stabilize qfile test output


Bugs: HIVE-20707
https://issues.apache.org/jira/browse/HIVE-20707


Repository: hive-git


Description
---

HIVE-20707: Automatic partition management


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java a458ac0 
  hbase-handler/src/test/results/positive/external_table_ppd.q.out edcbe7e 
  hbase-handler/src/test/results/positive/hbase_binary_storage_queries.q.out 
1209c88 
  hbase-handler/src/test/results/positive/hbase_ddl.q.out ccd4148 
  hbase-handler/src/test/results/positive/hbase_queries.q.out eeb97f0 
  hbase-handler/src/test/results/positive/hbasestats.q.out 5a4aea9 
  
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/TestCompactor.java
 a9d7468 
  ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 807f159 
  ql/src/java/org/apache/hadoop/hive/ql/exec/ExplainTask.java 46bf088 
  ql/src/java/org/apache/hadoop/hive/ql/metadata/CheckResult.java 0b4240f 
  ql/src/java/org/apache/hadoop/hive/ql/metadata/HiveMetaStoreChecker.java 
598bb2e 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java cff32d3 
  ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java bba7d6c 
  ql/src/java/org/apache/hadoop/hive/ql/plan/CreateTableDesc.java 27f677e 
  
ql/src/test/org/apache/hadoop/hive/ql/exec/TestMsckCreatePartitionsInBatches.java
 ce2b186 
  
ql/src/test/org/apache/hadoop/hive/ql/exec/TestMsckDropPartitionsInBatches.java 
9480d38 
  ql/src/test/org/apache/hadoop/hive/ql/metadata/TestHiveMetaStoreChecker.java 
a2a0583 
  ql/src/test/queries/clientpositive/msck_repair_acid.q PRE-CREATION 
  ql/src/test/queries/clientpositive/partition_discovery.q PRE-CREATION 
  ql/src/test/results/clientpositive/create_like.q.out f4a5ed5 
  ql/src/test/results/clientpositive/create_like_view.q.out 870f280 
  ql/src/test/results/clientpositive/default_file_format.q.out 0adf5ae 
  ql/src/test/results/clientpositive/druid/druidkafkamini_basic.q.out 883994c 
  ql/src/test/results/clientpositive/druid/druidmini_expressions.q.out 9c9af44 
  ql/src/test/results/clientpositive/druid_topn.q.out 179902a 
  ql/src/test/results/clientpositive/explain_locks.q.out ed7f1e8 
  ql/src/test/results/clientpositive/llap/external_table_purge.q.out 24c778e 
  ql/src/test/results/clientpositive/llap/mm_exim.q.out ee6cf06 
  ql/src/test/results/clientpositive/llap/strict_managed_tables2.q.out f3b6152 
  ql/src/test/results/clientpositive/llap/whroot_external1.q.out cac158c 
  ql/src/test/results/clientpositive/msck_repair_acid.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/msck_repair_drop.q.out 2456734 
  ql/src/test/results/clientpositive/partition_discovery.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/rename_external_partition_location.q.out 
02cd814 
  ql/src/test/results/clientpositive/repl_2_exim_basic.q.out b2bcd51 
  ql/src/test/results/clientpositive/show_create_table_alter.q.out 2c75c36 
  ql/src/test/results/clientpositive/show_create_table_partitioned.q.out 
e554a18 
  ql/src/test/results/clientpositive/show_create_table_serde.q.out 8b95c9b 
  ql/src/test/results/clientpositive/spark/stats_noscan_2.q.out 2d713a8 
  ql/src/test/results/clientpositive/stats_noscan_2.q.out 182820f 
  ql/src/test/results/clientpositive/temp_table_display_colstats_tbllvl.q.out 
2a442b4 
  
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/CheckResult.java
 PRE-CREATION 
  
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/Warehouse.java
 294dfb7 
  
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/api/MetastoreException.java
 PRE-CREATION 
  
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java
 7b01678 
  
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/utils/MetaStoreUtils.java
 8fb1fa7 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreChecker.java
 PRE-CREATION 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/Msck.java
 PRE-CREATION 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MsckInfo.java
 PRE-CREATION 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MsckPartitionExpressionProxy.java
 PRE-CREATION 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
 b98b4b4

Re: Review Request 68946: HIVE-20707: Automatic MSCK REPAIR for external tables

2018-10-16 Thread j . prasanth . j


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68946/
---

(Updated Oct. 16, 2018, 7:02 a.m.)


Review request for hive, Ashutosh Chauhan and Jason Dere.


Changes
---

Addressed review comment and updated druid test golden files.


Bugs: HIVE-20707
https://issues.apache.org/jira/browse/HIVE-20707


Repository: hive-git


Description
---

HIVE-20707: Automatic partition management


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 92a1c31 
  hbase-handler/src/test/results/positive/external_table_ppd.q.out edcbe7e 
  hbase-handler/src/test/results/positive/hbase_binary_storage_queries.q.out 
1209c88 
  hbase-handler/src/test/results/positive/hbase_ddl.q.out ccd4148 
  hbase-handler/src/test/results/positive/hbase_queries.q.out eeb97f0 
  hbase-handler/src/test/results/positive/hbasestats.q.out 5a4aea9 
  
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/TestCompactor.java
 a9d7468 
  ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 807f159 
  ql/src/java/org/apache/hadoop/hive/ql/exec/ExplainTask.java 46bf088 
  ql/src/java/org/apache/hadoop/hive/ql/metadata/CheckResult.java 0b4240f 
  ql/src/java/org/apache/hadoop/hive/ql/metadata/HiveMetaStoreChecker.java 
598bb2e 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java cff32d3 
  ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java 29f6ecf 
  ql/src/java/org/apache/hadoop/hive/ql/plan/CreateTableDesc.java 27f677e 
  
ql/src/test/org/apache/hadoop/hive/ql/exec/TestMsckCreatePartitionsInBatches.java
 ce2b186 
  
ql/src/test/org/apache/hadoop/hive/ql/exec/TestMsckDropPartitionsInBatches.java 
9480d38 
  ql/src/test/org/apache/hadoop/hive/ql/metadata/TestHiveMetaStoreChecker.java 
a2a0583 
  ql/src/test/queries/clientpositive/msck_repair_acid.q PRE-CREATION 
  ql/src/test/queries/clientpositive/partition_discovery.q PRE-CREATION 
  ql/src/test/results/clientpositive/create_like.q.out f4a5ed5 
  ql/src/test/results/clientpositive/create_like_view.q.out 870f280 
  ql/src/test/results/clientpositive/default_file_format.q.out 0adf5ae 
  ql/src/test/results/clientpositive/druid/druidkafkamini_basic.q.out 883994c 
  ql/src/test/results/clientpositive/druid/druidmini_expressions.q.out 9c9af44 
  ql/src/test/results/clientpositive/druid_topn.q.out 179902a 
  ql/src/test/results/clientpositive/explain_locks.q.out ed7f1e8 
  ql/src/test/results/clientpositive/llap/external_table_purge.q.out 24c778e 
  ql/src/test/results/clientpositive/llap/mm_exim.q.out ee6cf06 
  ql/src/test/results/clientpositive/llap/strict_managed_tables2.q.out f3b6152 
  ql/src/test/results/clientpositive/llap/whroot_external1.q.out cac158c 
  ql/src/test/results/clientpositive/msck_repair_acid.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/msck_repair_drop.q.out 2456734 
  ql/src/test/results/clientpositive/partition_discovery.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/rename_external_partition_location.q.out 
02cd814 
  ql/src/test/results/clientpositive/repl_2_exim_basic.q.out b2bcd51 
  ql/src/test/results/clientpositive/show_create_table_alter.q.out 2c75c36 
  ql/src/test/results/clientpositive/show_create_table_partitioned.q.out 
e554a18 
  ql/src/test/results/clientpositive/show_create_table_serde.q.out 8b95c9b 
  ql/src/test/results/clientpositive/spark/stats_noscan_2.q.out 2d713a8 
  ql/src/test/results/clientpositive/stats_noscan_2.q.out 182820f 
  ql/src/test/results/clientpositive/temp_table_display_colstats_tbllvl.q.out 
2a442b4 
  
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/CheckResult.java
 PRE-CREATION 
  
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/Warehouse.java
 294dfb7 
  
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/api/MetastoreException.java
 PRE-CREATION 
  
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java
 7b01678 
  
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/utils/MetaStoreUtils.java
 16f4a50 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreChecker.java
 PRE-CREATION 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/Msck.java
 PRE-CREATION 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MsckInfo.java
 PRE-CREATION 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MsckPartitionExpressionProxy.java
 PRE-CREATION 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
 66977d7

Re: Review Request 68946: HIVE-20707: Automatic MSCK REPAIR for external tables

2018-10-16 Thread j . prasanth . j



> On Oct. 16, 2018, 2:28 a.m., Jason Dere wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java
> > Lines 4761 (patched)
> > 
> >
> > Should this be on by default? If there are a lot of external tables 
> > (especially on s3), the metastore could be spending a lot of time doing 
> > auto discover. Could also affect the running of other MetastoreTaskThreads.
> 
> Prasanth_J wrote:
> Yeah. I think this should be default. This will remove manual msck step 
> or periodic msck query (via cron job). This thread kicks in once every 5 
> minutes but if the previous attempt is not done yet it will skip an attempt 
> so as to avoid queue'ing up of background tasks. Also it will use high batch 
> size by default so that in most case there should be 1 MS request per table. 
> MSCK thread also runs in a thread pool. The only place this background thread 
> could be blocked is when exclusive lock is obtained on a table (which gets 
> released after txn timeout of 300s). 
> We could probably restrict this only for EXTERNAL table types (currently 
> it defaults to both EXTERNAL and MANAGED). Since managed is ACID by default 
> we can avoid scanning managed tables. Changing ACID table layout out of band 
> is shooting themselves in the foot anyway.

Actually, thinking about it again. I think I will leave the table types 
(metastore.partition.management.table.types) as such. Required for partition 
retention. If user adds retention period to managed table, they don't have to 
do anything. 

Added a fix in new patch that won't acquire lock unless it is required (change 
detected, adding or dropping partition).


- Prasanth_J


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68946/#review209582
---


On Oct. 16, 2018, 12:21 a.m., Prasanth_J wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/68946/
> ---
> 
> (Updated Oct. 16, 2018, 12:21 a.m.)
> 
> 
> Review request for hive, Ashutosh Chauhan and Jason Dere.
> 
> 
> Bugs: HIVE-20707
> https://issues.apache.org/jira/browse/HIVE-20707
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-20707: Automatic partition management
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 92a1c31 
>   hbase-handler/src/test/results/positive/external_table_ppd.q.out edcbe7e 
>   hbase-handler/src/test/results/positive/hbase_binary_storage_queries.q.out 
> 1209c88 
>   hbase-handler/src/test/results/positive/hbase_ddl.q.out ccd4148 
>   hbase-handler/src/test/results/positive/hbase_queries.q.out eeb97f0 
>   hbase-handler/src/test/results/positive/hbasestats.q.out 5a4aea9 
>   
> itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/TestCompactor.java
>  a9d7468 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 807f159 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/ExplainTask.java 46bf088 
>   ql/src/java/org/apache/hadoop/hive/ql/metadata/CheckResult.java 0b4240f 
>   ql/src/java/org/apache/hadoop/hive/ql/metadata/HiveMetaStoreChecker.java 
> 598bb2e 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java cff32d3 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java 
> 29f6ecf 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/CreateTableDesc.java 27f677e 
>   
> ql/src/test/org/apache/hadoop/hive/ql/exec/TestMsckCreatePartitionsInBatches.java
>  ce2b186 
>   
> ql/src/test/org/apache/hadoop/hive/ql/exec/TestMsckDropPartitionsInBatches.java
>  9480d38 
>   
> ql/src/test/org/apache/hadoop/hive/ql/metadata/TestHiveMetaStoreChecker.java 
> a2a0583 
>   ql/src/test/queries/clientpositive/msck_repair_acid.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/partition_discovery.q PRE-CREATION 
>   ql/src/test/results/clientpositive/create_like.q.out f4a5ed5 
>   ql/src/test/results/clientpositive/create_like_view.q.out 870f280 
>   ql/src/test/results/clientpositive/default_file_format.q.out 0adf5ae 
>   ql/src/test/results/clientpositive/druid_topn.q.out 179902a 
>   ql/src/test/results/clientpositive/explain_locks.q.out ed7f1e8 
>   ql/src/test/results/clientpositive/llap/external_table_purge.q.out 24c778e 
>   ql/src/test/results/clientpositive/llap/mm_exim.q.out ee6cf06 
>   ql/src/test/results/clientpositive/llap/strict_managed_tables2.q.out 
> f3b6152 
>   ql/src/test/results/clientpositive/llap/whroot_external1.q.out cac158c 
>   ql/src/test/results/clientpositive/msck_repair_acid.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/msck_repair_drop.q.out 2456734 
>   ql/src/test/results/clientpositive/partition_discovery.q.out PRE-CREATION

Re: Review Request 68946: HIVE-20707: Automatic MSCK REPAIR for external tables

2018-10-15 Thread j . prasanth . j



> On Oct. 16, 2018, 2:28 a.m., Jason Dere wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java
> > Lines 4761 (patched)
> > 
> >
> > Should this be on by default? If there are a lot of external tables 
> > (especially on s3), the metastore could be spending a lot of time doing 
> > auto discover. Could also affect the running of other MetastoreTaskThreads.

Yeah. I think this should be default. This will remove manual msck step or 
periodic msck query (via cron job). This thread kicks in once every 5 minutes 
but if the previous attempt is not done yet it will skip an attempt so as to 
avoid queue'ing up of background tasks. Also it will use high batch size by 
default so that in most case there should be 1 MS request per table. MSCK 
thread also runs in a thread pool. The only place this background thread could 
be blocked is when exclusive lock is obtained on a table (which gets released 
after txn timeout of 300s). 
We could probably restrict this only for EXTERNAL table types (currently it 
defaults to both EXTERNAL and MANAGED). Since managed is ACID by default we can 
avoid scanning managed tables. Changing ACID table layout out of band is 
shooting themselves in the foot anyway.


> On Oct. 16, 2018, 2:28 a.m., Jason Dere wrote:
> > ql/src/test/results/clientpositive/msck_repair_drop.q.out
> > Line 127 (original), 127 (patched)
> > 
> >
> > What is the new ordering of these messages? Looks like it could be 
> > potential issue when diffing golden files?

I think the ordering got changed as part of 
https://issues.apache.org/jira/browse/HIVE-20696 
I will check again to see if we are using hashset anywhere else. The ordering 
is based on partition name so prepending catalog name "hive" might have changed 
the order although not sure (either hashset issue or unstable sort).


> On Oct. 16, 2018, 2:28 a.m., Jason Dere wrote:
> > standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/PartitionManagementTask.java
> > Lines 141 (patched)
> > 
> >
> > Is this variable used? It's logged, but I think retentionSeconds should 
> > be used instead.

good catch. will fix it.


> On Oct. 16, 2018, 2:28 a.m., Jason Dere wrote:
> > standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/PartitionManagementTask.java
> > Lines 142 (patched)
> > 
> >
> > Might want to check for exception from TimeValidator.validate() in 
> > getRententionPeriodInSeconds, or else a bad setting in one table can fail 
> > here and prevent this from running for any tables.
> > But if you do skip that table, make sure the countdown latch is updated 
> > appropriately.

make sense. will fix it.


- Prasanth_J


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68946/#review209582
---


On Oct. 16, 2018, 12:21 a.m., Prasanth_J wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/68946/
> ---
> 
> (Updated Oct. 16, 2018, 12:21 a.m.)
> 
> 
> Review request for hive, Ashutosh Chauhan and Jason Dere.
> 
> 
> Bugs: HIVE-20707
> https://issues.apache.org/jira/browse/HIVE-20707
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-20707: Automatic partition management
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 92a1c31 
>   hbase-handler/src/test/results/positive/external_table_ppd.q.out edcbe7e 
>   hbase-handler/src/test/results/positive/hbase_binary_storage_queries.q.out 
> 1209c88 
>   hbase-handler/src/test/results/positive/hbase_ddl.q.out ccd4148 
>   hbase-handler/src/test/results/positive/hbase_queries.q.out eeb97f0 
>   hbase-handler/src/test/results/positive/hbasestats.q.out 5a4aea9 
>   
> itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/TestCompactor.java
>  a9d7468 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 807f159 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/ExplainTask.java 46bf088 
>   ql/src/java/org/apache/hadoop/hive/ql/metadata/CheckResult.java 0b4240f 
>   ql/src/java/org/apache/hadoop/hive/ql/metadata/HiveMetaStoreChecker.java 
> 598bb2e 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java cff32d3 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java 
> 29f6ecf 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/CreateTableDesc.java 27f677e 
>   
>

Re: Review Request 68946: HIVE-20707: Automatic MSCK REPAIR for external tables

2018-10-15 Thread j . prasanth . j


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68946/
---

(Updated Oct. 16, 2018, 12:21 a.m.)


Review request for hive, Ashutosh Chauhan and Jason Dere.


Changes
---

Some test fixes


Bugs: HIVE-20707
https://issues.apache.org/jira/browse/HIVE-20707


Repository: hive-git


Description
---

HIVE-20707: Automatic partition management


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 92a1c31 
  hbase-handler/src/test/results/positive/external_table_ppd.q.out edcbe7e 
  hbase-handler/src/test/results/positive/hbase_binary_storage_queries.q.out 
1209c88 
  hbase-handler/src/test/results/positive/hbase_ddl.q.out ccd4148 
  hbase-handler/src/test/results/positive/hbase_queries.q.out eeb97f0 
  hbase-handler/src/test/results/positive/hbasestats.q.out 5a4aea9 
  
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/TestCompactor.java
 a9d7468 
  ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 807f159 
  ql/src/java/org/apache/hadoop/hive/ql/exec/ExplainTask.java 46bf088 
  ql/src/java/org/apache/hadoop/hive/ql/metadata/CheckResult.java 0b4240f 
  ql/src/java/org/apache/hadoop/hive/ql/metadata/HiveMetaStoreChecker.java 
598bb2e 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java cff32d3 
  ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java 29f6ecf 
  ql/src/java/org/apache/hadoop/hive/ql/plan/CreateTableDesc.java 27f677e 
  
ql/src/test/org/apache/hadoop/hive/ql/exec/TestMsckCreatePartitionsInBatches.java
 ce2b186 
  
ql/src/test/org/apache/hadoop/hive/ql/exec/TestMsckDropPartitionsInBatches.java 
9480d38 
  ql/src/test/org/apache/hadoop/hive/ql/metadata/TestHiveMetaStoreChecker.java 
a2a0583 
  ql/src/test/queries/clientpositive/msck_repair_acid.q PRE-CREATION 
  ql/src/test/queries/clientpositive/partition_discovery.q PRE-CREATION 
  ql/src/test/results/clientpositive/create_like.q.out f4a5ed5 
  ql/src/test/results/clientpositive/create_like_view.q.out 870f280 
  ql/src/test/results/clientpositive/default_file_format.q.out 0adf5ae 
  ql/src/test/results/clientpositive/druid_topn.q.out 179902a 
  ql/src/test/results/clientpositive/explain_locks.q.out ed7f1e8 
  ql/src/test/results/clientpositive/llap/external_table_purge.q.out 24c778e 
  ql/src/test/results/clientpositive/llap/mm_exim.q.out ee6cf06 
  ql/src/test/results/clientpositive/llap/strict_managed_tables2.q.out f3b6152 
  ql/src/test/results/clientpositive/llap/whroot_external1.q.out cac158c 
  ql/src/test/results/clientpositive/msck_repair_acid.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/msck_repair_drop.q.out 2456734 
  ql/src/test/results/clientpositive/partition_discovery.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/rename_external_partition_location.q.out 
02cd814 
  ql/src/test/results/clientpositive/repl_2_exim_basic.q.out b2bcd51 
  ql/src/test/results/clientpositive/show_create_table_alter.q.out 2c75c36 
  ql/src/test/results/clientpositive/show_create_table_partitioned.q.out 
e554a18 
  ql/src/test/results/clientpositive/show_create_table_serde.q.out 8b95c9b 
  ql/src/test/results/clientpositive/spark/stats_noscan_2.q.out 2d713a8 
  ql/src/test/results/clientpositive/stats_noscan_2.q.out 182820f 
  ql/src/test/results/clientpositive/temp_table_display_colstats_tbllvl.q.out 
2a442b4 
  
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/CheckResult.java
 PRE-CREATION 
  
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/Warehouse.java
 294dfb7 
  
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/api/MetastoreException.java
 PRE-CREATION 
  
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java
 7b01678 
  
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/utils/MetaStoreUtils.java
 16f4a50 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreChecker.java
 PRE-CREATION 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/Msck.java
 PRE-CREATION 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MsckInfo.java
 PRE-CREATION 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MsckPartitionExpressionProxy.java
 PRE-CREATION 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
 66977d7 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/PartitionIterable.java
 PRE-CREATION 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/PartitionManagementTask.java
 PRE-CREATION

Review Request 69011: HIVE-20713: Use percentage for join conversion size thresholds

2018-10-12 Thread j . prasanth . j


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69011/
---

Review request for hive and Ashutosh Chauhan.


Bugs: HIVE-20713
https://issues.apache.org/jira/browse/HIVE-20713


Repository: hive-git


Description
---

HIVE-20713: Use percentage for join conversion size thresholds


Diffs
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 
cc6239c3135714fb65aa1afc2882852460b68b37 
  data/conf/hive-site.xml 0daf9adc717bc1c4413d2e34691c26a3e2585c77 
  data/conf/llap/hive-site.xml 44ca6c9daf092a35f1c58c26dfa3575c303464ce 
  data/conf/tez/hive-site.xml 236adc7087b43f4e9ab95b2fa57436cf75c679aa 
  itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/TestAcidOnTez.java 
40dd992455f2fa6bae85d9d02338bc820a370ebe 
  ql/src/java/org/apache/hadoop/hive/ql/exec/MemoryInfo.java PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/BytesBytesMultiHashMap.java
 a6b0dbc0dc956d81d027f08a55fbdf0ca452638f 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/HybridHashTableContainer.java
 54377428eafdb79e1bbdc8a182eafb46f8febd23 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/HashTableLoader.java 
151d1b39eb17bba336bb02ab06f216361e4f6ff2 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/ConvertJoinMapJoin.java 
5217208611a6ab63b218a585e820516221022ec6 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/correlation/CorrelationOptimizer.java
 7ff92edd91018f55dd90200f184869a74868dd46 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/correlation/ReduceSinkDeDuplication.java
 5269eb6f6a77ac031d144a96a2687331e3444f4b 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/CommonJoinResolver.java
 a84534a9495b44ac319ad52a1f841428861d62e9 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/CommonJoinTaskDispatcher.java
 a52e5e60d97d24996c813b463caffb946ce5b61a 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/LlapClusterStateForCompile.java
 0d1990a8cb3882c2c8b271c90e2675e88c79a7ea 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/MemoryDecider.java 
64f1e7b830f85e898252f964bdbfb73267224ef6 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/spark/SparkMapJoinOptimizer.java
 ad6db21fe6041e18adc646ca20759ca3cde58713 
  ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java 
22f3266c87f1d42c254893b424b68e757fb2953b 
  ql/src/java/org/apache/hadoop/hive/ql/parse/TezCompiler.java 
fdc963506cabd96c125ec15b6a1eacb71fd33d42 
  ql/src/test/org/apache/hadoop/hive/ql/exec/TestOperators.java 
c7cd4ad3f6deba320b343cda071640cd1c78743e 
  ql/src/test/queries/clientpositive/acid_mapjoin.q 
e3d14cc97d1e193a6bbb2a0e25e0741a192e2638 
  ql/src/test/queries/clientpositive/autoColumnStats_1.q 
9a62127dabee743b8880f0effdd5184fb24975d0 
  ql/src/test/queries/clientpositive/autoColumnStats_2.q 
51f252b303fb579d315aabcabb75b4429c7b5d4d 
  ql/src/test/queries/clientpositive/autoColumnStats_3.q 
7a419118902ffab767780057d8e83fbde3fbdba9 
  ql/src/test/queries/clientpositive/auto_join_reordering_values.q 
39b2e1d9a78f77c50d60a749085b05ba9acd6b20 
  ql/src/test/queries/clientpositive/auto_join_stats.q 
8b377bf4ca70ccaf43b625afc51b1439916f 
  ql/src/test/queries/clientpositive/auto_join_without_localtask.q 
15697f2f11ac4669c7bdf6e4144f02fd3cb465b2 
  ql/src/test/queries/clientpositive/auto_sortmerge_join_1.q 
1fbe8f79cff97ef7baf8eb8afce36665b425b23d 
  ql/src/test/queries/clientpositive/auto_sortmerge_join_10.q 
1b15a740120862a7d56e7b36e07a0e3c4d7c835d 
  ql/src/test/queries/clientpositive/auto_sortmerge_join_11.q 
76e615fbde1d8d52eb2a8de5d08d69d26e304a99 
  ql/src/test/queries/clientpositive/auto_sortmerge_join_12.q 
d68efea393af588527a698dda2a578433195472e 
  ql/src/test/queries/clientpositive/auto_sortmerge_join_13.q 
bbd7afab07b62d1e6b63b408a5cce71db5ef55f1 
  ql/src/test/queries/clientpositive/auto_sortmerge_join_14.q 
1e21c92e59f56688d05d3fbc21184a70c9779996 
  ql/src/test/queries/clientpositive/auto_sortmerge_join_15.q 
b3dd8e50d7cd4e3f95f8b595c10acbfb15732943 
  ql/src/test/queries/clientpositive/auto_sortmerge_join_16.q 
0a72ddfc6b859912962b30719637645648d625ad 
  ql/src/test/queries/clientpositive/auto_sortmerge_join_2.q 
c6d5318f43daa419b8ad0fedcf6782ea3f333f5d 
  ql/src/test/queries/clientpositive/auto_sortmerge_join_3.q 
f41b0970c9ff98f5d72bfa2b3ad07b4b7581e6ab 
  ql/src/test/queries/clientpositive/auto_sortmerge_join_4.q 
e28e0ab4a09cf0214ce946ec79bb34063705e9ad 
  ql/src/test/queries/clientpositive/auto_sortmerge_join_5.q 
9550e21382237f26f9b7187bc024053221fe365f 
  ql/src/test/queries/clientpositive/auto_sortmerge_join_6.q 
0aeec0e50fa3dcff0e20f77cf9541d67c141d4db 
  ql/src/test/queries/clientpositive/auto_sortmerge_join_7.q 
8c9dbacbbad8efe42986556a7d2f6cb62fd3b906 
  ql/src/test/queries/clientpositive/auto_sortmerge_join_8.q 
51647e86b2bc1d680b59c30aeeaef6f949f21427

Re: Review Request 68946: HIVE-20707: Automatic MSCK REPAIR for external tables

2018-10-11 Thread j . prasanth . j


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68946/
---

(Updated Oct. 12, 2018, 1:35 a.m.)


Review request for hive, Ashutosh Chauhan and Jason Dere.


Bugs: HIVE-20707
https://issues.apache.org/jira/browse/HIVE-20707


Repository: hive-git


Description (updated)
---

HIVE-20707: Automatic partition management


Diffs
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java cc6239c 
  
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/TestCompactor.java
 cffa21a 
  ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java e913462 
  ql/src/java/org/apache/hadoop/hive/ql/exec/ExplainTask.java 46bf088 
  ql/src/java/org/apache/hadoop/hive/ql/metadata/CheckResult.java 0b4240f 
  ql/src/java/org/apache/hadoop/hive/ql/metadata/HiveMetaStoreChecker.java 
598bb2e 
  ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java 29f6ecf 
  ql/src/java/org/apache/hadoop/hive/ql/plan/CreateTableDesc.java 27f677e 
  
ql/src/test/org/apache/hadoop/hive/ql/exec/TestMsckCreatePartitionsInBatches.java
 ce2b186 
  
ql/src/test/org/apache/hadoop/hive/ql/exec/TestMsckDropPartitionsInBatches.java 
7e768da 
  ql/src/test/org/apache/hadoop/hive/ql/metadata/TestHiveMetaStoreChecker.java 
a2a0583 
  ql/src/test/queries/clientpositive/msck_repair_0.q aeb4820 
  ql/src/test/queries/clientpositive/msck_repair_2.q be745b2 
  ql/src/test/queries/clientpositive/msck_repair_3.q 140a690 
  ql/src/test/queries/clientpositive/msck_repair_acid.q PRE-CREATION 
  ql/src/test/queries/clientpositive/msck_repair_batchsize.q 5a7afcc 
  ql/src/test/queries/clientpositive/msck_repair_drop.q 9923fb5 
  ql/src/test/queries/clientpositive/partition_discovery.q PRE-CREATION 
  ql/src/test/results/clientpositive/msck_repair_0.q.out fa6e4a9 
  ql/src/test/results/clientpositive/msck_repair_2.q.out 7fbd934 
  ql/src/test/results/clientpositive/msck_repair_3.q.out 0e153fb 
  ql/src/test/results/clientpositive/msck_repair_acid.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/msck_repair_batchsize.q.out ab4b831 
  ql/src/test/results/clientpositive/msck_repair_drop.q.out 971c138 
  ql/src/test/results/clientpositive/partition_discovery.q.out PRE-CREATION 
  
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/CheckResult.java
 PRE-CREATION 
  
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/Warehouse.java
 294dfb7 
  
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/api/MetastoreException.java
 PRE-CREATION 
  
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java
 7b01678 
  
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/utils/MetaStoreUtils.java
 16f4a50 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreChecker.java
 PRE-CREATION 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/Msck.java
 PRE-CREATION 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MsckInfo.java
 PRE-CREATION 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MsckPartitionExpressionProxy.java
 PRE-CREATION 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
 66977d7 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/PartitionIterable.java
 PRE-CREATION 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/PartitionManagementTask.java
 PRE-CREATION 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/utils/MetaStoreServerUtils.java
 f3b3866 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/utils/RetryUtilities.java
 PRE-CREATION 
  
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/TestPartitionManagement.java
 PRE-CREATION 


Diff: https://reviews.apache.org/r/68946/diff/2/


Testing
---


Thanks,

Prasanth_J

Re: Review Request 68946: HIVE-20707: Automatic MSCK REPAIR for external tables

2018-10-11 Thread j . prasanth . j


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68946/
---

(Updated Oct. 12, 2018, 1:35 a.m.)


Review request for hive, Ashutosh Chauhan and Jason Dere.


Changes
---

There is more scope added to this patch

1) Support managed and external tables. Managed tables will get X lock for MSCK 
REPAIR.

2) Retention period for partitions can be specified via table property.

3) MSCK repair can now handle retention periods via config.

4) All external tables will now automatically enable partition discovery.

On top of these there are fixes related to ACID, MSCK was acquiring wrong lock, 
explain locks was not working for msck, fixes related to catalog handling (some 
metastore APIs was not handling catalog names correctly).


Bugs: HIVE-20707
https://issues.apache.org/jira/browse/HIVE-20707


Repository: hive-git


Description
---

HIVE-20707: Automatic MSCK REPAIR for external tables


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java cc6239c 
  
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/TestCompactor.java
 cffa21a 
  ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java e913462 
  ql/src/java/org/apache/hadoop/hive/ql/exec/ExplainTask.java 46bf088 
  ql/src/java/org/apache/hadoop/hive/ql/metadata/CheckResult.java 0b4240f 
  ql/src/java/org/apache/hadoop/hive/ql/metadata/HiveMetaStoreChecker.java 
598bb2e 
  ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java 29f6ecf 
  ql/src/java/org/apache/hadoop/hive/ql/plan/CreateTableDesc.java 27f677e 
  
ql/src/test/org/apache/hadoop/hive/ql/exec/TestMsckCreatePartitionsInBatches.java
 ce2b186 
  
ql/src/test/org/apache/hadoop/hive/ql/exec/TestMsckDropPartitionsInBatches.java 
7e768da 
  ql/src/test/org/apache/hadoop/hive/ql/metadata/TestHiveMetaStoreChecker.java 
a2a0583 
  ql/src/test/queries/clientpositive/msck_repair_0.q aeb4820 
  ql/src/test/queries/clientpositive/msck_repair_2.q be745b2 
  ql/src/test/queries/clientpositive/msck_repair_3.q 140a690 
  ql/src/test/queries/clientpositive/msck_repair_acid.q PRE-CREATION 
  ql/src/test/queries/clientpositive/msck_repair_batchsize.q 5a7afcc 
  ql/src/test/queries/clientpositive/msck_repair_drop.q 9923fb5 
  ql/src/test/queries/clientpositive/partition_discovery.q PRE-CREATION 
  ql/src/test/results/clientpositive/msck_repair_0.q.out fa6e4a9 
  ql/src/test/results/clientpositive/msck_repair_2.q.out 7fbd934 
  ql/src/test/results/clientpositive/msck_repair_3.q.out 0e153fb 
  ql/src/test/results/clientpositive/msck_repair_acid.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/msck_repair_batchsize.q.out ab4b831 
  ql/src/test/results/clientpositive/msck_repair_drop.q.out 971c138 
  ql/src/test/results/clientpositive/partition_discovery.q.out PRE-CREATION 
  
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/CheckResult.java
 PRE-CREATION 
  
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/Warehouse.java
 294dfb7 
  
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/api/MetastoreException.java
 PRE-CREATION 
  
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java
 7b01678 
  
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/utils/MetaStoreUtils.java
 16f4a50 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreChecker.java
 PRE-CREATION 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/Msck.java
 PRE-CREATION 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MsckInfo.java
 PRE-CREATION 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MsckPartitionExpressionProxy.java
 PRE-CREATION 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
 66977d7 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/PartitionIterable.java
 PRE-CREATION 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/PartitionManagementTask.java
 PRE-CREATION 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/utils/MetaStoreServerUtils.java
 f3b3866 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/utils/RetryUtilities.java
 PRE-CREATION 
  
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/TestPartitionManagement.java
 PRE-CREATION 


Diff: https://reviews.apache.org/r/68946/diff/2/

Changes: https://reviews.apache.org/r/68946/diff/1-2/


Testing
---


Thanks,

Prasanth_J

Review Request 68946: HIVE-20707: Automatic MSCK REPAIR for external tables

2018-10-07 Thread j . prasanth . j


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68946/
---

Review request for hive and Ashutosh Chauhan.


Bugs: HIVE-20707
https://issues.apache.org/jira/browse/HIVE-20707


Repository: hive-git


Description
---

HIVE-20707: Automatic MSCK REPAIR for external tables


Diffs
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 
d0adc35544cb8ae9d007a1d2ccb9b9565eedca88 
  data/conf/hive-site.xml 0daf9adc717bc1c4413d2e34691c26a3e2585c77 
  
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/TestCompactor.java
 cffa21af33d5abb2162fa16b6b990a469075f03d 
  ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 
e91346228e8724b8253364114145a348a7cbee26 
  ql/src/java/org/apache/hadoop/hive/ql/metadata/CheckResult.java 
0b4240f5665f0b544b2fc5864fc098eb286a281e 
  ql/src/java/org/apache/hadoop/hive/ql/metadata/HiveMetaStoreChecker.java 
598bb2ee8b72f1b7f75be7802b4eaae0204c988d 
  
ql/src/test/org/apache/hadoop/hive/ql/exec/TestMsckCreatePartitionsInBatches.java
 ce2b186b4dceda780106776daa022f18388ec76f 
  
ql/src/test/org/apache/hadoop/hive/ql/exec/TestMsckDropPartitionsInBatches.java 
7e768dacb0b00a0f1a9e64efbe778f9c2daaa31b 
  ql/src/test/org/apache/hadoop/hive/ql/metadata/TestHiveMetaStoreChecker.java 
a2a0583d4dbdfe9aece1a14ecac24e0e6189cafa 
  ql/src/test/queries/clientpositive/auto_msck_repair_0.q PRE-CREATION 
  ql/src/test/queries/clientpositive/auto_msck_repair_1.q PRE-CREATION 
  ql/src/test/queries/clientpositive/auto_msck_repair_2.q PRE-CREATION 
  ql/src/test/queries/clientpositive/auto_msck_repair_3.q PRE-CREATION 
  ql/src/test/queries/clientpositive/auto_msck_repair_4.q PRE-CREATION 
  ql/src/test/queries/clientpositive/auto_msck_repair_batchsize.q PRE-CREATION 
  ql/src/test/queries/clientpositive/msck_repair_0.q 
aeb4820af5b6687f7ae4163a94bdd2be25a8b0cd 
  ql/src/test/queries/clientpositive/msck_repair_2.q 
be745b2d607d8c727b862c71f153f09d5622a8b5 
  ql/src/test/queries/clientpositive/msck_repair_3.q 
140a6904ddc98b165d71a8b24314c56888ccbb9c 
  ql/src/test/queries/clientpositive/msck_repair_batchsize.q 
5a7afcca5b86c1887308626c0dc4d99916811bea 
  ql/src/test/queries/clientpositive/msck_repair_drop.q 
9923fb50cbdbdf9e8e07276ccaec073c490770e6 
  ql/src/test/results/clientpositive/auto_msck_repair_0.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/auto_msck_repair_1.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/auto_msck_repair_2.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/auto_msck_repair_3.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/auto_msck_repair_4.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/auto_msck_repair_batchsize.q.out 
PRE-CREATION 
  ql/src/test/results/clientpositive/msck_repair_0.q.out 
fa6e4a988273a71b0f9dab64a48ddda6320d5f2f 
  ql/src/test/results/clientpositive/msck_repair_2.q.out 
7fbd934e118e81b9c5f028191c7ea6582a34db75 
  ql/src/test/results/clientpositive/msck_repair_3.q.out 
0e153fbe69ba39819fac4629ef1bf5f90c17f37f 
  ql/src/test/results/clientpositive/msck_repair_batchsize.q.out 
ab4b83137dcf1ce36846ce74e0a546528e81358b 
  ql/src/test/results/clientpositive/msck_repair_drop.q.out 
971c1381276fa626bd91d34488a65e3bfb2781ae 
  
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/Warehouse.java
 294dfb728e12efaa13d239ea7b8949587a50fe1f 
  
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/api/MetastoreException.java
 PRE-CREATION 
  
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java
 7b01678a10f4f0667844fec64ae76695d835bd6e 
  
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/utils/MetaStoreUtils.java
 16f4a50d69f9120d565f61d028b060d7776689fc 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/AutoMsckTask.java
 PRE-CREATION 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/Msck.java
 PRE-CREATION 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MsckInfo.java
 PRE-CREATION 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MsckPartitionExpressionProxy.java
 PRE-CREATION 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/PartitionIterable.java
 PRE-CREATION 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/utils/MetaStoreServerUtils.java
 f3b38665676391fec9b85eb9a405c14632340dc6 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/utils/RetryUtilities.java
 PRE-CREATION 


Diff: https://reviews.apache.org/r/68946/diff/1/


Testing
---


Thanks,

Prasanth_J

Re: Review Request 68709: HIVE-20547 HS2: support Tez sessions started by someone else (part 1)

2018-09-13 Thread j . prasanth . j


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68709/#review208608
---


Ship it!




Ship It!

- Prasanth_J


On Sept. 13, 2018, 8:11 p.m., Sergey Shelukhin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/68709/
> ---
> 
> (Updated Sept. 13, 2018, 8:11 p.m.)
> 
> 
> Review request for hive and Prasanth_J.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> see jira
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java aa58d7445c 
>   itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java 
> bfa3d5d7d2 
>   ql/pom.xml d73deba440 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 79cb54e552 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/tez/AbstractTriggerValidator.java 
> PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/tez/KillTriggerActionHandler.java 
> f357775c86 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezExternalSessionState.java 
> PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezExternalSessionsRegistryClient.java
>  PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSession.java PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionPool.java 
> 89954cba67 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionPoolManager.java 
> 2633390861 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionPoolSession.java 
> d3748edb86 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionState.java 
> 08e65a4a6d 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezTask.java f2ed07add5 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TriggerValidatorRunnable.java 
> 670184b0ac 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/WmTezSession.java fa2b02e591 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/WorkloadManager.java 
> 5326e3590f 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/tez/WorkloadManagerFederation.java 
> 4b5022a101 
>   ql/src/java/org/apache/hadoop/hive/ql/session/KillQuery.java 01dc7e2cd7 
>   ql/src/java/org/apache/hadoop/hive/ql/session/NullKillQuery.java eac2936719 
>   ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java 71e130b608 
>   ql/src/java/org/apache/hadoop/hive/ql/wm/SessionTriggerProvider.java 
> 16106f481b 
>   ql/src/test/org/apache/hadoop/hive/ql/exec/tez/SampleTezSessionState.java 
> f5ab981f26 
>   ql/src/test/org/apache/hadoop/hive/ql/exec/tez/TestTezSessionPool.java 
> d5b683f788 
>   ql/src/test/org/apache/hadoop/hive/ql/exec/tez/TestTezTask.java b67aec371d 
>   ql/src/test/org/apache/hadoop/hive/ql/exec/tez/TestWorkloadManager.java 
> 4659ecb97b 
>   service/src/java/org/apache/hive/service/server/KillQueryImpl.java 
> 490a04da67 
> 
> 
> Diff: https://reviews.apache.org/r/68709/diff/2/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Sergey Shelukhin
> 
>

Re: Review Request 68709: HIVE-20547 HS2: support Tez sessions started by someone else (part 1)

2018-09-13 Thread j . prasanth . j



> On Sept. 13, 2018, 8:03 a.m., Prasanth_J wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezExternalSessionsRegistryClient.java
> > Lines 37 (patched)
> > 
> >
> > Yes. Tez needs severals fixes in addition to registry client
> > 1) One that cuts of yarn/RM interaction
> > 2) Registry configs via TezConfiguration
> > 3) Handling DAG recovery for external sessions
> 
> Sergey Shelukhin wrote:
> can you file some Tez jiras for that?

https://issues.apache.org/jira/browse/TEZ-3991


- Prasanth_J


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68709/#review208584
---


On Sept. 13, 2018, 2:19 a.m., Sergey Shelukhin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/68709/
> ---
> 
> (Updated Sept. 13, 2018, 2:19 a.m.)
> 
> 
> Review request for hive and Prasanth_J.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> see jira
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java aa58d7445c 
>   itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java 
> bfa3d5d7d2 
>   ql/pom.xml a55cbe380d 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/tez/AbstractTriggerValidator.java 
> PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/tez/KillTriggerActionHandler.java 
> f357775c86 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezExternalSessionState.java 
> PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezExternalSessionsRegistryClient.java
>  PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSession.java PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionPool.java 
> 89954cba67 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionPoolManager.java 
> 2633390861 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionPoolSession.java 
> d3748edb86 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionState.java 
> 08e65a4a6d 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezTask.java f2ed07add5 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TriggerValidatorRunnable.java 
> 670184b0ac 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/WmTezSession.java fa2b02e591 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/WorkloadManager.java 
> 5326e3590f 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/tez/WorkloadManagerFederation.java 
> 4b5022a101 
>   ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java 71e130b608 
>   ql/src/java/org/apache/hadoop/hive/ql/wm/SessionTriggerProvider.java 
> 16106f481b 
>   ql/src/test/org/apache/hadoop/hive/ql/exec/tez/SampleTezSessionState.java 
> f5ab981f26 
>   ql/src/test/org/apache/hadoop/hive/ql/exec/tez/TestTezSessionPool.java 
> d5b683f788 
>   ql/src/test/org/apache/hadoop/hive/ql/exec/tez/TestTezTask.java b67aec371d 
>   ql/src/test/org/apache/hadoop/hive/ql/exec/tez/TestWorkloadManager.java 
> 4659ecb97b 
> 
> 
> Diff: https://reviews.apache.org/r/68709/diff/1/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Sergey Shelukhin
> 
>

Re: Review Request 68709: HIVE-20547 HS2: support Tez sessions started by someone else (part 1)

2018-09-13 Thread j . prasanth . j


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68709/#review208584
---




ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezExternalSessionState.java
Lines 72 (patched)


These resources may not be required. External sessions are assumed to have 
all the resources that it requires.



ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezExternalSessionState.java
Lines 139 (patched)


should it return the one that doesn't talk to yarn? does "external" also 
mean no yarn/RM interaction everywhere in hive side or will that be in a 
separate jira?



ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezExternalSessionsRegistryClient.java
Lines 37 (patched)


Yes. Tez needs severals fixes in addition to registry client
1) One that cuts of yarn/RM interaction
2) Registry configs via TezConfiguration
3) Handling DAG recovery for external sessions



ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionPoolSession.java
Lines 74 (patched)


nit: rename to baseSession?


- Prasanth_J


On Sept. 13, 2018, 2:19 a.m., Sergey Shelukhin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/68709/
> ---
> 
> (Updated Sept. 13, 2018, 2:19 a.m.)
> 
> 
> Review request for hive and Prasanth_J.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> see jira
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java aa58d7445c 
>   itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java 
> bfa3d5d7d2 
>   ql/pom.xml a55cbe380d 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/tez/AbstractTriggerValidator.java 
> PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/tez/KillTriggerActionHandler.java 
> f357775c86 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezExternalSessionState.java 
> PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezExternalSessionsRegistryClient.java
>  PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSession.java PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionPool.java 
> 89954cba67 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionPoolManager.java 
> 2633390861 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionPoolSession.java 
> d3748edb86 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionState.java 
> 08e65a4a6d 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezTask.java f2ed07add5 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TriggerValidatorRunnable.java 
> 670184b0ac 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/WmTezSession.java fa2b02e591 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/WorkloadManager.java 
> 5326e3590f 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/tez/WorkloadManagerFederation.java 
> 4b5022a101 
>   ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java 71e130b608 
>   ql/src/java/org/apache/hadoop/hive/ql/wm/SessionTriggerProvider.java 
> 16106f481b 
>   ql/src/test/org/apache/hadoop/hive/ql/exec/tez/SampleTezSessionState.java 
> f5ab981f26 
>   ql/src/test/org/apache/hadoop/hive/ql/exec/tez/TestTezSessionPool.java 
> d5b683f788 
>   ql/src/test/org/apache/hadoop/hive/ql/exec/tez/TestTezTask.java b67aec371d 
>   ql/src/test/org/apache/hadoop/hive/ql/exec/tez/TestWorkloadManager.java 
> 4659ecb97b 
> 
> 
> Diff: https://reviews.apache.org/r/68709/diff/1/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Sergey Shelukhin
> 
>

Re: Review Request 68115: HIVE-20274: HiveServer2 ObjectInspectorFactory leaks for Struct and List object inspectors

2018-08-01 Thread j . prasanth . j


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68115/
---

(Updated Aug. 1, 2018, 10:32 p.m.)


Review request for hive, Gopal V and Sergey Shelukhin.


Changes
---

made oi cache unbounded soft valued cache.


Bugs: HIVE-20274
https://issues.apache.org/jira/browse/HIVE-20274


Repository: hive-git


Description
---

HIVE-20274: HiveServer2 ObjectInspectorFactory leaks for Struct and List object 
inspectors


Diffs (updated)
-

  
serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ObjectInspectorFactory.java
 b45a9de 
  
serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ReflectionStructObjectInspector.java
 3a88a40 
  
serde/src/test/org/apache/hadoop/hive/serde2/objectinspector/TestReflectionObjectInspectors.java
 cb1440e 


Diff: https://reviews.apache.org/r/68115/diff/2/

Changes: https://reviews.apache.org/r/68115/diff/1-2/


Testing
---


Thanks,

Prasanth_J

Re: Review Request 68115: HIVE-20274: HiveServer2 ObjectInspectorFactory leaks for Struct and List object inspectors

2018-07-30 Thread j . prasanth . j



> On July 31, 2018, 12:17 a.m., Gopal V wrote:
> > serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ObjectInspectorFactory.java
> > Lines 76 (patched)
> > 
> >
> > The type -> ObjectInspector is the only one which I think looks like it 
> > is there for "intern-ing" of object inpsectors (So that object == can be 
> > used instead of a deep equals check).
> > 
> > Of course, I'm not sure that's a "real" problem unless a query takes > 
> > 5 minutes to compile.
> > 
> > If that doesn't show up in the leaks, it might better to leave that one 
> > alone.

but, unbounded cache adversely affect the query compile if HS2 goes in GC 
pause? It is still softvalues but that doesn't get cleaned up aggressively 
(until full GC). Being bounded also gives predictable perf (doesn't reach the 
point of GC). right?


- Prasanth_J


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68115/#review206627
---


On July 31, 2018, 12:08 a.m., Prasanth_J wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/68115/
> ---
> 
> (Updated July 31, 2018, 12:08 a.m.)
> 
> 
> Review request for hive, Gopal V and Sergey Shelukhin.
> 
> 
> Bugs: HIVE-20274
> https://issues.apache.org/jira/browse/HIVE-20274
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-20274: HiveServer2 ObjectInspectorFactory leaks for Struct and List 
> object inspectors
> 
> 
> Diffs
> -
> 
>   
> serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ObjectInspectorFactory.java
>  b45a9dee72f5fb3a139ee205a05e23c068b1fa0c 
>   
> serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ReflectionStructObjectInspector.java
>  3a88a400250afa0e29001ec118e006f937678089 
>   
> serde/src/test/org/apache/hadoop/hive/serde2/objectinspector/TestReflectionObjectInspectors.java
>  cb1440e98b8fad399c4c02a7440d2a30483455f4 
> 
> 
> Diff: https://reviews.apache.org/r/68115/diff/1/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Prasanth_J
> 
>

Re: Review Request 68115: HIVE-20274: HiveServer2 ObjectInspectorFactory leaks for Struct and List object inspectors

2018-07-30 Thread j . prasanth . j


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68115/
---

(Updated July 31, 2018, 12:08 a.m.)


Review request for hive, Gopal V and Sergey Shelukhin.


Bugs: HIVE-20274
https://issues.apache.org/jira/browse/HIVE-20274


Repository: hive-git


Description
---

HIVE-20274: HiveServer2 ObjectInspectorFactory leaks for Struct and List object 
inspectors


Diffs
-

  
serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ObjectInspectorFactory.java
 b45a9dee72f5fb3a139ee205a05e23c068b1fa0c 
  
serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ReflectionStructObjectInspector.java
 3a88a400250afa0e29001ec118e006f937678089 
  
serde/src/test/org/apache/hadoop/hive/serde2/objectinspector/TestReflectionObjectInspectors.java
 cb1440e98b8fad399c4c02a7440d2a30483455f4 


Diff: https://reviews.apache.org/r/68115/diff/1/


Testing
---


Thanks,

Prasanth_J

Review Request 68115: HIVE-20274: HiveServer2 ObjectInspectorFactory leaks for Struct and List object inspectors

2018-07-30 Thread j . prasanth . j


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68115/
---

Review request for hive, Gopal V and Sergey Shelukhin.


Repository: hive-git


Description
---

HIVE-20274: HiveServer2 ObjectInspectorFactory leaks for Struct and List object 
inspectors


Diffs
-

  
serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ObjectInspectorFactory.java
 b45a9dee72f5fb3a139ee205a05e23c068b1fa0c 
  
serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ReflectionStructObjectInspector.java
 3a88a400250afa0e29001ec118e006f937678089 
  
serde/src/test/org/apache/hadoop/hive/serde2/objectinspector/TestReflectionObjectInspectors.java
 cb1440e98b8fad399c4c02a7440d2a30483455f4 


Diff: https://reviews.apache.org/r/68115/diff/1/


Testing
---


Thanks,

Prasanth_J

Re: Review Request 68075: HIVE-20249: LLAP IO: NPE during refCount decrement

2018-07-26 Thread j . prasanth . j


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68075/
---

(Updated July 27, 2018, 12:21 a.m.)


Review request for hive and Sergey Shelukhin.


Changes
---

moved locked buffers dump to trace log


Bugs: HIVE-20249
https://issues.apache.org/jira/browse/HIVE-20249


Repository: hive-git


Description
---

HIVE-20249: LLAP IO: NPE during refCount decrement


Diffs (updated)
-

  llap-server/src/java/org/apache/hadoop/hive/llap/cache/BuddyAllocator.java 
fcfc22a 
  llap-server/src/java/org/apache/hadoop/hive/llap/cache/LowLevelCacheImpl.java 
53bdc2a 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/EncodedReaderImpl.java 
348f9df 


Diff: https://reviews.apache.org/r/68075/diff/2/

Changes: https://reviews.apache.org/r/68075/diff/1-2/


Testing
---


Thanks,

Prasanth_J

Review Request 68075: HIVE-20249: LLAP IO: NPE during refCount decrement

2018-07-26 Thread j . prasanth . j


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68075/
---

Review request for hive and Sergey Shelukhin.


Bugs: HIVE-20249
https://issues.apache.org/jira/browse/HIVE-20249


Repository: hive-git


Description
---

HIVE-20249: LLAP IO: NPE during refCount decrement


Diffs
-

  llap-server/src/java/org/apache/hadoop/hive/llap/cache/BuddyAllocator.java 
fcfc22a7128f3d17af24c538a452e7b91a5e21f0 
  llap-server/src/java/org/apache/hadoop/hive/llap/cache/LowLevelCacheImpl.java 
53bdc2a49838e76f507ec6c20eb3d01ed78f17a9 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/EncodedReaderImpl.java 
348f9df773ff26b42d377e19947afaa5b1f51255 


Diff: https://reviews.apache.org/r/68075/diff/1/


Testing
---


Thanks,

Prasanth_J

Re: Review Request 67966: HIVE-20202: Add profiler endpoint to httpserver

2018-07-21 Thread j . prasanth . j


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67966/
---

(Updated July 21, 2018, 9:47 a.m.)


Review request for hive and Gopal V.


Changes
---

if output is invalid the serving servlet adds refresh header


Bugs: HIVE-20202
https://issues.apache.org/jira/browse/HIVE-20202


Repository: hive-git


Description
---

HIVE-20202: Add profiler endpoint to httpserver


Diffs (updated)
-

  bin/hive 1ade51e 
  common/src/java/org/apache/hive/common/util/ProcessUtils.java PRE-CREATION 
  common/src/java/org/apache/hive/http/HttpServer.java 3cb7a33 
  common/src/java/org/apache/hive/http/ProfileOutputServlet.java PRE-CREATION 
  common/src/java/org/apache/hive/http/ProfileServlet.java PRE-CREATION 


Diff: https://reviews.apache.org/r/67966/diff/6/

Changes: https://reviews.apache.org/r/67966/diff/5-6/


Testing
---


Thanks,

Prasanth_J

Re: Review Request 67966: HIVE-20202: Add profiler endpoint to httpserver

2018-07-19 Thread j . prasanth . j


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67966/
---

(Updated July 19, 2018, 8:10 p.m.)


Review request for hive and Gopal V.


Changes
---

prof-output is the output dir and is added as resource base. /prof set refresh 
header to point to /prof-outpu/file.svg. More friendly output file name.


Bugs: HIVE-20202
https://issues.apache.org/jira/browse/HIVE-20202


Repository: hive-git


Description
---

HIVE-20202: Add profiler endpoint to httpserver


Diffs (updated)
-

  bin/hive 1ade51e 
  common/src/java/org/apache/hive/common/util/ProcessUtils.java PRE-CREATION 
  common/src/java/org/apache/hive/http/HttpServer.java 3cb7a33 
  common/src/java/org/apache/hive/http/ProfileServlet.java PRE-CREATION 


Diff: https://reviews.apache.org/r/67966/diff/5/

Changes: https://reviews.apache.org/r/67966/diff/4-5/


Testing
---


Thanks,

Prasanth_J

Re: Review Request 67966: HIVE-20202: Add profiler endpoint to httpserver

2018-07-19 Thread j . prasanth . j


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67966/
---

(Updated July 19, 2018, 7:05 a.m.)


Review request for hive and Gopal V.


Changes
---

changed default duration. minor fixes (review comments + no error stream 
redirection in runCmdSync).


Bugs: HIVE-20202
https://issues.apache.org/jira/browse/HIVE-20202


Repository: hive-git


Description
---

HIVE-20202: Add profiler endpoint to httpserver


Diffs (updated)
-

  bin/hive 1ade51e 
  common/src/java/org/apache/hive/common/util/ProcessUtils.java PRE-CREATION 
  common/src/java/org/apache/hive/http/HttpServer.java 3cb7a33 
  common/src/java/org/apache/hive/http/ProfileServlet.java PRE-CREATION 


Diff: https://reviews.apache.org/r/67966/diff/4/

Changes: https://reviews.apache.org/r/67966/diff/3-4/


Testing
---


Thanks,

Prasanth_J

Re: Review Request 67966: HIVE-20202: Add profiler endpoint to httpserver

2018-07-18 Thread j . prasanth . j


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67966/
---

(Updated July 19, 2018, 1:08 a.m.)


Review request for hive and Gopal V.


Changes
---

addressed review comments.


Bugs: HIVE-20202
https://issues.apache.org/jira/browse/HIVE-20202


Repository: hive-git


Description
---

HIVE-20202: Add profiler endpoint to httpserver


Diffs (updated)
-

  bin/hive 1ade51e 
  common/src/java/org/apache/hive/common/util/ProcessUtils.java PRE-CREATION 
  common/src/java/org/apache/hive/http/HttpServer.java 3cb7a33 
  common/src/java/org/apache/hive/http/ProfileServlet.java PRE-CREATION 


Diff: https://reviews.apache.org/r/67966/diff/3/

Changes: https://reviews.apache.org/r/67966/diff/2-3/


Testing
---


Thanks,

Prasanth_J

Re: Review Request 67966: HIVE-20202: Add profiler endpoint to httpserver

2018-07-18 Thread j . prasanth . j


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67966/
---

(Updated July 18, 2018, 6:42 p.m.)


Review request for hive and Gopal V.


Changes
---

minor comment fix


Bugs: HIVE-20202
https://issues.apache.org/jira/browse/HIVE-20202


Repository: hive-git


Description
---

HIVE-20202: Add profiler endpoint to httpserver


Diffs (updated)
-

  bin/hive 1ade51e 
  common/src/java/org/apache/hive/common/util/ProcessUtils.java PRE-CREATION 
  common/src/java/org/apache/hive/http/HttpServer.java 3cb7a33 
  common/src/java/org/apache/hive/http/ProfileServlet.java PRE-CREATION 


Diff: https://reviews.apache.org/r/67966/diff/2/

Changes: https://reviews.apache.org/r/67966/diff/1-2/


Testing
---


Thanks,

Prasanth_J

Review Request 67966: HIVE-20202: Add profiler endpoint to httpserver

2018-07-18 Thread j . prasanth . j


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67966/
---

Review request for hive and Gopal V.


Bugs: HIVE-20202
https://issues.apache.org/jira/browse/HIVE-20202


Repository: hive-git


Description
---

HIVE-20202: Add profiler endpoint to httpserver


Diffs
-

  bin/hive 1ade51eebd6eac18c8a2295dd995263988d03e97 
  common/src/java/org/apache/hive/common/util/ProcessUtils.java PRE-CREATION 
  common/src/java/org/apache/hive/http/HttpServer.java 
3cb7a33c4e753bced603d8ef1a5917748b7a2c0a 
  common/src/java/org/apache/hive/http/ProfileServlet.java PRE-CREATION 


Diff: https://reviews.apache.org/r/67966/diff/1/


Testing
---


Thanks,

Prasanth_J

Review Request 67858: HIVE-20126: OrcInputFormat does not pass conf to orc reader options

2018-07-09 Thread j . prasanth . j


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67858/
---

Review request for hive and Sergey Shelukhin.


Bugs: HIVE-20126
https://issues.apache.org/jira/browse/HIVE-20126


Repository: hive-git


Description
---

HIVE-20126: OrcInputFormat does not pass conf to orc reader options


Diffs
-

  data/files/studenttab10k PRE-CREATION 
  itests/src/test/resources/testconfiguration.properties 
2795e3deca407e45ec22b43767ff48c906cc3823 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java 
3c118474af67f2cc680a07747f142af9433db04a 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/VectorizedOrcInputFormat.java 
892fcc0843b1137317ac00343a5ba55419baadc5 
  ql/src/test/queries/clientpositive/orc_schema_evol_quoted.q PRE-CREATION 
  ql/src/test/results/clientpositive/llap/orc_schema_evol_quoted.q.out 
PRE-CREATION 
  ql/src/test/results/clientpositive/orc_schema_evol_quoted.q.out PRE-CREATION 


Diff: https://reviews.apache.org/r/67858/diff/1/


Testing
---


Thanks,

Prasanth_J

Re: Review Request 54094: HIVE-15190: Field names are not preserved in ORC files written with ACID

2018-06-14 Thread j . prasanth . j


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/54094/#review204793
---




ql/src/test/org/apache/hadoop/hive/ql/TestTxnCommands2.java
Lines 1759 (patched)


Can you also add a test for complex types (nested) to see if those column 
names are retained as well?


- Prasanth_J


On Nov. 26, 2016, 11:03 p.m., Anthony Hsu wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/54094/
> ---
> 
> (Updated Nov. 26, 2016, 11:03 p.m.)
> 
> 
> Review request for hive.
> 
> 
> Bugs: HIVE-15190
> https://issues.apache.org/jira/browse/HIVE-15190
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Previously, when writing to an ACID ORC table, the file written to disk would 
> have a schema of `struct<...(acid 
> columns)...,row:struct<_col0:int,_col1:string,...>>`, using virtual column 
> names `_col0`, `_col1`, etc., instead of the actual table column names. This 
> patch fixes this issue.
> 
> Having the actual table column names in the ORC file itself is needed when 
> doing schema evolution based on field names: 
> https://issues.apache.org/jira/browse/ORC-54
> 
> 
> Diffs
> -
> 
>   orc/src/java/org/apache/orc/impl/SchemaEvolution.java 
> 7379de93a7f39d734ef7695c197bd9f24bc84321 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcFile.java 
> 53660206e3f59c37be261b1a9796f04721a244f3 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcRawRecordMerger.java 
> efde2db482367f1037c486df9c5cabd67b1368ed 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcRecordUpdater.java 
> 492c64c29e8d4f38d857381bc375074e06868f7c 
>   
> ql/src/java/org/apache/hadoop/hive/ql/io/orc/VectorizedOrcAcidRowBatchReader.java
>  75c7680e267ab44e426d0b21c6fd6dce6a352bbd 
>   ql/src/test/org/apache/hadoop/hive/ql/TestTxnCommands2.java 
> 49ba6675bae5b3e6d8bf1fa2e9ed8d2a27b7f83a 
> 
> 
> Diff: https://reviews.apache.org/r/54094/diff/1/
> 
> 
> Testing
> ---
> 
> Added unit test. Also ran some of the existing ACID tests and they still 
> passed.
> 
> 
> Thanks,
> 
> Anthony Hsu
> 
>

Review Request 67497: HIVE-19794: Disable removing order by from subquery in GenericUDTFGetSplits

2018-06-07 Thread j . prasanth . j


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67497/
---

Review request for hive and Jason Dere.


Bugs: HIVE-19794
https://issues.apache.org/jira/browse/HIVE-19794


Repository: hive-git


Description
---

HIVE-19794: Disable removing order by from subquery in GenericUDTFGetSplits


Diffs
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 
dd42fd127e633304a2da499afa60f7b051d329a9 
  
itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestJdbcGenericUDTFGetSplits.java
 PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/HiveSplitGenerator.java 
57f6c66a56a88bb7383ebe5832bba75240dea554 
  ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDTFGetSplits.java 
20d09611ccdf863d5a5e7dc811efe091f7b4aba2 


Diff: https://reviews.apache.org/r/67497/diff/1/


Testing
---


Thanks,

Prasanth_J

Review Request 67474: HIVE-19817: Hive streaming API + dynamic partitioning + json/regex writer does not work

2018-06-06 Thread j . prasanth . j


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67474/
---

Review request for hive and Eugene Koifman.


Bugs: HIVE-19817
https://issues.apache.org/jira/browse/HIVE-19817


Repository: hive-git


Description
---

HIVE-19817: Hive streaming API + dynamic partitioning + json/regex writer does 
not work


Diffs
-

  serde/src/java/org/apache/hadoop/hive/serde2/JsonSerDe.java 
40b2e8e27ffde82436523d41dd8515c2dd1208e3 
  streaming/src/java/org/apache/hive/streaming/HiveStreamingConnection.java 
0f9260d00b7003579e9e63daab91f80e180a1545 
  streaming/src/java/org/apache/hive/streaming/StrictJsonWriter.java 
0f9b6520a3bd84c79b20ae6e98197e363388c9fe 
  streaming/src/java/org/apache/hive/streaming/StrictRegexWriter.java 
3651fa120ae5c753d2a833bc28c9dcfcd24be823 
  
streaming/src/test/org/apache/hive/streaming/TestStreamingDynamicPartitioning.java
 e51391595c19f044e4fc6aaa16f32285e33926ff 


Diff: https://reviews.apache.org/r/67474/diff/1/


Testing
---


Thanks,

Prasanth_J

Re: Review Request 67420: HIVE-19772: Streaming ingest V2 API can generate invalid orc file if interrupted

2018-06-05 Thread j . prasanth . j


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67420/
---

(Updated June 5, 2018, 6:39 a.m.)


Review request for hive, Eugene Koifman and Gopal V.


Changes
---

Creating filesystem once per partition.


Bugs: HIVE-19772
https://issues.apache.org/jira/browse/HIVE-19772


Repository: hive-git


Description
---

HIVE-19772: Streaming ingest V2 API can generate invalid orc file if interrupted


Diffs (updated)
-

  ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcRecordUpdater.java e187ce1 
  streaming/src/java/org/apache/hive/streaming/AbstractRecordWriter.java 
0866850 
  streaming/src/java/org/apache/hive/streaming/HiveStreamingConnection.java 
f697211 


Diff: https://reviews.apache.org/r/67420/diff/2/

Changes: https://reviews.apache.org/r/67420/diff/1-2/


Testing
---


Thanks,

Prasanth_J

Re: Review Request 67406: HIVE-19663 refactor LLAP IO report generation

2018-06-01 Thread j . prasanth . j


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67406/#review204216
---


Ship it!




Ship It!

- Prasanth_J


On May 31, 2018, 10:19 p.m., Sergey Shelukhin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/67406/
> ---
> 
> (Updated May 31, 2018, 10:19 p.m.)
> 
> 
> Review request for hive and Prasanth_J.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> see jira
> 
> 
> Diffs
> -
> 
>   llap-server/src/java/org/apache/hadoop/hive/llap/cache/BuddyAllocator.java 
> f4a549c529 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/cache/CacheContentsTracker.java
>  6a361fa4fc 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/cache/EvictionDispatcher.java
>  10a4bfbb58 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/cache/LlapOomDebugDump.java 
> 29ddf5c112 
>   llap-server/src/java/org/apache/hadoop/hive/llap/cache/LowLevelCache.java 
> af1b699771 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/cache/LowLevelCacheImpl.java 
> 5e102d93de 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/cache/LowLevelCacheMemoryManager.java
>  31cd45f103 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/cache/LowLevelCachePolicy.java
>  33236362b5 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/cache/LowLevelFifoCachePolicy.java
>  f7f80a8958 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/cache/LowLevelLrfuCachePolicy.java
>  7787cb4867 
>   llap-server/src/java/org/apache/hadoop/hive/llap/cache/MemoryManager.java 
> 65b1d4f654 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/cache/SerDeLowLevelCacheImpl.java
>  cb89d12e80 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/cache/SimpleBufferManager.java
>  a1b6caeb60 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/LlapIoImpl.java 
> 5003d9b5e5 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/io/metadata/MetadataCache.java
>  0184e3053f 
>   
> llap-server/src/test/org/apache/hadoop/hive/llap/cache/TestBuddyAllocator.java
>  714efef182 
>   
> llap-server/src/test/org/apache/hadoop/hive/llap/cache/TestLowLevelCacheImpl.java
>  b19cdcf531 
>   
> llap-server/src/test/org/apache/hadoop/hive/llap/cache/TestOrcMetadataCache.java
>  58c918c200 
> 
> 
> Diff: https://reviews.apache.org/r/67406/diff/1/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Sergey Shelukhin
> 
>

Review Request 67420: HIVE-19772: Streaming ingest V2 API can generate invalid orc file if interrupted

2018-06-01 Thread j . prasanth . j


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67420/
---

Review request for hive, Eugene Koifman and Gopal V.


Bugs: HIVE-19772
https://issues.apache.org/jira/browse/HIVE-19772


Repository: hive-git


Description
---

HIVE-19772: Streaming ingest V2 API can generate invalid orc file if interrupted


Diffs
-

  ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcRecordUpdater.java 
e187ce141b3c284a624dedd2a07e9d5f9ce87d0a 
  streaming/src/java/org/apache/hive/streaming/HiveStreamingConnection.java 
f69721123c5625f5a0cfa28239936ff37431b069 


Diff: https://reviews.apache.org/r/67420/diff/1/


Testing
---


Thanks,

Prasanth_J

Re: Review Request 67329: HIVE-19629: Enable Decimal64 reader after orc version upgrade

2018-05-25 Thread j . prasanth . j



> On May 25, 2018, 10:58 p.m., Prasanth_J wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/Vectorizer.java
> > Lines 4402 (patched)
> > 
> >
> > actually this is unused variable.. i left a note to fix some test case. 
> > will fixup this parent and child data type variation mismatch in next patch.

(d + 1) % (d / 2) where d is DECIMAL_64 exhibits this behaviour. Mod and Divide 
does not support DECIMAL_64 whereas Addition does. So this expression becomes 
(+: DECIMAL_64) (%: NONE) (/: NONE). Mod is parent whose output variation is 
NONE but one of its child (+) has DECIMAL_64 variation which should be casted 
up to avoid runtime ClassCastException.


- Prasanth_J


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67329/#review203920
---


On May 25, 2018, 8:25 p.m., Prasanth_J wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/67329/
> ---
> 
> (Updated May 25, 2018, 8:25 p.m.)
> 
> 
> Review request for hive and Matt McCline.
> 
> 
> Bugs: HIVE-19629
> https://issues.apache.org/jira/browse/HIVE-19629
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-19629: Enable Decimal64 reader after orc version upgrade
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 931533a 
>   itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/TestAcidOnTez.java 
> 0af91bd 
>   itests/src/test/resources/testconfiguration.properties d146f92 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/LlapInputFormat.java
>  6d29163 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/io/decode/GenericColumnVectorProducer.java
>  7af1b05 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/io/decode/OrcEncodedDataConsumer.java
>  feccb87 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/OrcEncodedDataReader.java
>  4033b37 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/SerDeEncodedDataReader.java
>  1cfe929 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/VectorDeserializeOrcWriter.java
>  de19b1d 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/io/metadata/ConsumerFileMetadata.java
>  bf139c0 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/io/metadata/OrcFileMetadata.java
>  0012afb 
>   pom.xml e48974b 
>   ql/pom.xml 06124f7 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/FetchOperator.java 2246901 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorizedInputFormatInterface.java
>  e74b185 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorizedRowBatchCtx.java 
> 6588385 
>   ql/src/java/org/apache/hadoop/hive/ql/io/NullRowsInputFormat.java e632d43 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java f461364 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcRawRecordMerger.java 
> 8c7c72e 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/Reader.java 7485e60 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/ReaderImpl.java 1a6db1f 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/RecordReaderImpl.java 5b001a0 
>   
> ql/src/java/org/apache/hadoop/hive/ql/io/orc/VectorizedOrcAcidRowBatchReader.java
>  d2e1a68 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/VectorizedOrcInputFormat.java 
> c581bba 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/WriterImpl.java 71682af 
>   
> ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/EncodedTreeReaderFactory.java
>  646b214 
>   
> ql/src/java/org/apache/hadoop/hive/ql/io/parquet/MapredParquetInputFormat.java
>  ed6d577 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/Vectorizer.java 
> 394f826 
>   ql/src/test/org/apache/hadoop/hive/ql/TestTxnNoBuckets.java af43b14 
>   ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestInputOutputFormat.java 
> fb2335a 
>   ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestOrcFile.java ef678a8 
>   ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestOrcRawRecordMerger.java 
> d8a7af8 
>   ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestOrcSerDeStats.java 1533ffa 
>   ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestVectorizedORCReader.java 
> 0c9c95d 
>   
> ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestVectorizedOrcAcidRowBatchReader.java
>  e478371 
>   ql/src/test/queries/clientpositive/llap_acid2.q a409c26 
>   ql/src/test/queries/clientpositive/llap_decimal64_reader.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/llap_uncompressed.q 875356c 
>   ql/src/test/results/clientpositive/acid_mapjoin.q.out 76a781e 
>   ql/src/test/results/clientpositive/acid_nullscan.q.out 6dad497 
>

Re: Review Request 67329: HIVE-19629: Enable Decimal64 reader after orc version upgrade

2018-05-25 Thread j . prasanth . j


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67329/#review203920
---




ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/Vectorizer.java
Lines 4402 (patched)


actually this is unused variable.. i left a note to fix some test case. 
will fixup this parent and child data type variation mismatch in next patch.


- Prasanth_J


On May 25, 2018, 8:25 p.m., Prasanth_J wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/67329/
> ---
> 
> (Updated May 25, 2018, 8:25 p.m.)
> 
> 
> Review request for hive and Matt McCline.
> 
> 
> Bugs: HIVE-19629
> https://issues.apache.org/jira/browse/HIVE-19629
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-19629: Enable Decimal64 reader after orc version upgrade
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 931533a 
>   itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/TestAcidOnTez.java 
> 0af91bd 
>   itests/src/test/resources/testconfiguration.properties d146f92 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/LlapInputFormat.java
>  6d29163 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/io/decode/GenericColumnVectorProducer.java
>  7af1b05 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/io/decode/OrcEncodedDataConsumer.java
>  feccb87 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/OrcEncodedDataReader.java
>  4033b37 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/SerDeEncodedDataReader.java
>  1cfe929 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/VectorDeserializeOrcWriter.java
>  de19b1d 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/io/metadata/ConsumerFileMetadata.java
>  bf139c0 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/io/metadata/OrcFileMetadata.java
>  0012afb 
>   pom.xml e48974b 
>   ql/pom.xml 06124f7 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/FetchOperator.java 2246901 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorizedInputFormatInterface.java
>  e74b185 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorizedRowBatchCtx.java 
> 6588385 
>   ql/src/java/org/apache/hadoop/hive/ql/io/NullRowsInputFormat.java e632d43 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java f461364 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcRawRecordMerger.java 
> 8c7c72e 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/Reader.java 7485e60 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/ReaderImpl.java 1a6db1f 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/RecordReaderImpl.java 5b001a0 
>   
> ql/src/java/org/apache/hadoop/hive/ql/io/orc/VectorizedOrcAcidRowBatchReader.java
>  d2e1a68 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/VectorizedOrcInputFormat.java 
> c581bba 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/WriterImpl.java 71682af 
>   
> ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/EncodedTreeReaderFactory.java
>  646b214 
>   
> ql/src/java/org/apache/hadoop/hive/ql/io/parquet/MapredParquetInputFormat.java
>  ed6d577 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/Vectorizer.java 
> 394f826 
>   ql/src/test/org/apache/hadoop/hive/ql/TestTxnNoBuckets.java af43b14 
>   ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestInputOutputFormat.java 
> fb2335a 
>   ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestOrcFile.java ef678a8 
>   ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestOrcRawRecordMerger.java 
> d8a7af8 
>   ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestOrcSerDeStats.java 1533ffa 
>   ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestVectorizedORCReader.java 
> 0c9c95d 
>   
> ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestVectorizedOrcAcidRowBatchReader.java
>  e478371 
>   ql/src/test/queries/clientpositive/llap_acid2.q a409c26 
>   ql/src/test/queries/clientpositive/llap_decimal64_reader.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/llap_uncompressed.q 875356c 
>   ql/src/test/results/clientpositive/acid_mapjoin.q.out 76a781e 
>   ql/src/test/results/clientpositive/acid_nullscan.q.out 6dad497 
>   ql/src/test/results/clientpositive/acid_table_stats.q.out 2596922 
>   ql/src/test/results/clientpositive/annotate_stats_part.q.out 9e45101 
>   ql/src/test/results/clientpositive/annotate_stats_table.q.out b502957 
>   ql/src/test/results/clientpositive/autoColumnStats_4.q.out 5be906e 
>   
> ql/src/test/results/clientpositive/beeline/materialized_view_create_rewrite.q.out
>  a717b80 
>   
> ql/src/test/results/clientpositive/columnStatsUpdateForStatsOptimizer_2.q.out 
> fa58add

Re: Review Request 67329: HIVE-19629: Enable Decimal64 reader after orc version upgrade

2018-05-25 Thread j . prasanth . j


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67329/
---

(Updated May 25, 2018, 8:24 p.m.)


Review request for hive.


Bugs: HIVE-19629
https://issues.apache.org/jira/browse/HIVE-19629


Repository: hive-git


Description
---

HIVE-19629: Enable Decimal64 reader after orc version upgrade


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 931533a 
  itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/TestAcidOnTez.java 
0af91bd 
  itests/src/test/resources/testconfiguration.properties d146f92 
  
llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/LlapInputFormat.java
 6d29163 
  
llap-server/src/java/org/apache/hadoop/hive/llap/io/decode/GenericColumnVectorProducer.java
 7af1b05 
  
llap-server/src/java/org/apache/hadoop/hive/llap/io/decode/OrcEncodedDataConsumer.java
 feccb87 
  
llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/OrcEncodedDataReader.java
 4033b37 
  
llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/SerDeEncodedDataReader.java
 1cfe929 
  
llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/VectorDeserializeOrcWriter.java
 de19b1d 
  
llap-server/src/java/org/apache/hadoop/hive/llap/io/metadata/ConsumerFileMetadata.java
 bf139c0 
  
llap-server/src/java/org/apache/hadoop/hive/llap/io/metadata/OrcFileMetadata.java
 0012afb 
  pom.xml e48974b 
  ql/pom.xml 06124f7 
  ql/src/java/org/apache/hadoop/hive/ql/exec/FetchOperator.java 2246901 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorizedInputFormatInterface.java
 e74b185 
  ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorizedRowBatchCtx.java 
6588385 
  ql/src/java/org/apache/hadoop/hive/ql/io/NullRowsInputFormat.java e632d43 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java f461364 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcRawRecordMerger.java 8c7c72e 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/Reader.java 7485e60 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/ReaderImpl.java 1a6db1f 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/RecordReaderImpl.java 5b001a0 
  
ql/src/java/org/apache/hadoop/hive/ql/io/orc/VectorizedOrcAcidRowBatchReader.java
 d2e1a68 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/VectorizedOrcInputFormat.java 
c581bba 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/WriterImpl.java 71682af 
  
ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/EncodedTreeReaderFactory.java
 646b214 
  
ql/src/java/org/apache/hadoop/hive/ql/io/parquet/MapredParquetInputFormat.java 
ed6d577 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/Vectorizer.java 
394f826 
  ql/src/test/org/apache/hadoop/hive/ql/TestTxnNoBuckets.java af43b14 
  ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestInputOutputFormat.java 
fb2335a 
  ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestOrcFile.java ef678a8 
  ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestOrcRawRecordMerger.java 
d8a7af8 
  ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestOrcSerDeStats.java 1533ffa 
  ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestVectorizedORCReader.java 
0c9c95d 
  
ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestVectorizedOrcAcidRowBatchReader.java
 e478371 
  ql/src/test/queries/clientpositive/llap_acid2.q a409c26 
  ql/src/test/queries/clientpositive/llap_decimal64_reader.q PRE-CREATION 
  ql/src/test/queries/clientpositive/llap_uncompressed.q 875356c 
  ql/src/test/results/clientpositive/acid_mapjoin.q.out 76a781e 
  ql/src/test/results/clientpositive/acid_nullscan.q.out 6dad497 
  ql/src/test/results/clientpositive/acid_table_stats.q.out 2596922 
  ql/src/test/results/clientpositive/annotate_stats_part.q.out 9e45101 
  ql/src/test/results/clientpositive/annotate_stats_table.q.out b502957 
  ql/src/test/results/clientpositive/autoColumnStats_4.q.out 5be906e 
  
ql/src/test/results/clientpositive/beeline/materialized_view_create_rewrite.q.out
 a717b80 
  ql/src/test/results/clientpositive/columnStatsUpdateForStatsOptimizer_2.q.out 
fa58add 
  ql/src/test/results/clientpositive/deleteAnalyze.q.out d98114b 
  ql/src/test/results/clientpositive/druid/druidmini_mv.q.out 35f4c1b 
  ql/src/test/results/clientpositive/extrapolate_part_stats_date.q.out f1cd05c 
  ql/src/test/results/clientpositive/extrapolate_part_stats_full.q.out bec6dd4 
  ql/src/test/results/clientpositive/extrapolate_part_stats_partial.q.out 
1a13b21 
  ql/src/test/results/clientpositive/llap/acid_bucket_pruning.q.out c89c22c 
  ql/src/test/results/clientpositive/llap/acid_vectorization_original.q.out 
38a9770 
  ql/src/test/results/clientpositive/llap/alter_merge_orc.q.out 7b33e8e 
  ql/src/test/results/clientpositive/llap/alter_merge_stats_orc.q.out c8b831d 
  ql/src/test/results/clientpositive/llap/column_table_stats_orc.q.out 11a9c0e

Review Request 67329: HIVE-19629: Enable Decimal64 reader after orc version upgrade

2018-05-25 Thread j . prasanth . j


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67329/
---

Review request for hive.


Bugs: HIVE-19629
https://issues.apache.org/jira/browse/HIVE-19629


Repository: hive-git


Description
---

HIVE-19629: Enable Decimal64 reader after orc version upgrade


Diffs
-

  itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/TestAcidOnTez.java 
0af91bda95b49d87aac186af020d9f3f301c4393 
  
llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/OrcEncodedDataReader.java
 4033b379defbef2ed952bee7e4f737149a6c5a9d 
  
llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/SerDeEncodedDataReader.java
 1cfe92978a029ef68aa1af6eeda5a5aacbad255e 
  
llap-server/src/java/org/apache/hadoop/hive/llap/io/metadata/ConsumerFileMetadata.java
 bf139c071ccee07be3094ab64b9efafcb95c7b7c 
  
llap-server/src/java/org/apache/hadoop/hive/llap/io/metadata/OrcFileMetadata.java
 0012afb3efa1fa67c8c897711ab102c833f1f0a0 
  pom.xml e48974bab4906d2fa5e81d048c596e084e43205a 
  ql/pom.xml 06124f738779141e97e67f2e07c1d13dd85425d0 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java 
f4613647314a1b70b6cde3005ddce9d61b154de4 
  
ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/EncodedTreeReaderFactory.java
 646b21424960e56b90e1320e137c6b7cd8143c52 
  ql/src/test/org/apache/hadoop/hive/ql/TestTxnNoBuckets.java 
af43b14abfc474dd59646b02e31a792a04ec86d6 
  ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestInputOutputFormat.java 
fb2335ab1f990ca8f731192afd016506287473c1 
  ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestOrcFile.java 
ef678a8eb323149ee11cbe77c5b55a7c651cf474 
  ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestOrcSerDeStats.java 
1533ffae2a84ca4c71168f744db90fd732ecba04 
  ql/src/test/results/clientpositive/acid_mapjoin.q.out 
76a781e94e11a845b1143962d820c70b72214d4b 
  ql/src/test/results/clientpositive/acid_nullscan.q.out 
6dad4974ae42fbab665a2968e38e37bcc0be40d7 
  ql/src/test/results/clientpositive/acid_table_stats.q.out 
2596922fff380746d5952e52e0922271b0968f5e 
  ql/src/test/results/clientpositive/annotate_stats_part.q.out 
9e45101fe6cf6927dfec3a2b7dccb42fbfadb5d8 
  ql/src/test/results/clientpositive/annotate_stats_table.q.out 
b502957e96e0faa68147059a3ad105a9d02a528a 
  ql/src/test/results/clientpositive/autoColumnStats_4.q.out 
5be906eee456c27953454d9f2326f6cf67c4e3c1 
  
ql/src/test/results/clientpositive/beeline/materialized_view_create_rewrite.q.out
 a717b80d045ecbcb9066c02fa1dae4985be36e9d 
  ql/src/test/results/clientpositive/columnStatsUpdateForStatsOptimizer_2.q.out 
fa58add84d720ad866a4df31a637ecc9ad8a8cdb 
  ql/src/test/results/clientpositive/deleteAnalyze.q.out 
d98114bff19d668a64befb888601eef548e48bc3 
  ql/src/test/results/clientpositive/druid/druidmini_mv.q.out 
35f4c1b50b58c83534597fca43074390cb20f38d 
  ql/src/test/results/clientpositive/extrapolate_part_stats_date.q.out 
f1cd05cb9bc00f8fc97eec97fa76cd3697aa7f25 
  ql/src/test/results/clientpositive/extrapolate_part_stats_full.q.out 
bec6dd4a0ee80304e4b6e3ad8e12a37d5365ce0d 
  ql/src/test/results/clientpositive/extrapolate_part_stats_partial.q.out 
1a13b21e4e87908ec83a80f482707bc651627ef1 
  ql/src/test/results/clientpositive/llap/acid_bucket_pruning.q.out 
c89c22cebaed12fef8f0c0caaf0e641887a37d2a 
  ql/src/test/results/clientpositive/llap/acid_vectorization_original.q.out 
38a97700e67b99e86d1a0b9af67693482cf76f65 
  ql/src/test/results/clientpositive/llap/alter_merge_orc.q.out 
7b33e8e93d1561205fce8fae4ad539ce8f875b8f 
  ql/src/test/results/clientpositive/llap/alter_merge_stats_orc.q.out 
c8b831df7975bd7267d2c2b07f08e097493414ed 
  ql/src/test/results/clientpositive/llap/column_table_stats_orc.q.out 
11a9c0ecbe636f24b67a6aa0c0c021da07529730 
  ql/src/test/results/clientpositive/llap/default_constraint.q.out 
cb69af7579543890c42702e09a5808396870f3f5 
  ql/src/test/results/clientpositive/llap/deleteAnalyze.q.out 
a125fd62290957b31a265c24199cdff8b73bdbb7 
  ql/src/test/results/clientpositive/llap/dynamic_semijoin_reduction.q.out 
22f2860cfa5571b43fc9d37f149fdef6dcb40556 
  ql/src/test/results/clientpositive/llap/dynamic_semijoin_user_level.q.out 
048712eed1175c468f0002387d78fd7e36028b1d 
  ql/src/test/results/clientpositive/llap/dynpart_sort_opt_vectorization.q.out 
709100f1182053e4bec5fa694534f52387e1b789 
  ql/src/test/results/clientpositive/llap/dynpart_sort_optimization2.q.out 
72f3b8bf63d06cd99b66653a76240387bfa89a9d 
  ql/src/test/results/clientpositive/llap/dynpart_sort_optimization_acid.q.out 
fa51dd3e27dd7cc82ec95c470d3f9ac73d28b3d7 
  ql/src/test/results/clientpositive/llap/enforce_constraint_notnull.q.out 
5a9d263267816bb21f2836265bc5b65cf6b84ffd 
  
ql/src/test/results/clientpositive/llap/extrapolate_part_stats_partial_ndv.q.out
 8a0da7d73d602f9cc0f86c1ebc935cab059ae000

Review Request 67245: HIVE-19633: Remove/Migrate Minimr tests

2018-05-22 Thread j . prasanth . j


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67245/
---

Review request for hive, Jesús Camacho Rodríguez and Sergey Shelukhin.


Bugs: HIVE-19633
https://issues.apache.org/jira/browse/HIVE-19633


Repository: hive-git


Description
---

HIVE-19633: Remove/Migrate Minimr tests


Diffs
-

  data/scripts/q_test_init_for_minimr.sql 
e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 
  
itests/qtest/src/test/java/org/apache/hadoop/hive/cli/TestMinimrCliDriver.java 
417687c5f5a133bbd35d760b844a82876b081d97 
  itests/src/test/resources/testconfiguration.properties 
6528ec6db0220a65b53149465da87ae0cabd9b13 
  itests/util/pom.xml 5f91523e6a174cc81a0f3b81ee8edf3a5d831da4 
  
itests/util/src/main/java/org/apache/hadoop/hive/ql/hooks/VerifyNumReducersHook.java
 6172ca967a1f6648142de15398731dd0f712f984 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/tez/monitoring/TezProgressMonitor.java
 b0c1659bfed72a3e4009249b4b686baa4923da0c 
  ql/src/test/queries/clientpositive/root_dir_external_table.q 
7763ce24322967aa0cc69f770bdf5e4bcf769268 
  ql/src/test/queries/clientpositive/udf_using.q 
d5e9206e87c5ca02e9933b7ce2b40a4956774c55 
  ql/src/test/results/clientpositive/infer_bucket_sort_dyn_part.q.out 
1af3ea528ca1cd27220d5cb445a297ba5610905b 
  ql/src/test/results/clientpositive/infer_bucket_sort_map_operators.q.out 
63f6f5c06a59dc34893b18a021f2a51b8cf55990 
  ql/src/test/results/clientpositive/llap/bucket_num_reducers_acid.q.out 
PRE-CREATION 
  ql/src/test/results/clientpositive/parallel_orderby.q.out 
73a103de505c50f20fe653c4eac93bc5bd9d6383 
  ql/src/test/results/clientpositive/root_dir_external_table.q.out 
70494062d2eadf5a0f64a28a84780f3dff372fc9 
  ql/src/test/results/clientpositive/scriptfile1.q.out 
f374501df076b402cb00eadf8848113fe8afd146 
  ql/src/test/results/clientpositive/udf_using.q.out 
8f92101613ab2b1ef8c9c3329fb9aaf05105e556 


Diff: https://reviews.apache.org/r/67245/diff/1/


Testing
---


Thanks,

Prasanth_J

Re: Review Request 67240: HIVE-19642 add cache dump when LLAP cache is full on branch-2 (and fix retry issues)

2018-05-21 Thread j . prasanth . j


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67240/#review203541
---




llap-server/src/java/org/apache/hadoop/hive/llap/cache/LowLevelCacheMemoryManager.java
Lines 41 (patched)


nit: remove useless todo



llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/LlapIoImpl.java
Lines 154 (patched)


follow up ticket?


- Prasanth_J


On May 21, 2018, 11:26 p.m., Sergey Shelukhin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/67240/
> ---
> 
> (Updated May 21, 2018, 11:26 p.m.)
> 
> 
> Review request for hive and Prasanth_J.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> see jira
> 
> 
> Diffs
> -
> 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/cache/LowLevelCacheMemoryManager.java
>  2132574fb54f6993c06c22b3ec57e198f3278caa 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/LlapIoImpl.java 
> 58535d7646eb5e5e4ed91f1431ca6d12e8480dea 
> 
> 
> Diff: https://reviews.apache.org/r/67240/diff/1/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Sergey Shelukhin
> 
>

Review Request 67229: HIVE-19614: GenericUDTFGetSplits does not honor ORDER BY

2018-05-18 Thread j . prasanth . j


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67229/
---

Review request for hive and Jason Dere.


Bugs: HIVE-19614
https://issues.apache.org/jira/browse/HIVE-19614


Repository: hive-git


Description
---

HIVE-19614: GenericUDTFGetSplits does not honor ORDER BY


Diffs
-

  ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDTFGetSplits.java 
e74a18853ced722a8a5d7f5b8e80015e461e4a0b 


Diff: https://reviews.apache.org/r/67229/diff/1/


Testing
---


Thanks,

Prasanth_J

Review Request 67222: HIVE-19613: GenericUDTFGetSplits should handle fetch task with temp table rewrite

2018-05-18 Thread j . prasanth . j


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67222/
---

Review request for hive and Jason Dere.


Bugs: HIVE-19613
https://issues.apache.org/jira/browse/HIVE-19613


Repository: hive-git


Description
---

HIVE-19613: GenericUDTFGetSplits should handle fetch task with temp table 
rewrite


Diffs
-

  ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDTFGetSplits.java 
e74a18853ced722a8a5d7f5b8e80015e461e4a0b 


Diff: https://reviews.apache.org/r/67222/diff/1/


Testing
---


Thanks,

Prasanth_J

Re: Review Request 67197: HIVE-19588: Several invocation of file listing when creating VectorizedOrcAcidRowBatchReader

2018-05-17 Thread j . prasanth . j


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67197/
---

(Updated May 18, 2018, 5:45 a.m.)


Review request for hive and Eugene Koifman.


Changes
---

Removed VectorizedOrcAcidRowBatchReader creation from inner loop for LLAP.


Bugs: HIVE-19588
https://issues.apache.org/jira/browse/HIVE-19588


Repository: hive-git


Description
---

HIVE-19588: Several invocation of file listing when creating 
VectorizedOrcAcidRowBatchReader


Diffs (updated)
-

  
llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/LlapRecordReader.java
 7451ea4 
  ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java 183515a 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java 2337a35 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcRawRecordMerger.java 5655ee9 
  
ql/src/java/org/apache/hadoop/hive/ql/io/orc/VectorizedOrcAcidRowBatchReader.java
 8caa265 
  ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestInputOutputFormat.java 
b28c126 
  
ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestVectorizedOrcAcidRowBatchReader.java
 3acc085 


Diff: https://reviews.apache.org/r/67197/diff/2/

Changes: https://reviews.apache.org/r/67197/diff/1-2/


Testing
---


Thanks,

Prasanth_J

Review Request 67197: HIVE-19588: Several invocation of file listing when creating VectorizedOrcAcidRowBatchReader

2018-05-17 Thread j . prasanth . j


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67197/
---

Review request for hive and Eugene Koifman.


Bugs: HIVE-19588
https://issues.apache.org/jira/browse/HIVE-19588


Repository: hive-git


Description
---

HIVE-19588: Several invocation of file listing when creating 
VectorizedOrcAcidRowBatchReader


Diffs
-

  ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java 
183515a0ed2a0f48740825a4737d4bee63d8a90a 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java 
2337a350e6bd44e52826b46707fb8727974e00bf 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcRawRecordMerger.java 
5655ee9407d78ebc28a6e2b9c39ba55efc603a54 
  
ql/src/java/org/apache/hadoop/hive/ql/io/orc/VectorizedOrcAcidRowBatchReader.java
 8caa265e8b9bd5674a35083ccd81b873c9fc2c70 
  ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestInputOutputFormat.java 
b28c126dbca8f1796dae5e8c5c5e256d18b1f86d 
  
ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestVectorizedOrcAcidRowBatchReader.java
 3acc0855fb07b51377ef665eea9abf9927c260c6 


Diff: https://reviews.apache.org/r/67197/diff/1/


Testing
---


Thanks,

Prasanth_J

Review Request 67140: HIVE-19560: Retry test runner and retry rule for flaky tests

2018-05-15 Thread j . prasanth . j


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67140/
---

Review request for hive and Jesús Camacho Rodríguez.


Bugs: HIVE-19560
https://issues.apache.org/jira/browse/HIVE-19560


Repository: hive-git


Description
---

HIVE-19560: Retry test runner and retry rule for flaky tests


Diffs
-

  common/src/test/org/apache/hive/common/util/Retry.java PRE-CREATION 
  common/src/test/org/apache/hive/common/util/RetryTestRunner.java PRE-CREATION 
  
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/TestCompactor.java
 7e17d5d888755c7fb19bc34a712db5bd6fee4d32 
  
itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestTriggersMoveWorkloadManager.java
 e017e6382f752580cb7e2d1704fab34ad9ac001e 


Diff: https://reviews.apache.org/r/67140/diff/1/


Testing
---


Thanks,

Prasanth_J

Review Request 67042: HIVE-19449: Create minimized uber jar for hive streaming module

2018-05-09 Thread j . prasanth . j


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67042/
---

Review request for hive, Eugene Koifman and Sergey Shelukhin.


Bugs: HIVE-19449
https://issues.apache.org/jira/browse/HIVE-19449


Repository: hive-git


Description
---

HIVE-19449: Create minimized uber jar for hive streaming module


Diffs
-

  itests/hive-unit/pom.xml 26e423c5ff2d39d729c122a459c33e42bd4e389c 
  streaming/pom.xml ccc55ebb0d36acba7719ef7bcf6d0c4954097187 
  streaming/src/java/org/apache/hive/streaming/StrictRegexWriter.java 
3651fa120ae5c753d2a833bc28c9dcfcd24be823 


Diff: https://reviews.apache.org/r/67042/diff/1/


Testing
---


Thanks,

Prasanth_J

Re: Review Request 67039: HIVE-19479 encoded stream seek is incorrect for 0-length RGs in LLAP IO

2018-05-09 Thread j . prasanth . j


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67039/#review202800
---


Ship it!




Ship It!

- Prasanth_J


On May 9, 2018, 8:55 p.m., Sergey Shelukhin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/67039/
> ---
> 
> (Updated May 9, 2018, 8:55 p.m.)
> 
> 
> Review request for hive and Prasanth_J.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> see jira
> 
> 
> Diffs
> -
> 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/io/decode/OrcEncodedDataConsumer.java
>  fc0c66a888 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/EncodedReaderImpl.java 
> 1d7eceb1ef 
>   
> ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/EncodedTreeReaderFactory.java
>  42532f9a0e 
> 
> 
> Diff: https://reviews.apache.org/r/67039/diff/2/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Sergey Shelukhin
> 
>

Re: Review Request 67039: HIVE-19479 encoded stream seek is incorrect for 0-length RGs in LLAP IO

2018-05-09 Thread j . prasanth . j



> On May 9, 2018, 8:43 p.m., Prasanth_J wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/EncodedTreeReaderFactory.java
> > Lines 1076 (patched)
> > 
> >
> > Positions are packed variable in protobuf. 
> > Also there is stream suppression. 
> > Same column in a stripe may have a stream suppressed (say all non-nulls 
> > values, isPresent gets suppressed) then its positions will not be recorded. 
> > Same column in another stripe might have all streams. So all this does is 
> > if the stream does not exist don't move on to next position.
> > 
> > [[0,0,0],[0,0],[0,0]] -> positions for isPresent, Data, Length..
> > [[0,0],[0,0]] -> positions for Data, Length.. (isPresent suppressed)
> 
> Sergey Shelukhin wrote:
> I'm not sure what you mean... I'm moving this code outside of the 
> 0-length check, and moving into utility method.
> All calls of the same method did the exact same thing before.
> 
> Prasanth_J wrote:
> 
> https://github.com/apache/orc/blob/master/java/core/src/java/org/apache/orc/impl/writer/TreeWriterBase.java#L248-L256
>  
> If a stream is suppressed, its positions are removed.
> 
> So I am not sure if it is safe to advance the positions without checking 
> if the stream exists/available or not.

Actually we remove the positions only for isPresent streams. I think it should 
be ok if we advance positions first before seek for other streams.


- Prasanth_J


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67039/#review202793
---


On May 9, 2018, 8:55 p.m., Sergey Shelukhin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/67039/
> ---
> 
> (Updated May 9, 2018, 8:55 p.m.)
> 
> 
> Review request for hive and Prasanth_J.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> see jira
> 
> 
> Diffs
> -
> 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/io/decode/OrcEncodedDataConsumer.java
>  fc0c66a888 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/EncodedReaderImpl.java 
> 1d7eceb1ef 
>   
> ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/EncodedTreeReaderFactory.java
>  42532f9a0e 
> 
> 
> Diff: https://reviews.apache.org/r/67039/diff/2/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Sergey Shelukhin
> 
>

Re: Review Request 67039: HIVE-19479 encoded stream seek is incorrect for 0-length RGs in LLAP IO

2018-05-09 Thread j . prasanth . j



> On May 9, 2018, 8:43 p.m., Prasanth_J wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/EncodedTreeReaderFactory.java
> > Lines 1076 (patched)
> > 
> >
> > Positions are packed variable in protobuf. 
> > Also there is stream suppression. 
> > Same column in a stripe may have a stream suppressed (say all non-nulls 
> > values, isPresent gets suppressed) then its positions will not be recorded. 
> > Same column in another stripe might have all streams. So all this does is 
> > if the stream does not exist don't move on to next position.
> > 
> > [[0,0,0],[0,0],[0,0]] -> positions for isPresent, Data, Length..
> > [[0,0],[0,0]] -> positions for Data, Length.. (isPresent suppressed)
> 
> Sergey Shelukhin wrote:
> I'm not sure what you mean... I'm moving this code outside of the 
> 0-length check, and moving into utility method.
> All calls of the same method did the exact same thing before.

https://github.com/apache/orc/blob/master/java/core/src/java/org/apache/orc/impl/writer/TreeWriterBase.java#L248-L256
 
If a stream is suppressed, its positions are removed.

So I am not sure if it is safe to advance the positions without checking if the 
stream exists/available or not.


- Prasanth_J


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67039/#review202793
---


On May 9, 2018, 8:55 p.m., Sergey Shelukhin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/67039/
> ---
> 
> (Updated May 9, 2018, 8:55 p.m.)
> 
> 
> Review request for hive and Prasanth_J.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> see jira
> 
> 
> Diffs
> -
> 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/io/decode/OrcEncodedDataConsumer.java
>  fc0c66a888 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/EncodedReaderImpl.java 
> 1d7eceb1ef 
>   
> ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/EncodedTreeReaderFactory.java
>  42532f9a0e 
> 
> 
> Diff: https://reviews.apache.org/r/67039/diff/2/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Sergey Shelukhin
> 
>

Re: Review Request 67039: HIVE-19479 encoded stream seek is incorrect for 0-length RGs in LLAP IO

2018-05-09 Thread j . prasanth . j


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67039/#review202793
---




ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/EncodedTreeReaderFactory.java
Lines 280 (patched)


the reader seeks mostly delegates to data stream seek and length stream 
seek (RLE seeks does more than just position seek - it has to seek within bytes 
or runs)



ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/EncodedTreeReaderFactory.java
Lines 1076 (patched)


Positions are packed variable in protobuf. 
Also there is stream suppression. 
Same column in a stripe may have a stream suppressed (say all non-nulls 
values, isPresent gets suppressed) then its positions will not be recorded. 
Same column in another stripe might have all streams. So all this does is if 
the stream does not exist don't move on to next position.

[[0,0,0],[0,0],[0,0]] -> positions for isPresent, Data, Length..
[[0,0],[0,0]] -> positions for Data, Length.. (isPresent suppressed)



ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/EncodedTreeReaderFactory.java
Lines 1894 (patched)


where is this skipSeek() implementation? I don't see it in the patch or 
master.


- Prasanth_J


On May 9, 2018, 7:12 p.m., Sergey Shelukhin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/67039/
> ---
> 
> (Updated May 9, 2018, 7:12 p.m.)
> 
> 
> Review request for hive and Prasanth_J.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> see jira
> 
> 
> Diffs
> -
> 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/io/decode/OrcEncodedDataConsumer.java
>  fc0c66a888 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/EncodedReaderImpl.java 
> 1d7eceb1ef 
>   
> ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/EncodedTreeReaderFactory.java
>  42532f9a0e 
> 
> 
> Diff: https://reviews.apache.org/r/67039/diff/1/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Sergey Shelukhin
> 
>

Re: Review Request 66645: HIVE-19211: New streaming ingest API and support for dynamic partitioning

2018-05-01 Thread j . prasanth . j


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66645/
---

(Updated May 1, 2018, 9:53 p.m.)


Review request for hive, Ashutosh Chauhan and Eugene Koifman.


Changes
---

Added abort under lock.


Bugs: HIVE-19211
https://issues.apache.org/jira/browse/HIVE-19211


Repository: hive-git


Description
---

HIVE-19211: New streaming ingest API and support for dynamic partitioning


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 5a13726 
  
hcatalog/streaming/src/test/org/apache/hive/hcatalog/streaming/TestStreaming.java
 90dbdac 
  itests/hive-unit/pom.xml 3ae7f2f 
  
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/TestCompactor.java
 8ee033d 
  metastore/src/java/org/apache/hadoop/hive/metastore/HiveClientCache.java 
PRE-CREATION 
  metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStoreUtils.java 
a66c135 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcRecordUpdater.java 09f8802 
  ql/src/java/org/apache/hadoop/hive/ql/lockmgr/DbTxnManager.java 76569d5 
  ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java f6608eb 
  serde/src/java/org/apache/hadoop/hive/serde2/JsonSerDe.java PRE-CREATION 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/utils/MetaStoreUtils.java
 8c159e9 
  streaming/pom.xml b58ec01 
  streaming/src/java/org/apache/hive/streaming/AbstractRecordWriter.java 
25998ae 
  streaming/src/java/org/apache/hive/streaming/ConnectionError.java 668bffb 
  streaming/src/java/org/apache/hive/streaming/ConnectionInfo.java PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/DelimitedInputWriter.java 
898b3f9 
  streaming/src/java/org/apache/hive/streaming/HeartBeatFailure.java b1f9520 
  streaming/src/java/org/apache/hive/streaming/HiveEndPoint.java b04e137 
  streaming/src/java/org/apache/hive/streaming/HiveStreamingConnection.java 
PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/ImpersonationFailed.java 23e17e7 
  streaming/src/java/org/apache/hive/streaming/InvalidColumn.java 0011b14 
  streaming/src/java/org/apache/hive/streaming/InvalidPartition.java f1f9804 
  streaming/src/java/org/apache/hive/streaming/InvalidTable.java ef1c91d 
  streaming/src/java/org/apache/hive/streaming/InvalidTransactionState.java 
PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/InvalidTrasactionState.java 
762f5f8 
  streaming/src/java/org/apache/hive/streaming/PartitionCreationFailed.java 
5f9aca6 
  streaming/src/java/org/apache/hive/streaming/PartitionHandler.java 
PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/PartitionInfo.java PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/QueryFailedException.java 
ccd3ae0 
  streaming/src/java/org/apache/hive/streaming/RecordWriter.java dc6d70e 
  streaming/src/java/org/apache/hive/streaming/SerializationError.java a57ba00 
  streaming/src/java/org/apache/hive/streaming/StreamingConnection.java 2f760ea 
  streaming/src/java/org/apache/hive/streaming/StreamingException.java a7f84c1 
  streaming/src/java/org/apache/hive/streaming/StreamingIOFailure.java 0dfbfa7 
  streaming/src/java/org/apache/hive/streaming/StrictDelimitedInputWriter.java 
PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/StrictJsonWriter.java 0077913 
  streaming/src/java/org/apache/hive/streaming/StrictRegexWriter.java c0b7324 
  streaming/src/java/org/apache/hive/streaming/TransactionBatch.java 2b05771 
  streaming/src/java/org/apache/hive/streaming/TransactionBatchUnAvailable.java 
a8c8cd4 
  streaming/src/java/org/apache/hive/streaming/TransactionError.java a331b20 
  streaming/src/test/org/apache/hive/streaming/TestDelimitedInputWriter.java 
f0843a1 
  streaming/src/test/org/apache/hive/streaming/TestStreaming.java 0ec3048 
  
streaming/src/test/org/apache/hive/streaming/TestStreamingDynamicPartitioning.java
 PRE-CREATION 


Diff: https://reviews.apache.org/r/66645/diff/12/

Changes: https://reviews.apache.org/r/66645/diff/11-12/


Testing
---


Thanks,

Prasanth_J

Re: Review Request 66805: HIVE-19311 : Partition and bucketing support for “load data” statement

2018-05-01 Thread j . prasanth . j


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66805/#review202219
---


Ship it!




Ship It!

- Prasanth_J


On May 1, 2018, 9:06 p.m., Deepak Jaiswal wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66805/
> ---
> 
> (Updated May 1, 2018, 9:06 p.m.)
> 
> 
> Review request for hive, Ashutosh Chauhan, Eugene Koifman, Jesús Camacho 
> Rodríguez, Prasanth_J, and Vineet Garg.
> 
> 
> Bugs: HIVE-19311
> https://issues.apache.org/jira/browse/HIVE-19311
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Currently, "load data" statement is very limited. It errors out if any of the 
> information is missing such as partitioning info if table is partitioned or 
> appropriate names when table is bucketed.
> It should be able to launch an insert job to load the data instead.
> 
> 
> Diffs
> -
> 
>   data/files/load_data_job/bucketing.txt PRE-CREATION 
>   data/files/load_data_job/load_data_1_partition.txt PRE-CREATION 
>   data/files/load_data_job/partitions/load_data_1_partition.txt PRE-CREATION 
>   data/files/load_data_job/partitions/load_data_2_partitions.txt PRE-CREATION 
>   data/files/load_data_job/partitions/subdir/load_data_1_partition.txt 
> PRE-CREATION 
>   data/files/load_data_job/partitions/subdir/load_data_2_partitions.txt 
> PRE-CREATION 
>   itests/src/test/resources/testconfiguration.properties 2ca7b5f63b 
>   ql/src/java/org/apache/hadoop/hive/ql/Context.java 0fedf0e76e 
>   ql/src/java/org/apache/hadoop/hive/ql/ErrorMsg.java 94dd63641d 
>   ql/src/java/org/apache/hadoop/hive/ql/metadata/Table.java abd678bb54 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/LoadSemanticAnalyzer.java 
> c07991d434 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 
> fad0e5c24a 
>   
> ql/src/java/org/apache/hadoop/hive/ql/parse/UpdateDeleteSemanticAnalyzer.java 
> 2f3b07f4af 
>   ql/src/test/org/apache/hadoop/hive/ql/TestTxnLoadData.java ec8c1507ec 
>   ql/src/test/queries/clientnegative/load_part_nospec.q 81517991b2 
>   ql/src/test/queries/clientnegative/nopart_load.q 966982fd5c 
>   ql/src/test/queries/clientpositive/load_data_using_job.q PRE-CREATION 
>   ql/src/test/results/clientnegative/load_part_nospec.q.out bebaf92311 
>   ql/src/test/results/clientnegative/nopart_load.q.out 881514640c 
>   ql/src/test/results/clientpositive/llap/load_data_using_job.q.out 
> PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/66805/diff/8/
> 
> 
> Testing
> ---
> 
> Added a unit test.
> 
> 
> Thanks,
> 
> Deepak Jaiswal
> 
>

Re: Review Request 66645: HIVE-19211: New streaming ingest API and support for dynamic partitioning

2018-05-01 Thread j . prasanth . j


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66645/
---

(Updated May 1, 2018, 9:01 p.m.)


Review request for hive, Ashutosh Chauhan and Eugene Koifman.


Changes
---

Addressed review comments.


Bugs: HIVE-19211
https://issues.apache.org/jira/browse/HIVE-19211


Repository: hive-git


Description
---

HIVE-19211: New streaming ingest API and support for dynamic partitioning


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 5a13726 
  
hcatalog/streaming/src/test/org/apache/hive/hcatalog/streaming/TestStreaming.java
 90dbdac 
  itests/hive-unit/pom.xml 3ae7f2f 
  
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/TestCompactor.java
 8ee033d 
  metastore/src/java/org/apache/hadoop/hive/metastore/HiveClientCache.java 
PRE-CREATION 
  metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStoreUtils.java 
a66c135 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcRecordUpdater.java 09f8802 
  ql/src/java/org/apache/hadoop/hive/ql/lockmgr/DbTxnManager.java 76569d5 
  ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java f6608eb 
  serde/src/java/org/apache/hadoop/hive/serde2/JsonSerDe.java PRE-CREATION 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/utils/MetaStoreUtils.java
 8c159e9 
  streaming/pom.xml b58ec01 
  streaming/src/java/org/apache/hive/streaming/AbstractRecordWriter.java 
25998ae 
  streaming/src/java/org/apache/hive/streaming/ConnectionError.java 668bffb 
  streaming/src/java/org/apache/hive/streaming/ConnectionInfo.java PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/DelimitedInputWriter.java 
898b3f9 
  streaming/src/java/org/apache/hive/streaming/HeartBeatFailure.java b1f9520 
  streaming/src/java/org/apache/hive/streaming/HiveEndPoint.java b04e137 
  streaming/src/java/org/apache/hive/streaming/HiveStreamingConnection.java 
PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/ImpersonationFailed.java 23e17e7 
  streaming/src/java/org/apache/hive/streaming/InvalidColumn.java 0011b14 
  streaming/src/java/org/apache/hive/streaming/InvalidPartition.java f1f9804 
  streaming/src/java/org/apache/hive/streaming/InvalidTable.java ef1c91d 
  streaming/src/java/org/apache/hive/streaming/InvalidTransactionState.java 
PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/InvalidTrasactionState.java 
762f5f8 
  streaming/src/java/org/apache/hive/streaming/PartitionCreationFailed.java 
5f9aca6 
  streaming/src/java/org/apache/hive/streaming/PartitionHandler.java 
PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/PartitionInfo.java PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/QueryFailedException.java 
ccd3ae0 
  streaming/src/java/org/apache/hive/streaming/RecordWriter.java dc6d70e 
  streaming/src/java/org/apache/hive/streaming/SerializationError.java a57ba00 
  streaming/src/java/org/apache/hive/streaming/StreamingConnection.java 2f760ea 
  streaming/src/java/org/apache/hive/streaming/StreamingException.java a7f84c1 
  streaming/src/java/org/apache/hive/streaming/StreamingIOFailure.java 0dfbfa7 
  streaming/src/java/org/apache/hive/streaming/StrictDelimitedInputWriter.java 
PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/StrictJsonWriter.java 0077913 
  streaming/src/java/org/apache/hive/streaming/StrictRegexWriter.java c0b7324 
  streaming/src/java/org/apache/hive/streaming/TransactionBatch.java 2b05771 
  streaming/src/java/org/apache/hive/streaming/TransactionBatchUnAvailable.java 
a8c8cd4 
  streaming/src/java/org/apache/hive/streaming/TransactionError.java a331b20 
  streaming/src/test/org/apache/hive/streaming/TestDelimitedInputWriter.java 
f0843a1 
  streaming/src/test/org/apache/hive/streaming/TestStreaming.java 0ec3048 
  
streaming/src/test/org/apache/hive/streaming/TestStreamingDynamicPartitioning.java
 PRE-CREATION 


Diff: https://reviews.apache.org/r/66645/diff/11/

Changes: https://reviews.apache.org/r/66645/diff/10-11/


Testing
---


Thanks,

Prasanth_J

Re: Review Request 66805: HIVE-19311 : Partition and bucketing support for “load data” statement

2018-05-01 Thread j . prasanth . j


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66805/#review202211
---




ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java
Lines 1255 (patched)


Instead of doing instanceof check here, you would get the same behaviour if 
you Override getAllOutputs() methods in LoadSemanticAnalyzer right?


- Prasanth_J


On May 1, 2018, 7:56 p.m., Deepak Jaiswal wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66805/
> ---
> 
> (Updated May 1, 2018, 7:56 p.m.)
> 
> 
> Review request for hive, Ashutosh Chauhan, Eugene Koifman, Jesús Camacho 
> Rodríguez, Prasanth_J, and Vineet Garg.
> 
> 
> Bugs: HIVE-19311
> https://issues.apache.org/jira/browse/HIVE-19311
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Currently, "load data" statement is very limited. It errors out if any of the 
> information is missing such as partitioning info if table is partitioned or 
> appropriate names when table is bucketed.
> It should be able to launch an insert job to load the data instead.
> 
> 
> Diffs
> -
> 
>   data/files/load_data_job/bucketing.txt PRE-CREATION 
>   data/files/load_data_job/load_data_1_partition.txt PRE-CREATION 
>   data/files/load_data_job/partitions/load_data_1_partition.txt PRE-CREATION 
>   data/files/load_data_job/partitions/load_data_2_partitions.txt PRE-CREATION 
>   data/files/load_data_job/partitions/subdir/load_data_1_partition.txt 
> PRE-CREATION 
>   data/files/load_data_job/partitions/subdir/load_data_2_partitions.txt 
> PRE-CREATION 
>   itests/src/test/resources/testconfiguration.properties 2ca7b5f63b 
>   ql/src/java/org/apache/hadoop/hive/ql/Context.java 0fedf0e76e 
>   ql/src/java/org/apache/hadoop/hive/ql/ErrorMsg.java 94dd63641d 
>   ql/src/java/org/apache/hadoop/hive/ql/metadata/Table.java abd678bb54 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/LoadSemanticAnalyzer.java 
> c07991d434 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 
> fad0e5c24a 
>   
> ql/src/java/org/apache/hadoop/hive/ql/parse/UpdateDeleteSemanticAnalyzer.java 
> 2f3b07f4af 
>   ql/src/test/org/apache/hadoop/hive/ql/TestTxnLoadData.java ec8c1507ec 
>   ql/src/test/queries/clientnegative/load_part_nospec.q 81517991b2 
>   ql/src/test/queries/clientnegative/nopart_load.q 966982fd5c 
>   ql/src/test/queries/clientpositive/load_data_using_job.q PRE-CREATION 
>   ql/src/test/results/clientnegative/load_part_nospec.q.out bebaf92311 
>   ql/src/test/results/clientnegative/nopart_load.q.out 881514640c 
>   ql/src/test/results/clientpositive/llap/load_data_using_job.q.out 
> PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/66805/diff/7/
> 
> 
> Testing
> ---
> 
> Added a unit test.
> 
> 
> Thanks,
> 
> Deepak Jaiswal
> 
>

Re: Review Request 66645: HIVE-19211: New streaming ingest API and support for dynamic partitioning

2018-05-01 Thread j . prasanth . j


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66645/#review202207
---




streaming/src/java/org/apache/hive/streaming/HiveStreamingConnection.java
Line 490 (original), 414 (patched)


Won't be required with volatile change.



streaming/src/java/org/apache/hive/streaming/HiveStreamingConnection.java
Lines 442 (patched)


This class was primarily added so that min and max of range can be 
atomically changed. But you had brought up a good point that maxTxn does not 
change which doesn't warrant this class. I will fix it by changing minTxn a 
volatile variable.



streaming/src/test/org/apache/hive/streaming/TestStreaming.java
Lines 589 (patched)


HIVE-19366



itests/hive-unit/pom.xml
Lines 79 (patched)


Parametrized test to support old vs new api.



ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcRecordUpdater.java
Lines 471 (patched)


HIVE-19206 added automatic memory management which will flush open files 
periodically without actually committing or closing it. Will add more comments 
regd. the same.


- Prasanth_J


On April 30, 2018, 11:10 p.m., Prasanth_J wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66645/
> ---
> 
> (Updated April 30, 2018, 11:10 p.m.)
> 
> 
> Review request for hive, Ashutosh Chauhan and Eugene Koifman.
> 
> 
> Bugs: HIVE-19211
> https://issues.apache.org/jira/browse/HIVE-19211
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-19211: New streaming ingest API and support for dynamic partitioning
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 6e35653 
>   
> hcatalog/streaming/src/test/org/apache/hive/hcatalog/streaming/TestStreaming.java
>  90dbdac 
>   itests/hive-unit/pom.xml 3ae7f2f 
>   
> itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/TestCompactor.java
>  8ee033d 
>   metastore/src/java/org/apache/hadoop/hive/metastore/HiveClientCache.java 
> PRE-CREATION 
>   metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStoreUtils.java 
> a66c135 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcRecordUpdater.java 09f8802 
>   ql/src/java/org/apache/hadoop/hive/ql/lockmgr/DbTxnManager.java 76569d5 
>   ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java 4661881 
>   serde/src/java/org/apache/hadoop/hive/serde2/JsonSerDe.java PRE-CREATION 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/utils/MetaStoreUtils.java
>  8c159e9 
>   streaming/pom.xml b58ec01 
>   streaming/src/java/org/apache/hive/streaming/AbstractRecordWriter.java 
> 25998ae 
>   streaming/src/java/org/apache/hive/streaming/ConnectionError.java 668bffb 
>   streaming/src/java/org/apache/hive/streaming/ConnectionInfo.java 
> PRE-CREATION 
>   streaming/src/java/org/apache/hive/streaming/DelimitedInputWriter.java 
> 898b3f9 
>   streaming/src/java/org/apache/hive/streaming/HeartBeatFailure.java b1f9520 
>   streaming/src/java/org/apache/hive/streaming/HiveEndPoint.java b04e137 
>   streaming/src/java/org/apache/hive/streaming/HiveStreamingConnection.java 
> PRE-CREATION 
>   streaming/src/java/org/apache/hive/streaming/ImpersonationFailed.java 
> 23e17e7 
>   streaming/src/java/org/apache/hive/streaming/InvalidColumn.java 0011b14 
>   streaming/src/java/org/apache/hive/streaming/InvalidPartition.java f1f9804 
>   streaming/src/java/org/apache/hive/streaming/InvalidTable.java ef1c91d 
>   streaming/src/java/org/apache/hive/streaming/InvalidTransactionState.java 
> PRE-CREATION 
>   streaming/src/java/org/apache/hive/streaming/InvalidTrasactionState.java 
> 762f5f8 
>   streaming/src/java/org/apache/hive/streaming/PartitionCreationFailed.java 
> 5f9aca6 
>   streaming/src/java/org/apache/hive/streaming/PartitionHandler.java 
> PRE-CREATION 
>   streaming/src/java/org/apache/hive/streaming/PartitionInfo.java 
> PRE-CREATION 
>   streaming/src/java/org/apache/hive/streaming/QueryFailedException.java 
> ccd3ae0 
>   streaming/src/java/org/apache/hive/streaming/RecordWriter.java dc6d70e 
>   streaming/src/java/org/apache/hive/streaming/SerializationError.java 
> a57ba00 
>   streaming/src/java/org/apache/hive/streaming/StreamingConnection.java 
> 2f760ea 
>   streaming/src/java/org/apache/hive/streaming/StreamingException.java 
> a7f84c1 
>   streaming/src/java/org/apache/hive/streaming/StreamingIOFailure.java 
> 0dfbfa7 
>   
>

Re: Review Request 66805: HIVE-19311 : Partition and bucketing support for “load data” statement

2018-05-01 Thread j . prasanth . j


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66805/#review202190
---




ql/src/java/org/apache/hadoop/hive/ql/parse/LoadSemanticAnalyzer.java
Lines 241 (patched)


Is useSuper required here? I can see why you are doing this. But can you 
make this decision based on where tempTable is set in the ctx or not? If 
tempTable is set in the ctx then it is rewritten load statement. Alternatively 
you could use a separate boolean in ctx to say if load query is rewritten or 
not based on which you can construct the if condition.



ql/src/java/org/apache/hadoop/hive/ql/parse/LoadSemanticAnalyzer.java
Line 255 (original), 254 (patched)


This can be inlined at the reparse method.



ql/src/java/org/apache/hadoop/hive/ql/parse/LoadSemanticAnalyzer.java
Lines 483 (patched)


Why is this required? Isn't load data "insert" operation type?



ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java
Lines 1254 (patched)


could you add a comment to say why does load require a shallow copy?



ql/src/test/queries/clientpositive/load_data_using_job.q
Lines 72 (patched)


Could you also add a case for multi-column bucketing?
I think Eric also asked the same question, does this handle multi-level 
subdirs? IIRC load only recurses to 1 or 2 levels.


- Prasanth_J


On May 1, 2018, 4:50 a.m., Deepak Jaiswal wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66805/
> ---
> 
> (Updated May 1, 2018, 4:50 a.m.)
> 
> 
> Review request for hive, Ashutosh Chauhan, Eugene Koifman, Jesús Camacho 
> Rodríguez, Prasanth_J, and Vineet Garg.
> 
> 
> Bugs: HIVE-19311
> https://issues.apache.org/jira/browse/HIVE-19311
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Currently, "load data" statement is very limited. It errors out if any of the 
> information is missing such as partitioning info if table is partitioned or 
> appropriate names when table is bucketed.
> It should be able to launch an insert job to load the data instead.
> 
> 
> Diffs
> -
> 
>   data/files/load_data_job/bucketing.txt PRE-CREATION 
>   data/files/load_data_job/load_data_1_partition.txt PRE-CREATION 
>   data/files/load_data_job/partitions/load_data_1_partition.txt PRE-CREATION 
>   data/files/load_data_job/partitions/load_data_2_partitions.txt PRE-CREATION 
>   data/files/load_data_job/partitions/subdir/load_data_1_partition.txt 
> PRE-CREATION 
>   data/files/load_data_job/partitions/subdir/load_data_2_partitions.txt 
> PRE-CREATION 
>   itests/src/test/resources/testconfiguration.properties 2ca7b5f63b 
>   ql/src/java/org/apache/hadoop/hive/ql/Context.java 0fedf0e76e 
>   ql/src/java/org/apache/hadoop/hive/ql/ErrorMsg.java 94dd63641d 
>   ql/src/java/org/apache/hadoop/hive/ql/metadata/Table.java abd678bb54 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/LoadSemanticAnalyzer.java 
> c07991d434 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 
> fad0e5c24a 
>   
> ql/src/java/org/apache/hadoop/hive/ql/parse/UpdateDeleteSemanticAnalyzer.java 
> 2f3b07f4af 
>   ql/src/test/org/apache/hadoop/hive/ql/TestTxnLoadData.java ec8c1507ec 
>   ql/src/test/queries/clientnegative/load_part_nospec.q 81517991b2 
>   ql/src/test/queries/clientnegative/nopart_load.q 966982fd5c 
>   ql/src/test/queries/clientpositive/load_data_using_job.q PRE-CREATION 
>   ql/src/test/results/clientnegative/load_part_nospec.q.out bebaf92311 
>   ql/src/test/results/clientnegative/nopart_load.q.out 881514640c 
>   ql/src/test/results/clientpositive/llap/load_data_using_job.q.out 
> PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/66805/diff/6/
> 
> 
> Testing
> ---
> 
> Added a unit test.
> 
> 
> Thanks,
> 
> Deepak Jaiswal
> 
>

Re: Review Request 66645: HIVE-19211: New streaming ingest API and support for dynamic partitioning

2018-04-30 Thread j . prasanth . j


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66645/
---

(Updated April 30, 2018, 11:10 p.m.)


Review request for hive, Ashutosh Chauhan and Eugene Koifman.


Changes
---

Rebased patch to latest master.


Bugs: HIVE-19211
https://issues.apache.org/jira/browse/HIVE-19211


Repository: hive-git


Description
---

HIVE-19211: New streaming ingest API and support for dynamic partitioning


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 6e35653 
  
hcatalog/streaming/src/test/org/apache/hive/hcatalog/streaming/TestStreaming.java
 90dbdac 
  itests/hive-unit/pom.xml 3ae7f2f 
  
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/TestCompactor.java
 8ee033d 
  metastore/src/java/org/apache/hadoop/hive/metastore/HiveClientCache.java 
PRE-CREATION 
  metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStoreUtils.java 
a66c135 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcRecordUpdater.java 09f8802 
  ql/src/java/org/apache/hadoop/hive/ql/lockmgr/DbTxnManager.java 76569d5 
  ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java 4661881 
  serde/src/java/org/apache/hadoop/hive/serde2/JsonSerDe.java PRE-CREATION 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/utils/MetaStoreUtils.java
 8c159e9 
  streaming/pom.xml b58ec01 
  streaming/src/java/org/apache/hive/streaming/AbstractRecordWriter.java 
25998ae 
  streaming/src/java/org/apache/hive/streaming/ConnectionError.java 668bffb 
  streaming/src/java/org/apache/hive/streaming/ConnectionInfo.java PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/DelimitedInputWriter.java 
898b3f9 
  streaming/src/java/org/apache/hive/streaming/HeartBeatFailure.java b1f9520 
  streaming/src/java/org/apache/hive/streaming/HiveEndPoint.java b04e137 
  streaming/src/java/org/apache/hive/streaming/HiveStreamingConnection.java 
PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/ImpersonationFailed.java 23e17e7 
  streaming/src/java/org/apache/hive/streaming/InvalidColumn.java 0011b14 
  streaming/src/java/org/apache/hive/streaming/InvalidPartition.java f1f9804 
  streaming/src/java/org/apache/hive/streaming/InvalidTable.java ef1c91d 
  streaming/src/java/org/apache/hive/streaming/InvalidTransactionState.java 
PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/InvalidTrasactionState.java 
762f5f8 
  streaming/src/java/org/apache/hive/streaming/PartitionCreationFailed.java 
5f9aca6 
  streaming/src/java/org/apache/hive/streaming/PartitionHandler.java 
PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/PartitionInfo.java PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/QueryFailedException.java 
ccd3ae0 
  streaming/src/java/org/apache/hive/streaming/RecordWriter.java dc6d70e 
  streaming/src/java/org/apache/hive/streaming/SerializationError.java a57ba00 
  streaming/src/java/org/apache/hive/streaming/StreamingConnection.java 2f760ea 
  streaming/src/java/org/apache/hive/streaming/StreamingException.java a7f84c1 
  streaming/src/java/org/apache/hive/streaming/StreamingIOFailure.java 0dfbfa7 
  streaming/src/java/org/apache/hive/streaming/StrictDelimitedInputWriter.java 
PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/StrictJsonWriter.java 0077913 
  streaming/src/java/org/apache/hive/streaming/StrictRegexWriter.java c0b7324 
  streaming/src/java/org/apache/hive/streaming/TransactionBatch.java 2b05771 
  streaming/src/java/org/apache/hive/streaming/TransactionBatchUnAvailable.java 
a8c8cd4 
  streaming/src/java/org/apache/hive/streaming/TransactionError.java a331b20 
  streaming/src/test/org/apache/hive/streaming/TestDelimitedInputWriter.java 
f0843a1 
  streaming/src/test/org/apache/hive/streaming/TestStreaming.java 0ec3048 
  
streaming/src/test/org/apache/hive/streaming/TestStreamingDynamicPartitioning.java
 PRE-CREATION 


Diff: https://reviews.apache.org/r/66645/diff/10/

Changes: https://reviews.apache.org/r/66645/diff/9-10/


Testing
---


Thanks,

Prasanth_J

Re: Review Request 66645: HIVE-19211: New streaming ingest API and support for dynamic partitioning

2018-04-27 Thread j . prasanth . j


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66645/
---

(Updated April 27, 2018, 9:36 p.m.)


Review request for hive, Ashutosh Chauhan and Eugene Koifman.


Changes
---

Rebased patch.


Bugs: HIVE-19211
https://issues.apache.org/jira/browse/HIVE-19211


Repository: hive-git


Description
---

HIVE-19211: New streaming ingest API and support for dynamic partitioning


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 6e35653 
  itests/hive-unit/pom.xml 3ae7f2f 
  
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/TestCompactor.java
 4ebd096 
  metastore/src/java/org/apache/hadoop/hive/metastore/HiveClientCache.java 
PRE-CREATION 
  metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStoreUtils.java 
a66c135 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcRecordUpdater.java 09f8802 
  ql/src/java/org/apache/hadoop/hive/ql/lockmgr/DbTxnManager.java 76569d5 
  ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java 4661881 
  serde/src/java/org/apache/hadoop/hive/serde2/JsonSerDe.java PRE-CREATION 
  streaming/pom.xml b58ec01 
  streaming/src/java/org/apache/hive/streaming/AbstractRecordWriter.java 
25998ae 
  streaming/src/java/org/apache/hive/streaming/ConnectionError.java 668bffb 
  streaming/src/java/org/apache/hive/streaming/ConnectionInfo.java PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/DelimitedInputWriter.java 
898b3f9 
  streaming/src/java/org/apache/hive/streaming/HeartBeatFailure.java b1f9520 
  streaming/src/java/org/apache/hive/streaming/HiveEndPoint.java b04e137 
  streaming/src/java/org/apache/hive/streaming/HiveStreamingConnection.java 
PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/ImpersonationFailed.java 23e17e7 
  streaming/src/java/org/apache/hive/streaming/InvalidColumn.java 0011b14 
  streaming/src/java/org/apache/hive/streaming/InvalidPartition.java f1f9804 
  streaming/src/java/org/apache/hive/streaming/InvalidTable.java ef1c91d 
  streaming/src/java/org/apache/hive/streaming/InvalidTransactionState.java 
PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/InvalidTrasactionState.java 
762f5f8 
  streaming/src/java/org/apache/hive/streaming/PartitionCreationFailed.java 
5f9aca6 
  streaming/src/java/org/apache/hive/streaming/PartitionHandler.java 
PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/PartitionInfo.java PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/QueryFailedException.java 
ccd3ae0 
  streaming/src/java/org/apache/hive/streaming/RecordWriter.java dc6d70e 
  streaming/src/java/org/apache/hive/streaming/SerializationError.java a57ba00 
  streaming/src/java/org/apache/hive/streaming/StreamingConnection.java 2f760ea 
  streaming/src/java/org/apache/hive/streaming/StreamingException.java a7f84c1 
  streaming/src/java/org/apache/hive/streaming/StreamingIOFailure.java 0dfbfa7 
  streaming/src/java/org/apache/hive/streaming/StrictDelimitedInputWriter.java 
PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/StrictJsonWriter.java 0077913 
  streaming/src/java/org/apache/hive/streaming/StrictRegexWriter.java c0b7324 
  streaming/src/java/org/apache/hive/streaming/TransactionBatch.java 2b05771 
  streaming/src/java/org/apache/hive/streaming/TransactionBatchUnAvailable.java 
a8c8cd4 
  streaming/src/java/org/apache/hive/streaming/TransactionError.java a331b20 
  streaming/src/test/org/apache/hive/streaming/TestDelimitedInputWriter.java 
f0843a1 
  streaming/src/test/org/apache/hive/streaming/TestStreaming.java 3343d10 
  
streaming/src/test/org/apache/hive/streaming/TestStreamingDynamicPartitioning.java
 PRE-CREATION 


Diff: https://reviews.apache.org/r/66645/diff/9/

Changes: https://reviews.apache.org/r/66645/diff/8-9/


Testing
---


Thanks,

Prasanth_J

Review Request 66848: HIVE-19211: New streaming ingest API and support for dynamic partitioning

2018-04-27 Thread j . prasanth . j


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66848/
---

Review request for hive and Gopal V.


Bugs: HIVE-19211
https://issues.apache.org/jira/browse/HIVE-19211


Repository: hive-git


Description
---

HIVE-19206: Automatic memory management for open streaming writers


Diffs
-

  common/src/java/org/apache/hadoop/hive/common/HeapMemoryMonitor.java 
PRE-CREATION 
  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 
f40c60606ca704d3f8c434c83107d63652ad359b 
  itests/hive-unit/pom.xml 3ae7f2feeec2df8156f89bdc204e2c35976485e2 
  
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/TestCompactor.java
 82ba77528678d01517dec58c3dc5d6b07077a965 
  metastore/src/java/org/apache/hadoop/hive/metastore/HiveClientCache.java 
PRE-CREATION 
  metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStoreUtils.java 
a66c13507abef42977dfdb315ff7d69404f67ac3 
  ql/src/java/org/apache/hadoop/hive/ql/io/RecordUpdater.java 
0aed172b6978847cdefe97878d2bc91648e37e0b 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcOutputFormat.java 
e69d1a06e2e001a6d706c3e6d92605c5d6a3d678 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcRecordUpdater.java 
09f880221ce74cbe5477e6d514b4a6915532a491 
  ql/src/java/org/apache/hadoop/hive/ql/lockmgr/DbTxnManager.java 
7b7fd5d19842ec691390880dcd1366539e9a6993 
  ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java 
466188130184516459dbd307e9cd0cf22844b46d 
  ql/src/test/org/apache/hadoop/hive/ql/exec/TestFileSinkOperator.java 
7f6077c460504a9a6bf2404e07905a0f6b0da654 
  serde/src/java/org/apache/hadoop/hive/serde2/JsonSerDe.java PRE-CREATION 
  streaming/pom.xml b58ec015aa69e29aacdc0a165ead9439ea2e4b26 
  streaming/src/java/org/apache/hive/streaming/AbstractRecordWriter.java 
25998ae31a3a829aab45f9e526aa03d94feff5e0 
  streaming/src/java/org/apache/hive/streaming/ConnectionError.java 
668bffb1ab17558dec33d599bddd6e28a06b3c5a 
  streaming/src/java/org/apache/hive/streaming/ConnectionInfo.java PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/ConnectionStats.java 
PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/DelimitedInputWriter.java 
898b3f9bb1d1c483cae8c1dd4f2338fc453d514b 
  streaming/src/java/org/apache/hive/streaming/HeartBeatFailure.java 
b1f9520814d260a3d2df23e6050e72d803874da9 
  streaming/src/java/org/apache/hive/streaming/HiveEndPoint.java 
b04e13784485ca097153bbec86f80d22e15e5cdc 
  streaming/src/java/org/apache/hive/streaming/HiveStreamingConnection.java 
PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/ImpersonationFailed.java 
23e17e76237036d8f9419bef2255f4f82c5b18a1 
  streaming/src/java/org/apache/hive/streaming/InvalidColumn.java 
0011b1454f8815816be931bf67cc13e7e78c9c0d 
  streaming/src/java/org/apache/hive/streaming/InvalidPartition.java 
f1f980430f3aceeb044bb549cc1a37a33c144750 
  streaming/src/java/org/apache/hive/streaming/InvalidTable.java 
ef1c91dbeb84b325b019318122fdd1f45b927414 
  streaming/src/java/org/apache/hive/streaming/InvalidTrasactionState.java 
762f5f86fc0df4a59cb54812a5dc79c1e2bc9489 
  streaming/src/java/org/apache/hive/streaming/PartitionCreationFailed.java 
5f9aca66ea0f2a7b2c3d2f6fb805fa1760b69e44 
  streaming/src/java/org/apache/hive/streaming/PartitionInfo.java PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/QueryFailedException.java 
ccd3ae0c98ea6ced0290f1ab027ad6337453fca2 
  streaming/src/java/org/apache/hive/streaming/RecordWriter.java 
dc6d70e92438e037d764099c82f5f654d5f5d801 
  streaming/src/java/org/apache/hive/streaming/SerializationError.java 
a57ba00ba401283aedd3f685171ef6bd810b11cd 
  streaming/src/java/org/apache/hive/streaming/StreamingConnection.java 
2f760ea86eecbbc96db08509405a369abf7d89d5 
  streaming/src/java/org/apache/hive/streaming/StreamingException.java 
a7f84c14f30f2e4753bd99b3d2d1dcb236b0197b 
  streaming/src/java/org/apache/hive/streaming/StreamingIOFailure.java 
0dfbfa71c50215d8f3e25298c8d11634a3cbedc4 
  streaming/src/java/org/apache/hive/streaming/StrictDelimitedInputWriter.java 
PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/StrictJsonWriter.java 
0077913cd1f0afbafe4608c4378398f61e254424 
  streaming/src/java/org/apache/hive/streaming/StrictRegexWriter.java 
c0b732482d35305ceaba1adfff09659e193ab098 
  streaming/src/java/org/apache/hive/streaming/TransactionBatch.java 
2b057718f58dec6de3e2b329a43bb5a06ce7c9ed 
  streaming/src/java/org/apache/hive/streaming/TransactionBatchUnAvailable.java 
a8c8cd48726421003df186fa1e0c2ecd18bdd5b4 
  streaming/src/java/org/apache/hive/streaming/TransactionError.java 
a331b20463e8328148fb08d85cf3ce77a7463062 
  streaming/src/test/org/apache/hive/streaming/TestDelimitedInputWriter.java 
f0843a1748d956ea99dd4807cf0b4ffbe0ef9cba 
  streaming/src/test/org/apache/hive/streaming/TestStreaming.java

Re: Review Request 66848: HIVE-19211: New streaming ingest API and support for dynamic partitioning

2018-04-27 Thread j . prasanth . j


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66848/
---

(Updated April 27, 2018, 9:43 a.m.)


Review request for hive and Gopal V.


Bugs: HIVE-19206
https://issues.apache.org/jira/browse/HIVE-19206


Repository: hive-git


Description
---

HIVE-19206: Automatic memory management for open streaming writers


Diffs
-

  common/src/java/org/apache/hadoop/hive/common/HeapMemoryMonitor.java 
PRE-CREATION 
  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 
f40c60606ca704d3f8c434c83107d63652ad359b 
  itests/hive-unit/pom.xml 3ae7f2feeec2df8156f89bdc204e2c35976485e2 
  
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/TestCompactor.java
 82ba77528678d01517dec58c3dc5d6b07077a965 
  metastore/src/java/org/apache/hadoop/hive/metastore/HiveClientCache.java 
PRE-CREATION 
  metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStoreUtils.java 
a66c13507abef42977dfdb315ff7d69404f67ac3 
  ql/src/java/org/apache/hadoop/hive/ql/io/RecordUpdater.java 
0aed172b6978847cdefe97878d2bc91648e37e0b 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcOutputFormat.java 
e69d1a06e2e001a6d706c3e6d92605c5d6a3d678 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcRecordUpdater.java 
09f880221ce74cbe5477e6d514b4a6915532a491 
  ql/src/java/org/apache/hadoop/hive/ql/lockmgr/DbTxnManager.java 
7b7fd5d19842ec691390880dcd1366539e9a6993 
  ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java 
466188130184516459dbd307e9cd0cf22844b46d 
  ql/src/test/org/apache/hadoop/hive/ql/exec/TestFileSinkOperator.java 
7f6077c460504a9a6bf2404e07905a0f6b0da654 
  serde/src/java/org/apache/hadoop/hive/serde2/JsonSerDe.java PRE-CREATION 
  streaming/pom.xml b58ec015aa69e29aacdc0a165ead9439ea2e4b26 
  streaming/src/java/org/apache/hive/streaming/AbstractRecordWriter.java 
25998ae31a3a829aab45f9e526aa03d94feff5e0 
  streaming/src/java/org/apache/hive/streaming/ConnectionError.java 
668bffb1ab17558dec33d599bddd6e28a06b3c5a 
  streaming/src/java/org/apache/hive/streaming/ConnectionInfo.java PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/ConnectionStats.java 
PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/DelimitedInputWriter.java 
898b3f9bb1d1c483cae8c1dd4f2338fc453d514b 
  streaming/src/java/org/apache/hive/streaming/HeartBeatFailure.java 
b1f9520814d260a3d2df23e6050e72d803874da9 
  streaming/src/java/org/apache/hive/streaming/HiveEndPoint.java 
b04e13784485ca097153bbec86f80d22e15e5cdc 
  streaming/src/java/org/apache/hive/streaming/HiveStreamingConnection.java 
PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/ImpersonationFailed.java 
23e17e76237036d8f9419bef2255f4f82c5b18a1 
  streaming/src/java/org/apache/hive/streaming/InvalidColumn.java 
0011b1454f8815816be931bf67cc13e7e78c9c0d 
  streaming/src/java/org/apache/hive/streaming/InvalidPartition.java 
f1f980430f3aceeb044bb549cc1a37a33c144750 
  streaming/src/java/org/apache/hive/streaming/InvalidTable.java 
ef1c91dbeb84b325b019318122fdd1f45b927414 
  streaming/src/java/org/apache/hive/streaming/InvalidTrasactionState.java 
762f5f86fc0df4a59cb54812a5dc79c1e2bc9489 
  streaming/src/java/org/apache/hive/streaming/PartitionCreationFailed.java 
5f9aca66ea0f2a7b2c3d2f6fb805fa1760b69e44 
  streaming/src/java/org/apache/hive/streaming/PartitionInfo.java PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/QueryFailedException.java 
ccd3ae0c98ea6ced0290f1ab027ad6337453fca2 
  streaming/src/java/org/apache/hive/streaming/RecordWriter.java 
dc6d70e92438e037d764099c82f5f654d5f5d801 
  streaming/src/java/org/apache/hive/streaming/SerializationError.java 
a57ba00ba401283aedd3f685171ef6bd810b11cd 
  streaming/src/java/org/apache/hive/streaming/StreamingConnection.java 
2f760ea86eecbbc96db08509405a369abf7d89d5 
  streaming/src/java/org/apache/hive/streaming/StreamingException.java 
a7f84c14f30f2e4753bd99b3d2d1dcb236b0197b 
  streaming/src/java/org/apache/hive/streaming/StreamingIOFailure.java 
0dfbfa71c50215d8f3e25298c8d11634a3cbedc4 
  streaming/src/java/org/apache/hive/streaming/StrictDelimitedInputWriter.java 
PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/StrictJsonWriter.java 
0077913cd1f0afbafe4608c4378398f61e254424 
  streaming/src/java/org/apache/hive/streaming/StrictRegexWriter.java 
c0b732482d35305ceaba1adfff09659e193ab098 
  streaming/src/java/org/apache/hive/streaming/TransactionBatch.java 
2b057718f58dec6de3e2b329a43bb5a06ce7c9ed 
  streaming/src/java/org/apache/hive/streaming/TransactionBatchUnAvailable.java 
a8c8cd48726421003df186fa1e0c2ecd18bdd5b4 
  streaming/src/java/org/apache/hive/streaming/TransactionError.java 
a331b20463e8328148fb08d85cf3ce77a7463062 
  streaming/src/test/org/apache/hive/streaming/TestDelimitedInputWriter.java 
f0843a1748d956ea99dd4807cf0b4ffbe0ef9cba

Re: Review Request 66645: HIVE-19211: New streaming ingest API and support for dynamic partitioning

2018-04-26 Thread j . prasanth . j


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66645/
---

(Updated April 26, 2018, 10:55 p.m.)


Review request for hive, Ashutosh Chauhan and Eugene Koifman.


Changes
---

Addressed review comments.


Bugs: HIVE-19211
https://issues.apache.org/jira/browse/HIVE-19211


Repository: hive-git


Description
---

HIVE-19211: New streaming ingest API and support for dynamic partitioning


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java f40c606 
  itests/hive-unit/pom.xml 3ae7f2f 
  
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/TestCompactor.java
 82ba775 
  metastore/src/java/org/apache/hadoop/hive/metastore/HiveClientCache.java 
PRE-CREATION 
  metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStoreUtils.java 
a66c135 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcRecordUpdater.java 09f8802 
  ql/src/java/org/apache/hadoop/hive/ql/lockmgr/DbTxnManager.java 7b7fd5d 
  ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java 4661881 
  serde/src/java/org/apache/hadoop/hive/serde2/JsonSerDe.java PRE-CREATION 
  streaming/pom.xml b58ec01 
  streaming/src/java/org/apache/hive/streaming/AbstractRecordWriter.java 
25998ae 
  streaming/src/java/org/apache/hive/streaming/ConnectionError.java 668bffb 
  streaming/src/java/org/apache/hive/streaming/ConnectionInfo.java PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/DelimitedInputWriter.java 
898b3f9 
  streaming/src/java/org/apache/hive/streaming/HeartBeatFailure.java b1f9520 
  streaming/src/java/org/apache/hive/streaming/HiveEndPoint.java b04e137 
  streaming/src/java/org/apache/hive/streaming/HiveStreamingConnection.java 
PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/ImpersonationFailed.java 23e17e7 
  streaming/src/java/org/apache/hive/streaming/InvalidColumn.java 0011b14 
  streaming/src/java/org/apache/hive/streaming/InvalidPartition.java f1f9804 
  streaming/src/java/org/apache/hive/streaming/InvalidTable.java ef1c91d 
  streaming/src/java/org/apache/hive/streaming/InvalidTransactionState.java 
PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/InvalidTrasactionState.java 
762f5f8 
  streaming/src/java/org/apache/hive/streaming/PartitionCreationFailed.java 
5f9aca6 
  streaming/src/java/org/apache/hive/streaming/PartitionHandler.java 
PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/PartitionInfo.java PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/QueryFailedException.java 
ccd3ae0 
  streaming/src/java/org/apache/hive/streaming/RecordWriter.java dc6d70e 
  streaming/src/java/org/apache/hive/streaming/SerializationError.java a57ba00 
  streaming/src/java/org/apache/hive/streaming/StreamingConnection.java 2f760ea 
  streaming/src/java/org/apache/hive/streaming/StreamingException.java a7f84c1 
  streaming/src/java/org/apache/hive/streaming/StreamingIOFailure.java 0dfbfa7 
  streaming/src/java/org/apache/hive/streaming/StrictDelimitedInputWriter.java 
PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/StrictJsonWriter.java 0077913 
  streaming/src/java/org/apache/hive/streaming/StrictRegexWriter.java c0b7324 
  streaming/src/java/org/apache/hive/streaming/TransactionBatch.java 2b05771 
  streaming/src/java/org/apache/hive/streaming/TransactionBatchUnAvailable.java 
a8c8cd4 
  streaming/src/java/org/apache/hive/streaming/TransactionError.java a331b20 
  streaming/src/test/org/apache/hive/streaming/TestDelimitedInputWriter.java 
f0843a1 
  streaming/src/test/org/apache/hive/streaming/TestStreaming.java 3343d10 
  
streaming/src/test/org/apache/hive/streaming/TestStreamingDynamicPartitioning.java
 PRE-CREATION 


Diff: https://reviews.apache.org/r/66645/diff/8/

Changes: https://reviews.apache.org/r/66645/diff/7-8/


Testing
---


Thanks,

Prasanth_J

Re: Review Request 66645: HIVE-19211: New streaming ingest API and support for dynamic partitioning

2018-04-26 Thread j . prasanth . j



> On April 26, 2018, 2:09 a.m., Eugene Koifman wrote:
> > itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/TestCompactor.java
> > Lines 426 (patched)
> > 
> >
> > This seems to switch the test from 1 api to another.  Could it be 
> > parametrized so the old one doesn't loose coverage?

Parametrized.


> On April 26, 2018, 2:09 a.m., Eugene Koifman wrote:
> > streaming/src/java/org/apache/hive/streaming/AbstractRecordWriter.java
> > Lines 113 (patched)
> > 
> >
> > Is there a plan to support MM tables?  (with batch size of 1 it can be 
> > done).  They won't implement AcidOutputFormat

Will take this up in a follow up.


> On April 26, 2018, 2:09 a.m., Eugene Koifman wrote:
> > streaming/src/java/org/apache/hive/streaming/HiveStreamingConnection.java
> > Lines 76 (patched)
> > 
> >
> > It should explain the threading model somewhere.  I assume everything 
> > is meant to be single threaded, though it's common for client code to 
> > process cancel type of event in a separate thread.  here it would 
> > presumably map to close() (perhaps abortTransaction())

Updated javadoc. Currently event close() and abortTransaction() are not thread 
safe. If a separate thread has to close or abort then it has to coordinate via 
external variables.


> On April 26, 2018, 2:09 a.m., Eugene Koifman wrote:
> > streaming/src/java/org/apache/hive/streaming/HiveStreamingConnection.java
> > Lines 283 (patched)
> > 
> >
> > Not sure we want to expose it in such an obvious way.
> > It may be better if there is a table prop for this or perhaps the doc 
> > should say that this is 'advisory' and we may ignore the value.  Longer 
> > term we'd like to consider an option where we force batch size = 1.  This 
> > is the only option for some cloud stores and generally simplifies acid if 
> > we don't have deltas_x_y with x<>y which may contain aborted txns.

I changed the default batch size to 1. Also marked this API as evolving with a 
note.


> On April 26, 2018, 2:09 a.m., Eugene Koifman wrote:
> > streaming/src/java/org/apache/hive/streaming/HiveStreamingConnection.java
> > Lines 377 (patched)
> > 
> >
> > why does this need Alter Table?  You already have a metastore 
> > connection and you know that you are adding a partiton w/o data.  In other 
> > words, why not just msClient.add_partition() and handle 
> > AlreadyExistsException or add_partitions(
> >   List partitions, boolean ifNotExists, boolean 
> > needResults).
> >  
> > This seems simpler and more efficient.

This was retained from old code. Updated to use add_partition() which make is 
simpler and efficient.


> On April 26, 2018, 2:09 a.m., Eugene Koifman wrote:
> > streaming/src/java/org/apache/hive/streaming/HiveStreamingConnection.java
> > Lines 399 (patched)
> > 
> >
> > use Warehouse.makePartName()?

Note required anymore.


> On April 26, 2018, 2:09 a.m., Eugene Koifman wrote:
> > streaming/src/java/org/apache/hive/streaming/HiveStreamingConnection.java
> > Lines 996 (patched)
> > 
> >
> > "we don'table wait"

Some weird IDE rename all over. Fixed it.


> On April 26, 2018, 2:09 a.m., Eugene Koifman wrote:
> > streaming/src/java/org/apache/hive/streaming/HiveStreamingConnection.java
> > Lines 1050 (patched)
> > 
> >
> > is this for testing?  How can the user control these?

Made it user configurable with a note advising against changing it.


> On April 26, 2018, 2:09 a.m., Eugene Koifman wrote:
> > streaming/src/java/org/apache/hive/streaming/RecordWriter.java
> > Lines 53 (patched)
> > 
> >
> > maybe make this onNewBatch() since it's not starting a batch

Removed this API. since the min and max write id doesn't change for a 
transaction batch, I am passing them via init() API now.


> On April 26, 2018, 2:09 a.m., Eugene Koifman wrote:
> > streaming/src/java/org/apache/hive/streaming/RecordWriter.java
> > Line 42 (original), 60 (patched)
> > 
> >
> > close() or closeWriter()?

Renamed it to close()


> On April 26, 2018, 2:09 a.m., Eugene Koifman wrote:
> > streaming/src/test/org/apache/hive/streaming/TestStreamingDynamicPartitioning.java
> > Lines 261 (patched)
> >

Re: Review Request 66645: HIVE-19211: New streaming ingest API and support for dynamic partitioning

2018-04-24 Thread j . prasanth . j


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66645/
---

(Updated April 24, 2018, 9:43 p.m.)


Review request for hive, Ashutosh Chauhan and Eugene Koifman.


Changes
---

Added some more tests for writer after close/commit/abort etc. test cases. Also 
rebased patch.


Bugs: HIVE-19211
https://issues.apache.org/jira/browse/HIVE-19211


Repository: hive-git


Description
---

HIVE-19211: New streaming ingest API and support for dynamic partitioning


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java f40c606 
  
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/TestCompactor.java
 82ba775 
  metastore/src/java/org/apache/hadoop/hive/metastore/HiveClientCache.java 
PRE-CREATION 
  metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStoreUtils.java 
a66c135 
  ql/src/java/org/apache/hadoop/hive/ql/lockmgr/DbTxnManager.java 7b7fd5d 
  serde/src/java/org/apache/hadoop/hive/serde2/JsonSerDe.java PRE-CREATION 
  streaming/pom.xml b58ec01 
  streaming/src/java/org/apache/hive/streaming/AbstractRecordWriter.java 
25998ae 
  streaming/src/java/org/apache/hive/streaming/ConnectionError.java 668bffb 
  streaming/src/java/org/apache/hive/streaming/ConnectionInfo.java PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/DelimitedInputWriter.java 
898b3f9 
  streaming/src/java/org/apache/hive/streaming/HeartBeatFailure.java b1f9520 
  streaming/src/java/org/apache/hive/streaming/HiveEndPoint.java b04e137 
  streaming/src/java/org/apache/hive/streaming/HiveStreamingConnection.java 
PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/ImpersonationFailed.java 23e17e7 
  streaming/src/java/org/apache/hive/streaming/InvalidColumn.java 0011b14 
  streaming/src/java/org/apache/hive/streaming/InvalidPartition.java f1f9804 
  streaming/src/java/org/apache/hive/streaming/InvalidTable.java ef1c91d 
  streaming/src/java/org/apache/hive/streaming/InvalidTransactionState.java 
PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/InvalidTrasactionState.java 
762f5f8 
  streaming/src/java/org/apache/hive/streaming/PartitionCreationFailed.java 
5f9aca6 
  streaming/src/java/org/apache/hive/streaming/PartitionHandler.java 
PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/QueryFailedException.java 
ccd3ae0 
  streaming/src/java/org/apache/hive/streaming/RecordWriter.java dc6d70e 
  streaming/src/java/org/apache/hive/streaming/SerializationError.java a57ba00 
  streaming/src/java/org/apache/hive/streaming/StreamingConnection.java 2f760ea 
  streaming/src/java/org/apache/hive/streaming/StreamingException.java a7f84c1 
  streaming/src/java/org/apache/hive/streaming/StreamingIOFailure.java 0dfbfa7 
  streaming/src/java/org/apache/hive/streaming/StrictDelimitedInputWriter.java 
PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/StrictJsonWriter.java 0077913 
  streaming/src/java/org/apache/hive/streaming/StrictRegexWriter.java c0b7324 
  streaming/src/java/org/apache/hive/streaming/TransactionBatch.java 2b05771 
  streaming/src/java/org/apache/hive/streaming/TransactionBatchUnAvailable.java 
a8c8cd4 
  streaming/src/java/org/apache/hive/streaming/TransactionError.java a331b20 
  streaming/src/test/org/apache/hive/streaming/TestDelimitedInputWriter.java 
f0843a1 
  streaming/src/test/org/apache/hive/streaming/TestStreaming.java 3343d10 
  
streaming/src/test/org/apache/hive/streaming/TestStreamingDynamicPartitioning.java
 PRE-CREATION 


Diff: https://reviews.apache.org/r/66645/diff/7/

Changes: https://reviews.apache.org/r/66645/diff/6-7/


Testing
---


Thanks,

Prasanth_J

Re: Review Request 66645: HIVE-19211: New streaming ingest API and support for dynamic partitioning

2018-04-24 Thread j . prasanth . j


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66645/
---

(Updated April 24, 2018, 10:43 a.m.)


Review request for hive, Ashutosh Chauhan and Eugene Koifman.


Changes
---

Small fix


Bugs: HIVE-19211
https://issues.apache.org/jira/browse/HIVE-19211


Repository: hive-git


Description
---

HIVE-19211: New streaming ingest API and support for dynamic partitioning


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 2403d7a 
  
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/TestCompactor.java
 82ba775 
  metastore/src/java/org/apache/hadoop/hive/metastore/HiveClientCache.java 
PRE-CREATION 
  metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStoreUtils.java 
a66c135 
  ql/src/java/org/apache/hadoop/hive/ql/lockmgr/DbTxnManager.java 7b7fd5d 
  serde/src/java/org/apache/hadoop/hive/serde2/JsonSerDe.java PRE-CREATION 
  streaming/pom.xml b58ec01 
  streaming/src/java/org/apache/hive/streaming/AbstractRecordWriter.java 
25998ae 
  streaming/src/java/org/apache/hive/streaming/ConnectionError.java 668bffb 
  streaming/src/java/org/apache/hive/streaming/ConnectionInfo.java PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/DelimitedInputWriter.java 
898b3f9 
  streaming/src/java/org/apache/hive/streaming/HeartBeatFailure.java b1f9520 
  streaming/src/java/org/apache/hive/streaming/HiveEndPoint.java b04e137 
  streaming/src/java/org/apache/hive/streaming/HiveStreamingConnection.java 
PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/ImpersonationFailed.java 23e17e7 
  streaming/src/java/org/apache/hive/streaming/InvalidColumn.java 0011b14 
  streaming/src/java/org/apache/hive/streaming/InvalidPartition.java f1f9804 
  streaming/src/java/org/apache/hive/streaming/InvalidTable.java ef1c91d 
  streaming/src/java/org/apache/hive/streaming/InvalidTransactionState.java 
PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/InvalidTrasactionState.java 
762f5f8 
  streaming/src/java/org/apache/hive/streaming/PartitionCreationFailed.java 
5f9aca6 
  streaming/src/java/org/apache/hive/streaming/PartitionHandler.java 
PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/QueryFailedException.java 
ccd3ae0 
  streaming/src/java/org/apache/hive/streaming/RecordWriter.java dc6d70e 
  streaming/src/java/org/apache/hive/streaming/SerializationError.java a57ba00 
  streaming/src/java/org/apache/hive/streaming/StreamingConnection.java 2f760ea 
  streaming/src/java/org/apache/hive/streaming/StreamingException.java a7f84c1 
  streaming/src/java/org/apache/hive/streaming/StreamingIOFailure.java 0dfbfa7 
  streaming/src/java/org/apache/hive/streaming/StrictDelimitedInputWriter.java 
PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/StrictJsonWriter.java 0077913 
  streaming/src/java/org/apache/hive/streaming/StrictRegexWriter.java c0b7324 
  streaming/src/java/org/apache/hive/streaming/TransactionBatch.java 2b05771 
  streaming/src/java/org/apache/hive/streaming/TransactionBatchUnAvailable.java 
a8c8cd4 
  streaming/src/java/org/apache/hive/streaming/TransactionError.java a331b20 
  streaming/src/test/org/apache/hive/streaming/TestDelimitedInputWriter.java 
f0843a1 
  streaming/src/test/org/apache/hive/streaming/TestStreaming.java e5dd3b3 
  
streaming/src/test/org/apache/hive/streaming/TestStreamingDynamicPartitioning.java
 PRE-CREATION 


Diff: https://reviews.apache.org/r/66645/diff/6/

Changes: https://reviews.apache.org/r/66645/diff/5-6/


Testing
---


Thanks,

Prasanth_J

Re: Review Request 66645: HIVE-19211: New streaming ingest API and support for dynamic partitioning

2018-04-24 Thread j . prasanth . j


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66645/#review201821
---




streaming/src/java/org/apache/hive/streaming/HiveStreamingConnection.java
Lines 500 (patched)


these are the variables that has to be locked. Will remove this comment in 
next patch or before commit.


- Prasanth_J


On April 24, 2018, 10:24 a.m., Prasanth_J wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66645/
> ---
> 
> (Updated April 24, 2018, 10:24 a.m.)
> 
> 
> Review request for hive, Ashutosh Chauhan and Eugene Koifman.
> 
> 
> Bugs: HIVE-19211
> https://issues.apache.org/jira/browse/HIVE-19211
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-19211: New streaming ingest API and support for dynamic partitioning
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 2403d7a 
>   
> itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/TestCompactor.java
>  82ba775 
>   metastore/src/java/org/apache/hadoop/hive/metastore/HiveClientCache.java 
> PRE-CREATION 
>   metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStoreUtils.java 
> a66c135 
>   ql/src/java/org/apache/hadoop/hive/ql/lockmgr/DbTxnManager.java 7b7fd5d 
>   serde/src/java/org/apache/hadoop/hive/serde2/JsonSerDe.java PRE-CREATION 
>   streaming/pom.xml b58ec01 
>   streaming/src/java/org/apache/hive/streaming/AbstractRecordWriter.java 
> 25998ae 
>   streaming/src/java/org/apache/hive/streaming/ConnectionError.java 668bffb 
>   streaming/src/java/org/apache/hive/streaming/ConnectionInfo.java 
> PRE-CREATION 
>   streaming/src/java/org/apache/hive/streaming/DelimitedInputWriter.java 
> 898b3f9 
>   streaming/src/java/org/apache/hive/streaming/HeartBeatFailure.java b1f9520 
>   streaming/src/java/org/apache/hive/streaming/HiveEndPoint.java b04e137 
>   streaming/src/java/org/apache/hive/streaming/HiveStreamingConnection.java 
> PRE-CREATION 
>   streaming/src/java/org/apache/hive/streaming/ImpersonationFailed.java 
> 23e17e7 
>   streaming/src/java/org/apache/hive/streaming/InvalidColumn.java 0011b14 
>   streaming/src/java/org/apache/hive/streaming/InvalidPartition.java f1f9804 
>   streaming/src/java/org/apache/hive/streaming/InvalidTable.java ef1c91d 
>   streaming/src/java/org/apache/hive/streaming/InvalidTransactionState.java 
> PRE-CREATION 
>   streaming/src/java/org/apache/hive/streaming/InvalidTrasactionState.java 
> 762f5f8 
>   streaming/src/java/org/apache/hive/streaming/PartitionCreationFailed.java 
> 5f9aca6 
>   streaming/src/java/org/apache/hive/streaming/PartitionHandler.java 
> PRE-CREATION 
>   streaming/src/java/org/apache/hive/streaming/QueryFailedException.java 
> ccd3ae0 
>   streaming/src/java/org/apache/hive/streaming/RecordWriter.java dc6d70e 
>   streaming/src/java/org/apache/hive/streaming/SerializationError.java 
> a57ba00 
>   streaming/src/java/org/apache/hive/streaming/StreamingConnection.java 
> 2f760ea 
>   streaming/src/java/org/apache/hive/streaming/StreamingException.java 
> a7f84c1 
>   streaming/src/java/org/apache/hive/streaming/StreamingIOFailure.java 
> 0dfbfa7 
>   
> streaming/src/java/org/apache/hive/streaming/StrictDelimitedInputWriter.java 
> PRE-CREATION 
>   streaming/src/java/org/apache/hive/streaming/StrictJsonWriter.java 0077913 
>   streaming/src/java/org/apache/hive/streaming/StrictRegexWriter.java c0b7324 
>   streaming/src/java/org/apache/hive/streaming/TransactionBatch.java 2b05771 
>   
> streaming/src/java/org/apache/hive/streaming/TransactionBatchUnAvailable.java 
> a8c8cd4 
>   streaming/src/java/org/apache/hive/streaming/TransactionError.java a331b20 
>   streaming/src/test/org/apache/hive/streaming/TestDelimitedInputWriter.java 
> f0843a1 
>   streaming/src/test/org/apache/hive/streaming/TestStreaming.java e5dd3b3 
>   
> streaming/src/test/org/apache/hive/streaming/TestStreamingDynamicPartitioning.java
>  PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/66645/diff/5/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Prasanth_J
> 
>

Re: Review Request 66645: HIVE-19211: New streaming ingest API and support for dynamic partitioning

2018-04-24 Thread j . prasanth . j


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66645/
---

(Updated April 24, 2018, 10:24 a.m.)


Review request for hive, Ashutosh Chauhan and Eugene Koifman.


Changes
---

Addressed review comments.


Bugs: HIVE-19211
https://issues.apache.org/jira/browse/HIVE-19211


Repository: hive-git


Description
---

HIVE-19211: New streaming ingest API and support for dynamic partitioning


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 2403d7a 
  
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/TestCompactor.java
 82ba775 
  metastore/src/java/org/apache/hadoop/hive/metastore/HiveClientCache.java 
PRE-CREATION 
  metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStoreUtils.java 
a66c135 
  ql/src/java/org/apache/hadoop/hive/ql/lockmgr/DbTxnManager.java 7b7fd5d 
  serde/src/java/org/apache/hadoop/hive/serde2/JsonSerDe.java PRE-CREATION 
  streaming/pom.xml b58ec01 
  streaming/src/java/org/apache/hive/streaming/AbstractRecordWriter.java 
25998ae 
  streaming/src/java/org/apache/hive/streaming/ConnectionError.java 668bffb 
  streaming/src/java/org/apache/hive/streaming/ConnectionInfo.java PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/DelimitedInputWriter.java 
898b3f9 
  streaming/src/java/org/apache/hive/streaming/HeartBeatFailure.java b1f9520 
  streaming/src/java/org/apache/hive/streaming/HiveEndPoint.java b04e137 
  streaming/src/java/org/apache/hive/streaming/HiveStreamingConnection.java 
PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/ImpersonationFailed.java 23e17e7 
  streaming/src/java/org/apache/hive/streaming/InvalidColumn.java 0011b14 
  streaming/src/java/org/apache/hive/streaming/InvalidPartition.java f1f9804 
  streaming/src/java/org/apache/hive/streaming/InvalidTable.java ef1c91d 
  streaming/src/java/org/apache/hive/streaming/InvalidTransactionState.java 
PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/InvalidTrasactionState.java 
762f5f8 
  streaming/src/java/org/apache/hive/streaming/PartitionCreationFailed.java 
5f9aca6 
  streaming/src/java/org/apache/hive/streaming/PartitionHandler.java 
PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/QueryFailedException.java 
ccd3ae0 
  streaming/src/java/org/apache/hive/streaming/RecordWriter.java dc6d70e 
  streaming/src/java/org/apache/hive/streaming/SerializationError.java a57ba00 
  streaming/src/java/org/apache/hive/streaming/StreamingConnection.java 2f760ea 
  streaming/src/java/org/apache/hive/streaming/StreamingException.java a7f84c1 
  streaming/src/java/org/apache/hive/streaming/StreamingIOFailure.java 0dfbfa7 
  streaming/src/java/org/apache/hive/streaming/StrictDelimitedInputWriter.java 
PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/StrictJsonWriter.java 0077913 
  streaming/src/java/org/apache/hive/streaming/StrictRegexWriter.java c0b7324 
  streaming/src/java/org/apache/hive/streaming/TransactionBatch.java 2b05771 
  streaming/src/java/org/apache/hive/streaming/TransactionBatchUnAvailable.java 
a8c8cd4 
  streaming/src/java/org/apache/hive/streaming/TransactionError.java a331b20 
  streaming/src/test/org/apache/hive/streaming/TestDelimitedInputWriter.java 
f0843a1 
  streaming/src/test/org/apache/hive/streaming/TestStreaming.java e5dd3b3 
  
streaming/src/test/org/apache/hive/streaming/TestStreamingDynamicPartitioning.java
 PRE-CREATION 


Diff: https://reviews.apache.org/r/66645/diff/5/

Changes: https://reviews.apache.org/r/66645/diff/4-5/


Testing
---


Thanks,

Prasanth_J

Re: Review Request 66645: HIVE-19211: New streaming ingest API and support for dynamic partitioning

2018-04-20 Thread j . prasanth . j



> On April 20, 2018, 7:44 p.m., Eugene Koifman wrote:
> > common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
> > Lines 2499 (patched)
> > 
> >
> > Heartbeat for each txn should be sent at ConfVars.HIVE_TXN_TIMEOUT/2 
> > intervals.  HIVE_TXN_TIMEOUT is what the server uses to timeout a txn.  
> > Sending heartbets more frequently loads the metastore unnecessarily.  See 
> > DbTxnManger.getHeartbeatInterval() for example.  This also guarantees that 
> > server and client are in sync.

Ok. Will fix it. Will make heartbeat interval default to txn_timeout/2.


> On April 20, 2018, 7:44 p.m., Eugene Koifman wrote:
> > streaming/src/java/org/apache/hive/streaming/ConnectionInfo.java
> > Lines 54 (patched)
> > 
> >
> > Why is it useful to make this distinction?  Since the imple is able to 
> > route all events to appropriate partition, why should it not do that all 
> > the time.

This avoids baking the static partition in the record. Static partitions take 
fast path in which it does not decode the row to extract partition values out.


> On April 20, 2018, 7:44 p.m., Eugene Koifman wrote:
> > streaming/src/java/org/apache/hive/streaming/ConnectionInfo.java
> > Lines 68 (patched)
> > 
> >
> > if isStaticPartitioning() == true, can isDynamicPartitioning == ture, 
> > i.e. why 2 methods

This was just for convenience. I don't think it is required or used other than 
logging. Will remove it.


> On April 20, 2018, 7:44 p.m., Eugene Koifman wrote:
> > streaming/src/java/org/apache/hive/streaming/HiveStreamingConnection.java
> > Lines 121 (patched)
> > 
> >
> > if all operations on hms client are mutexted via txnBatchLock, why have 
> > 2 hms clients?
> > 
> > More generally, having 1 thread per connection may be rather heavy 
> > weight.  Each thread will send 1 msg every few miniutes, i.e. will be 
> > mostly idle.  DbTxnManager uses a static pool of 5 or so threads which does 
> > heartbeats for the whole VM (HS2).  I think that would be a good model here 
> > as well.

Only the state updates are mutexed. Actual heartbeat is outside the lock. The 
write path which happens more frequently is not mutexed. The transaction 
handling is mutexed so that heartbeat thread gets to see the correct state. 
Once heartbeat thread gets the first and last transaction id that is open, 
actual hearbeat is done via second ms client outside of the mutex. And with 
dynamic partitioning, we don't expect to see many streaming connections. In the 
earlier model, each partition was handled by a separate connection. In this new 
model, there can be only one connection per VM to write using streaming 
connection in which case I think having a static pool for heartbeat threads 
might not be required.


> On April 20, 2018, 7:44 p.m., Eugene Koifman wrote:
> > streaming/src/java/org/apache/hive/streaming/HiveStreamingConnection.java
> > Lines 309 (patched)
> > 
> >
> > Shoule each streamingConnection have a UUID to include here?  I can 
> > imagine mulitple instances writing to the same table which are not 
> > distiguishable by db.table

Will piggyback on the agentInfo. If agentInfo is not explicitly, will create 
UUID.


> On April 20, 2018, 7:44 p.m., Eugene Koifman wrote:
> > streaming/src/java/org/apache/hive/streaming/HiveStreamingConnection.java
> > Lines 326 (patched)
> > 
> >
> > this should be in try/catch.  SessionState.start() can sometimes throw, 
> > but after it creates a Session object and ataches it to a threadLocal which 
> > then makes every user of that thread fail.  There is an Apache bug 
> > describing some situations where this can happen.

Will fix it.


> On April 20, 2018, 7:44 p.m., Eugene Koifman wrote:
> > streaming/src/java/org/apache/hive/streaming/HiveStreamingConnection.java
> > Lines 337 (patched)
> > 
> >
> > should include db.table in all exceptions/message

Will fix it.


> On April 20, 2018, 7:44 p.m., Eugene Koifman wrote:
> > streaming/src/java/org/apache/hive/streaming/HiveStreamingConnection.java
> > Lines 422 (patched)
> > 
> >
> > The API in general doesn't expose 'catalog' - should it?

Is catalog work already part of 3.0.0? If so I will include it.


> On April 20, 2018, 7:44 p.m., Eugene Koifman wrote:
> > streaming/src/java/org/apache/hive/streaming/HiveStreamingConnection.java
> > Lines 538 (patched)
> >

Review Request 66729: HIVE-19214: High throughput ingest ORC format

2018-04-19 Thread j . prasanth . j


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66729/
---

Review request for hive and Gopal V.


Bugs: HIVE-19214
https://issues.apache.org/jira/browse/HIVE-19214


Repository: hive-git


Description
---

HIVE-19214: High throughput ingest ORC format


Diffs
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 
73492ff99c0aad6f6aade4fb04cd80a9c982b979 
  
llap-server/src/java/org/apache/hadoop/hive/llap/io/decode/OrcEncodedDataConsumer.java
 9e8ae103ee7de59c368eb76fd49a33b866bd30fe 
  
llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/OrcEncodedDataReader.java
 afb8fc51f22f768e3a465cda65b4ea9fcb635949 
  
llap-server/src/java/org/apache/hadoop/hive/llap/io/metadata/OrcStripeMetadata.java
 3d9e99cfae8fc7b658582277c9a630f7a801953d 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java 
fe109d7b9668f686b999df37c60016a93a251d22 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcRecordUpdater.java 
d850062377d182e33a6191268d50d0008d7c77de 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/EncodedReaderImpl.java 
4e17394653caf51cb29b4017656518a5facab8ee 
  ql/src/test/queries/clientpositive/orc_ppd_exception.q 
1513d911201c30130d611f49d1877cdacf2892fc 
  ql/src/test/queries/clientpositive/vector_acid3.q 
d4313f4ba5a8358ccf6a266564777823047640e9 
  ql/src/test/results/clientpositive/llap/vector_acid3.q.out 
46c82fcbb0d8dfd5889a4e258ba8af2670d33373 
  ql/src/test/results/clientpositive/orc_ppd_exception.q.out 
e03eb71068ebd80cc8e7453bd6129d10b18336c3 
  streaming/src/test/org/apache/hive/streaming/TestStreaming.java 
6f63bfb43e5dbe4c9529dfc80787a95ba6524c01 


Diff: https://reviews.apache.org/r/66729/diff/1/


Testing
---


Thanks,

Prasanth_J

Re: Review Request 66645: HIVE-19211: New streaming ingest API and support for dynamic partitioning

2018-04-18 Thread j . prasanth . j


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66645/
---

(Updated April 19, 2018, 2:36 a.m.)


Review request for hive, Ashutosh Chauhan and Eugene Koifman.


Changes
---

Some concurrency fixes.


Bugs: HIVE-19211
https://issues.apache.org/jira/browse/HIVE-19211


Repository: hive-git


Description
---

HIVE-19211: New streaming ingest API and support for dynamic partitioning


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 73492ff 
  
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/TestCompactor.java
 82ba775 
  metastore/src/java/org/apache/hadoop/hive/metastore/HiveClientCache.java 
PRE-CREATION 
  metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStoreUtils.java 
a66c135 
  serde/src/java/org/apache/hadoop/hive/serde2/JsonSerDe.java PRE-CREATION 
  streaming/pom.xml b58ec01 
  streaming/src/java/org/apache/hive/streaming/AbstractRecordWriter.java 
25998ae 
  streaming/src/java/org/apache/hive/streaming/ConnectionError.java 668bffb 
  streaming/src/java/org/apache/hive/streaming/ConnectionInfo.java PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/DelimitedInputWriter.java 
898b3f9 
  streaming/src/java/org/apache/hive/streaming/HeartBeatFailure.java b1f9520 
  streaming/src/java/org/apache/hive/streaming/HiveEndPoint.java b04e137 
  streaming/src/java/org/apache/hive/streaming/HiveStreamingConnection.java 
PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/ImpersonationFailed.java 23e17e7 
  streaming/src/java/org/apache/hive/streaming/InvalidColumn.java 0011b14 
  streaming/src/java/org/apache/hive/streaming/InvalidPartition.java f1f9804 
  streaming/src/java/org/apache/hive/streaming/InvalidTable.java ef1c91d 
  streaming/src/java/org/apache/hive/streaming/InvalidTransactionState.java 
PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/InvalidTrasactionState.java 
762f5f8 
  streaming/src/java/org/apache/hive/streaming/PartitionCreationFailed.java 
5f9aca6 
  streaming/src/java/org/apache/hive/streaming/PartitionHandler.java 
PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/QueryFailedException.java 
ccd3ae0 
  streaming/src/java/org/apache/hive/streaming/RecordWriter.java dc6d70e 
  streaming/src/java/org/apache/hive/streaming/SerializationError.java a57ba00 
  streaming/src/java/org/apache/hive/streaming/StreamingConnection.java 2f760ea 
  streaming/src/java/org/apache/hive/streaming/StreamingException.java a7f84c1 
  streaming/src/java/org/apache/hive/streaming/StreamingIOFailure.java 0dfbfa7 
  streaming/src/java/org/apache/hive/streaming/StrictDelimitedInputWriter.java 
PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/StrictJsonWriter.java 0077913 
  streaming/src/java/org/apache/hive/streaming/StrictRegexWriter.java c0b7324 
  streaming/src/java/org/apache/hive/streaming/TransactionBatch.java 2b05771 
  streaming/src/java/org/apache/hive/streaming/TransactionBatchUnAvailable.java 
a8c8cd4 
  streaming/src/java/org/apache/hive/streaming/TransactionError.java a331b20 
  streaming/src/test/org/apache/hive/streaming/TestDelimitedInputWriter.java 
f0843a1 
  streaming/src/test/org/apache/hive/streaming/TestStreaming.java 6f63bfb 
  
streaming/src/test/org/apache/hive/streaming/TestStreamingDynamicPartitioning.java
 PRE-CREATION 


Diff: https://reviews.apache.org/r/66645/diff/4/

Changes: https://reviews.apache.org/r/66645/diff/3-4/


Testing
---


Thanks,

Prasanth_J

Re: Review Request 66645: HIVE-19211: New streaming ingest API and support for dynamic partitioning

2018-04-18 Thread j . prasanth . j


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66645/#review201484
---




streaming/src/java/org/apache/hive/streaming/AbstractRecordWriter.java
Lines 384 (patched)


This is perf issue. Will cache the partitionPath in the next patch.


- Prasanth_J


On April 18, 2018, 9:42 p.m., Prasanth_J wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66645/
> ---
> 
> (Updated April 18, 2018, 9:42 p.m.)
> 
> 
> Review request for hive, Ashutosh Chauhan and Eugene Koifman.
> 
> 
> Bugs: HIVE-19211
> https://issues.apache.org/jira/browse/HIVE-19211
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-19211: New streaming ingest API and support for dynamic partitioning
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 73492ff 
>   
> itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/TestCompactor.java
>  82ba775 
>   metastore/src/java/org/apache/hadoop/hive/metastore/HiveClientCache.java 
> PRE-CREATION 
>   metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStoreUtils.java 
> a66c135 
>   serde/src/java/org/apache/hadoop/hive/serde2/JsonSerDe.java PRE-CREATION 
>   streaming/pom.xml b58ec01 
>   streaming/src/java/org/apache/hive/streaming/AbstractRecordWriter.java 
> 25998ae 
>   streaming/src/java/org/apache/hive/streaming/ConnectionError.java 668bffb 
>   streaming/src/java/org/apache/hive/streaming/ConnectionInfo.java 
> PRE-CREATION 
>   streaming/src/java/org/apache/hive/streaming/DelimitedInputWriter.java 
> 898b3f9 
>   streaming/src/java/org/apache/hive/streaming/HeartBeatFailure.java b1f9520 
>   streaming/src/java/org/apache/hive/streaming/HiveEndPoint.java b04e137 
>   streaming/src/java/org/apache/hive/streaming/HiveStreamingConnection.java 
> PRE-CREATION 
>   streaming/src/java/org/apache/hive/streaming/ImpersonationFailed.java 
> 23e17e7 
>   streaming/src/java/org/apache/hive/streaming/InvalidColumn.java 0011b14 
>   streaming/src/java/org/apache/hive/streaming/InvalidPartition.java f1f9804 
>   streaming/src/java/org/apache/hive/streaming/InvalidTable.java ef1c91d 
>   streaming/src/java/org/apache/hive/streaming/InvalidTransactionState.java 
> PRE-CREATION 
>   streaming/src/java/org/apache/hive/streaming/InvalidTrasactionState.java 
> 762f5f8 
>   streaming/src/java/org/apache/hive/streaming/PartitionCreationFailed.java 
> 5f9aca6 
>   streaming/src/java/org/apache/hive/streaming/PartitionHandler.java 
> PRE-CREATION 
>   streaming/src/java/org/apache/hive/streaming/QueryFailedException.java 
> ccd3ae0 
>   streaming/src/java/org/apache/hive/streaming/RecordWriter.java dc6d70e 
>   streaming/src/java/org/apache/hive/streaming/SerializationError.java 
> a57ba00 
>   streaming/src/java/org/apache/hive/streaming/StreamingConnection.java 
> 2f760ea 
>   streaming/src/java/org/apache/hive/streaming/StreamingException.java 
> a7f84c1 
>   streaming/src/java/org/apache/hive/streaming/StreamingIOFailure.java 
> 0dfbfa7 
>   
> streaming/src/java/org/apache/hive/streaming/StrictDelimitedInputWriter.java 
> PRE-CREATION 
>   streaming/src/java/org/apache/hive/streaming/StrictJsonWriter.java 0077913 
>   streaming/src/java/org/apache/hive/streaming/StrictRegexWriter.java c0b7324 
>   streaming/src/java/org/apache/hive/streaming/TransactionBatch.java 2b05771 
>   
> streaming/src/java/org/apache/hive/streaming/TransactionBatchUnAvailable.java 
> a8c8cd4 
>   streaming/src/java/org/apache/hive/streaming/TransactionError.java a331b20 
>   streaming/src/test/org/apache/hive/streaming/TestDelimitedInputWriter.java 
> f0843a1 
>   streaming/src/test/org/apache/hive/streaming/TestStreaming.java 6f63bfb 
>   
> streaming/src/test/org/apache/hive/streaming/TestStreamingDynamicPartitioning.java
>  PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/66645/diff/3/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Prasanth_J
> 
>

Re: Review Request 66645: HIVE-19211: New streaming ingest API and support for dynamic partitioning

2018-04-18 Thread j . prasanth . j


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66645/
---

(Updated April 18, 2018, 9:42 p.m.)


Review request for hive, Ashutosh Chauhan and Eugene Koifman.


Changes
---

Added automatic heartbeating of open transactions.


Bugs: HIVE-19211
https://issues.apache.org/jira/browse/HIVE-19211


Repository: hive-git


Description
---

HIVE-19211: New streaming ingest API and support for dynamic partitioning


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 73492ff 
  
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/TestCompactor.java
 82ba775 
  metastore/src/java/org/apache/hadoop/hive/metastore/HiveClientCache.java 
PRE-CREATION 
  metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStoreUtils.java 
a66c135 
  serde/src/java/org/apache/hadoop/hive/serde2/JsonSerDe.java PRE-CREATION 
  streaming/pom.xml b58ec01 
  streaming/src/java/org/apache/hive/streaming/AbstractRecordWriter.java 
25998ae 
  streaming/src/java/org/apache/hive/streaming/ConnectionError.java 668bffb 
  streaming/src/java/org/apache/hive/streaming/ConnectionInfo.java PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/DelimitedInputWriter.java 
898b3f9 
  streaming/src/java/org/apache/hive/streaming/HeartBeatFailure.java b1f9520 
  streaming/src/java/org/apache/hive/streaming/HiveEndPoint.java b04e137 
  streaming/src/java/org/apache/hive/streaming/HiveStreamingConnection.java 
PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/ImpersonationFailed.java 23e17e7 
  streaming/src/java/org/apache/hive/streaming/InvalidColumn.java 0011b14 
  streaming/src/java/org/apache/hive/streaming/InvalidPartition.java f1f9804 
  streaming/src/java/org/apache/hive/streaming/InvalidTable.java ef1c91d 
  streaming/src/java/org/apache/hive/streaming/InvalidTransactionState.java 
PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/InvalidTrasactionState.java 
762f5f8 
  streaming/src/java/org/apache/hive/streaming/PartitionCreationFailed.java 
5f9aca6 
  streaming/src/java/org/apache/hive/streaming/PartitionHandler.java 
PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/QueryFailedException.java 
ccd3ae0 
  streaming/src/java/org/apache/hive/streaming/RecordWriter.java dc6d70e 
  streaming/src/java/org/apache/hive/streaming/SerializationError.java a57ba00 
  streaming/src/java/org/apache/hive/streaming/StreamingConnection.java 2f760ea 
  streaming/src/java/org/apache/hive/streaming/StreamingException.java a7f84c1 
  streaming/src/java/org/apache/hive/streaming/StreamingIOFailure.java 0dfbfa7 
  streaming/src/java/org/apache/hive/streaming/StrictDelimitedInputWriter.java 
PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/StrictJsonWriter.java 0077913 
  streaming/src/java/org/apache/hive/streaming/StrictRegexWriter.java c0b7324 
  streaming/src/java/org/apache/hive/streaming/TransactionBatch.java 2b05771 
  streaming/src/java/org/apache/hive/streaming/TransactionBatchUnAvailable.java 
a8c8cd4 
  streaming/src/java/org/apache/hive/streaming/TransactionError.java a331b20 
  streaming/src/test/org/apache/hive/streaming/TestDelimitedInputWriter.java 
f0843a1 
  streaming/src/test/org/apache/hive/streaming/TestStreaming.java 6f63bfb 
  
streaming/src/test/org/apache/hive/streaming/TestStreamingDynamicPartitioning.java
 PRE-CREATION 


Diff: https://reviews.apache.org/r/66645/diff/3/

Changes: https://reviews.apache.org/r/66645/diff/2-3/


Testing
---


Thanks,

Prasanth_J

Re: Review Request 66645: HIVE-19211: New streaming ingest API and support for dynamic partitioning

2018-04-16 Thread j . prasanth . j


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66645/
---

(Updated April 16, 2018, 10:45 p.m.)


Review request for hive, Ashutosh Chauhan and Eugene Koifman.


Changes
---

Removed HIVE-19214 changes.


Bugs: HIVE-19211
https://issues.apache.org/jira/browse/HIVE-19211


Repository: hive-git


Description
---

HIVE-19211: New streaming ingest API and support for dynamic partitioning


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java e533ee6 
  metastore/src/java/org/apache/hadoop/hive/metastore/HiveClientCache.java 
PRE-CREATION 
  metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStoreUtils.java 
a66c135 
  serde/src/java/org/apache/hadoop/hive/serde2/JsonSerDe.java PRE-CREATION 
  streaming/pom.xml b58ec01 
  streaming/src/java/org/apache/hive/streaming/AbstractRecordWriter.java 
25998ae 
  streaming/src/java/org/apache/hive/streaming/ConnectionError.java 668bffb 
  streaming/src/java/org/apache/hive/streaming/ConnectionInfo.java PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/DelimitedInputWriter.java 
898b3f9 
  streaming/src/java/org/apache/hive/streaming/HeartBeatFailure.java b1f9520 
  streaming/src/java/org/apache/hive/streaming/HiveEndPoint.java b04e137 
  streaming/src/java/org/apache/hive/streaming/HiveStreamingConnection.java 
PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/ImpersonationFailed.java 23e17e7 
  streaming/src/java/org/apache/hive/streaming/InvalidColumn.java 0011b14 
  streaming/src/java/org/apache/hive/streaming/InvalidPartition.java f1f9804 
  streaming/src/java/org/apache/hive/streaming/InvalidTable.java ef1c91d 
  streaming/src/java/org/apache/hive/streaming/InvalidTransactionState.java 
PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/InvalidTrasactionState.java 
762f5f8 
  streaming/src/java/org/apache/hive/streaming/PartitionCreationFailed.java 
5f9aca6 
  streaming/src/java/org/apache/hive/streaming/PartitionHandler.java 
PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/QueryFailedException.java 
ccd3ae0 
  streaming/src/java/org/apache/hive/streaming/RecordWriter.java dc6d70e 
  streaming/src/java/org/apache/hive/streaming/SerializationError.java a57ba00 
  streaming/src/java/org/apache/hive/streaming/StreamingConnection.java 2f760ea 
  streaming/src/java/org/apache/hive/streaming/StreamingException.java a7f84c1 
  streaming/src/java/org/apache/hive/streaming/StreamingIOFailure.java 0dfbfa7 
  streaming/src/java/org/apache/hive/streaming/StrictDelimitedInputWriter.java 
PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/StrictJsonWriter.java 0077913 
  streaming/src/java/org/apache/hive/streaming/StrictRegexWriter.java c0b7324 
  streaming/src/java/org/apache/hive/streaming/TransactionBatch.java 2b05771 
  streaming/src/java/org/apache/hive/streaming/TransactionBatchUnAvailable.java 
a8c8cd4 
  streaming/src/java/org/apache/hive/streaming/TransactionError.java a331b20 
  streaming/src/test/org/apache/hive/streaming/TestDelimitedInputWriter.java 
f0843a1 
  streaming/src/test/org/apache/hive/streaming/TestStreaming.java 6f63bfb 
  
streaming/src/test/org/apache/hive/streaming/TestStreamingDynamicPartitioning.java
 PRE-CREATION 


Diff: https://reviews.apache.org/r/66645/diff/2/

Changes: https://reviews.apache.org/r/66645/diff/1-2/


Testing
---


Thanks,

Prasanth_J

Re: Review Request 66645: HIVE-19211: New streaming ingest API and support for dynamic partitioning

2018-04-16 Thread j . prasanth . j


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66645/#review201274
---




ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcRecordUpdater.java
Lines 297 (patched)


This is covered HIVE-19214. Will be removed after HIVE-19214 is committed.


- Prasanth_J


On April 16, 2018, 10:42 p.m., Prasanth_J wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66645/
> ---
> 
> (Updated April 16, 2018, 10:42 p.m.)
> 
> 
> Review request for hive, Ashutosh Chauhan and Eugene Koifman.
> 
> 
> Bugs: HIVE-19211
> https://issues.apache.org/jira/browse/HIVE-19211
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-19211: New streaming ingest API and support for dynamic partitioning
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 
> e533ee627fd38e8ddcdcedf00f0c2d3ae150c530 
>   metastore/src/java/org/apache/hadoop/hive/metastore/HiveClientCache.java 
> PRE-CREATION 
>   metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStoreUtils.java 
> a66c13507abef42977dfdb315ff7d69404f67ac3 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcRecordUpdater.java 
> d850062377d182e33a6191268d50d0008d7c77de 
>   serde/src/java/org/apache/hadoop/hive/serde2/JsonSerDe.java PRE-CREATION 
>   streaming/pom.xml b58ec015aa69e29aacdc0a165ead9439ea2e4b26 
>   streaming/src/java/org/apache/hive/streaming/AbstractRecordWriter.java 
> 25998ae31a3a829aab45f9e526aa03d94feff5e0 
>   streaming/src/java/org/apache/hive/streaming/ConnectionError.java 
> 668bffb1ab17558dec33d599bddd6e28a06b3c5a 
>   streaming/src/java/org/apache/hive/streaming/ConnectionInfo.java 
> PRE-CREATION 
>   streaming/src/java/org/apache/hive/streaming/DelimitedInputWriter.java 
> 898b3f9bb1d1c483cae8c1dd4f2338fc453d514b 
>   streaming/src/java/org/apache/hive/streaming/HeartBeatFailure.java 
> b1f9520814d260a3d2df23e6050e72d803874da9 
>   streaming/src/java/org/apache/hive/streaming/HiveEndPoint.java 
> b04e13784485ca097153bbec86f80d22e15e5cdc 
>   streaming/src/java/org/apache/hive/streaming/HiveStreamingConnection.java 
> PRE-CREATION 
>   streaming/src/java/org/apache/hive/streaming/ImpersonationFailed.java 
> 23e17e76237036d8f9419bef2255f4f82c5b18a1 
>   streaming/src/java/org/apache/hive/streaming/InvalidColumn.java 
> 0011b1454f8815816be931bf67cc13e7e78c9c0d 
>   streaming/src/java/org/apache/hive/streaming/InvalidPartition.java 
> f1f980430f3aceeb044bb549cc1a37a33c144750 
>   streaming/src/java/org/apache/hive/streaming/InvalidTable.java 
> ef1c91dbeb84b325b019318122fdd1f45b927414 
>   streaming/src/java/org/apache/hive/streaming/InvalidTrasactionState.java 
> 762f5f86fc0df4a59cb54812a5dc79c1e2bc9489 
>   streaming/src/java/org/apache/hive/streaming/PartitionCreationFailed.java 
> 5f9aca66ea0f2a7b2c3d2f6fb805fa1760b69e44 
>   streaming/src/java/org/apache/hive/streaming/QueryFailedException.java 
> ccd3ae0c98ea6ced0290f1ab027ad6337453fca2 
>   streaming/src/java/org/apache/hive/streaming/RecordWriter.java 
> dc6d70e92438e037d764099c82f5f654d5f5d801 
>   streaming/src/java/org/apache/hive/streaming/SerializationError.java 
> a57ba00ba401283aedd3f685171ef6bd810b11cd 
>   streaming/src/java/org/apache/hive/streaming/StreamingConnection.java 
> 2f760ea86eecbbc96db08509405a369abf7d89d5 
>   streaming/src/java/org/apache/hive/streaming/StreamingException.java 
> a7f84c14f30f2e4753bd99b3d2d1dcb236b0197b 
>   streaming/src/java/org/apache/hive/streaming/StreamingIOFailure.java 
> 0dfbfa71c50215d8f3e25298c8d11634a3cbedc4 
>   
> streaming/src/java/org/apache/hive/streaming/StrictDelimitedInputWriter.java 
> PRE-CREATION 
>   streaming/src/java/org/apache/hive/streaming/StrictJsonWriter.java 
> 0077913cd1f0afbafe4608c4378398f61e254424 
>   streaming/src/java/org/apache/hive/streaming/StrictRegexWriter.java 
> c0b732482d35305ceaba1adfff09659e193ab098 
>   streaming/src/java/org/apache/hive/streaming/TransactionBatch.java 
> 2b057718f58dec6de3e2b329a43bb5a06ce7c9ed 
>   
> streaming/src/java/org/apache/hive/streaming/TransactionBatchUnAvailable.java 
> a8c8cd48726421003df186fa1e0c2ecd18bdd5b4 
>   streaming/src/java/org/apache/hive/streaming/TransactionError.java 
> a331b20463e8328148fb08d85cf3ce77a7463062 
>   streaming/src/test/org/apache/hive/streaming/TestDelimitedInputWriter.java 
> f0843a1748d956ea99dd4807cf0b4ffbe0ef9cba 
>   streaming/src/test/org/apache/hive/streaming/TestStreaming.java 
> 6f63bfb43e5dbe4c9529dfc80787a95ba6524c01 
>   
> streaming/src/test/org/apache/hive/streaming/TestStreamingDynamicPartitioning.java
>  PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/66645/diff/1/
> 
> 
> Testing
> ---
> 
> 
> Thanks,

Review Request 66645: HIVE-19211: New streaming ingest API and support for dynamic partitioning

2018-04-16 Thread j . prasanth . j


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66645/
---

Review request for hive, Ashutosh Chauhan and Eugene Koifman.


Bugs: HIVE-19211
https://issues.apache.org/jira/browse/HIVE-19211


Repository: hive-git


Description
---

HIVE-19211: New streaming ingest API and support for dynamic partitioning


Diffs
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 
e533ee627fd38e8ddcdcedf00f0c2d3ae150c530 
  metastore/src/java/org/apache/hadoop/hive/metastore/HiveClientCache.java 
PRE-CREATION 
  metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStoreUtils.java 
a66c13507abef42977dfdb315ff7d69404f67ac3 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcRecordUpdater.java 
d850062377d182e33a6191268d50d0008d7c77de 
  serde/src/java/org/apache/hadoop/hive/serde2/JsonSerDe.java PRE-CREATION 
  streaming/pom.xml b58ec015aa69e29aacdc0a165ead9439ea2e4b26 
  streaming/src/java/org/apache/hive/streaming/AbstractRecordWriter.java 
25998ae31a3a829aab45f9e526aa03d94feff5e0 
  streaming/src/java/org/apache/hive/streaming/ConnectionError.java 
668bffb1ab17558dec33d599bddd6e28a06b3c5a 
  streaming/src/java/org/apache/hive/streaming/ConnectionInfo.java PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/DelimitedInputWriter.java 
898b3f9bb1d1c483cae8c1dd4f2338fc453d514b 
  streaming/src/java/org/apache/hive/streaming/HeartBeatFailure.java 
b1f9520814d260a3d2df23e6050e72d803874da9 
  streaming/src/java/org/apache/hive/streaming/HiveEndPoint.java 
b04e13784485ca097153bbec86f80d22e15e5cdc 
  streaming/src/java/org/apache/hive/streaming/HiveStreamingConnection.java 
PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/ImpersonationFailed.java 
23e17e76237036d8f9419bef2255f4f82c5b18a1 
  streaming/src/java/org/apache/hive/streaming/InvalidColumn.java 
0011b1454f8815816be931bf67cc13e7e78c9c0d 
  streaming/src/java/org/apache/hive/streaming/InvalidPartition.java 
f1f980430f3aceeb044bb549cc1a37a33c144750 
  streaming/src/java/org/apache/hive/streaming/InvalidTable.java 
ef1c91dbeb84b325b019318122fdd1f45b927414 
  streaming/src/java/org/apache/hive/streaming/InvalidTrasactionState.java 
762f5f86fc0df4a59cb54812a5dc79c1e2bc9489 
  streaming/src/java/org/apache/hive/streaming/PartitionCreationFailed.java 
5f9aca66ea0f2a7b2c3d2f6fb805fa1760b69e44 
  streaming/src/java/org/apache/hive/streaming/QueryFailedException.java 
ccd3ae0c98ea6ced0290f1ab027ad6337453fca2 
  streaming/src/java/org/apache/hive/streaming/RecordWriter.java 
dc6d70e92438e037d764099c82f5f654d5f5d801 
  streaming/src/java/org/apache/hive/streaming/SerializationError.java 
a57ba00ba401283aedd3f685171ef6bd810b11cd 
  streaming/src/java/org/apache/hive/streaming/StreamingConnection.java 
2f760ea86eecbbc96db08509405a369abf7d89d5 
  streaming/src/java/org/apache/hive/streaming/StreamingException.java 
a7f84c14f30f2e4753bd99b3d2d1dcb236b0197b 
  streaming/src/java/org/apache/hive/streaming/StreamingIOFailure.java 
0dfbfa71c50215d8f3e25298c8d11634a3cbedc4 
  streaming/src/java/org/apache/hive/streaming/StrictDelimitedInputWriter.java 
PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/StrictJsonWriter.java 
0077913cd1f0afbafe4608c4378398f61e254424 
  streaming/src/java/org/apache/hive/streaming/StrictRegexWriter.java 
c0b732482d35305ceaba1adfff09659e193ab098 
  streaming/src/java/org/apache/hive/streaming/TransactionBatch.java 
2b057718f58dec6de3e2b329a43bb5a06ce7c9ed 
  streaming/src/java/org/apache/hive/streaming/TransactionBatchUnAvailable.java 
a8c8cd48726421003df186fa1e0c2ecd18bdd5b4 
  streaming/src/java/org/apache/hive/streaming/TransactionError.java 
a331b20463e8328148fb08d85cf3ce77a7463062 
  streaming/src/test/org/apache/hive/streaming/TestDelimitedInputWriter.java 
f0843a1748d956ea99dd4807cf0b4ffbe0ef9cba 
  streaming/src/test/org/apache/hive/streaming/TestStreaming.java 
6f63bfb43e5dbe4c9529dfc80787a95ba6524c01 
  
streaming/src/test/org/apache/hive/streaming/TestStreamingDynamicPartitioning.java
 PRE-CREATION 


Diff: https://reviews.apache.org/r/66645/diff/1/


Testing
---


Thanks,

Prasanth_J

Re: Review Request 66614: HIVE-19210: Create separate module for streaming ingest

2018-04-13 Thread j . prasanth . j


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66614/
---

(Updated April 13, 2018, 11:58 p.m.)


Review request for hive, Ashutosh Chauhan and EugeneBK EugeneBK.


Changes
---

Not moving mutate package from hcat.


Bugs: HIVE-19210
https://issues.apache.org/jira/browse/HIVE-19210


Repository: hive-git


Description
---

HIVE-19210: Create separate module for streaming ingest


Diffs (updated)
-

  
hcatalog/streaming/src/java/org/apache/hive/hcatalog/streaming/package-info.java
 PRE-CREATION 
  pom.xml 5802bd3 
  streaming/pom.xml PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/AbstractRecordWriter.java 
PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/ConnectionError.java 
PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/DelimitedInputWriter.java 
PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/HeartBeatFailure.java 
PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/HiveEndPoint.java PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/ImpersonationFailed.java 
PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/InvalidColumn.java PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/InvalidPartition.java 
PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/InvalidTable.java PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/InvalidTrasactionState.java 
PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/PartitionCreationFailed.java 
PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/QueryFailedException.java 
PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/RecordWriter.java PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/SerializationError.java 
PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/StreamingConnection.java 
PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/StreamingException.java 
PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/StreamingIOFailure.java 
PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/StrictJsonWriter.java 
PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/StrictRegexWriter.java 
PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/TransactionBatch.java 
PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/TransactionBatchUnAvailable.java 
PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/TransactionError.java 
PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/package.html PRE-CREATION 
  streaming/src/test/org/apache/hive/streaming/TestDelimitedInputWriter.java 
PRE-CREATION 
  streaming/src/test/org/apache/hive/streaming/TestStreaming.java PRE-CREATION 


Diff: https://reviews.apache.org/r/66614/diff/2/

Changes: https://reviews.apache.org/r/66614/diff/1-2/


Testing
---


Thanks,

Prasanth_J

Re: Review Request 66614: HIVE-19210: Create separate module for streaming ingest

2018-04-13 Thread j . prasanth . j


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66614/
---

(Updated April 13, 2018, 10:49 p.m.)


Review request for hive, Ashutosh Chauhan and EugeneBK EugeneBK.


Bugs: HIVE-19210
https://issues.apache.org/jira/browse/HIVE-19210


Repository: hive-git


Description
---

HIVE-19210: Create separate module for streaming ingest


Diffs
-

  
hcatalog/streaming/src/java/org/apache/hive/hcatalog/streaming/package-info.java
 PRE-CREATION 
  pom.xml 5802bd3a354ce6a78fd786e43847c7475a3f965e 
  streaming/pom.xml PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/AbstractRecordWriter.java 
PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/ConnectionError.java 
PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/DelimitedInputWriter.java 
PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/HeartBeatFailure.java 
PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/HiveEndPoint.java PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/ImpersonationFailed.java 
PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/InvalidColumn.java PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/InvalidPartition.java 
PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/InvalidTable.java PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/InvalidTrasactionState.java 
PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/PartitionCreationFailed.java 
PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/QueryFailedException.java 
PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/RecordWriter.java PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/SerializationError.java 
PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/StreamingConnection.java 
PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/StreamingException.java 
PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/StreamingIOFailure.java 
PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/StreamingUtil.java PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/StrictJsonWriter.java 
PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/StrictRegexWriter.java 
PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/TransactionBatch.java 
PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/TransactionBatchUnAvailable.java 
PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/TransactionError.java 
PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/mutate/HiveConfFactory.java 
PRE-CREATION 
  
streaming/src/java/org/apache/hive/streaming/mutate/UgiMetaStoreClientFactory.java
 PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/mutate/client/AcidTable.java 
PRE-CREATION 
  
streaming/src/java/org/apache/hive/streaming/mutate/client/AcidTableSerializer.java
 PRE-CREATION 
  
streaming/src/java/org/apache/hive/streaming/mutate/client/ClientException.java 
PRE-CREATION 
  
streaming/src/java/org/apache/hive/streaming/mutate/client/ConnectionException.java
 PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/mutate/client/MutatorClient.java 
PRE-CREATION 
  
streaming/src/java/org/apache/hive/streaming/mutate/client/MutatorClientBuilder.java
 PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/mutate/client/TableType.java 
PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/mutate/client/Transaction.java 
PRE-CREATION 
  
streaming/src/java/org/apache/hive/streaming/mutate/client/TransactionException.java
 PRE-CREATION 
  
streaming/src/java/org/apache/hive/streaming/mutate/client/lock/HeartbeatFactory.java
 PRE-CREATION 
  
streaming/src/java/org/apache/hive/streaming/mutate/client/lock/HeartbeatTimerTask.java
 PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/mutate/client/lock/Lock.java 
PRE-CREATION 
  
streaming/src/java/org/apache/hive/streaming/mutate/client/lock/LockException.java
 PRE-CREATION 
  
streaming/src/java/org/apache/hive/streaming/mutate/client/lock/LockFailureListener.java
 PRE-CREATION 
  
streaming/src/java/org/apache/hive/streaming/mutate/doc-files/system-overview.dot
 PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/mutate/package.html PRE-CREATION 
  
streaming/src/java/org/apache/hive/streaming/mutate/worker/BucketIdException.java
 PRE-CREATION 
  
streaming/src/java/org/apache/hive/streaming/mutate/worker/BucketIdResolver.java
 PRE-CREATION 
  
streaming/src/java/org/apache/hive/streaming/mutate/worker/BucketIdResolverImpl.java
 PRE-CREATION 
  
streaming/src/java/org/apache/hive/streaming/mutate/worker/GroupRevisitedException.java
 PRE-CREATION 
  
streaming/src/java/org/apache/hive/streaming/mutate/worker/GroupingValidator.java
 PRE-CREATION

Review Request 66614: HIVE-19210: Create separate module for streaming ingest

2018-04-13 Thread j . prasanth . j


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66614/
---

Review request for hive, Ashutosh Chauhan and EugeneBK EugeneBK.


Repository: hive-git


Description
---

HIVE-19210: Create separate module for streaming ingest


Diffs
-

  
hcatalog/streaming/src/java/org/apache/hive/hcatalog/streaming/package-info.java
 PRE-CREATION 
  pom.xml 5802bd3a354ce6a78fd786e43847c7475a3f965e 
  streaming/pom.xml PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/AbstractRecordWriter.java 
PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/ConnectionError.java 
PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/DelimitedInputWriter.java 
PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/HeartBeatFailure.java 
PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/HiveEndPoint.java PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/ImpersonationFailed.java 
PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/InvalidColumn.java PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/InvalidPartition.java 
PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/InvalidTable.java PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/InvalidTrasactionState.java 
PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/PartitionCreationFailed.java 
PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/QueryFailedException.java 
PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/RecordWriter.java PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/SerializationError.java 
PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/StreamingConnection.java 
PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/StreamingException.java 
PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/StreamingIOFailure.java 
PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/StreamingUtil.java PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/StrictJsonWriter.java 
PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/StrictRegexWriter.java 
PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/TransactionBatch.java 
PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/TransactionBatchUnAvailable.java 
PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/TransactionError.java 
PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/mutate/HiveConfFactory.java 
PRE-CREATION 
  
streaming/src/java/org/apache/hive/streaming/mutate/UgiMetaStoreClientFactory.java
 PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/mutate/client/AcidTable.java 
PRE-CREATION 
  
streaming/src/java/org/apache/hive/streaming/mutate/client/AcidTableSerializer.java
 PRE-CREATION 
  
streaming/src/java/org/apache/hive/streaming/mutate/client/ClientException.java 
PRE-CREATION 
  
streaming/src/java/org/apache/hive/streaming/mutate/client/ConnectionException.java
 PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/mutate/client/MutatorClient.java 
PRE-CREATION 
  
streaming/src/java/org/apache/hive/streaming/mutate/client/MutatorClientBuilder.java
 PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/mutate/client/TableType.java 
PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/mutate/client/Transaction.java 
PRE-CREATION 
  
streaming/src/java/org/apache/hive/streaming/mutate/client/TransactionException.java
 PRE-CREATION 
  
streaming/src/java/org/apache/hive/streaming/mutate/client/lock/HeartbeatFactory.java
 PRE-CREATION 
  
streaming/src/java/org/apache/hive/streaming/mutate/client/lock/HeartbeatTimerTask.java
 PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/mutate/client/lock/Lock.java 
PRE-CREATION 
  
streaming/src/java/org/apache/hive/streaming/mutate/client/lock/LockException.java
 PRE-CREATION 
  
streaming/src/java/org/apache/hive/streaming/mutate/client/lock/LockFailureListener.java
 PRE-CREATION 
  
streaming/src/java/org/apache/hive/streaming/mutate/doc-files/system-overview.dot
 PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/mutate/package.html PRE-CREATION 
  
streaming/src/java/org/apache/hive/streaming/mutate/worker/BucketIdException.java
 PRE-CREATION 
  
streaming/src/java/org/apache/hive/streaming/mutate/worker/BucketIdResolver.java
 PRE-CREATION 
  
streaming/src/java/org/apache/hive/streaming/mutate/worker/BucketIdResolverImpl.java
 PRE-CREATION 
  
streaming/src/java/org/apache/hive/streaming/mutate/worker/GroupRevisitedException.java
 PRE-CREATION 
  
streaming/src/java/org/apache/hive/streaming/mutate/worker/GroupingValidator.java
 PRE-CREATION 
  
streaming/src/java/org/apache/hive/streaming/mutate/worker/MetaStorePartitionHelper.java
 PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/mutate/worker/Mutator.java

Re: Review Request 66236: HIVE-19033: Provide an option to purge LLAP IO cache

2018-04-02 Thread j . prasanth . j


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66236/
---

(Updated April 3, 2018, 1:25 a.m.)


Review request for hive, Gopal V and Sergey Shelukhin.


Changes
---

Fixes test failure.


Bugs: HIVE-19033
https://issues.apache.org/jira/browse/HIVE-19033


Repository: hive-git


Description
---

HIVE-19033: Provide an option to purge LLAP IO cache


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 02367eb 
  itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestActivePassiveHA.java 
d2d0bee 
  
itests/hive-unit/src/test/java/org/apache/hive/jdbc/authorization/TestJdbcWithSQLAuthorization.java
 9e021ea 
  llap-client/src/java/org/apache/hadoop/hive/llap/io/api/LlapIo.java 6e6f5b9 
  
llap-common/src/gen/protobuf/gen-java/org/apache/hadoop/hive/llap/daemon/rpc/LlapDaemonProtocolProtos.java
 4753812 
  
llap-common/src/java/org/apache/hadoop/hive/llap/impl/LlapManagementProtocolClientImpl.java
 af760b1 
  
llap-common/src/java/org/apache/hadoop/hive/llap/protocol/LlapManagementProtocolPB.java
 ff215d4 
  llap-common/src/protobuf/LlapDaemonProtocol.proto 12beca5 
  
llap-server/src/java/org/apache/hadoop/hive/llap/cache/CacheContentsTracker.java
 4fbaac1 
  
llap-server/src/java/org/apache/hadoop/hive/llap/cache/LowLevelCachePolicy.java 
2cd70b9 
  
llap-server/src/java/org/apache/hadoop/hive/llap/cache/LowLevelFifoCachePolicy.java
 50a2411 
  
llap-server/src/java/org/apache/hadoop/hive/llap/cache/LowLevelLrfuCachePolicy.java
 b42f761 
  
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapProtocolServerImpl.java
 81785f0 
  
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/services/impl/LlapIoMemoryServlet.java
 153ab35 
  llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/LlapIoImpl.java 
e5bc3c2 
  
llap-server/src/test/org/apache/hadoop/hive/llap/cache/TestLowLevelCacheImpl.java
 2c87bc2 
  
llap-server/src/test/org/apache/hadoop/hive/llap/cache/TestOrcMetadataCache.java
 f7ebff2 
  ql/src/java/org/apache/hadoop/hive/ql/processors/CommandProcessorFactory.java 
74a34b3 
  ql/src/java/org/apache/hadoop/hive/ql/processors/CommandUtil.java 0dcef08 
  ql/src/java/org/apache/hadoop/hive/ql/processors/HiveCommand.java c45563d 
  
ql/src/java/org/apache/hadoop/hive/ql/processors/LlapCacheResourceProcessor.java
 PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/processors/LlapClusterResourceProcessor.java
 PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/security/authorization/plugin/HiveOperationType.java
 5d6905a 
  
ql/src/java/org/apache/hadoop/hive/ql/security/authorization/plugin/sqlstd/Operation2Privilege.java
 a55e66b 
  
ql/src/java/org/apache/hadoop/hive/ql/security/authorization/plugin/sqlstd/SQLStdHiveAuthorizationValidator.java
 4e456e7 
  
ql/src/test/org/apache/hadoop/hive/ql/processors/TestCommandProcessorFactory.java
 748d639 
  
service/src/java/org/apache/hive/service/cli/operation/HiveCommandOperation.java
 e3a5922 


Diff: https://reviews.apache.org/r/66236/diff/5/

Changes: https://reviews.apache.org/r/66236/diff/4-5/


Testing
---


Thanks,

Prasanth_J

Re: Review Request 66236: HIVE-19033: Provide an option to purge LLAP IO cache

2018-03-28 Thread j . prasanth . j


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66236/
---

(Updated March 28, 2018, 8:46 p.m.)


Review request for hive, Gopal V and Sergey Shelukhin.


Changes
---

Changes for Ranger. Addressed review comments.


Bugs: HIVE-19033
https://issues.apache.org/jira/browse/HIVE-19033


Repository: hive-git


Description
---

HIVE-19033: Provide an option to purge LLAP IO cache


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java a6866e7 
  
itests/hive-unit/src/test/java/org/apache/hive/jdbc/authorization/TestJdbcWithSQLAuthorization.java
 9e021ea 
  llap-client/src/java/org/apache/hadoop/hive/llap/io/api/LlapIo.java 6e6f5b9 
  
llap-common/src/gen/protobuf/gen-java/org/apache/hadoop/hive/llap/daemon/rpc/LlapDaemonProtocolProtos.java
 4753812 
  
llap-common/src/java/org/apache/hadoop/hive/llap/impl/LlapManagementProtocolClientImpl.java
 af760b1 
  
llap-common/src/java/org/apache/hadoop/hive/llap/protocol/LlapManagementProtocolPB.java
 ff215d4 
  llap-common/src/protobuf/LlapDaemonProtocol.proto 12beca5 
  
llap-server/src/java/org/apache/hadoop/hive/llap/cache/CacheContentsTracker.java
 4fbaac1 
  
llap-server/src/java/org/apache/hadoop/hive/llap/cache/LowLevelCachePolicy.java 
2cd70b9 
  
llap-server/src/java/org/apache/hadoop/hive/llap/cache/LowLevelFifoCachePolicy.java
 50a2411 
  
llap-server/src/java/org/apache/hadoop/hive/llap/cache/LowLevelLrfuCachePolicy.java
 b42f761 
  
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapProtocolServerImpl.java
 81785f0 
  
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/services/impl/LlapIoMemoryServlet.java
 153ab35 
  llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/LlapIoImpl.java 
e5bc3c2 
  
llap-server/src/test/org/apache/hadoop/hive/llap/cache/TestLowLevelCacheImpl.java
 2c87bc2 
  
llap-server/src/test/org/apache/hadoop/hive/llap/cache/TestOrcMetadataCache.java
 f7ebff2 
  ql/src/java/org/apache/hadoop/hive/ql/processors/CommandProcessorFactory.java 
74a34b3 
  ql/src/java/org/apache/hadoop/hive/ql/processors/CommandUtil.java 0dcef08 
  ql/src/java/org/apache/hadoop/hive/ql/processors/HiveCommand.java c45563d 
  
ql/src/java/org/apache/hadoop/hive/ql/processors/LlapCacheResourceProcessor.java
 PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/processors/LlapClusterResourceProcessor.java
 PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/security/authorization/plugin/HiveOperationType.java
 5d6905a 
  
ql/src/java/org/apache/hadoop/hive/ql/security/authorization/plugin/sqlstd/Operation2Privilege.java
 a55e66b 
  
ql/src/java/org/apache/hadoop/hive/ql/security/authorization/plugin/sqlstd/SQLStdHiveAuthorizationValidator.java
 4e456e7 
  
service/src/java/org/apache/hive/service/cli/operation/HiveCommandOperation.java
 e3a5922 


Diff: https://reviews.apache.org/r/66236/diff/4/

Changes: https://reviews.apache.org/r/66236/diff/3-4/


Testing
---


Thanks,

Prasanth_J

Re: Review Request 66236: HIVE-19033: Provide an option to purge LLAP IO cache

2018-03-27 Thread j . prasanth . j


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66236/
---

(Updated March 27, 2018, 11:11 p.m.)


Review request for hive, Gopal V and Sergey Shelukhin.


Changes
---

Minor review comment update


Bugs: HIVE-19033
https://issues.apache.org/jira/browse/HIVE-19033


Repository: hive-git


Description
---

HIVE-19033: Provide an option to purge LLAP IO cache


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 5888eaa 
  
itests/hive-unit/src/test/java/org/apache/hive/jdbc/authorization/TestJdbcWithSQLAuthorization.java
 9e021ea 
  llap-client/src/java/org/apache/hadoop/hive/llap/io/api/LlapIo.java 6e6f5b9 
  
llap-common/src/gen/protobuf/gen-java/org/apache/hadoop/hive/llap/daemon/rpc/LlapDaemonProtocolProtos.java
 4753812 
  
llap-common/src/java/org/apache/hadoop/hive/llap/impl/LlapManagementProtocolClientImpl.java
 af760b1 
  
llap-common/src/java/org/apache/hadoop/hive/llap/protocol/LlapManagementProtocolPB.java
 ff215d4 
  llap-common/src/protobuf/LlapDaemonProtocol.proto 12beca5 
  
llap-server/src/java/org/apache/hadoop/hive/llap/cache/CacheContentsTracker.java
 4fbaac1 
  
llap-server/src/java/org/apache/hadoop/hive/llap/cache/LowLevelCachePolicy.java 
2cd70b9 
  
llap-server/src/java/org/apache/hadoop/hive/llap/cache/LowLevelFifoCachePolicy.java
 50a2411 
  
llap-server/src/java/org/apache/hadoop/hive/llap/cache/LowLevelLrfuCachePolicy.java
 b42f761 
  
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapProtocolServerImpl.java
 81785f0 
  llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/LlapIoImpl.java 
e5bc3c2 
  
llap-server/src/test/org/apache/hadoop/hive/llap/cache/TestLowLevelCacheImpl.java
 2c87bc2 
  
llap-server/src/test/org/apache/hadoop/hive/llap/cache/TestOrcMetadataCache.java
 f7ebff2 
  ql/src/java/org/apache/hadoop/hive/ql/processors/CommandProcessorFactory.java 
74a34b3 
  ql/src/java/org/apache/hadoop/hive/ql/processors/HiveCommand.java c45563d 
  
ql/src/java/org/apache/hadoop/hive/ql/processors/LlapCacheResourceProcessor.java
 PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/processors/LlapClusterResourceProcessor.java
 PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/security/authorization/plugin/HiveOperationType.java
 5d6905a 
  
ql/src/java/org/apache/hadoop/hive/ql/security/authorization/plugin/sqlstd/Operation2Privilege.java
 a55e66b 
  
service/src/java/org/apache/hive/service/cli/operation/HiveCommandOperation.java
 e3a5922 


Diff: https://reviews.apache.org/r/66236/diff/3/

Changes: https://reviews.apache.org/r/66236/diff/2-3/


Testing
---


Thanks,

Prasanth_J

Re: Review Request 66236: HIVE-19033: Provide an option to purge LLAP IO cache

2018-03-27 Thread j . prasanth . j


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66236/
---

(Updated March 27, 2018, 11:05 p.m.)


Review request for hive, Gopal V and Sergey Shelukhin.


Changes
---

Removed cli option
Removed web endpoint for purge
Added LLAP commands that are AuthZ'd


Bugs: HIVE-19033
https://issues.apache.org/jira/browse/HIVE-19033


Repository: hive-git


Description
---

HIVE-19033: Provide an option to purge LLAP IO cache


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 5888eaa 
  
itests/hive-unit/src/test/java/org/apache/hive/jdbc/authorization/TestJdbcWithSQLAuthorization.java
 9e021ea 
  llap-client/src/java/org/apache/hadoop/hive/llap/io/api/LlapIo.java 6e6f5b9 
  
llap-common/src/gen/protobuf/gen-java/org/apache/hadoop/hive/llap/daemon/rpc/LlapDaemonProtocolProtos.java
 4753812 
  
llap-common/src/java/org/apache/hadoop/hive/llap/impl/LlapManagementProtocolClientImpl.java
 af760b1 
  
llap-common/src/java/org/apache/hadoop/hive/llap/protocol/LlapManagementProtocolPB.java
 ff215d4 
  llap-common/src/protobuf/LlapDaemonProtocol.proto 12beca5 
  
llap-server/src/java/org/apache/hadoop/hive/llap/cache/CacheContentsTracker.java
 4fbaac1 
  
llap-server/src/java/org/apache/hadoop/hive/llap/cache/LowLevelCachePolicy.java 
2cd70b9 
  
llap-server/src/java/org/apache/hadoop/hive/llap/cache/LowLevelFifoCachePolicy.java
 50a2411 
  
llap-server/src/java/org/apache/hadoop/hive/llap/cache/LowLevelLrfuCachePolicy.java
 b42f761 
  
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapProtocolServerImpl.java
 81785f0 
  llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/LlapIoImpl.java 
e5bc3c2 
  
llap-server/src/test/org/apache/hadoop/hive/llap/cache/TestLowLevelCacheImpl.java
 2c87bc2 
  
llap-server/src/test/org/apache/hadoop/hive/llap/cache/TestOrcMetadataCache.java
 f7ebff2 
  ql/src/java/org/apache/hadoop/hive/ql/processors/CommandProcessorFactory.java 
74a34b3 
  ql/src/java/org/apache/hadoop/hive/ql/processors/HiveCommand.java c45563d 
  
ql/src/java/org/apache/hadoop/hive/ql/processors/LlapCacheResourceProcessor.java
 PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/processors/LlapClusterResourceProcessor.java
 PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/security/authorization/plugin/HiveOperationType.java
 5d6905a 
  
ql/src/java/org/apache/hadoop/hive/ql/security/authorization/plugin/sqlstd/Operation2Privilege.java
 a55e66b 
  
service/src/java/org/apache/hive/service/cli/operation/HiveCommandOperation.java
 e3a5922 


Diff: https://reviews.apache.org/r/66236/diff/2/

Changes: https://reviews.apache.org/r/66236/diff/1-2/


Testing
---


Thanks,

Prasanth_J

Re: Review Request 66317: HIVE-19061 WM needs to output an event for allocation update

2018-03-27 Thread j . prasanth . j


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66317/#review200065
---


Ship it!




Ship It!

- Prasanth_J


On March 27, 2018, 8:29 p.m., Sergey Shelukhin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66317/
> ---
> 
> (Updated March 27, 2018, 8:29 p.m.)
> 
> 
> Review request for hive and Prasanth_J.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> see jira
> 
> 
> Diffs
> -
> 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/tez/GuaranteedTasksAllocator.java 
> a52928cc7a 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/WmEvent.java 33341ad4a9 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/WmTezSession.java 1cf5493959 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/WorkloadManager.java 
> f0e620c684 
>   ql/src/test/org/apache/hadoop/hive/ql/exec/tez/TestWorkloadManager.java 
> 20a5947291 
> 
> 
> Diff: https://reviews.apache.org/r/66317/diff/1/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Sergey Shelukhin
> 
>

Re: Review Request 66237: HIVE-18971 add HS2 WM metrics for use in Grafana and such

2018-03-26 Thread j . prasanth . j



> On March 24, 2018, 1:34 a.m., Prasanth_J wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/exec/tez/WmPoolMetrics.java
> > Lines 159 (patched)
> > 
> >
> > why isn't this tag alone be sufficient? With this tag we can get all 
> > metrics associated/registered under pool. right?
> > 
> > instead of emitting metrics like
> > WM__
> > 
> > why not build a wrapper around this getMetrics() which gets all pool 
> > names and set the tag. So we will have something like
> > 
> > {
> > "tag.SessionId": "6020e225-f36e-470b-a170-b18e69af6fc8",
> > "tag.Poolname": "llap",
> > "NumExecutors": 2,
> > "NumSessions": 2
> > }
> > 
> > If you try to run 2 LLAP daemons on the same host, you would get 2 
> > different metrics with different SessionId. This looks similar to that 
> > except that only thing that changes here is poolName. Am I missing 
> > something?
> 
> Sergey Shelukhin wrote:
> I'm not sure what you mean. I just used session ID as a standard tag, 
> since there's nothing else to put in there.
> Note that metrics are emitted into Hadoop metrics and also codahale 
> (mostly for HS2 JMX).
> Only codahale one uses silly names (which seems to be a common pattern if 
> you see HS2 JMX), the tagged metrics from Hadoop metrics should all have the 
> same name.

make sense.. missed the part about handling codahale vs hadoop metrics which 
supports tags.


- Prasanth_J


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66237/#review199922
---


On March 26, 2018, 9:30 p.m., Sergey Shelukhin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66237/
> ---
> 
> (Updated March 26, 2018, 9:30 p.m.)
> 
> 
> Review request for hive and Prasanth_J.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> .
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/common/metrics/LegacyMetrics.java 
> effe26b6b6 
>   common/src/java/org/apache/hadoop/hive/common/metrics/common/Metrics.java 
> 88c513b8cd 
>   
> common/src/java/org/apache/hadoop/hive/common/metrics/metrics2/CodahaleMetrics.java
>  a43b09db8c 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 8d9b5a3194 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/LlapRecordReader.java
>  3a2c19a3e6 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/tez/GuaranteedTasksAllocator.java 
> a52928cc7a 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/QueryAllocationManager.java 
> 9885ce7221 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/WmPoolMetrics.java 
> PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/WorkloadManager.java 
> f0e620c684 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/tez/monitoring/TezProgressMonitor.java
>  a14cdb609a 
>   ql/src/test/org/apache/hadoop/hive/ql/exec/tez/TestWorkloadManager.java 
> 20a5947291 
> 
> 
> Diff: https://reviews.apache.org/r/66237/diff/4/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Sergey Shelukhin
> 
>

Re: Review Request 66237: HIVE-18971 add HS2 WM metrics for use in Grafana and such

2018-03-26 Thread j . prasanth . j


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66237/#review20
---


Ship it!




Ship It!

- Prasanth_J


On March 26, 2018, 9:30 p.m., Sergey Shelukhin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66237/
> ---
> 
> (Updated March 26, 2018, 9:30 p.m.)
> 
> 
> Review request for hive and Prasanth_J.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> .
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/common/metrics/LegacyMetrics.java 
> effe26b6b6 
>   common/src/java/org/apache/hadoop/hive/common/metrics/common/Metrics.java 
> 88c513b8cd 
>   
> common/src/java/org/apache/hadoop/hive/common/metrics/metrics2/CodahaleMetrics.java
>  a43b09db8c 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 8d9b5a3194 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/LlapRecordReader.java
>  3a2c19a3e6 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/tez/GuaranteedTasksAllocator.java 
> a52928cc7a 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/QueryAllocationManager.java 
> 9885ce7221 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/WmPoolMetrics.java 
> PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/WorkloadManager.java 
> f0e620c684 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/tez/monitoring/TezProgressMonitor.java
>  a14cdb609a 
>   ql/src/test/org/apache/hadoop/hive/ql/exec/tez/TestWorkloadManager.java 
> 20a5947291 
> 
> 
> Diff: https://reviews.apache.org/r/66237/diff/4/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Sergey Shelukhin
> 
>

Review Request 66266: HIVE-18994: Handle client connections on failover

2018-03-23 Thread j . prasanth . j


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66266/
---

Review request for hive and Sergey Shelukhin.


Bugs: HIVE-18994
https://issues.apache.org/jira/browse/HIVE-18994


Repository: hive-git


Description
---

HIVE-18994: Handle client connections on failover


Diffs
-

  itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestActivePassiveHA.java 
e53826df1e65edbb7f84ea9987208e3018858bce 
  itests/util/src/main/java/org/apache/hive/jdbc/miniHS2/MiniHS2.java 
fa5edec1a660d0eedca7f822a8e2d79358b2bc17 
  service/src/java/org/apache/hive/service/server/HiveServer2.java 
90ba752bff6b493c4d0010617bea0a7a9da29876 


Diff: https://reviews.apache.org/r/66266/diff/1/


Testing
---


Thanks,

Prasanth_J

Re: Review Request 66237: HIVE-18971 add HS2 WM metrics for use in Grafana and such

2018-03-23 Thread j . prasanth . j


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66237/#review199922
---




common/src/java/org/apache/hadoop/hive/common/metrics/metrics2/CodahaleMetrics.java
Lines 304 (patched)


why do we need to remove the gauge names? why not add a guage and forget 
about it.



ql/src/java/org/apache/hadoop/hive/ql/exec/tez/GuaranteedTasksAllocator.java
Lines 103 (patched)


nit: whitespace in many places.



ql/src/java/org/apache/hadoop/hive/ql/exec/tez/WmPoolMetrics.java
Lines 159 (patched)


why isn't this tag alone be sufficient? With this tag we can get all 
metrics associated/registered under pool. right?

instead of emitting metrics like
WM__

why not build a wrapper around this getMetrics() which gets all pool names 
and set the tag. So we will have something like

{
"tag.SessionId": "6020e225-f36e-470b-a170-b18e69af6fc8",
"tag.Poolname": "llap",
"NumExecutors": 2,
"NumSessions": 2
}

If you try to run 2 LLAP daemons on the same host, you would get 2 
different metrics with different SessionId. This looks similar to that except 
that only thing that changes here is poolName. Am I missing something?


- Prasanth_J


On March 23, 2018, 8:14 p.m., Sergey Shelukhin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66237/
> ---
> 
> (Updated March 23, 2018, 8:14 p.m.)
> 
> 
> Review request for hive and Prasanth_J.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> .
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/common/metrics/LegacyMetrics.java 
> effe26b6b6 
>   common/src/java/org/apache/hadoop/hive/common/metrics/common/Metrics.java 
> 88c513b8cd 
>   
> common/src/java/org/apache/hadoop/hive/common/metrics/metrics2/CodahaleMetrics.java
>  a43b09db8c 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 8d9b5a3194 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/LlapRecordReader.java
>  3a2c19a3e6 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/tez/GuaranteedTasksAllocator.java 
> a52928cc7a 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/QueryAllocationManager.java 
> 9885ce7221 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/WmPoolMetrics.java 
> PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/WorkloadManager.java 
> f0e620c684 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/tez/monitoring/TezProgressMonitor.java
>  a14cdb609a 
>   ql/src/test/org/apache/hadoop/hive/ql/exec/tez/TestWorkloadManager.java 
> 20a5947291 
> 
> 
> Diff: https://reviews.apache.org/r/66237/diff/2/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Sergey Shelukhin
> 
>

Re: Review Request 66236: HIVE-19033: Provide an option to purge LLAP IO cache

2018-03-22 Thread j . prasanth . j



> On March 23, 2018, 2:27 a.m., Sergey Shelukhin wrote:
> > llap-server/src/java/org/apache/hadoop/hive/llap/cli/LlapStatusServiceDriver.java
> > Lines 1105 (patched)
> > 
> >
> > is it possible to do this thru management endpoint that has appropriate 
> > acls and security?

I can add a management endpoint as well. I will leave the HTTP endpoint for 
secure cluster access only (maybe it can be useful if someone builds UI for 
this).


- Prasanth_J


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66236/#review199835
---


On March 23, 2018, 2 a.m., Prasanth_J wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66236/
> ---
> 
> (Updated March 23, 2018, 2 a.m.)
> 
> 
> Review request for hive, Gopal V and Sergey Shelukhin.
> 
> 
> Bugs: HIVE-19033
> https://issues.apache.org/jira/browse/HIVE-19033
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-19033: Provide an option to purge LLAP IO cache
> 
> 
> Diffs
> -
> 
>   llap-client/src/java/org/apache/hadoop/hive/llap/io/api/LlapIo.java 
> 6e6f5b92c998bb1b3055421a06155fad9b22e8e4 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/cache/CacheContentsTracker.java
>  4fbaac1db0a545e591f4b8af53b729d6a10dd679 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/cache/LowLevelCachePolicy.java
>  2cd70b97b2dfad94fb7e05f78d268fb6aefe4e6e 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/cache/LowLevelFifoCachePolicy.java
>  50a2411b4b14306342d451b6a62c629440a0cfba 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/cache/LowLevelLrfuCachePolicy.java
>  b42f76184cb08be0f6db1be8e64f86593a89c372 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/cli/LlapStatusOptionsProcessor.java
>  e88c819b2c26bbc4fef92d4030206853bdb33184 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/cli/LlapStatusServiceDriver.java
>  65b4d81000d1b4ac9041a51e5e8d8377efa25a71 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/services/impl/LlapIoMemoryServlet.java
>  153ab3554bbd5437aba39e980700551b11b333f9 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/LlapIoImpl.java 
> e5bc3c2bfa8615187eb1d1a3b80afb16a42b2c08 
>   
> llap-server/src/test/org/apache/hadoop/hive/llap/cache/TestLowLevelCacheImpl.java
>  2c87bc28e51918d34c73edcdbb6874ec84db86cb 
>   
> llap-server/src/test/org/apache/hadoop/hive/llap/cache/TestOrcMetadataCache.java
>  f7ebff2028dcde66d3345873a96ba1becc5ef59c 
> 
> 
> Diff: https://reviews.apache.org/r/66236/diff/1/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Prasanth_J
> 
>

1 2 3 4 >

1 - 100 of 393 matches

Mail list logo