Re: Review Request 63528: HIVE-17969: Metastore to alter table in batches of partitions when renaming table

[Barna Zsombor Klara]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63528/#review190166
---


Ship it!




Ship It!

- Barna Zsombor Klara


On Nov. 2, 2017, 10:02 p.m., Adam Szita wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/63528/
> ---
> 
> (Updated Nov. 2, 2017, 10:02 p.m.)
> 
> 
> Review request for hive, Peter Vary and Barna Zsombor Klara.
> 
> 
> Bugs: HIVE-17969
> https://issues.apache.org/jira/browse/HIVE-17969
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Refactoring alter table code to use batching of partitions when calling the 
> heavy removeUnusedColumnDescriptor method
> 
> 
> Diffs
> -
> 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveAlterHandler.java
>  ccadac1ada6aaae884ab39f5d99e91b8c542404e 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
>  62801c53853dbafb7c425cff943ec819dcee4800 
> 
> 
> Diff: https://reviews.apache.org/r/63528/diff/1/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Adam Szita
> 
>

Re: Review Request 63486: HIVE-17953 Metrics should move to destination atomically

[Barna Zsombor Klara]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63486/#review189889
---


Ship it!




Ship It!

- Barna Zsombor Klara


On nov. 1, 2017, 11:25 du, Alexander Kolbasov wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/63486/
> ---
> 
> (Updated nov. 1, 2017, 11:25 du)
> 
> 
> Review request for hive, Sahil Takiar and Vihang Karajgaonkar.
> 
> 
> Bugs: HIVE-17953
> https://issues.apache.org/jira/browse/HIVE-17953
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-17953 Metrics should move to destination atomically
> 
> 
> Diffs
> -
> 
>   
> common/src/java/org/apache/hadoop/hive/common/metrics/metrics2/JsonFileMetricsReporter.java
>  b9be8bd18285df42a82385f7ce73d9e1924fee4c 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/metrics/JsonReporter.java
>  04a5f02a91b76ed3775f7c1c8e0122a135564c70 
> 
> 
> Diff: https://reviews.apache.org/r/63486/diff/1/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Alexander Kolbasov
> 
>

Re: Review Request 62810: HIVE-17300 WebUI query plan graphs

[Barna Zsombor Klara]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62810/#review189606
---


Ship it!




Ship It!

- Barna Zsombor Klara


On Oct. 30, 2017, 4:23 p.m., Peter Vary wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/62810/
> ---
> 
> (Updated Oct. 30, 2017, 4:23 p.m.)
> 
> 
> Review request for hive, Karen Coppage, Xuefu Zhang, and Xuefu Zhang.
> 
> 
> Bugs: HIVE-17300
> https://issues.apache.org/jira/browse/HIVE-17300
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Moving the review here, since could not change Karen's original one
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/common/LogUtils.java 0a3e0c7 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 6631a6e 
>   ql/src/java/org/apache/hadoop/hive/ql/Driver.java 6c6ad92 
>   ql/src/java/org/apache/hadoop/hive/ql/MapRedStats.java 4b60514 
>   ql/src/java/org/apache/hadoop/hive/ql/QueryDisplay.java 132bec6 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/mr/HadoopJobExecHelper.java 
> 2d2eafd 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/mr/MapRedTask.java 41a1ef1 
>   service/src/jamon/org/apache/hive/tmpl/QueryProfileTmpl.jamon ff7476e 
>   service/src/resources/hive-webapps/static/css/query-plan-graph.css 
> PRE-CREATION 
>   service/src/resources/hive-webapps/static/js/query-plan-graph.js 
> PRE-CREATION 
>   service/src/resources/hive-webapps/static/js/vis.min.js PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/62810/diff/2/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Peter Vary
> 
>

Re: Review Request 62810: HIVE-17300 WebUI query plan graphs

[Barna Zsombor Klara]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62810/#review189575
---



Thank you for the patch Peter (and Karen). I have 3 minor comments if you fixed 
those, then we can ship it.


common/src/java/org/apache/hadoop/hive/common/LogUtils.java
Lines 239-240 (patched)
<https://reviews.apache.org/r/62810/#comment266728>

Can you please check that this cast is always correct? An if with an 
instance of check should be enough.



ql/src/java/org/apache/hadoop/hive/ql/QueryDisplay.java
Lines 159 (patched)
<https://reviews.apache.org/r/62810/#comment266730>

I would prefer an iterator with a type parameter. This way we can avoid the 
explicit cast 2 lines below.



ql/src/java/org/apache/hadoop/hive/ql/QueryDisplay.java
Lines 162 (patched)
<https://reviews.apache.org/r/62810/#comment266731>

Same as before, please use a typed iterator if possible.


- Barna Zsombor Klara


On Oct. 6, 2017, 3:37 p.m., Peter Vary wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/62810/
> ---
> 
> (Updated Oct. 6, 2017, 3:37 p.m.)
> 
> 
> Review request for hive, Karen Coppage, Xuefu Zhang, and Xuefu Zhang.
> 
> 
> Bugs: HIVE-17300
> https://issues.apache.org/jira/browse/HIVE-17300
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Moving the review here, since could not change Karen's original one
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/common/LogUtils.java 0a3e0c7 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java d2afc2c 
>   ql/src/java/org/apache/hadoop/hive/ql/Driver.java 1943c6d 
>   ql/src/java/org/apache/hadoop/hive/ql/MapRedStats.java 4b60514 
>   ql/src/java/org/apache/hadoop/hive/ql/QueryDisplay.java bf6cb91 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/mr/HadoopJobExecHelper.java 
> 3c07197 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/mr/MapRedTask.java 41a1ef1 
>   service/src/jamon/org/apache/hive/tmpl/QueryProfileTmpl.jamon ff7476e 
>   service/src/resources/hive-webapps/static/css/query-plan-graph.css 
> PRE-CREATION 
>   service/src/resources/hive-webapps/static/js/query-plan-graph.js 
> PRE-CREATION 
>   service/src/resources/hive-webapps/static/js/vis.min.js PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/62810/diff/1/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Peter Vary
> 
>

Re: Review Request 63144: HIVE-16748: Integreate YETUS to Pre-Commit

[Barna Zsombor Klara]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63144/#review188676
---




testutils/ptest2/src/main/java/org/apache/hive/ptest/execution/YetusPhase.java
Lines 72-85 (patched)
<https://reviews.apache.org/r/63144/#comment265693>

I agree that this could go into a shell script generated by a velocity 
template. It would also make this a bit more testable.


- Barna Zsombor Klara


On okt. 19, 2017, 12:40 du, Adam Szita wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/63144/
> ---
> 
> (Updated okt. 19, 2017, 12:40 du)
> 
> 
> Review request for hive, Peter Vary and Barna Zsombor Klara.
> 
> 
> Bugs: HIVE-16748
> https://issues.apache.org/jira/browse/HIVE-16748
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> We already have Yetus check script ready for run in dev-support. We should 
> integrate this with the automated ptest infrastructure so that during 
> Precommit test we get a Yetus result back as Jira comment
> 
> 
> Diffs
> -
> 
>   dev-support/hive-personality.sh f3247aac6284b8dd863691b4819a10c3a896d50c 
>   testutils/ptest2/src/main/java/org/apache/hive/ptest/execution/PTest.java 
> 65a8216f6a076b0ee7baee11ca557f5e9f746316 
>   
> testutils/ptest2/src/main/java/org/apache/hive/ptest/execution/YetusPhase.java
>  PRE-CREATION 
>   testutils/ptest2/src/main/resources/source-prep.vm 
> 7ad50248af02dfaeb6524a61d4895f1a8efba211 
> 
> 
> Diff: https://reviews.apache.org/r/63144/diff/1/
> 
> 
> Testing
> ---
> 
> Tested on Cloudera sponsored sandbox ptest server.
> 
> 
> Thanks,
> 
> Adam Szita
> 
>

Re: Review Request 62782: HIVE-17706 Add a possibility to run the BeeLine tests on the default database

[Barna Zsombor Klara]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62782/#review187240
---


Ship it!




Ship It!

- Barna Zsombor Klara


On Oct. 5, 2017, 4:39 p.m., Peter Vary wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/62782/
> ---
> 
> (Updated Oct. 5, 2017, 4:39 p.m.)
> 
> 
> Review request for hive and Barna Zsombor Klara.
> 
> 
> Bugs: HIVE-17706
> https://issues.apache.org/jira/browse/HIVE-17706
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> - Added a cleanup method which removed unkonwn databases, tables and view. 
> This makes it is possible to run the tests sequentially using 'default' 
> database.
> - Renamed the test.rewrite.source.tables parameter to 
> test.beeline.run.parallel so it makes more sense.
> - Copied some masking from QTestUtil
> - Enhanced the table name rewriting regexp a little, so mixed case tablenames 
> are kept as it is
> - In the QFile made it possible to not call create/drop database command, if 
> not needed.
> 
> I would value any advice where the cleanup logic should be kept.
> - I this solution when the tests are parallel then the cleanup is in 
> QFileBeeLineClient (create/drop database), when the tests are sequential then 
> the cleanup is in the CoreBeeLineDriver.
> 
> Would it be a good idea to move every cleanup related stuff to 
> CoreBeeLineDriver? Like:
> - QFileBeeLineClient.beforeExecute
> - QFileBeeLineClient.afterExecute
> Both of these are need QFile specific info, and an existing BeeLineClient, 
> which we currently do not have in hand in the CoreBeeLineDriver, and more 
> refactoring is needed
> 
> Or would it be a good idea to move every cleanup related stuff to 
> QFileBeeLineClient? Like:
> - CoreBeeLineDriver.runCleanup
> This is really cleanup stuff
> 
> Or we should leave as it is :)
> 
> Any comments are welcome.
> Thanks,
> Peter
> 
> 
> Diffs
> -
> 
>   
> itests/util/src/main/java/org/apache/hadoop/hive/cli/control/CoreBeeLineDriver.java
>  9dfc253 
>   itests/util/src/main/java/org/apache/hive/beeline/QFile.java e70ac38 
>   itests/util/src/main/java/org/apache/hive/beeline/QFileBeeLineClient.java 
> 2f91834 
> 
> 
> Diff: https://reviews.apache.org/r/62782/diff/2/
> 
> 
> Testing
> ---
> 
> Run the tests manually with, and without the test.beeline.run.parallel 
> parameter
> 
> 
> Thanks,
> 
> Peter Vary
> 
>

Re: Review Request 62748: HIVE-17078: Add more logs to MapredLocalTask

[Barna Zsombor Klara]

> On Oct. 5, 2017, 11:28 a.m., Peter Vary wrote:
> > common/src/java/org/apache/hadoop/hive/common/log/LogRedirector.java
> > Lines 67 (patched)
> > <https://reviews.apache.org/r/62748/diff/1/?file=1843648#file1843648line67>
> >
> > Is it worthwile to make MAX_LINES configurable?

Don't think so. This only gathers error lines to be concatenated into an error 
message.


- Barna Zsombor


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62748/#review187167
---


On Oct. 3, 2017, 12:29 p.m., Barna Zsombor Klara wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/62748/
> ---
> 
> (Updated Oct. 3, 2017, 12:29 p.m.)
> 
> 
> Review request for hive, Peter Vary, Adam Szita, and Yibing Shi.
> 
> 
> Bugs: HIVE-17078
> https://issues.apache.org/jira/browse/HIVE-17078
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-17078: Add more logs to MapredLocalTask
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/common/log/LogRedirector.java 
> PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/mr/MapredLocalTask.java 
> 9dfefeece56d7c3127c6921b5ad880fcaa926d56 
>   ql/src/test/results/clientpositive/auto_join25.q.out 
> 534bdb6ff0461614deab6e6fb84409cd34422f96 
>   ql/src/test/results/clientpositive/auto_join_without_localtask.q.out 
> 57f00674de30087143ef579cd0cff2bc024f529f 
>   ql/src/test/results/clientpositive/bucketsortoptimize_insert_8.q.out 
> f0e77f00afe0c753fbaa402c27662b2559c3e4d7 
>   ql/src/test/results/clientpositive/infer_bucket_sort_convert_join.q.out 
> 52ebe5aa8db035434c1bb0978b7b3ac2a25e062a 
>   ql/src/test/results/clientpositive/mapjoin_hook.q.out 
> a9f9be3a4dca764f70303412cc07dd31f7cece1c 
>   
> spark-client/src/main/java/org/apache/hive/spark/client/SparkClientImpl.java 
> 03e773a803f1dc7edfb9be8d795aba32535bd433 
> 
> 
> Diff: https://reviews.apache.org/r/62748/diff/2/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Barna Zsombor Klara
> 
>

Re: Review Request 62782: HIVE-17706 Add a possibility to run the BeeLine tests on the default database

[Barna Zsombor Klara]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62782/#review187168
---



Thank you for the patch Peter, looking good.
I don't have a clear preference on where the cleanup should go, but I would 
like it in one class. CoreBeeLineDriver or QFileBeeLineClient doesn't matter to 
me much, whichever is easier to refactor.


itests/util/src/main/java/org/apache/hadoop/hive/cli/control/CoreBeeLineDriver.java
Line 119 (original), 122 (patched)
<https://reviews.apache.org/r/62782/#comment264079>

This may be OK, but the naming confuses me. The method is called 
setTestSpecificDatabase so I would expect a database name to test, yet we feed 
it something called parallel?



itests/util/src/main/java/org/apache/hive/beeline/QFileBeeLineClient.java
Lines 80 (patched)
<https://reviews.apache.org/r/62782/#comment264082>

Please add javadoc for public APIs.


- Barna Zsombor Klara


On Oct. 5, 2017, 11:08 a.m., Peter Vary wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/62782/
> ---
> 
> (Updated Oct. 5, 2017, 11:08 a.m.)
> 
> 
> Review request for hive and Barna Zsombor Klara.
> 
> 
> Bugs: HIVE-17706
> https://issues.apache.org/jira/browse/HIVE-17706
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> - Added a cleanup method which removed unkonwn databases, tables and view. 
> This makes it is possible to run the tests sequentially using 'default' 
> database.
> - Renamed the test.rewrite.source.tables parameter to 
> test.beeline.run.parallel so it makes more sense.
> - Copied some masking from QTestUtil
> - Enhanced the table name rewriting regexp a little, so mixed case tablenames 
> are kept as it is
> - In the QFile made it possible to not call create/drop database command, if 
> not needed.
> 
> I would value any advice where the cleanup logic should be kept.
> - I this solution when the tests are parallel then the cleanup is in 
> QFileBeeLineClient (create/drop database), when the tests are sequential then 
> the cleanup is in the CoreBeeLineDriver.
> 
> Would it be a good idea to move every cleanup related stuff to 
> CoreBeeLineDriver? Like:
> - QFileBeeLineClient.beforeExecute
> - QFileBeeLineClient.afterExecute
> Both of these are need QFile specific info, and an existing BeeLineClient, 
> which we currently do not have in hand in the CoreBeeLineDriver, and more 
> refactoring is needed
> 
> Or would it be a good idea to move every cleanup related stuff to 
> QFileBeeLineClient? Like:
> - CoreBeeLineDriver.runCleanup
> This is really cleanup stuff
> 
> Or we should leave as it is :)
> 
> Any comments are welcome.
> Thanks,
> Peter
> 
> 
> Diffs
> -
> 
>   
> itests/util/src/main/java/org/apache/hadoop/hive/cli/control/CliAdapter.java 
> b89d6e7 
>   
> itests/util/src/main/java/org/apache/hadoop/hive/cli/control/CoreBeeLineDriver.java
>  9dfc253 
>   itests/util/src/main/java/org/apache/hive/beeline/QFile.java e70ac38 
>   itests/util/src/main/java/org/apache/hive/beeline/QFileBeeLineClient.java 
> 2f91834 
> 
> 
> Diff: https://reviews.apache.org/r/62782/diff/1/
> 
> 
> Testing
> ---
> 
> Run the tests manually with, and without the test.beeline.run.parallel 
> parameter
> 
> 
> Thanks,
> 
> Peter Vary
> 
>

Review Request 62748: HIVE-17078: Add more logs to MapredLocalTask

[Barna Zsombor Klara]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62748/
---

Review request for hive, Peter Vary and Adam Szita.


Bugs: HIVE-17078
https://issues.apache.org/jira/browse/HIVE-17078


Repository: hive-git


Description
---

HIVE-17078: Add more logs to MapredLocalTask


Diffs
-

  common/src/java/org/apache/hadoop/hive/common/log/LogRedirector.java 
PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/exec/mr/MapredLocalTask.java 
9dfefeece56d7c3127c6921b5ad880fcaa926d56 
  ql/src/test/results/clientpositive/auto_join25.q.out 
534bdb6ff0461614deab6e6fb84409cd34422f96 
  ql/src/test/results/clientpositive/auto_join_without_localtask.q.out 
57f00674de30087143ef579cd0cff2bc024f529f 
  ql/src/test/results/clientpositive/bucketsortoptimize_insert_8.q.out 
f0e77f00afe0c753fbaa402c27662b2559c3e4d7 
  ql/src/test/results/clientpositive/infer_bucket_sort_convert_join.q.out 
52ebe5aa8db035434c1bb0978b7b3ac2a25e062a 
  ql/src/test/results/clientpositive/mapjoin_hook.q.out 
a9f9be3a4dca764f70303412cc07dd31f7cece1c 
  spark-client/src/main/java/org/apache/hive/spark/client/SparkClientImpl.java 
03e773a803f1dc7edfb9be8d795aba32535bd433 


Diff: https://reviews.apache.org/r/62748/diff/1/


Testing
---


Thanks,

Barna Zsombor Klara

Re: [Announce] New committer: Sankar Hariappan

[Barna Zsombor Klara]

Congrats Sankar!

Rajesh Balamohan  (időpont: 2017. szept. 30., Szo,
2:24) ezt írta:

> Congrats Sankar!!
>
> ~Rajesh.B
>
> On Sat, Sep 30, 2017 at 4:30 AM, Vaibhav Gumashta <
> vgumas...@hortonworks.com
> > wrote:
>
> > Congratulations Sankar!
> >
> > On 9/29/17, 3:58 PM, "Thejas Nair"  wrote:
> >
> > >Congrats Sankar!
> > >
> > >On Fri, Sep 29, 2017 at 11:36 AM, Peter Vary 
> wrote:
> > >
> > >> Congratulations Sankar!
> > >>
> > >> > On Sep 29, 2017, at 7:56 PM, Ashutosh Chauhan  >
> > >> wrote:
> > >> >
> > >> > The Project Management Committee (PMC) for Apache Hive has invited
> > >>Sankar
> > >> > Harriapan to become a committer and we are pleased to announce that
> he
> > >> has
> > >> > accepted.
> > >> >
> > >> > Welcome, Sankar!
> > >> >
> > >> > Thanks,
> > >> > Ashutosh
> > >>
> > >>
> >
> >
>

Re: [Announce] New committer: Anishek Agarwal

[Barna Zsombor Klara]

Congratulations Anishek!

Rajesh Balamohan  (időpont: 2017. szept. 30., Szo,
2:25) ezt írta:

> Congrats Anishek!!
>
> ~Rajesh.B
>
> On Sat, Sep 30, 2017 at 4:30 AM, Vaibhav Gumashta <
> vgumas...@hortonworks.com
> > wrote:
>
> > Congratulations Anishek!
> >
> >
> > On 9/29/17, 3:57 PM, "Thejas Nair"  wrote:
> >
> > >Congrats Anishek!
> > >
> > >On Fri, Sep 29, 2017 at 11:36 AM, Peter Vary 
> wrote:
> > >
> > >> Congratulations Anishek!
> > >>
> > >> > On Sep 29, 2017, at 7:55 PM, Ashutosh Chauhan  >
> > >> wrote:
> > >> >
> > >> > The Project Management Committee (PMC) for Apache Hive has invited
> > >> Anishek
> > >> > Agarwal to become a committer and we are pleased to announce that he
> > >>has
> > >> > accepted.
> > >> >
> > >> > Welcome, Anishek!
> > >> >
> > >> > Thanks,
> > >> > Ashutosh
> > >>
> > >>
> >
> >
>

Re: Review Request 62404: HIVE-17556: Fixing the output of the udf_mask_hash.q test

[Barna Zsombor Klara]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62404/#review185674
---


Ship it!




Ship It!

- Barna Zsombor Klara


On Sept. 19, 2017, 11:55 a.m., Marta Kuczora wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/62404/
> ---
> 
> (Updated Sept. 19, 2017, 11:55 a.m.)
> 
> 
> Review request for hive and Barna Zsombor Klara.
> 
> 
> Bugs: HIVE-17556
> https://issues.apache.org/jira/browse/HIVE-17556
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Regenerated the out file of udf_mask_hash.q. In HIVE-17515 the used hash 
> function in GenericUDFMaskHash has been changed, but the output file for the 
> udf_mask_hash.q test has not been adapted.
> 
> 
> Diffs
> -
> 
>   ql/src/test/results/clientpositive/udf_mask_hash.q.out 200f817 
> 
> 
> Diff: https://reviews.apache.org/r/62404/diff/1/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Marta Kuczora
> 
>

Re: New committer : Barna Zsombor Klara

[Barna Zsombor Klara]

Thank you everyone! :)

On Tue, Sep 19, 2017 at 7:41 AM, Vaibhav Gumashta <vgumas...@hortonworks.com
> wrote:

> Congratulations Zsombor!
> 
> From: Prasanth Jayachandran <pjayachand...@hortonworks.com>
> Sent: Monday, September 18, 2017 5:52 PM
> To: dev@hive.apache.org
> Subject: Re: New committer : Barna Zsombor Klara
>
> Congratulations Zsombor!!
>
> Thanks
> Prasanth
> > On Sep 18, 2017, at 5:50 PM, Chetna C <chetna@gmail.com> wrote:
> >
> > Congratulations Zsombor!
> >
> > Thanks,
> > Chetna Chaudhari
> >
> > On 19 September 2017 at 10:37, Sergio Pena <sergio.p...@cloudera.com>
> wrote:
> >
> >> Great, congratulations Zsombor !!!
> >>
> >> On Mon, Sep 18, 2017 at 12:44 PM, Sahil Takiar <takiar.sa...@gmail.com>
> >> wrote:
> >>
> >>> Congrats Zsombor!
> >>>
> >>> On Mon, Sep 18, 2017 at 9:50 AM, Vihang Karajgaonkar <
> >> vih...@cloudera.com>
> >>> wrote:
> >>>
> >>>> Congratulations Zsombor!
> >>>>
> >>>> On Mon, Sep 18, 2017 at 8:15 AM, Zoltan Haindrich <
> >>>> zhaindr...@hortonworks.com> wrote:
> >>>>
> >>>>> Congratulations!!
> >>>>>
> >>>>> On 16 Sep 2017 00:52, Ashutosh Chauhan <hashut...@apache.org> wrote:
> >>>>> The Project Management Committee (PMC) for Apache Hive has invited
> >>> Barna
> >>>>> Zsombor Klara to become a committer and we are pleased to announce
> >> that
> >>>> he
> >>>>> has accepted.
> >>>>>
> >>>>> Barna, thank you for your contributions, and we look forward your
> >>>>> further interactions
> >>>>> with the community!
> >>>>>
> >>>>> Welcome, Barna!
> >>>>>
> >>>>> Thanks,
> >>>>> Ashutosh (on behalf of Apache Hive PMC)
> >>>>>
> >>>>>
> >>>>
> >>>
> >>>
> >>>
> >>> --
> >>> Sahil Takiar
> >>> Software Engineer at Cloudera
> >>> takiar.sa...@gmail.com | (510) 673-0309
> >>>
> >>
>
>
>
>

Re: Review Request 62152: HIVE-17317: Make Dbcp configurable using hive properties in hive-site.xml

[Barna Zsombor Klara]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62152/
---

(Updated Sept. 8, 2017, 3 p.m.)


Review request for hive and Peter Vary.


Changes
---

Thanks for the review Peter!


Bugs: HIVE-17317
https://issues.apache.org/jira/browse/HIVE-17317


Repository: hive-git


Description
---

HIVE-17317: Make Dbcp configurable using hive properties in hive-site.xml


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/Constants.java 
794b697dc005802a3403bd39499e13bcd8cb2f99 
  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 
cf3f50ba64a28e63b58badcc2bce7738bf434245 
  metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java 
0db1bc059c0f6a36e721d441dbd466736d270eca 
  
metastore/src/java/org/apache/hadoop/hive/metastore/datasource/BoneCPDataSourceProvider.java
 34765b0b2f34698a3ba29751a65a108e4c997502 
  
metastore/src/java/org/apache/hadoop/hive/metastore/datasource/DataSourceProviderFactory.java
 1eb792ce4503dfd82ce5660a39a5f33c1db86913 
  
metastore/src/java/org/apache/hadoop/hive/metastore/datasource/DbCPDataSourceProvider.java
 PRE-CREATION 
  
metastore/src/java/org/apache/hadoop/hive/metastore/datasource/HikariCPDataSourceProvider.java
 9b3d6d5d7078301254a4cff0a0d8e5de44d03bc3 
  metastore/src/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java 
1887c052be1e535539cc5ba4c634fa28dfc22f9d 
  
metastore/src/test/org/apache/hadoop/hive/metastore/datasource/TestDataSourceProviderFactory.java
 daea544c7126fad26f02e39a95ea0bc0e4847387 


Diff: https://reviews.apache.org/r/62152/diff/2/

Changes: https://reviews.apache.org/r/62152/diff/1-2/


Testing
---


Thanks,

Barna Zsombor Klara

Review Request 62152: HIVE-17317: Make Dbcp configurable using hive properties in hive-site.xml

[Barna Zsombor Klara]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62152/
---

Review request for hive and Peter Vary.


Bugs: HIVE-17317
https://issues.apache.org/jira/browse/HIVE-17317


Repository: hive-git


Description
---

HIVE-17317: Make Dbcp configurable using hive properties in hive-site.xml


Diffs
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 
cf3f50ba64a28e63b58badcc2bce7738bf434245 
  metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java 
0db1bc059c0f6a36e721d441dbd466736d270eca 
  
metastore/src/java/org/apache/hadoop/hive/metastore/datasource/BoneCPDataSourceProvider.java
 34765b0b2f34698a3ba29751a65a108e4c997502 
  
metastore/src/java/org/apache/hadoop/hive/metastore/datasource/DataSourceProviderFactory.java
 1eb792ce4503dfd82ce5660a39a5f33c1db86913 
  
metastore/src/java/org/apache/hadoop/hive/metastore/datasource/DbCPDataSourceProvider.java
 PRE-CREATION 
  
metastore/src/java/org/apache/hadoop/hive/metastore/datasource/HikariCPDataSourceProvider.java
 9b3d6d5d7078301254a4cff0a0d8e5de44d03bc3 
  metastore/src/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java 
1887c052be1e535539cc5ba4c634fa28dfc22f9d 
  
metastore/src/test/org/apache/hadoop/hive/metastore/datasource/TestDataSourceProviderFactory.java
 daea544c7126fad26f02e39a95ea0bc0e4847387 


Diff: https://reviews.apache.org/r/62152/diff/1/


Testing
---


Thanks,

Barna Zsombor Klara

[jira] [Created] (HIVE-17389) Yetus is always failing on rat checks

[Barna Zsombor Klara (JIRA)]

Barna Zsombor Klara created HIVE-17389:
--

 Summary: Yetus is always failing on rat checks
 Key: HIVE-17389
 URL: https://issues.apache.org/jira/browse/HIVE-17389
 Project: Hive
  Issue Type: Bug
Affects Versions: 3.0.0
Reporter: Barna Zsombor Klara
Assignee: Barna Zsombor Klara
Priority: Trivial


Rat checks are failing on metastore_db/dblock and files under patchprocess 
created by Yetus itself.
Both directories should be excluded from rat checks.

CC: [~pvary] [~kgyrtkirk]



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Review Request 61885: HIVE-17382: Change startsWith relation introduced in HIVE-17316

[Barna Zsombor Klara]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61885/
---

Review request for hive and Peter Vary.


Bugs: HIVE-17382
https://issues.apache.org/jira/browse/HIVE-17382


Repository: hive-git


Description
---

HIVE-17382: Change startsWith relation introduced in HIVE-17316


Diffs
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 
2fb004c0ecea6d371a3a18e4d0eab19e5b80af55 
  common/src/test/org/apache/hadoop/hive/conf/TestHiveConf.java 
c914d2332dc645af2de5f85c6ce8b36c6e09b3dc 
  
metastore/src/test/org/apache/hadoop/hive/metastore/datasource/TestDataSourceProviderFactory.java
 628460482dc646f8a38f607e815eddaa3cc2a831 


Diff: https://reviews.apache.org/r/61885/diff/1/


Testing
---


Thanks,

Barna Zsombor Klara

[jira] [Created] (HIVE-17382) Change startsWith relation introduced in HIVE-17316

[Barna Zsombor Klara (JIRA)]

Barna Zsombor Klara created HIVE-17382:
--

 Summary: Change startsWith relation introduced in HIVE-17316
 Key: HIVE-17382
 URL: https://issues.apache.org/jira/browse/HIVE-17382
 Project: Hive
  Issue Type: Sub-task
Affects Versions: 3.0.0
Reporter: Barna Zsombor Klara
Assignee: Barna Zsombor Klara
 Fix For: 3.0.0


In HiveConf the new name should be checked if it starts with a 
restricted/hidden variable prefix and not vice-versa.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: Review Request 61851: HIVE-17318: Make Hikari CP configurable using hive properties in hive-site.xml

[Barna Zsombor Klara]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61851/
---

(Updated Aug. 24, 2017, 9:43 a.m.)


Review request for hive and Peter Vary.


Changes
---

Patch updated. Thanks Peter for spotting the missing restricted list setting.


Summary (updated)
-

HIVE-17318: Make Hikari CP configurable using hive properties in hive-site.xml


Bugs: HIVE-17318
https://issues.apache.org/jira/browse/HIVE-17318


Repository: hive-git


Description (updated)
---

HIVE-17318: Make Hikari CP configurable using hive properties in hive-site.xml


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 
2fb004c0ecea6d371a3a18e4d0eab19e5b80af55 
  
metastore/src/java/org/apache/hadoop/hive/metastore/datasource/DataSourceProviderFactory.java
 fa6bb1cc5252a71ccc6f011d5dd062a952de5b8a 
  
metastore/src/java/org/apache/hadoop/hive/metastore/datasource/HikariCPDataSourceProvider.java
 PRE-CREATION 
  metastore/src/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java 
a6d56137b3c4501b74b1e9a3c5d4085b55919710 
  
metastore/src/test/org/apache/hadoop/hive/metastore/datasource/TestDataSourceProviderFactory.java
 628460482dc646f8a38f607e815eddaa3cc2a831 


Diff: https://reviews.apache.org/r/61851/diff/3/

Changes: https://reviews.apache.org/r/61851/diff/2-3/


Testing
---


Thanks,

Barna Zsombor Klara

Re: Review Request 61851: HIVE-17318: Make Hikari configurable using hive properties in hive-site.xml

[Barna Zsombor Klara]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61851/
---

(Updated Aug. 23, 2017, 4:45 p.m.)


Review request for hive and Peter Vary.


Changes
---

Updated changes to TxnHandler.java.


Bugs: HIVE-17318
https://issues.apache.org/jira/browse/HIVE-17318


Repository: hive-git


Description
---

HIVE-17318: Make Hikari configurable using hive properties in hive-site.xml


Diffs (updated)
-

  
metastore/src/java/org/apache/hadoop/hive/metastore/datasource/DataSourceProviderFactory.java
 fa6bb1cc5252a71ccc6f011d5dd062a952de5b8a 
  
metastore/src/java/org/apache/hadoop/hive/metastore/datasource/HikariCPDataSourceProvider.java
 PRE-CREATION 
  metastore/src/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java 
a6d56137b3c4501b74b1e9a3c5d4085b55919710 
  
metastore/src/test/org/apache/hadoop/hive/metastore/datasource/TestDataSourceProviderFactory.java
 628460482dc646f8a38f607e815eddaa3cc2a831 


Diff: https://reviews.apache.org/r/61851/diff/2/

Changes: https://reviews.apache.org/r/61851/diff/1-2/


Testing
---


Thanks,

Barna Zsombor Klara

Review Request 61851: HIVE-17318: Make Hikari configurable using hive properties in hive-site.xml

[Barna Zsombor Klara]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61851/
---

Review request for hive and Peter Vary.


Bugs: HIVE-17318
https://issues.apache.org/jira/browse/HIVE-17318


Repository: hive-git


Description
---

HIVE-17318: Make Hikari configurable using hive properties in hive-site.xml


Diffs
-

  
metastore/src/java/org/apache/hadoop/hive/metastore/datasource/DataSourceProviderFactory.java
 fa6bb1cc5252a71ccc6f011d5dd062a952de5b8a 
  
metastore/src/java/org/apache/hadoop/hive/metastore/datasource/HikariCPDataSourceProvider.java
 PRE-CREATION 
  
metastore/src/test/org/apache/hadoop/hive/metastore/datasource/TestDataSourceProviderFactory.java
 628460482dc646f8a38f607e815eddaa3cc2a831 


Diff: https://reviews.apache.org/r/61851/diff/1/


Testing
---


Thanks,

Barna Zsombor Klara

Re: Review Request 61712: HIVE-17319: Make BoneCp configurable using hive properties in hive-site.xml

[Barna Zsombor Klara]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61712/
---

(Updated Aug. 21, 2017, 12:59 p.m.)


Review request for hive, Eugene Koifman, Peter Vary, and Thejas Nair.


Changes
---

Patch has been rebased and HiveConf variables substituted with MetastoreConf.


Bugs: HIVE-17319
https://issues.apache.org/jira/browse/HIVE-17319


Repository: hive-git


Description
---

HIVE-17319: Make BoneCp configurable using hive properties in hive-site.xml


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 
8724930b77da9f8ae81053000d24e386b5a581c8 
  metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java 
b87811502beac1143fbc0ba1e8daf2079562e7b3 
  
metastore/src/java/org/apache/hadoop/hive/metastore/datasource/BoneCPDataSourceProvider.java
 PRE-CREATION 
  
metastore/src/java/org/apache/hadoop/hive/metastore/datasource/DataSourceProvider.java
 PRE-CREATION 
  
metastore/src/java/org/apache/hadoop/hive/metastore/datasource/DataSourceProviderFactory.java
 PRE-CREATION 
  
metastore/src/java/org/apache/hadoop/hive/metastore/datasource/package-info.java
 PRE-CREATION 
  metastore/src/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java 
f3968e45c4bef2676c2000c17b3e8e6ea5761017 
  
metastore/src/test/org/apache/hadoop/hive/metastore/datasource/TestDataSourceProviderFactory.java
 PRE-CREATION 


Diff: https://reviews.apache.org/r/61712/diff/3/

Changes: https://reviews.apache.org/r/61712/diff/2-3/


Testing
---


Thanks,

Barna Zsombor Klara

Re: Review Request 61712: HIVE-17319: Make BoneCp configurable using hive properties in hive-site.xml

[Barna Zsombor Klara]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61712/
---

(Updated Aug. 18, 2017, 2:37 p.m.)


Review request for hive, Eugene Koifman, Peter Vary, and Thejas Nair.


Changes
---

Updated the patch based on the review and Yetus report. Thank you for the 
review Peter.


Bugs: HIVE-17319
https://issues.apache.org/jira/browse/HIVE-17319


Repository: hive-git


Description
---

HIVE-17319: Make BoneCp configurable using hive properties in hive-site.xml


Diffs (updated)
-

  metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java 
897fc4efd4c541ee2da30516c933647f5d4a0af4 
  
metastore/src/java/org/apache/hadoop/hive/metastore/datasource/BoneCPDataSourceProvider.java
 PRE-CREATION 
  
metastore/src/java/org/apache/hadoop/hive/metastore/datasource/DataSourceProvider.java
 PRE-CREATION 
  
metastore/src/java/org/apache/hadoop/hive/metastore/datasource/DataSourceProviderFactory.java
 PRE-CREATION 
  
metastore/src/java/org/apache/hadoop/hive/metastore/datasource/package-info.java
 PRE-CREATION 
  metastore/src/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java 
b722af6ceb8375c76c3a393cee79ed2387555b69 
  
metastore/src/test/org/apache/hadoop/hive/metastore/datasource/TestDataSourceProviderFactory.java
 PRE-CREATION 


Diff: https://reviews.apache.org/r/61712/diff/2/

Changes: https://reviews.apache.org/r/61712/diff/1-2/


Testing
---


Thanks,

Barna Zsombor Klara

[jira] [Created] (HIVE-17356) Missing ASF headers 3 classes

[Barna Zsombor Klara (JIRA)]

Barna Zsombor Klara created HIVE-17356:
--

 Summary: Missing ASF headers 3 classes
 Key: HIVE-17356
 URL: https://issues.apache.org/jira/browse/HIVE-17356
 Project: Hive
  Issue Type: Bug
Reporter: Barna Zsombor Klara
Assignee: Barna Zsombor Klara
Priority: Trivial


JSONAddNotNullConstraintMessage.java, BucketCodec.java, TaskTrackerTest.java 
are missing the ASF header that should be added.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Review Request 61712: HIVE-17319: Make BoneCp configurable using hive properties in hive-site.xml

[Barna Zsombor Klara]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61712/
---

Review request for hive, Eugene Koifman, Peter Vary, and Thejas Nair.


Bugs: HIVE-17319
https://issues.apache.org/jira/browse/HIVE-17319


Repository: hive-git


Description
---

HIVE-17319: Make BoneCp configurable using hive properties in hive-site.xml


Diffs
-

  metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java 
897fc4efd4c541ee2da30516c933647f5d4a0af4 
  
metastore/src/java/org/apache/hadoop/hive/metastore/datasource/BoneCPDataSourceProvider.java
 PRE-CREATION 
  
metastore/src/java/org/apache/hadoop/hive/metastore/datasource/DataSourceProvider.java
 PRE-CREATION 
  
metastore/src/java/org/apache/hadoop/hive/metastore/datasource/DataSourceProviderFactory.java
 PRE-CREATION 
  metastore/src/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java 
b722af6ceb8375c76c3a393cee79ed2387555b69 
  
metastore/src/test/org/apache/hadoop/hive/metastore/datasource/TestDataSourceProviderFactory.java
 PRE-CREATION 


Diff: https://reviews.apache.org/r/61712/diff/1/


Testing
---


Thanks,

Barna Zsombor Klara

Re: Review Request 60085: HIVE-14747: Remove JAVA paths from profiles by sending them from ptest-client

[Barna Zsombor Klara]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/60085/
---

(Updated Aug. 17, 2017, 7:56 a.m.)


Review request for hive, Marta Kuczora, Peter Vary, and Sergio Pena.


Changes
---

Added server-env.properties.example.


Bugs: HIVE-14747
https://issues.apache.org/jira/browse/HIVE-14747


Repository: hive-git


Description
---

HIVE-14747: Remove JAVA paths from profiles by sending them from ptest-client


Diffs (updated)
-

  testutils/ptest2/conf/server-env.properties.example PRE-CREATION 
  
testutils/ptest2/src/main/java/org/apache/hive/ptest/api/server/ExecutionController.java
 2f96ad03023e9f51d44d203f34edd04183605a22 
  
testutils/ptest2/src/main/java/org/apache/hive/ptest/api/server/TestExecutor.java
 b2c61f03c5bf5f170894141848c89fc26129115a 
  testutils/ptest2/src/main/java/org/apache/hive/ptest/execution/PTest.java 
1cdfdb309acd8282e593abd7ed10c87721926c60 
  
testutils/ptest2/src/main/java/org/apache/hive/ptest/execution/conf/Context.java
 14984bafdd18fb7636e729cc7fbbfa349b0f043e 
  
testutils/ptest2/src/main/java/org/apache/hive/ptest/execution/conf/ExecutionContextConfiguration.java
 35ddd44accf34be1f5957c6df31802ee8c8022b5 
  
testutils/ptest2/src/main/java/org/apache/hive/ptest/execution/conf/TestConfiguration.java
 e584f9c105fa134e3e267d6c6817d441b4c6b249 
  
testutils/ptest2/src/main/java/org/apache/hive/ptest/execution/conf/TestParser.java
 a243774e52f3f5fda4a082bb99387cf5808c307b 
  
testutils/ptest2/src/test/java/org/apache/hive/ptest/execution/TestJIRAService.java
 b97b890dfe855539de2696788327ba9b4a841ff3 
  
testutils/ptest2/src/test/java/org/apache/hive/ptest/execution/conf/TestContext.java
 PRE-CREATION 
  
testutils/ptest2/src/test/java/org/apache/hive/ptest/execution/conf/TestTestConfiguration.java
 848faf27af1ed8945d7013b6562bab544605e4bc 


Diff: https://reviews.apache.org/r/60085/diff/6/

Changes: https://reviews.apache.org/r/60085/diff/5-6/


Testing
---

Added unit tests for the Context class.


Thanks,

Barna Zsombor Klara

[jira] [Created] (HIVE-17337) Investigate the source of the flakiness of BeeLineTests when executed in parallel

[Barna Zsombor Klara (JIRA)]

Barna Zsombor Klara created HIVE-17337:
--

 Summary: Investigate the source of the flakiness of BeeLineTests 
when executed in parallel
 Key: HIVE-17337
 URL: https://issues.apache.org/jira/browse/HIVE-17337
 Project: Hive
  Issue Type: Improvement
Affects Versions: 3.0.0
Reporter: Barna Zsombor Klara


BeeLine qtests occasionally fail with unexplained diffs in the output when run 
in parallel.
HIVE-17322 removed the parallelisation, but we should investigate the root 
cause of the issue before we create many more BeeLine tests.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: Review Request 59402: HIVE-16718: Provide a way to pass in user supplied maven build and test arguments to Ptest

[Barna Zsombor Klara]

> On July 28, 2017, 7:01 p.m., Sahil Takiar wrote:
> > Overall it looks good. What about other Maven options though? Would it be 
> > difficult to update this patch so that any option for `mvn` can be passed 
> > in? Not just options specified by `-D`?

Sorry for the very late reply Sahil. I'm not sure how useful it would be to 
propagate any/other maven args, I can't think of a use case where we would want 
a specific maven arg on a per test profile or per test branch basis. And if one 
day we decide to globally change something, e.g. to increase the memory needed 
or the thread number, then we only need to edit one vm template in PTest. I 
guess that would not be a significantly bigger effort than to edit the shell 
script kicking off the PTest execution. But if you can share a more useful 
usecase, then of course I'd be happy to update the patch.


- Barna Zsombor


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59402/#review181710
---


On May 19, 2017, 3:10 p.m., Barna Zsombor Klara wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/59402/
> ---
> 
> (Updated May 19, 2017, 3:10 p.m.)
> 
> 
> Review request for hive, Sergio Pena and Sahil Takiar.
> 
> 
> Bugs: HIVE-16718
> https://issues.apache.org/jira/browse/HIVE-16718
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-16718: Provide a way to pass in user supplied maven build and test 
> arguments to Ptest
> 
> 
> Diffs
> -
> 
>   testutils/ptest2/src/main/java/org/apache/hive/ptest/execution/PTest.java 
> 1cdfdb309acd8282e593abd7ed10c87721926c60 
>   
> testutils/ptest2/src/main/java/org/apache/hive/ptest/execution/conf/TestConfiguration.java
>  e584f9c105fa134e3e267d6c6817d441b4c6b249 
> 
> 
> Diff: https://reviews.apache.org/r/59402/diff/1/
> 
> 
> Testing
> ---
> 
> Tested manually that I can pass in 0, 1 or 2 arguments to both mvnBuildArgs 
> and mvnTestArgs in PTest.
> Arguments will be key value pairs separated by '='. The '-D' will be added 
> automatically.
> 
> 
> Thanks,
> 
> Barna Zsombor Klara
> 
>

Re: Review Request 61656: HIVE-17316: Use regular expressions for the hidden configuration variables

[Barna Zsombor Klara]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61656/
---

(Updated Aug. 15, 2017, 3:30 p.m.)


Review request for hive and Peter Vary.


Changes
---

Fixed failing tests and updated based on review comments. Thank you Peter for 
the review!


Repository: hive-git


Description
---

HIVE-17316: Use regular expressions for the hidden configuration variables


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 
056f2d78346b6b306d34dfb610e3a7fed4ca68aa 
  common/src/java/org/apache/hadoop/hive/conf/HiveConfUtil.java 
9084fed9fa5121285709874923b3e03e887cb20d 
  common/src/test/org/apache/hadoop/hive/conf/TestHiveConfRestrictList.java 
980fa761541d72a0b37bc5d687e1fbc4551b5282 
  common/src/test/org/apache/hadoop/hive/conf/TestHiveConfUtil.java 
PRE-CREATION 
  
service/src/test/org/apache/hive/service/cli/session/TestSessionGlobalInitFile.java
 5760e66786b1521e439f7abc3597776449b6336d 


Diff: https://reviews.apache.org/r/61656/diff/2/

Changes: https://reviews.apache.org/r/61656/diff/1-2/


Testing
---

Added unit tests and checked that the current set of restricted variables will 
not restrict any new ones due to the change in the check from equals to 
contains.


Thanks,

Barna Zsombor Klara

Review Request 61656: HIVE-17316: Use regular expressions for the hidden configuration variables

[Barna Zsombor Klara]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61656/
---

Review request for hive and Peter Vary.


Repository: hive-git


Description
---

HIVE-17316: Use regular expressions for the hidden configuration variables


Diffs
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 
056f2d78346b6b306d34dfb610e3a7fed4ca68aa 
  common/src/java/org/apache/hadoop/hive/conf/HiveConfUtil.java 
9084fed9fa5121285709874923b3e03e887cb20d 
  common/src/test/org/apache/hadoop/hive/conf/TestHiveConfRestrictList.java 
980fa761541d72a0b37bc5d687e1fbc4551b5282 
  common/src/test/org/apache/hadoop/hive/conf/TestHiveConfUntil.java 
PRE-CREATION 


Diff: https://reviews.apache.org/r/61656/diff/1/


Testing
---

Added unit tests and checked that the current set of restricted variables will 
not restrict any new ones due to the change in the check from equals to 
contains.


Thanks,

Barna Zsombor Klara

[jira] [Created] (HIVE-17322) Execute BeeLine qtests in a serial manner to prevent flakyness

[Barna Zsombor Klara (JIRA)]

Barna Zsombor Klara created HIVE-17322:
--

 Summary: Execute BeeLine qtests in a serial manner to prevent 
flakyness
 Key: HIVE-17322
 URL: https://issues.apache.org/jira/browse/HIVE-17322
 Project: Hive
  Issue Type: Bug
Affects Versions: 3.0.0
Reporter: Barna Zsombor Klara
Assignee: Barna Zsombor Klara
Priority: Minor






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (HIVE-17319) Make BoneCp configurable using hive properties in hive-site.xml

[Barna Zsombor Klara (JIRA)]

Barna Zsombor Klara created HIVE-17319:
--

 Summary: Make BoneCp configurable using hive properties in 
hive-site.xml
 Key: HIVE-17319
 URL: https://issues.apache.org/jira/browse/HIVE-17319
 Project: Hive
  Issue Type: Sub-task
Reporter: Barna Zsombor Klara






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (HIVE-17318) Make Hikari configurable using hive properties in hive-site.xml

[Barna Zsombor Klara (JIRA)]

Barna Zsombor Klara created HIVE-17318:
--

 Summary: Make Hikari configurable using hive properties in 
hive-site.xml
 Key: HIVE-17318
 URL: https://issues.apache.org/jira/browse/HIVE-17318
 Project: Hive
  Issue Type: Sub-task
Reporter: Barna Zsombor Klara






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (HIVE-17317) Make Dbcp configurable using hive properties in hive-site.xml

[Barna Zsombor Klara (JIRA)]

Barna Zsombor Klara created HIVE-17317:
--

 Summary: Make Dbcp configurable using hive properties in 
hive-site.xml
 Key: HIVE-17317
 URL: https://issues.apache.org/jira/browse/HIVE-17317
 Project: Hive
  Issue Type: Sub-task
Reporter: Barna Zsombor Klara






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (HIVE-17316) Use regular expressions for the hidden configuration variables

[Barna Zsombor Klara (JIRA)]

Barna Zsombor Klara created HIVE-17316:
--

 Summary: Use regular expressions for the hidden configuration 
variables
 Key: HIVE-17316
 URL: https://issues.apache.org/jira/browse/HIVE-17316
 Project: Hive
  Issue Type: Sub-task
Reporter: Barna Zsombor Klara
Assignee: Barna Zsombor Klara


Currently HiveConf variables which should not be displayed to the user need to 
be enumerated. We should enhance this to be able to set regular expressions and 
any variable matching it should be hidden.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (HIVE-17315) Make the DataSource used by the DataNucleus in the HMS configurable using Hive properties

[Barna Zsombor Klara (JIRA)]

Barna Zsombor Klara created HIVE-17315:
--

 Summary: Make the DataSource used by the DataNucleus in the HMS 
configurable using Hive properties
 Key: HIVE-17315
 URL: https://issues.apache.org/jira/browse/HIVE-17315
 Project: Hive
  Issue Type: New Feature
Affects Versions: 3.0.0
Reporter: Barna Zsombor Klara
Assignee: Barna Zsombor Klara


Currently we may use several connection pool implementations in the backend 
(hikari, dbCp, boneCp) but these can only be configured using proprietary xml 
files and not through hive-site.xml like DataNucleus.
We should make them configurable just like DataNucleus, by allowing Hive 
properties prefix by hikari, dbcp, bonecp to be set in the hive-site.xml. 
However since these configurations may contain sensitive information 
(passwords) these properties should not be displayable or manually settable.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (HIVE-17305) New insert overwrite dynamic partitions qtest need to have the golden file regenerated

[Barna Zsombor Klara (JIRA)]

Barna Zsombor Klara created HIVE-17305:
--

 Summary: New insert overwrite dynamic partitions qtest need to 
have the golden file regenerated
 Key: HIVE-17305
 URL: https://issues.apache.org/jira/browse/HIVE-17305
 Project: Hive
  Issue Type: Bug
  Components: Tests
Affects Versions: 3.0.0
Reporter: Barna Zsombor Klara
Assignee: Barna Zsombor Klara
Priority: Trivial






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: Review Request 61497: HIVE-17267: Make HMS Notification Listeners typesafe

[Barna Zsombor Klara]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61497/
---

(Updated Aug. 8, 2017, 3:47 p.m.)


Review request for hive and Peter Vary.


Changes
---

Addressed comments. Thank you for the review Peter!


Bugs: HIVE-17267
https://issues.apache.org/jira/browse/HIVE-17267


Repository: hive-git


Description
---

HIVE-17267: Make HMS Notification Listeners typesafe


Diffs (updated)
-

  
hcatalog/server-extensions/src/main/java/org/apache/hive/hcatalog/listener/DbNotificationListener.java
 351546c3bc9edb69a435f04795b5ea6c3421f5b0 
  metastore/src/java/org/apache/hadoop/hive/metastore/HiveAlterHandler.java 
ea90a8e4a2d8a79f4356818b11cdbf9eb870256d 
  metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java 
e2a7d7df3fc072ca7eb7addb1fff4378d592df77 
  
metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreListenerNotifier.java
 37327f8bd2ebf3f883fb3bd408d63f776b26406e 
  
metastore/src/java/org/apache/hadoop/hive/metastore/TransactionalMetaStoreEventListener.java
 PRE-CREATION 


Diff: https://reviews.apache.org/r/61497/diff/2/

Changes: https://reviews.apache.org/r/61497/diff/1-2/


Testing
---


Thanks,

Barna Zsombor Klara

Re: Review Request 60085: HIVE-14747: Remove JAVA paths from profiles by sending them from ptest-client

[Barna Zsombor Klara]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/60085/
---

(Updated Aug. 8, 2017, 12:50 p.m.)


Review request for hive, Marta Kuczora, Peter Vary, and Sergio Pena.


Changes
---

Addressed comments from Sergio Pena. Thank you for the review!


Bugs: HIVE-14747
https://issues.apache.org/jira/browse/HIVE-14747


Repository: hive-git


Description
---

HIVE-14747: Remove JAVA paths from profiles by sending them from ptest-client


Diffs (updated)
-

  
testutils/ptest2/src/main/java/org/apache/hive/ptest/api/server/ExecutionController.java
 2f96ad03023e9f51d44d203f34edd04183605a22 
  
testutils/ptest2/src/main/java/org/apache/hive/ptest/api/server/TestExecutor.java
 b2c61f03c5bf5f170894141848c89fc26129115a 
  testutils/ptest2/src/main/java/org/apache/hive/ptest/execution/PTest.java 
1cdfdb309acd8282e593abd7ed10c87721926c60 
  
testutils/ptest2/src/main/java/org/apache/hive/ptest/execution/conf/Context.java
 14984bafdd18fb7636e729cc7fbbfa349b0f043e 
  
testutils/ptest2/src/main/java/org/apache/hive/ptest/execution/conf/ExecutionContextConfiguration.java
 35ddd44accf34be1f5957c6df31802ee8c8022b5 
  
testutils/ptest2/src/main/java/org/apache/hive/ptest/execution/conf/TestConfiguration.java
 e584f9c105fa134e3e267d6c6817d441b4c6b249 
  
testutils/ptest2/src/main/java/org/apache/hive/ptest/execution/conf/TestParser.java
 a243774e52f3f5fda4a082bb99387cf5808c307b 
  
testutils/ptest2/src/test/java/org/apache/hive/ptest/execution/TestJIRAService.java
 b97b890dfe855539de2696788327ba9b4a841ff3 
  
testutils/ptest2/src/test/java/org/apache/hive/ptest/execution/conf/TestContext.java
 PRE-CREATION 
  
testutils/ptest2/src/test/java/org/apache/hive/ptest/execution/conf/TestTestConfiguration.java
 848faf27af1ed8945d7013b6562bab544605e4bc 


Diff: https://reviews.apache.org/r/60085/diff/4/

Changes: https://reviews.apache.org/r/60085/diff/3-4/


Testing
---

Added unit tests for the Context class.


Thanks,

Barna Zsombor Klara

Review Request 61497: HIVE-17267: Make HMS Notification Listeners typesafe

[Barna Zsombor Klara]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61497/
---

Review request for hive and Peter Vary.


Bugs: HIVE-17267
https://issues.apache.org/jira/browse/HIVE-17267


Repository: hive-git


Description
---

HIVE-17267: Make HMS Notification Listeners typesafe


Diffs
-

  
hcatalog/server-extensions/src/main/java/org/apache/hive/hcatalog/listener/DbNotificationListener.java
 351546c3bc9edb69a435f04795b5ea6c3421f5b0 
  metastore/src/java/org/apache/hadoop/hive/metastore/HiveAlterHandler.java 
ea90a8e4a2d8a79f4356818b11cdbf9eb870256d 
  metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java 
6a6fd439d72fd5e24c881554c86480b0b3e19574 
  
metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreListenerNotifier.java
 37327f8bd2ebf3f883fb3bd408d63f776b26406e 
  
metastore/src/java/org/apache/hadoop/hive/metastore/TransactionalMetaStoreEventListener.java
 PRE-CREATION 


Diff: https://reviews.apache.org/r/61497/diff/1/


Testing
---


Thanks,

Barna Zsombor Klara

[jira] [Created] (HIVE-17267) Make HMS Notification Listeners typesafe

[Barna Zsombor Klara (JIRA)]

Barna Zsombor Klara created HIVE-17267:
--

 Summary: Make HMS Notification Listeners typesafe
 Key: HIVE-17267
 URL: https://issues.apache.org/jira/browse/HIVE-17267
 Project: Hive
  Issue Type: Bug
Reporter: Barna Zsombor Klara
Assignee: Barna Zsombor Klara


Currently in the HMS we support two types of notification listeners, 
transactional and non-transactional ones. Transactional listeners will only be 
invoked if the jdbc transaction finished successfully while non-transactional 
ones are supposed to be resilient and will be invoked in any case, even for 
failures.
Having the same type for these two is a source of confusion and opens the door 
for misconfigurations. We should try to fix this.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (HIVE-17263) Reduce debug logging for S3 tables

[Barna Zsombor Klara (JIRA)]

Barna Zsombor Klara created HIVE-17263:
--

 Summary: Reduce debug logging for S3 tables
 Key: HIVE-17263
 URL: https://issues.apache.org/jira/browse/HIVE-17263
 Project: Hive
  Issue Type: Improvement
Reporter: Barna Zsombor Klara
Assignee: Barna Zsombor Klara
Priority: Minor


When log level is set to debug operations accessing tables on amazon s3 will 
output a significant amount of logs, a lot of which is about the http 
communication (http headers and requests) which may not be that useful even for 
debugging purposes.
Since some ZooKeeper, Hadoop, DataNucleus etc. loggers are by default set to 
INFO+ levels I suggest we do the same for Apache Http and AWS. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (HIVE-17262) Use Hdfs API instead of directory listing to check snapshots for a path

[Barna Zsombor Klara (JIRA)]

Barna Zsombor Klara created HIVE-17262:
--

 Summary: Use Hdfs API instead of directory listing to check 
snapshots for a path
 Key: HIVE-17262
 URL: https://issues.apache.org/jira/browse/HIVE-17262
 Project: Hive
  Issue Type: Improvement
Affects Versions: 3.0.0
Reporter: Barna Zsombor Klara


HIVE-11667 introduced a check for snapshots of a Hdfs directory to enable 
truncating snapshotted Hive tables.
https://issues.apache.org/jira/browse/HDFS-12257 will introduce a public API to 
check for the same. Once it is resolved we should switch to it in Hive.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: Review Request 61380: HIVE-14786: Beeline displays binary column data as string instead of byte array

[Barna Zsombor Klara]

> On Aug. 3, 2017, 11:52 a.m., Peter Vary wrote:
> > beeline/src/java/org/apache/hive/beeline/Rows.java
> > Lines 164 (patched)
> > <https://reviews.apache.org/r/61380/diff/1/?file=1788407#file1788407line165>
> >
> > Why not calling the same o.toString() as before?

I'm not sure where you would like to have the o.toString. Arrays don't have 
their toString overridden so a direct call would result in nonsensical output.


- Barna Zsombor


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61380/#review182090
-------


On Aug. 2, 2017, 3:56 p.m., Barna Zsombor Klara wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61380/
> ---
> 
> (Updated Aug. 2, 2017, 3:56 p.m.)
> 
> 
> Review request for hive, Marta Kuczora and Peter Vary.
> 
> 
> Bugs: HIVE-14786
> https://issues.apache.org/jira/browse/HIVE-14786
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-14786: Beeline displays binary column data as string instead of byte 
> array
> 
> 
> Diffs
> -
> 
>   beeline/src/java/org/apache/hive/beeline/BeeLineOpts.java 
> 3ebbc9af9ba1a99dfc1d0af63ba362bae5eb2df4 
>   beeline/src/java/org/apache/hive/beeline/Rows.java 
> 924b9519a64427936101a9dc4bbe1831719194e6 
>   beeline/src/main/resources/BeeLine.properties 
> 3b8e3e6e9c94d88e5b05b136012aaa0e605262f1 
>   beeline/src/test/org/apache/hive/beeline/TestBufferedRows.java 
> f3f3d3a20cfd751b544636d86ad95e8ad7a2341d 
>   
> beeline/src/test/org/apache/hive/beeline/TestIncrementalRowsWithNormalization.java
>  68da841f850d2e97bf4b89071ec6d20ce8cf5d10 
>   beeline/src/test/org/apache/hive/beeline/TestTableOutputFormat.java 
> c7d9f8095cf56df957ebe2f50ed033a09bd4e31b 
> 
> 
> Diff: https://reviews.apache.org/r/61380/diff/2/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Barna Zsombor Klara
> 
>

Review Request 61380: HIVE-14786: Beeline displays binary column data as string instead of byte array

[Barna Zsombor Klara]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61380/
---

Review request for hive, Marta Kuczora and Peter Vary.


Bugs: HIVE-14786
https://issues.apache.org/jira/browse/HIVE-14786


Repository: hive-git


Description
---

HIVE-14786: Beeline displays binary column data as string instead of byte array


Diffs
-

  beeline/src/java/org/apache/hive/beeline/BeeLineOpts.java 
3ebbc9af9ba1a99dfc1d0af63ba362bae5eb2df4 
  beeline/src/java/org/apache/hive/beeline/Rows.java 
924b9519a64427936101a9dc4bbe1831719194e6 
  
beeline/src/test/org/apache/hive/beeline/TestIncrementalRowsWithNormalization.java
 68da841f850d2e97bf4b89071ec6d20ce8cf5d10 


Diff: https://reviews.apache.org/r/61380/diff/1/


Testing
---


Thanks,

Barna Zsombor Klara

Review Request 61379: HIVE-16294: Support snapshot for truncate table

[Barna Zsombor Klara]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61379/
---

Review request for hive, Marta Kuczora and Peter Vary.


Bugs: HIVE-16294
https://issues.apache.org/jira/browse/HIVE-16294


Repository: hive-git


Description
---

HIVE-16294: Support snapshot for truncate table


Diffs
-

  common/src/java/org/apache/hadoop/hive/common/FileUtils.java 
e8a3a7a49e31d02ba7ccb8774ea59c2cf0fea536 
  metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java 
6a6fd439d72fd5e24c881554c86480b0b3e19574 


Diff: https://reviews.apache.org/r/61379/diff/1/


Testing
---

Manual testing as automated testing would entail the creation of snapshots 
using hadoop which as far as I know is not supported with the current Hive 
testing framework.


Thanks,

Barna Zsombor Klara

Re: Review Request 60085: HIVE-14747: Remove JAVA paths from profiles by sending them from ptest-client

[Barna Zsombor Klara]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/60085/
---

(Updated July 31, 2017, 8:05 a.m.)


Review request for hive, Marta Kuczora, Peter Vary, and Sergio Pena.


Changes
---

Refactored based on offline comments from Sergio Pena.


Repository: hive-git


Description
---

HIVE-14747: Remove JAVA paths from profiles by sending them from ptest-client


Diffs (updated)
-

  
testutils/ptest2/src/main/java/org/apache/hive/ptest/api/server/ExecutionController.java
 2f96ad03023e9f51d44d203f34edd04183605a22 
  
testutils/ptest2/src/main/java/org/apache/hive/ptest/api/server/TestExecutor.java
 b2c61f03c5bf5f170894141848c89fc26129115a 
  testutils/ptest2/src/main/java/org/apache/hive/ptest/execution/PTest.java 
1cdfdb309acd8282e593abd7ed10c87721926c60 
  
testutils/ptest2/src/main/java/org/apache/hive/ptest/execution/conf/Context.java
 14984bafdd18fb7636e729cc7fbbfa349b0f043e 
  
testutils/ptest2/src/main/java/org/apache/hive/ptest/execution/conf/ExecutionContextConfiguration.java
 35ddd44accf34be1f5957c6df31802ee8c8022b5 
  
testutils/ptest2/src/main/java/org/apache/hive/ptest/execution/conf/TestConfiguration.java
 e584f9c105fa134e3e267d6c6817d441b4c6b249 
  
testutils/ptest2/src/main/java/org/apache/hive/ptest/execution/conf/TestParser.java
 a243774e52f3f5fda4a082bb99387cf5808c307b 
  
testutils/ptest2/src/test/java/org/apache/hive/ptest/execution/TestJIRAService.java
 b97b890dfe855539de2696788327ba9b4a841ff3 
  
testutils/ptest2/src/test/java/org/apache/hive/ptest/execution/conf/TestContext.java
 PRE-CREATION 
  
testutils/ptest2/src/test/java/org/apache/hive/ptest/execution/conf/TestTestConfiguration.java
 848faf27af1ed8945d7013b6562bab544605e4bc 


Diff: https://reviews.apache.org/r/60085/diff/3/

Changes: https://reviews.apache.org/r/60085/diff/2-3/


Testing (updated)
---

Added unit tests for the Context class.


Thanks,

Barna Zsombor Klara

Re: Review Request 60433: HIVE-16784: Missing lineage information when hive.blobstore.optimizations.enabled is true

[Barna Zsombor Klara]

> On July 25, 2017, 9:29 p.m., Sahil Takiar wrote:
> > ql/src/test/org/apache/hadoop/hive/ql/optimizer/TestGenMapRedUtilsCreateConditionalTask.java
> > Lines 57 (patched)
> > <https://reviews.apache.org/r/60433/diff/2/?file=1777640#file1777640line57>
> >
> > Whats this change for?

This is needed to prevent an NPE in GenMapRedUtils#mergeMovePaths. If the 
Session is not initialized Session.get() will return null.


- Barna Zsombor


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/60433/#review181386
---


On July 17, 2017, 2:08 p.m., Barna Zsombor Klara wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/60433/
> ---
> 
> (Updated July 17, 2017, 2:08 p.m.)
> 
> 
> Review request for hive, Marta Kuczora, Sergio Pena, and Sahil Takiar.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-16784: Missing lineage information when 
> hive.blobstore.optimizations.enabled is true
> 
> 
> Diffs
> -
> 
>   
> itests/hive-blobstore/src/test/results/clientpositive/insert_blobstore_to_blobstore.q.out
>  46518999358be9ec5191ad2c3b37f555efbec263 
>   
> itests/hive-blobstore/src/test/results/clientpositive/insert_empty_into_blobstore.q.out
>  8e5e09627b3a0f5c66a01fdd096b4b17b0b10689 
>   
> itests/hive-blobstore/src/test/results/clientpositive/insert_into_table.q.out 
> 663a57274840a84a5f04777540d09d9f9bcfb5da 
>   
> itests/hive-blobstore/src/test/results/clientpositive/insert_overwrite_table.q.out
>  8052fd86eefaff1fe7d024d3f58f6896e7223dd9 
>   
> itests/hive-blobstore/src/test/results/clientpositive/orc_format_nonpart.q.out
>  f4705687ff4b7b29d9bd6b81dfcb4065364c2965 
>   itests/hive-blobstore/src/test/results/clientpositive/orc_format_part.q.out 
> 1ef98104c5a711ef4b6553ca06fccf9003b99361 
>   
> itests/hive-blobstore/src/test/results/clientpositive/orc_nonstd_partitions_loc.q.out
>  9de4190072cc779b34e556a0e774ce7403324525 
>   
> itests/hive-blobstore/src/test/results/clientpositive/rcfile_format_nonpart.q.out
>  44a1f11fa2c7bd32c636b9bad4e4e35d79989dcf 
>   
> itests/hive-blobstore/src/test/results/clientpositive/rcfile_format_part.q.out
>  defca3bb78d4d97a327c389650d256a2cd880899 
>   
> itests/hive-blobstore/src/test/results/clientpositive/rcfile_nonstd_partitions_loc.q.out
>  5db9c7ef90c81524eae3eb6ec5346321e38b2a9e 
>   
> itests/hive-blobstore/src/test/results/clientpositive/zero_rows_blobstore.q.out
>  590f947add9b34d3adb05a7e22b0eb7d011c9a0d 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java 
> 88bf8253346bc92dac17c07373e6360be232 
>   ql/src/java/org/apache/hadoop/hive/ql/session/LineageState.java 
> 0f9506376c283ec6e6ab659b20a5718807268969 
>   
> ql/src/test/org/apache/hadoop/hive/ql/optimizer/TestGenMapRedUtilsCreateConditionalTask.java
>  68ccda9ef61744e369638ceec5c083c77524c2db 
> 
> 
> Diff: https://reviews.apache.org/r/60433/diff/2/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Barna Zsombor Klara
> 
>

Re: Review Request 60433: HIVE-16784: Missing lineage information when hive.blobstore.optimizations.enabled is true

[Barna Zsombor Klara]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/60433/#review181440
---




ql/src/test/org/apache/hadoop/hive/ql/optimizer/TestGenMapRedUtilsCreateConditionalTask.java
Lines 57 (patched)
<https://reviews.apache.org/r/60433/#comment257014>

This is needed to prevent an NPE in GenMapRedUtils#mergeMovePaths. If the 
Session is not initialized Session.get() will return null.


- Barna Zsombor Klara


On July 17, 2017, 2:08 p.m., Barna Zsombor Klara wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/60433/
> ---
> 
> (Updated July 17, 2017, 2:08 p.m.)
> 
> 
> Review request for hive, Marta Kuczora, Sergio Pena, and Sahil Takiar.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-16784: Missing lineage information when 
> hive.blobstore.optimizations.enabled is true
> 
> 
> Diffs
> -
> 
>   
> itests/hive-blobstore/src/test/results/clientpositive/insert_blobstore_to_blobstore.q.out
>  46518999358be9ec5191ad2c3b37f555efbec263 
>   
> itests/hive-blobstore/src/test/results/clientpositive/insert_empty_into_blobstore.q.out
>  8e5e09627b3a0f5c66a01fdd096b4b17b0b10689 
>   
> itests/hive-blobstore/src/test/results/clientpositive/insert_into_table.q.out 
> 663a57274840a84a5f04777540d09d9f9bcfb5da 
>   
> itests/hive-blobstore/src/test/results/clientpositive/insert_overwrite_table.q.out
>  8052fd86eefaff1fe7d024d3f58f6896e7223dd9 
>   
> itests/hive-blobstore/src/test/results/clientpositive/orc_format_nonpart.q.out
>  f4705687ff4b7b29d9bd6b81dfcb4065364c2965 
>   itests/hive-blobstore/src/test/results/clientpositive/orc_format_part.q.out 
> 1ef98104c5a711ef4b6553ca06fccf9003b99361 
>   
> itests/hive-blobstore/src/test/results/clientpositive/orc_nonstd_partitions_loc.q.out
>  9de4190072cc779b34e556a0e774ce7403324525 
>   
> itests/hive-blobstore/src/test/results/clientpositive/rcfile_format_nonpart.q.out
>  44a1f11fa2c7bd32c636b9bad4e4e35d79989dcf 
>   
> itests/hive-blobstore/src/test/results/clientpositive/rcfile_format_part.q.out
>  defca3bb78d4d97a327c389650d256a2cd880899 
>   
> itests/hive-blobstore/src/test/results/clientpositive/rcfile_nonstd_partitions_loc.q.out
>  5db9c7ef90c81524eae3eb6ec5346321e38b2a9e 
>   
> itests/hive-blobstore/src/test/results/clientpositive/zero_rows_blobstore.q.out
>  590f947add9b34d3adb05a7e22b0eb7d011c9a0d 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java 
> 88bf8253346bc92dac17c07373e6360be232 
>   ql/src/java/org/apache/hadoop/hive/ql/session/LineageState.java 
> 0f9506376c283ec6e6ab659b20a5718807268969 
>   
> ql/src/test/org/apache/hadoop/hive/ql/optimizer/TestGenMapRedUtilsCreateConditionalTask.java
>  68ccda9ef61744e369638ceec5c083c77524c2db 
> 
> 
> Diff: https://reviews.apache.org/r/60433/diff/2/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Barna Zsombor Klara
> 
>

[jira] [Created] (HIVE-17149) Hdfs directory is not cleared if partition creation failed on HMS

[Barna Zsombor Klara (JIRA)]

Barna Zsombor Klara created HIVE-17149:
--

 Summary: Hdfs directory is not cleared if partition creation 
failed on HMS
 Key: HIVE-17149
 URL: https://issues.apache.org/jira/browse/HIVE-17149
 Project: Hive
  Issue Type: Bug
  Components: Metastore
Affects Versions: 3.0.0
Reporter: Barna Zsombor Klara
Assignee: Barna Zsombor Klara


Hive#loadPartition will load a directory into a Hive Table Partition. It will 
alter the existing content of
the partition with the new contents and create a new partition if one does not 
exist.
The file move is performed before the partition creation and if the creation 
failes, the moved files are not cleared.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: Review Request 60433: HIVE-16784: Missing lineage information when hive.blobstore.optimizations.enabled is true

[Barna Zsombor Klara]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/60433/
---

(Updated July 17, 2017, 2:08 p.m.)


Review request for hive, Marta Kuczora, Sergio Pena, and Sahil Takiar.


Changes
---

It seems we cannot just rewrite the path for the lineage information as it may 
have been used already. Instead we will duplicate it any cleaner suggestion 
would be much appreciated.


Repository: hive-git


Description
---

HIVE-16784: Missing lineage information when 
hive.blobstore.optimizations.enabled is true


Diffs (updated)
-

  
itests/hive-blobstore/src/test/results/clientpositive/insert_blobstore_to_blobstore.q.out
 46518999358be9ec5191ad2c3b37f555efbec263 
  
itests/hive-blobstore/src/test/results/clientpositive/insert_empty_into_blobstore.q.out
 8e5e09627b3a0f5c66a01fdd096b4b17b0b10689 
  itests/hive-blobstore/src/test/results/clientpositive/insert_into_table.q.out 
663a57274840a84a5f04777540d09d9f9bcfb5da 
  
itests/hive-blobstore/src/test/results/clientpositive/insert_overwrite_table.q.out
 8052fd86eefaff1fe7d024d3f58f6896e7223dd9 
  
itests/hive-blobstore/src/test/results/clientpositive/orc_format_nonpart.q.out 
f4705687ff4b7b29d9bd6b81dfcb4065364c2965 
  itests/hive-blobstore/src/test/results/clientpositive/orc_format_part.q.out 
1ef98104c5a711ef4b6553ca06fccf9003b99361 
  
itests/hive-blobstore/src/test/results/clientpositive/orc_nonstd_partitions_loc.q.out
 9de4190072cc779b34e556a0e774ce7403324525 
  
itests/hive-blobstore/src/test/results/clientpositive/rcfile_format_nonpart.q.out
 44a1f11fa2c7bd32c636b9bad4e4e35d79989dcf 
  
itests/hive-blobstore/src/test/results/clientpositive/rcfile_format_part.q.out 
defca3bb78d4d97a327c389650d256a2cd880899 
  
itests/hive-blobstore/src/test/results/clientpositive/rcfile_nonstd_partitions_loc.q.out
 5db9c7ef90c81524eae3eb6ec5346321e38b2a9e 
  
itests/hive-blobstore/src/test/results/clientpositive/zero_rows_blobstore.q.out 
590f947add9b34d3adb05a7e22b0eb7d011c9a0d 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java 
88bf8253346bc92dac17c07373e6360be232 
  ql/src/java/org/apache/hadoop/hive/ql/session/LineageState.java 
0f9506376c283ec6e6ab659b20a5718807268969 
  
ql/src/test/org/apache/hadoop/hive/ql/optimizer/TestGenMapRedUtilsCreateConditionalTask.java
 68ccda9ef61744e369638ceec5c083c77524c2db 


Diff: https://reviews.apache.org/r/60433/diff/2/

Changes: https://reviews.apache.org/r/60433/diff/1-2/


Testing
---


Thanks,

Barna Zsombor Klara

Re: Review Request 60589: HIVE-17001: Insert overwrite table doesn't clean partition directory on HDFS if partition is missing from HMS

[Barna Zsombor Klara]

> On July 5, 2017, 7:03 p.m., Vihang Karajgaonkar wrote:
> > ql/src/test/queries/clientpositive/insert_overwrite_table.q
> > Lines 1-10 (patched)
> > <https://reviews.apache.org/r/60589/diff/1/?file=1768234#file1768234line1>
> >
> > I don't understand this test case completely. The table is defined as 
> > external so it is expected that the drop partition will not delete the HDFS 
> > file. The DFS operation is performed without the knowledge of Hive so when 
> > it returned 2 rows instead of 1 isn't it the expected behavior?
> > 
> > I think the right way to solve this problem to throw an exception when 
> > we do a insert overwrite on an external table. Just like truncate table 
> > command on an external table doesn't work, I think insert overwrite should 
> > also fail on a external table. The behavior of external table is 
> > inconsistent in my opinion. We allow it to be overwritten but not truncated.
> > 
> > When the table is a managed table, the test works as expected since 
> > Hive cleans up the directory after drop partition command.

The issue here is not really about external tables, I used it because I had to 
manually move/delete files for the testcase and I could not do it with tables 
on hdfs. The dfs command by default will look for the files on the local file 
system, and if I add the URI then I need to provide a host/port of a running 
HDFS instance which I don't have in a qtest. I could not find a working example 
in other qtests either so I went with the external table.


- Barna Zsombor


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/60589/#review179680
---


On July 3, 2017, 9:05 a.m., Barna Zsombor Klara wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/60589/
> ---
> 
> (Updated July 3, 2017, 9:05 a.m.)
> 
> 
> Review request for hive.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-17001: Insert overwrite table doesn't clean partition directory on HDFS 
> if partition is missing from HMS
> 
> 
> Diffs
> -
> 
>   ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java 
> 73710a7c2917b5268f788f22baaee2d87846961b 
>   ql/src/test/queries/clientpositive/insert_overwrite_table.q PRE-CREATION 
>   ql/src/test/results/clientpositive/insert_overwrite_table.q.out 
> PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/60589/diff/1/
> 
> 
> Testing
> ---
> 
> Manual testing and qtests.
> 
> 
> Thanks,
> 
> Barna Zsombor Klara
> 
>

[jira] [Created] (HIVE-17052) Remove logging of predicate filters

[Barna Zsombor Klara (JIRA)]

Barna Zsombor Klara created HIVE-17052:
--

 Summary: Remove logging of predicate filters
 Key: HIVE-17052
 URL: https://issues.apache.org/jira/browse/HIVE-17052
 Project: Hive
  Issue Type: Bug
  Components: Hive
Affects Versions: 3.0.0
Reporter: Barna Zsombor Klara


HIVE-16869 added the filter predicate to the debug log of HS2, but since these 
filters may contain sensitive information they should not be logged out.
The log statement should be changed back to the original form.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Review Request 60589: HIVE-17001: Insert overwrite table doesn't clean partition directory on HDFS if partition is missing from HMS

[Barna Zsombor Klara]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/60589/
---

Review request for hive.


Repository: hive-git


Description
---

HIVE-17001: Insert overwrite table doesn't clean partition directory on HDFS if 
partition is missing from HMS


Diffs
-

  ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java 
73710a7c2917b5268f788f22baaee2d87846961b 
  ql/src/test/queries/clientpositive/insert_overwrite_table.q PRE-CREATION 
  ql/src/test/results/clientpositive/insert_overwrite_table.q.out PRE-CREATION 


Diff: https://reviews.apache.org/r/60589/diff/1/


Testing
---

Manual testing and qtests.


Thanks,

Barna Zsombor Klara

[jira] [Created] (HIVE-17001) Insert overwrite table doesn't clean partition directory on HDFS if partition is missing from HMS

[Barna Zsombor Klara (JIRA)]

Barna Zsombor Klara created HIVE-17001:
--

 Summary: Insert overwrite table doesn't clean partition directory 
on HDFS if partition is missing from HMS
 Key: HIVE-17001
 URL: https://issues.apache.org/jira/browse/HIVE-17001
 Project: Hive
  Issue Type: Bug
  Components: HiveServer2, Metastore
Reporter: Barna Zsombor Klara
Assignee: Barna Zsombor Klara


Insert overwrite table should clear existing data before creating the new data 
files.
For a partitioned table we will clean any folder of existing partitions on 
HDFS, however if the partition folder exists only on HDFS and the partition 
definition is missing in HMS, the folder is not cleared.
Reproduction steps:
1. CREATE TABLE test( col1 string) PARTITIONED BY (ds string);
2. INSERT INTO test PARTITION(ds='p1') values ('a');
3. Copy the data to a different folder with different name.
4. ALTER TABLE test DROP PARTITION (ds='p1');
5. Recreate the partition directory, copy and rename the data file back
6. INSERT INTO test PARTITION(ds='p1') values ('b');
7. SELECT * from test;
will result in 2 records being returned instead of 1.




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: Review Request 60355: HIVE-16929 User-defined UDF functions can be registered as invariant functions

[Barna Zsombor Klara]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/60355/#review179085
---



Sorry for getting back this late and thank you for the updates. I don't want to 
be too nitpicky but I did have another comment about rewording a log line, 
sorry.
As for testing, you do have jars on the classpath during testing. So for 
example you can be pretty sure that the junit jar will be on your classpath 
somewhere, and you could write tests against ClassUtil using the junit packages.

But the patch LGTM.


ql/src/java/org/apache/hadoop/hive/ql/util/ClassUtil.java
Lines 85 (patched)
<https://reviews.apache.org/r/60355/#comment253520>

Nit:
I think what you meant should be one of the following:
- Exception occured while executing getJarFile
- Exception occured during the execution of getJarFile
- getJarFile encountered an exception

Same on line 90.


- Barna Zsombor Klara


On June 23, 2017, 3:56 a.m., ZhangBing Lin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/60355/
> ---
> 
> (Updated June 23, 2017, 3:56 a.m.)
> 
> 
> Review request for hive, cheng xu and Xuefu Zhang.
> 
> 
> Bugs: HIVE-16929
> https://issues.apache.org/jira/browse/HIVE-16929
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Add a configuration item "hive.aux.udf.package.name.list" in hive-site.xml, 
> which is a scan corresponding to the $HIVE_HOME/auxlib/ directory jar package 
> that contains the corresponding configuration package name under the class 
> registered as a constant function.
> Such as,
> 
>   hive.aux.udf.package.name.list
>   com.sample.udf,com.test.udf
> 
> Instructions:
>1, upload your jar file to $ HIVE_HOME/auxlib
>2, configure your UDF function corresponding to the package to the 
> following configuration parameters
>
> hive.aux.udf.package.name.list
> com.sample.udf
>
>   
>3, the configuration items need to be placed in the hive-site.xml file
>4, restart the Hive service to take effect
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java a8bdefdad6 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java 9795f3ef98 
>   ql/src/java/org/apache/hadoop/hive/ql/util/ClassUtil.java PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/util/UDFRegister.java PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/60355/diff/4/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> ZhangBing Lin
> 
>

Review Request 60433: HIVE-16784: Missing lineage information when hive.blobstore.optimizations.enabled is true

[Barna Zsombor Klara]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/60433/
---

Review request for hive, Marta Kuczora, Sergio Pena, and Sahil Takiar.


Repository: hive-git


Description
---

HIVE-16784: Missing lineage information when 
hive.blobstore.optimizations.enabled is true


Diffs
-

  
itests/hive-blobstore/src/test/results/clientpositive/insert_blobstore_to_blobstore.q.out
 46518999358be9ec5191ad2c3b37f555efbec263 
  
itests/hive-blobstore/src/test/results/clientpositive/insert_empty_into_blobstore.q.out
 8e5e09627b3a0f5c66a01fdd096b4b17b0b10689 
  itests/hive-blobstore/src/test/results/clientpositive/insert_into_table.q.out 
663a57274840a84a5f04777540d09d9f9bcfb5da 
  
itests/hive-blobstore/src/test/results/clientpositive/insert_overwrite_table.q.out
 8052fd86eefaff1fe7d024d3f58f6896e7223dd9 
  
itests/hive-blobstore/src/test/results/clientpositive/orc_format_nonpart.q.out 
f4705687ff4b7b29d9bd6b81dfcb4065364c2965 
  itests/hive-blobstore/src/test/results/clientpositive/orc_format_part.q.out 
1ef98104c5a711ef4b6553ca06fccf9003b99361 
  
itests/hive-blobstore/src/test/results/clientpositive/orc_nonstd_partitions_loc.q.out
 9de4190072cc779b34e556a0e774ce7403324525 
  
itests/hive-blobstore/src/test/results/clientpositive/rcfile_format_nonpart.q.out
 44a1f11fa2c7bd32c636b9bad4e4e35d79989dcf 
  
itests/hive-blobstore/src/test/results/clientpositive/rcfile_format_part.q.out 
defca3bb78d4d97a327c389650d256a2cd880899 
  
itests/hive-blobstore/src/test/results/clientpositive/rcfile_nonstd_partitions_loc.q.out
 5db9c7ef90c81524eae3eb6ec5346321e38b2a9e 
  
itests/hive-blobstore/src/test/results/clientpositive/zero_rows_blobstore.q.out 
590f947add9b34d3adb05a7e22b0eb7d011c9a0d 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java 
88bf8253346bc92dac17c07373e6360be232 
  ql/src/java/org/apache/hadoop/hive/ql/session/LineageState.java 
0f9506376c283ec6e6ab659b20a5718807268969 
  
ql/src/test/org/apache/hadoop/hive/ql/optimizer/TestGenMapRedUtilsCreateConditionalTask.java
 68ccda9ef61744e369638ceec5c083c77524c2db 


Diff: https://reviews.apache.org/r/60433/diff/1/


Testing
---


Thanks,

Barna Zsombor Klara

Re: Review Request 60432: HIVE-16845: INSERT OVERWRITE a table with dynamic partitions on S3 fails with NPE

[Barna Zsombor Klara]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/60432/#review178891
---



Thank you for the review Marta, I just had one comment.
Also do you think it makes sense to add tests for HoS as well or is this 
problem only MR related?


ql/src/java/org/apache/hadoop/hive/ql/plan/ConditionalResolverMergeFiles.java
Lines 304 (patched)
<https://reviews.apache.org/r/60432/#comment253227>

If this all depends on the *hive.blobstore.optimizations.enabled* could we 
use the HiveConf object and check that value instead of introducing a boolean 
instance variable?


- Barna Zsombor Klara


On June 26, 2017, 1:56 p.m., Marta Kuczora wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/60432/
> ---
> 
> (Updated June 26, 2017, 1:56 p.m.)
> 
> 
> Review request for hive and Sergio Pena.
> 
> 
> Bugs: HIVE-16845
> https://issues.apache.org/jira/browse/HIVE-16845
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> The following steps lead to the NPE in the 
> ConditionalResolverMergeFiles.generateActualTasks method:
> 
> In the GenMapRedUtils.createCondTask method, the tasks for the merge, move 
> and "merge and move" use cases are created and set as task list to the 
> ConditionalWork. Originally the moveOnlyMoveTask and the mergeAndMoveMoveTask 
> was created from the same moveWork, which was the dummyWork created like this 
> in the createMRWorkForMergingFiles method:
> 
> MoveWork dummyMv = new MoveWork(null, null, null,
>  new LoadFileDesc(fsInputDesc.getFinalDirName(), finalName, true, 
> null, null), false);
> 
> 
> Then in the ConditionalResolverMergeFiles.generateActualTasks method we get 
> these tasks and use them to create result "resTsks" list.
> 
> For the "merge and move" use case, the code looks like this:
> 
> if (toMove.size() > 0) {
> resTsks.add(mrAndMvTask);
> 
> MoveWork mvWork = (MoveWork) mvTask.getWork();
> LoadFileDesc lfd = mvWork.getLoadFileWork();
> 
> ...
> 
> LoadMultiFilesDesc lmfd = new LoadMultiFilesDesc(toMove,
> targetDirs, lfd.getIsDfsDir(), lfd.getColumns(), 
> lfd.getColumnTypes());
> mvWork.setLoadFileWork(null);
> mvWork.setLoadTableWork(null);
> mvWork.setMultiFilesDesc(lmfd);
>   }
> 
> It adds the mrAndMvTask task to the resTsks list and modifies the move work 
> to move all necessary files in one-step. The mrAndMvTask contains a move task 
> as child task, which is the same as the mvWork work. 
> 
> With the blobstore optimization on, the moveOnlyMoveTask task is created from 
> a different move work, not from the dummyMoveWork as before:
> 
> MoveWork workForMoveOnlyTask;
> if (shouldMergeMovePaths) {
>   workForMoveOnlyTask = mergeMovePaths(condInputPath, 
> moveTaskToLink.getWork());
> } else {
>   workForMoveOnlyTask = dummyMoveWork;
> }
> 
> ...
> 
> Task mergeOnlyMergeTask = 
> TaskFactory.get(mergeWork, conf);
> Task moveOnlyMoveTask = 
> TaskFactory.get(workForMoveOnlyTask, conf);
> Task mergeAndMoveMergeTask = 
> TaskFactory.get(mergeWork, conf);
> Task mergeAndMoveMoveTask = 
> TaskFactory.get(dummyMoveWork, conf);
> 
> Because of this the mvWork in the 
> ConditionalResolverMergeFiles.generateActualTasks method will also be 
> different. It has the LoadTableDesc variable set and not the LoadFileDesc, 
> that causes the NPE.
> 
> When the blobstore optimization is on and the move work is changed, we should 
> use the child move task of the mrAndMvTask in the generateActualTasks method, 
> instead of the mvTask. Not just to avoid the NPE, but because this is the 
> correct move task for the "merge and move" use case.
> 
> 
> Diffs
> -
> 
>   
> itests/hive-blobstore/src/test/queries/clientpositive/insert_overwrite_dynamic_partitions_merge_move.q
>  PRE-CREATION 
>   
> itests/hive-blobstore/src/test/queries/clientpositive/insert_overwrite_dynamic_partitions_merge_only.q
>  PRE-CREATION 
>   
> itests/hive-blobstore/src/test/queries/clientpositive/insert_overwrite_dynamic_partitions_move_only.q
>  PRE-CREATION 
>   
> itests/hive-blobstore/src/test/results/clientpositive/insert_overwrite_dynamic_partitions_merge_move.q.out
>  PRE-CREATION 
>   
> itests/hive-blobstore/src/test/results/clientpos

[jira] [Created] (HIVE-16959) Flaky Test : TestBeeLineDriver.testCliDriver[insert_overwrite_local_directory_1]

[Barna Zsombor Klara (JIRA)]

Barna Zsombor Klara created HIVE-16959:
--

 Summary: Flaky Test : 
TestBeeLineDriver.testCliDriver[insert_overwrite_local_directory_1]
 Key: HIVE-16959
 URL: https://issues.apache.org/jira/browse/HIVE-16959
 Project: Hive
  Issue Type: Sub-task
  Components: Hive
Reporter: Barna Zsombor Klara


Test failed on the pre-commit but runs locally.
Error Message

Client result comparison failed with error code = 1 while executing 
fname=insert_overwrite_local_directory_1
1172d1171
< k21=v21#k22=v22#k31=v31:foo2



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: Review Request 59446: HIVE-16559: Parquet schema evolution for partitioned tables may break if table and partition serdes differ

[Barna Zsombor Klara]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59446/
---

(Updated June 26, 2017, 11:48 a.m.)


Review request for hive and Sergio Pena.


Changes
---

Removed the new error message and added a warning to clarify the error before 
the exception.


Bugs: HIVE-16559
https://issues.apache.org/jira/browse/HIVE-16559


Repository: hive-git


Description
---

HIVE-16559: Parquet schema evolution for partitioned tables may break if table 
and partition serdes differ


Diffs (updated)
-

  ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 
87928ee930b5ee974d5e4144a584773a243f8d6f 
  ql/src/test/queries/clientnegative/parquet_alter_part_table_drop_columns.q 
PRE-CREATION 
  
ql/src/test/results/clientnegative/parquet_alter_part_table_drop_columns.q.out 
PRE-CREATION 


Diff: https://reviews.apache.org/r/59446/diff/3/

Changes: https://reviews.apache.org/r/59446/diff/2-3/


Testing
---

Added a negative qtest. Manually tested that no regression is caused for avro 
and textfile SerDes when columns are added or replaced in a partitioned table.


Thanks,

Barna Zsombor Klara

Re: Review Request 60355: HIVE-16929 User-defined UDF functions can be registered as invariant functions

[Barna Zsombor Klara]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/60355/#review178639
---



Thank you for the patch ZhangBing Lin. I only had a few minor comments and nits.
Since you are adding several utility methods, do you think it would be possible 
to add a few unit tests?
Thanks!


common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
Lines 3475 (patched)
<https://reviews.apache.org/r/60355/#comment252787>

nit: packagename*s*. I think you meant to use the plural.



ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java
Line 22 (original), 22 (patched)
<https://reviews.apache.org/r/60355/#comment252785>

Let's respect the Hive coding standards, please list all the imports 
instead of using the wildcard.



ql/src/java/org/apache/hadoop/hive/ql/util/ClassUtil.java
Lines 38 (patched)
<https://reviews.apache.org/r/60355/#comment252795>

Could we get a javadoc here? Based on the method at a glance I'm not sure 
if this method will list all implementing classes or just the ones from the 
same package.



ql/src/java/org/apache/hadoop/hive/ql/util/ClassUtil.java
Lines 56 (patched)
<https://reviews.apache.org/r/60355/#comment252797>

nit: Could we return an empty list instead on null to avoid NPEs?



ql/src/java/org/apache/hadoop/hive/ql/util/ClassUtil.java
Lines 110 (patched)
<https://reviews.apache.org/r/60355/#comment252796>

Can we log this out instead of just writing to the err stream? Same on line 
117,122 and 149.



ql/src/java/org/apache/hadoop/hive/ql/util/UDFRegister.java
Lines 40 (patched)
<https://reviews.apache.org/r/60355/#comment252791>

nit: Should this be a warning instead of info?


- Barna Zsombor Klara


On June 22, 2017, 3:18 a.m., ZhangBing Lin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/60355/
> ---
> 
> (Updated June 22, 2017, 3:18 a.m.)
> 
> 
> Review request for hive, cheng xu and Xuefu Zhang.
> 
> 
> Bugs: HIVE-16929
> https://issues.apache.org/jira/browse/HIVE-16929
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Add a configuration item "hive.aux.udf.package.name.list" in hive-site.xml, 
> which is a scan corresponding to the $HIVE_HOME/auxlib/ directory jar package 
> that contains the corresponding configuration package name under the class 
> registered as a constant function.
> Such as,
> 
>   hive.aux.udf.package.name.list
>   com.sample.udf,com.test.udf
> 
> Instructions:
>1, upload your jar file to $ HIVE_HOME/auxlib
>2, configure your UDF function corresponding to the package to the 
> following configuration parameters
>
> hive.aux.udf.package.name.list
> com.sample.udf
>
>   
>3, the configuration items need to be placed in the hive-site.xml file
>4, restart the Hive service to take effect
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java a8bdefdad6 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java 9795f3ef98 
>   ql/src/java/org/apache/hadoop/hive/ql/util/ClassUtil.java PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/util/UDFRegister.java PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/60355/diff/1/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> ZhangBing Lin
> 
>

[jira] [Created] (HIVE-16931) Flaky test: TestPerfCliDriver.query14, query16, query94

[Barna Zsombor Klara (JIRA)]

Barna Zsombor Klara created HIVE-16931:
--

 Summary: Flaky test: TestPerfCliDriver.query14, query16, query94
 Key: HIVE-16931
 URL: https://issues.apache.org/jira/browse/HIVE-16931
 Project: Hive
  Issue Type: Sub-task
Reporter: Barna Zsombor Klara


Diffs are similar for all 3 tests:
{code}
1c1
< Warning: Shuffle Join MERGEJOIN[107][tables = [$hdt$_2, $hdt$_3, $hdt$_1, 
$hdt$_4]] in Stage 'Reducer 17' is a cross product
---
> Warning: Shuffle Join MERGEJOIN[113][tables = [$hdt$_2, $hdt$_3, $hdt$_1, 
> $hdt$_4]] in Stage 'Reducer 18' is a cross product
65,74c65,74
< Reducer 13 <- Map 12 (SIMPLE_EDGE)
< Reducer 15 <- Map 14 (SIMPLE_EDGE), Reducer 18 (SIMPLE_EDGE)
< Reducer 16 <- Reducer 15 (SIMPLE_EDGE)
< Reducer 17 <- Map 14 (CUSTOM_SIMPLE_EDGE), Map 19 (CUSTOM_SIMPLE_EDGE), Map 
20 (CUSTOM_SIMPLE_EDGE), Map 21 (CUSTOM_SIMPLE_EDGE)
< Reducer 18 <- Reducer 17 (SIMPLE_EDGE)
< Reducer 2 <- Map 1 (SIMPLE_EDGE), Map 9 (SIMPLE_EDGE)
< Reducer 3 <- Map 10 (SIMPLE_EDGE), Reducer 2 (SIMPLE_EDGE)
< Reducer 4 <- Map 11 (SIMPLE_EDGE), Reducer 3 (SIMPLE_EDGE)
< Reducer 5 <- Reducer 13 (SIMPLE_EDGE), Reducer 4 (SIMPLE_EDGE)
< Reducer 6 <- Reducer 16 (SIMPLE_EDGE), Reducer 5 (SIMPLE_EDGE)
---
> Reducer 14 <- Map 13 (SIMPLE_EDGE)
> Reducer 16 <- Map 15 (SIMPLE_EDGE), Reducer 19 (SIMPLE_EDGE)
> Reducer 17 <- Reducer 16 (SIMPLE_EDGE)
> Reducer 18 <- Map 15 (CUSTOM_SIMPLE_EDGE), Map 20 (CUSTOM_SIMPLE_EDGE), Map 
> 21 (CUSTOM_SIMPLE_EDGE), Map 22 (CUSTOM_SIMPLE_EDGE)
> Reducer 19 <- Reducer 18 (SIMPLE_EDGE)
> Reducer 2 <- Map 1 (SIMPLE_EDGE), Map 10 (SIMPLE_EDGE)
> Reducer 3 <- Map 11 (SIMPLE_EDGE), Reducer 2 (SIMPLE_EDGE)
> Reducer 4 <- Map 12 (SIMPLE_EDGE), Reducer 3 (SIMPLE_EDGE)
> Reducer 5 <- Reducer 14 (SIMPLE_EDGE), Reducer 4 (SIMPLE_EDGE)
> Reducer 6 <- Reducer 17 (SIMPLE_EDGE), Reducer 5 (SIMPLE_EDGE)
76c76,77
< Reducer 8 <- Reducer 7 (SIMPLE_EDGE)
---
> Reducer 8 <- Reducer 7 (CUSTOM_SIMPLE_EDGE)
> Reducer 9 <- Reducer 8 (SIMPLE_EDGE)
82c83
{code}




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: Review Request 59446: HIVE-16559: Parquet schema evolution for partitioned tables may break if table and partition serdes differ

[Barna Zsombor Klara]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59446/
---

(Updated June 19, 2017, 9:52 a.m.)


Review request for hive and Sergio Pena.


Bugs: HIVE-16559
https://issues.apache.org/jira/browse/HIVE-16559


Repository: hive-git


Description
---

HIVE-16559: Parquet schema evolution for partitioned tables may break if table 
and partition serdes differ


Diffs (updated)
-

  ql/src/java/org/apache/hadoop/hive/ql/ErrorMsg.java 
6651900e79a5c3d4ad8329afbe3894544ce9f46e 
  ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 
87928ee930b5ee974d5e4144a584773a243f8d6f 
  ql/src/test/queries/clientnegative/parquet_alter_part_table_drop_columns.q 
PRE-CREATION 
  
ql/src/test/results/clientnegative/parquet_alter_part_table_drop_columns.q.out 
PRE-CREATION 


Diff: https://reviews.apache.org/r/59446/diff/2/

Changes: https://reviews.apache.org/r/59446/diff/1-2/


Testing (updated)
---

Added a negative qtest. Manually tested that no regression is caused for avro 
and textfile SerDes when columns are added or replaced in a partitioned table.


Thanks,

Barna Zsombor Klara

Re: Review Request 60006: HIVE-14746: Remove branch and repositories from profiles by sending them from ptest-client

[Barna Zsombor Klara]

> On June 14, 2017, 4:03 p.m., Sergio Pena wrote:
> > testutils/ptest2/src/main/java/org/apache/hive/ptest/api/client/PTestClient.java
> > Line 322 (original), 324 (patched)
> > <https://reviews.apache.org/r/60006/diff/2/?file=1749727#file1749727line324>
> >
> > Should we check that BRANCH is not empty before starting the test? 
> > 
> > Is this a required argument or optional?
> > 
> > If it is optional, what branch will use as default? We're removing the 
> > branch option from the profiles, so this will not exist.

Based on the scripts and the properties it seems like PTest also supports SVN 
as the SCM system alongside Git, but there branch is a not supported parameter. 
If we want to keep the compatibility with SVN then we cannot make branch 
mandatory (since at this point we don't know the SCM type in the PTestClient).


> On June 14, 2017, 4:03 p.m., Sergio Pena wrote:
> > testutils/ptest2/src/main/java/org/apache/hive/ptest/execution/PTest.java
> > Lines 108 (patched)
> > <https://reviews.apache.org/r/60006/diff/2/?file=1749730#file1749730line108>
> >
> > Why is a workingDirectoryWrapper needed?
> > 
> > Currently, Ptest works on a 'working' directory for any branch detected 
> > on the profile. Why do we want to change that for different branches?

I did this based on Siddharth Seth's comments on the Jira, where he mentioned 
that associating the working dir with the branch could prevent frequent 
rebasing when the checkout occurs. We don't need it for the original intent of 
the Jira. We can discuss if we want it or if the rebase would not be that 
problematic.


- Barna Zsombor


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/60006/#review177910
---


On June 13, 2017, 2:28 p.m., Barna Zsombor Klara wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/60006/
> ---
> 
> (Updated June 13, 2017, 2:28 p.m.)
> 
> 
> Review request for hive, Sergio Pena and Siddharth Seth.
> 
> 
> Bugs: HIVE-14746
> https://issues.apache.org/jira/browse/HIVE-14746
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-14746: Remove branch and repositories from profiles by sending them from 
> ptest-client
> 
> 
> Diffs
> -
> 
>   
> testutils/ptest2/src/main/java/org/apache/hive/ptest/api/client/PTestClient.java
>  8e2604d372ac29b94445b269f08423b058308efe 
>   
> testutils/ptest2/src/main/java/org/apache/hive/ptest/api/request/TestStartRequest.java
>  8deed52ae0307d4fc075654a4d75e6cb09a5d9db 
>   
> testutils/ptest2/src/main/java/org/apache/hive/ptest/api/server/TestExecutor.java
>  b2c61f03c5bf5f170894141848c89fc26129115a 
>   testutils/ptest2/src/main/java/org/apache/hive/ptest/execution/PTest.java 
> 1cdfdb309acd8282e593abd7ed10c87721926c60 
>   
> testutils/ptest2/src/main/java/org/apache/hive/ptest/execution/context/CloudExecutionContextProvider.java
>  8b82497bdaf43694e0e1552e125b5ffdce40f56c 
>   
> testutils/ptest2/src/main/java/org/apache/hive/ptest/execution/context/ExecutionContext.java
>  b09de1d4d930cf2d4d26b500f3457cea3fffa9ce 
>   
> testutils/ptest2/src/main/java/org/apache/hive/ptest/execution/context/FixedExecutionContextProvider.java
>  f7b50d6a61962d2727b5181215be9de2e64b05b7 
>   
> testutils/ptest2/src/main/java/org/apache/hive/ptest/execution/context/WorkingDirWrapper.java
>  PRE-CREATION 
>   
> testutils/ptest2/src/test/java/org/apache/hive/ptest/api/server/TestTestExecutor.java
>  a4a789b579305d9ed573d8c1fd0b6ce75787d50f 
>   
> testutils/ptest2/src/test/java/org/apache/hive/ptest/execution/conf/TestTestConfiguration.java
>  848faf27af1ed8945d7013b6562bab544605e4bc 
> 
> 
> Diff: https://reviews.apache.org/r/60006/diff/2/
> 
> 
> Testing
> ---
> 
> Manually tested the PTestClient with and without the branch argument.
> Updated and ran the unit tests.
> 
> 
> Thanks,
> 
> Barna Zsombor Klara
> 
>

Re: Review Request 60085: HIVE-14747: Remove JAVA paths from profiles by sending them from ptest-client

[Barna Zsombor Klara]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/60085/
---

(Updated June 14, 2017, 2:38 p.m.)


Review request for hive, Marta Kuczora, Peter Vary, and Sergio Pena.


Changes
---

Updated comments and javadoc.


Repository: hive-git


Description
---

HIVE-14747: Remove JAVA paths from profiles by sending them from ptest-client


Diffs (updated)
-

  testutils/ptest2/conf/deployed/master-mr2.properties 
05c040599172caf9d67c4e27affce12a335dab4b 
  
testutils/ptest2/src/main/java/org/apache/hive/ptest/api/client/PTestClient.java
 8e2604d372ac29b94445b269f08423b058308efe 
  
testutils/ptest2/src/main/java/org/apache/hive/ptest/api/request/TestStartRequest.java
 8deed52ae0307d4fc075654a4d75e6cb09a5d9db 
  
testutils/ptest2/src/main/java/org/apache/hive/ptest/api/server/TestExecutor.java
 b2c61f03c5bf5f170894141848c89fc26129115a 
  testutils/ptest2/src/main/java/org/apache/hive/ptest/execution/PTest.java 
1cdfdb309acd8282e593abd7ed10c87721926c60 
  
testutils/ptest2/src/main/java/org/apache/hive/ptest/execution/conf/TestConfiguration.java
 e584f9c105fa134e3e267d6c6817d441b4c6b249 
  
testutils/ptest2/src/test/java/org/apache/hive/ptest/execution/conf/TestTestConfiguration.java
 848faf27af1ed8945d7013b6562bab544605e4bc 


Diff: https://reviews.apache.org/r/60085/diff/2/

Changes: https://reviews.apache.org/r/60085/diff/1-2/


Testing
---

Manually tested PTestClient.
Updated unit tests for the code in TestConfiguration.


Thanks,

Barna Zsombor Klara

Review Request 60085: HIVE-14747: Remove JAVA paths from profiles by sending them from ptest-client

[Barna Zsombor Klara]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/60085/
---

Review request for hive, Peter Vary and Sergio Pena.


Repository: hive-git


Description
---

HIVE-14747: Remove JAVA paths from profiles by sending them from ptest-client


Diffs
-

  testutils/ptest2/conf/deployed/master-mr2.properties 
05c040599172caf9d67c4e27affce12a335dab4b 
  
testutils/ptest2/src/main/java/org/apache/hive/ptest/api/client/PTestClient.java
 8e2604d372ac29b94445b269f08423b058308efe 
  
testutils/ptest2/src/main/java/org/apache/hive/ptest/api/request/TestStartRequest.java
 8deed52ae0307d4fc075654a4d75e6cb09a5d9db 
  
testutils/ptest2/src/main/java/org/apache/hive/ptest/api/server/TestExecutor.java
 b2c61f03c5bf5f170894141848c89fc26129115a 
  testutils/ptest2/src/main/java/org/apache/hive/ptest/execution/PTest.java 
1cdfdb309acd8282e593abd7ed10c87721926c60 
  
testutils/ptest2/src/main/java/org/apache/hive/ptest/execution/conf/TestConfiguration.java
 e584f9c105fa134e3e267d6c6817d441b4c6b249 
  
testutils/ptest2/src/test/java/org/apache/hive/ptest/execution/conf/TestTestConfiguration.java
 848faf27af1ed8945d7013b6562bab544605e4bc 


Diff: https://reviews.apache.org/r/60085/diff/1/


Testing
---

Manually tested PTestClient.
Updated unit tests for the code in TestConfiguration.


Thanks,

Barna Zsombor Klara

Re: Review Request 59629: HIVE-14745: Remove jira user/password from profiles by using another command to submit results to jira

[Barna Zsombor Klara]

> On June 13, 2017, 10:20 a.m., Peter Vary wrote:
> > testutils/ptest2/src/main/java/org/apache/hive/ptest/api/client/PTestClient.java
> > Lines 92-93 (original), 120-121 (patched)
> > <https://reviews.apache.org/r/59629/diff/1/?file=1746782#file1746782line120>
> >
> > nit: If there is any useful comment, please changed the order back to 
> > the original :D

The HttpClient is no longer final, so I took it our from the list of final 
variables.


- Barna Zsombor


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59629/#review177723
-----------


On June 13, 2017, 3 p.m., Barna Zsombor Klara wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/59629/
> ---
> 
> (Updated June 13, 2017, 3 p.m.)
> 
> 
> Review request for hive, Sergio Pena and Siddharth Seth.
> 
> 
> Bugs: HIVE-14745
> https://issues.apache.org/jira/browse/HIVE-14745
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Added the ability to write data needed for the jira comment into json.
> Refactored the TestStatus response to contain the generated jira comment.
> Refactored the PTestClient to take the jira comment from the last TestStatus 
> response which is either a success or a failure and post it to the Jira 
> Service.
> 
> 
> Diffs
> -
> 
>   testutils/ptest2/conf/deployed/master-mr2.properties 
> 05c040599172caf9d67c4e27affce12a335dab4b 
>   
> testutils/ptest2/src/main/java/org/apache/hive/ptest/api/client/PTestClient.java
>  8e2604d372ac29b94445b269f08423b058308efe 
>   
> testutils/ptest2/src/main/java/org/apache/hive/ptest/api/common/JiraCommentPoster.java
>  PRE-CREATION 
>   
> testutils/ptest2/src/main/java/org/apache/hive/ptest/api/response/TestStatus.java
>  7dfadb42edbb8526b169d3034e01d0e8e8739fdf 
>   
> testutils/ptest2/src/main/java/org/apache/hive/ptest/api/server/ExecutionController.java
>  2f96ad03023e9f51d44d203f34edd04183605a22 
>   testutils/ptest2/src/main/java/org/apache/hive/ptest/api/server/Test.java 
> 7fbabd96d8c9a443bb5cfd2defe7a10510cf9a15 
>   
> testutils/ptest2/src/main/java/org/apache/hive/ptest/api/server/TestExecutor.java
>  b2c61f03c5bf5f170894141848c89fc26129115a 
>   
> testutils/ptest2/src/main/java/org/apache/hive/ptest/execution/JIRAService.java
>  ff1dbd4bafaba9a0e88321c6507a47bbe0421833 
>   testutils/ptest2/src/main/java/org/apache/hive/ptest/execution/PTest.java 
> 1cdfdb309acd8282e593abd7ed10c87721926c60 
>   
> testutils/ptest2/src/main/java/org/apache/hive/ptest/execution/conf/TestConfiguration.java
>  e584f9c105fa134e3e267d6c6817d441b4c6b249 
>   
> testutils/ptest2/src/test/java/org/apache/hive/ptest/api/client/TestPTestClient.java
>  PRE-CREATION 
>   
> testutils/ptest2/src/test/java/org/apache/hive/ptest/api/server/TestTestExecutor.java
>  a4a789b579305d9ed573d8c1fd0b6ce75787d50f 
>   
> testutils/ptest2/src/test/java/org/apache/hive/ptest/execution/TestJIRAService.java
>  b97b890dfe855539de2696788327ba9b4a841ff3 
> 
> 
> Diff: https://reviews.apache.org/r/59629/diff/2/
> 
> 
> Testing
> ---
> 
> Added unit tests for PTestClient and JiraService.
> Ran a test with a heavily dumbed down CloudExecutionContextProvider that 
> would create the drowns on the local machine isntead of a cloud povider to 
> test integration between PTestClient and the WebService.
> 
> 
> Thanks,
> 
> Barna Zsombor Klara
> 
>

Re: Review Request 59629: HIVE-14745: Remove jira user/password from profiles by using another command to submit results to jira

[Barna Zsombor Klara]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59629/
---

(Updated June 13, 2017, 3 p.m.)


Review request for hive, Sergio Pena and Siddharth Seth.


Changes
---

Updated based on Peter's review. Thank you!


Bugs: HIVE-14745
https://issues.apache.org/jira/browse/HIVE-14745


Repository: hive-git


Description
---

Added the ability to write data needed for the jira comment into json.
Refactored the TestStatus response to contain the generated jira comment.
Refactored the PTestClient to take the jira comment from the last TestStatus 
response which is either a success or a failure and post it to the Jira Service.


Diffs (updated)
-

  testutils/ptest2/conf/deployed/master-mr2.properties 
05c040599172caf9d67c4e27affce12a335dab4b 
  
testutils/ptest2/src/main/java/org/apache/hive/ptest/api/client/PTestClient.java
 8e2604d372ac29b94445b269f08423b058308efe 
  
testutils/ptest2/src/main/java/org/apache/hive/ptest/api/common/JiraCommentPoster.java
 PRE-CREATION 
  
testutils/ptest2/src/main/java/org/apache/hive/ptest/api/response/TestStatus.java
 7dfadb42edbb8526b169d3034e01d0e8e8739fdf 
  
testutils/ptest2/src/main/java/org/apache/hive/ptest/api/server/ExecutionController.java
 2f96ad03023e9f51d44d203f34edd04183605a22 
  testutils/ptest2/src/main/java/org/apache/hive/ptest/api/server/Test.java 
7fbabd96d8c9a443bb5cfd2defe7a10510cf9a15 
  
testutils/ptest2/src/main/java/org/apache/hive/ptest/api/server/TestExecutor.java
 b2c61f03c5bf5f170894141848c89fc26129115a 
  
testutils/ptest2/src/main/java/org/apache/hive/ptest/execution/JIRAService.java 
ff1dbd4bafaba9a0e88321c6507a47bbe0421833 
  testutils/ptest2/src/main/java/org/apache/hive/ptest/execution/PTest.java 
1cdfdb309acd8282e593abd7ed10c87721926c60 
  
testutils/ptest2/src/main/java/org/apache/hive/ptest/execution/conf/TestConfiguration.java
 e584f9c105fa134e3e267d6c6817d441b4c6b249 
  
testutils/ptest2/src/test/java/org/apache/hive/ptest/api/client/TestPTestClient.java
 PRE-CREATION 
  
testutils/ptest2/src/test/java/org/apache/hive/ptest/api/server/TestTestExecutor.java
 a4a789b579305d9ed573d8c1fd0b6ce75787d50f 
  
testutils/ptest2/src/test/java/org/apache/hive/ptest/execution/TestJIRAService.java
 b97b890dfe855539de2696788327ba9b4a841ff3 


Diff: https://reviews.apache.org/r/59629/diff/2/

Changes: https://reviews.apache.org/r/59629/diff/1-2/


Testing
---

Added unit tests for PTestClient and JiraService.
Ran a test with a heavily dumbed down CloudExecutionContextProvider that would 
create the drowns on the local machine isntead of a cloud povider to test 
integration between PTestClient and the WebService.


Thanks,

Barna Zsombor Klara

Re: Review Request 60006: HIVE-14746: Remove branch and repositories from profiles by sending them from ptest-client

[Barna Zsombor Klara]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/60006/
---

(Updated June 13, 2017, 2:28 p.m.)


Review request for hive, Sergio Pena and Siddharth Seth.


Changes
---

Updated based on Peter's comments. Thank you!


Bugs: HIVE-14746
https://issues.apache.org/jira/browse/HIVE-14746


Repository: hive-git


Description
---

HIVE-14746: Remove branch and repositories from profiles by sending them from 
ptest-client


Diffs (updated)
-

  
testutils/ptest2/src/main/java/org/apache/hive/ptest/api/client/PTestClient.java
 8e2604d372ac29b94445b269f08423b058308efe 
  
testutils/ptest2/src/main/java/org/apache/hive/ptest/api/request/TestStartRequest.java
 8deed52ae0307d4fc075654a4d75e6cb09a5d9db 
  
testutils/ptest2/src/main/java/org/apache/hive/ptest/api/server/TestExecutor.java
 b2c61f03c5bf5f170894141848c89fc26129115a 
  testutils/ptest2/src/main/java/org/apache/hive/ptest/execution/PTest.java 
1cdfdb309acd8282e593abd7ed10c87721926c60 
  
testutils/ptest2/src/main/java/org/apache/hive/ptest/execution/context/CloudExecutionContextProvider.java
 8b82497bdaf43694e0e1552e125b5ffdce40f56c 
  
testutils/ptest2/src/main/java/org/apache/hive/ptest/execution/context/ExecutionContext.java
 b09de1d4d930cf2d4d26b500f3457cea3fffa9ce 
  
testutils/ptest2/src/main/java/org/apache/hive/ptest/execution/context/FixedExecutionContextProvider.java
 f7b50d6a61962d2727b5181215be9de2e64b05b7 
  
testutils/ptest2/src/main/java/org/apache/hive/ptest/execution/context/WorkingDirWrapper.java
 PRE-CREATION 
  
testutils/ptest2/src/test/java/org/apache/hive/ptest/api/server/TestTestExecutor.java
 a4a789b579305d9ed573d8c1fd0b6ce75787d50f 
  
testutils/ptest2/src/test/java/org/apache/hive/ptest/execution/conf/TestTestConfiguration.java
 848faf27af1ed8945d7013b6562bab544605e4bc 


Diff: https://reviews.apache.org/r/60006/diff/2/

Changes: https://reviews.apache.org/r/60006/diff/1-2/


Testing
---

Manually tested the PTestClient with and without the branch argument.
Updated and ran the unit tests.


Thanks,

Barna Zsombor Klara

Re: Review Request 60006: HIVE-14746: Remove branch and repositories from profiles by sending them from ptest-client

[Barna Zsombor Klara]

> On June 13, 2017, 10:21 a.m., Peter Vary wrote:
> > testutils/ptest2/src/main/java/org/apache/hive/ptest/api/server/TestExecutor.java
> > Lines 118-120 (patched)
> > <https://reviews.apache.org/r/60006/diff/1/?file=1748269#file1748269line118>
> >
> > Is it possible to have a request without branch? Should not we use a 
> > default for this so we have a cleaner directory layout?

PTest can technically work with svn as well not just git, and the branch 
parameter is not supported for it. So always having a sensible default is a bit 
difficult.


- Barna Zsombor


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/60006/#review177720
-------


On June 13, 2017, 2:28 p.m., Barna Zsombor Klara wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/60006/
> ---
> 
> (Updated June 13, 2017, 2:28 p.m.)
> 
> 
> Review request for hive, Sergio Pena and Siddharth Seth.
> 
> 
> Bugs: HIVE-14746
> https://issues.apache.org/jira/browse/HIVE-14746
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-14746: Remove branch and repositories from profiles by sending them from 
> ptest-client
> 
> 
> Diffs
> -
> 
>   
> testutils/ptest2/src/main/java/org/apache/hive/ptest/api/client/PTestClient.java
>  8e2604d372ac29b94445b269f08423b058308efe 
>   
> testutils/ptest2/src/main/java/org/apache/hive/ptest/api/request/TestStartRequest.java
>  8deed52ae0307d4fc075654a4d75e6cb09a5d9db 
>   
> testutils/ptest2/src/main/java/org/apache/hive/ptest/api/server/TestExecutor.java
>  b2c61f03c5bf5f170894141848c89fc26129115a 
>   testutils/ptest2/src/main/java/org/apache/hive/ptest/execution/PTest.java 
> 1cdfdb309acd8282e593abd7ed10c87721926c60 
>   
> testutils/ptest2/src/main/java/org/apache/hive/ptest/execution/context/CloudExecutionContextProvider.java
>  8b82497bdaf43694e0e1552e125b5ffdce40f56c 
>   
> testutils/ptest2/src/main/java/org/apache/hive/ptest/execution/context/ExecutionContext.java
>  b09de1d4d930cf2d4d26b500f3457cea3fffa9ce 
>   
> testutils/ptest2/src/main/java/org/apache/hive/ptest/execution/context/FixedExecutionContextProvider.java
>  f7b50d6a61962d2727b5181215be9de2e64b05b7 
>   
> testutils/ptest2/src/main/java/org/apache/hive/ptest/execution/context/WorkingDirWrapper.java
>  PRE-CREATION 
>   
> testutils/ptest2/src/test/java/org/apache/hive/ptest/api/server/TestTestExecutor.java
>  a4a789b579305d9ed573d8c1fd0b6ce75787d50f 
>   
> testutils/ptest2/src/test/java/org/apache/hive/ptest/execution/conf/TestTestConfiguration.java
>  848faf27af1ed8945d7013b6562bab544605e4bc 
> 
> 
> Diff: https://reviews.apache.org/r/60006/diff/2/
> 
> 
> Testing
> ---
> 
> Manually tested the PTestClient with and without the branch argument.
> Updated and ran the unit tests.
> 
> 
> Thanks,
> 
> Barna Zsombor Klara
> 
>

Review Request 60006: HIVE-14746: Remove branch and repositories from profiles by sending them from ptest-client

[Barna Zsombor Klara]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/60006/
---

Review request for hive, Sergio Pena and Siddharth Seth.


Bugs: HIVE-14746
https://issues.apache.org/jira/browse/HIVE-14746


Repository: hive-git


Description
---

HIVE-14746: Remove branch and repositories from profiles by sending them from 
ptest-client


Diffs
-

  
testutils/ptest2/src/main/java/org/apache/hive/ptest/api/client/PTestClient.java
 8e2604d372ac29b94445b269f08423b058308efe 
  
testutils/ptest2/src/main/java/org/apache/hive/ptest/api/request/TestStartRequest.java
 8deed52ae0307d4fc075654a4d75e6cb09a5d9db 
  
testutils/ptest2/src/main/java/org/apache/hive/ptest/api/server/TestExecutor.java
 b2c61f03c5bf5f170894141848c89fc26129115a 
  testutils/ptest2/src/main/java/org/apache/hive/ptest/execution/PTest.java 
1cdfdb309acd8282e593abd7ed10c87721926c60 
  
testutils/ptest2/src/main/java/org/apache/hive/ptest/execution/context/CloudExecutionContextProvider.java
 8b82497bdaf43694e0e1552e125b5ffdce40f56c 
  
testutils/ptest2/src/main/java/org/apache/hive/ptest/execution/context/ExecutionContext.java
 b09de1d4d930cf2d4d26b500f3457cea3fffa9ce 
  
testutils/ptest2/src/main/java/org/apache/hive/ptest/execution/context/FixedExecutionContextProvider.java
 f7b50d6a61962d2727b5181215be9de2e64b05b7 
  
testutils/ptest2/src/main/java/org/apache/hive/ptest/execution/context/WorkingDirWrapper.java
 PRE-CREATION 
  
testutils/ptest2/src/test/java/org/apache/hive/ptest/api/server/TestTestExecutor.java
 a4a789b579305d9ed573d8c1fd0b6ce75787d50f 
  
testutils/ptest2/src/test/java/org/apache/hive/ptest/execution/conf/TestTestConfiguration.java
 848faf27af1ed8945d7013b6562bab544605e4bc 


Diff: https://reviews.apache.org/r/60006/diff/1/


Testing
---

Manually tested the PTestClient with and without the branch argument.
Updated and ran the unit tests.


Thanks,

Barna Zsombor Klara

Review Request 59629: HIVE-14745: Remove jira user/password from profiles by using another command to submit results to jira

[Barna Zsombor Klara]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59629/
---

Review request for hive, Sergio Pena and Siddharth Seth.


Bugs: HIVE-14745
https://issues.apache.org/jira/browse/HIVE-14745


Repository: hive-git


Description
---

Added the ability to write data needed for the jira comment into json.
Refactored the TestStatus response to contain the generated jira comment.
Refactored the PTestClient to take the jira comment from the last TestStatus 
response which is either a success or a failure and post it to the Jira Service.


Diffs
-

  testutils/ptest2/conf/deployed/master-mr2.properties 
05c040599172caf9d67c4e27affce12a335dab4b 
  
testutils/ptest2/src/main/java/org/apache/hive/ptest/api/client/PTestClient.java
 8e2604d372ac29b94445b269f08423b058308efe 
  
testutils/ptest2/src/main/java/org/apache/hive/ptest/api/common/JiraCommentPoster.java
 PRE-CREATION 
  
testutils/ptest2/src/main/java/org/apache/hive/ptest/api/response/TestStatus.java
 7dfadb42edbb8526b169d3034e01d0e8e8739fdf 
  
testutils/ptest2/src/main/java/org/apache/hive/ptest/api/server/ExecutionController.java
 2f96ad03023e9f51d44d203f34edd04183605a22 
  testutils/ptest2/src/main/java/org/apache/hive/ptest/api/server/Test.java 
7fbabd96d8c9a443bb5cfd2defe7a10510cf9a15 
  
testutils/ptest2/src/main/java/org/apache/hive/ptest/api/server/TestExecutor.java
 b2c61f03c5bf5f170894141848c89fc26129115a 
  
testutils/ptest2/src/main/java/org/apache/hive/ptest/execution/JIRAService.java 
ff1dbd4bafaba9a0e88321c6507a47bbe0421833 
  testutils/ptest2/src/main/java/org/apache/hive/ptest/execution/PTest.java 
1cdfdb309acd8282e593abd7ed10c87721926c60 
  
testutils/ptest2/src/main/java/org/apache/hive/ptest/execution/conf/TestConfiguration.java
 e584f9c105fa134e3e267d6c6817d441b4c6b249 
  
testutils/ptest2/src/test/java/org/apache/hive/ptest/api/client/TestPTestClient.java
 PRE-CREATION 
  
testutils/ptest2/src/test/java/org/apache/hive/ptest/api/server/TestTestExecutor.java
 a4a789b579305d9ed573d8c1fd0b6ce75787d50f 
  
testutils/ptest2/src/test/java/org/apache/hive/ptest/execution/TestJIRAService.java
 b97b890dfe855539de2696788327ba9b4a841ff3 


Diff: https://reviews.apache.org/r/59629/diff/1/


Testing
---

Added unit tests for PTestClient and JiraService.
Ran a test with a heavily dumbed down CloudExecutionContextProvider that would 
create the drowns on the local machine isntead of a cloud povider to test 
integration between PTestClient and the WebService.


Thanks,

Barna Zsombor Klara

Review Request 59446: HIVE-16559: Parquet schema evolution for partitioned tables may break if table and partition serdes differ

[Barna Zsombor Klara]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59446/
---

Review request for hive and Sergio Pena.


Bugs: HIVE-16559
https://issues.apache.org/jira/browse/HIVE-16559


Repository: hive-git


Description
---

HIVE-16559: Parquet schema evolution for partitioned tables may break if table 
and partition serdes differ


Diffs
-

  ql/src/test/queries/clientpositive/parquet_schema_evolution.q 
e767b8186ead8e005ef4ee40f583a8afc48fa175 
  ql/src/test/results/clientpositive/parquet_schema_evolution.q.out 
43d75dc105ff104946d093d7c5e3c9de1cdcb196 
  serde/src/java/org/apache/hadoop/hive/serde2/AbstractSerDe.java 
a2a85b3dc44d7d7d676ad17eb476edbb5560f1cc 
  serde/src/java/org/apache/hadoop/hive/serde2/SerDeUtils.java 
b9d23dd72580a684788747bfdff4c35b08726fba 
  serde/src/test/org/apache/hadoop/hive/serde2/TestSerDeUtils.java PRE-CREATION 


Diff: https://reviews.apache.org/r/59446/diff/1/


Testing
---

Added unit tests and a qtest. Manually tested that no regression is caused for 
avro and textfile SerDes when columns are added or replaced in a partitioned 
table.


Thanks,

Barna Zsombor Klara

Review Request 59402: HIVE-16718: Provide a way to pass in user supplied maven build and test arguments to Ptest

[Barna Zsombor Klara]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59402/
---

Review request for hive, Sergio Pena and Sahil Takiar.


Bugs: HIVE-16718
https://issues.apache.org/jira/browse/HIVE-16718


Repository: hive-git


Description
---

HIVE-16718: Provide a way to pass in user supplied maven build and test 
arguments to Ptest


Diffs
-

  testutils/ptest2/src/main/java/org/apache/hive/ptest/execution/PTest.java 
1cdfdb309acd8282e593abd7ed10c87721926c60 
  
testutils/ptest2/src/main/java/org/apache/hive/ptest/execution/conf/TestConfiguration.java
 e584f9c105fa134e3e267d6c6817d441b4c6b249 


Diff: https://reviews.apache.org/r/59402/diff/1/


Testing
---

Tested manually that I can pass in 0, 1 or 2 arguments to both mvnBuildArgs and 
mvnTestArgs in PTest.
Arguments will be key value pairs separated by '='. The '-D' will be added 
automatically.


Thanks,

Barna Zsombor Klara

[jira] [Created] (HIVE-16718) Provide a way to pass in user supplied maven build and test arguments to Ptest

[Barna Zsombor Klara (JIRA)]

Barna Zsombor Klara created HIVE-16718:
--

 Summary: Provide a way to pass in user supplied maven build and 
test arguments to Ptest
 Key: HIVE-16718
 URL: https://issues.apache.org/jira/browse/HIVE-16718
 Project: Hive
  Issue Type: New Feature
Reporter: Barna Zsombor Klara
Assignee: Barna Zsombor Klara
Priority: Minor
 Fix For: 3.0.0


Currently we can only pass in maven build and test arguments from the 
properties file, so all of them need to be hardcoded.
We should find a way to pass in arguments from the command line.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (HIVE-16679) Missing ASF header on properties file in ptest2 project

[Barna Zsombor Klara (JIRA)]

Barna Zsombor Klara created HIVE-16679:
--

 Summary: Missing ASF header on properties file in ptest2 project
 Key: HIVE-16679
 URL: https://issues.apache.org/jira/browse/HIVE-16679
 Project: Hive
  Issue Type: Bug
Reporter: Barna Zsombor Klara
Assignee: Barna Zsombor Klara
Priority: Trivial
 Fix For: 3.0.0


The ASF header is missing on 
{{testutils/ptest2//conf/deployed/master-mr2.properties}} causing the build of 
the ptest2 project to fail on a RAT check.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: Review Request 59096: HIVE-16607 ColumnStatsAutoGatherContext regenerates HiveConf.HIVEQUERYID

[Barna Zsombor Klara]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59096/#review174438
---


Ship it!




Ship It!

- Barna Zsombor Klara


On May 10, 2017, 9:17 a.m., Peter Vary wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/59096/
> ---
> 
> (Updated May 10, 2017, 9:17 a.m.)
> 
> 
> Review request for hive, Aihua Xu and pengcheng xiong.
> 
> 
> Bugs: HIVE-16607
> https://issues.apache.org/jira/browse/HIVE-16607
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> When creating a QueryState object the caller could specify if new QueryID 
> should be created or the exisiting should be used.
> Created a QueryStateBuilder to make the QueryState object creation more 
> readable.
> New QueryId is only created in two places:
> - Driver constructor
> - Operation constructor
> Otherwise the existing queryId is used
> 
> 
> Diffs
> -
> 
>   
> hcatalog/core/src/test/java/org/apache/hive/hcatalog/mapreduce/TestHCatMultiOutputFormat.java
>  6ff48ee 
>   itests/src/test/resources/testconfiguration.properties 5ab3076 
>   itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java b897ffa 
>   itests/util/src/main/java/org/apache/hive/beeline/QFile.java 3d9ca99 
>   itests/util/src/main/java/org/apache/hive/beeline/QFileBeeLineClient.java 
> 7c50e18 
>   ql/src/java/org/apache/hadoop/hive/ql/Driver.java 29cce9a 
>   ql/src/java/org/apache/hadoop/hive/ql/QueryState.java 6dfaa9f 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java cf575de 
>   ql/src/java/org/apache/hadoop/hive/ql/io/rcfile/stats/PartialScanTask.java 
> 77bce97 
>   
> ql/src/java/org/apache/hadoop/hive/ql/metadata/HiveMaterializedViewsRegistry.java
>  b121eea 
>   
> ql/src/java/org/apache/hadoop/hive/ql/parse/ColumnStatsAutoGatherContext.java 
> 3b719af 
>   ql/src/test/org/apache/hadoop/hive/ql/exec/TestExecDriver.java c7266bc 
>   ql/src/test/org/apache/hadoop/hive/ql/parse/TestMacroSemanticAnalyzer.java 
> c734988 
>   ql/src/test/org/apache/hadoop/hive/ql/parse/TestQBCompact.java 201622e 
>   
> ql/src/test/org/apache/hadoop/hive/ql/parse/TestQBJoinTreeApplyPredicate.java 
> e607f10 
>   ql/src/test/org/apache/hadoop/hive/ql/parse/TestQBSubQuery.java 2674835 
>   
> ql/src/test/org/apache/hadoop/hive/ql/parse/TestReplicationSemanticAnalyzer.java
>  80865bd 
>   
> ql/src/test/org/apache/hadoop/hive/ql/parse/TestSemanticAnalyzerFactory.java 
> 5849950 
>   
> ql/src/test/org/apache/hadoop/hive/ql/parse/TestUpdateDeleteSemanticAnalyzer.java
>  a573808 
>   
> ql/src/test/org/apache/hadoop/hive/ql/parse/authorization/TestHiveAuthorizationTaskFactory.java
>  58cb4b4 
>   
> ql/src/test/org/apache/hadoop/hive/ql/parse/authorization/TestPrivilegesV1.java
>  5d01080 
>   
> ql/src/test/org/apache/hadoop/hive/ql/parse/authorization/TestPrivilegesV2.java
>  c552ba7 
>   
> ql/src/test/results/clientpositive/beeline/materialized_view_create_rewrite.q.out
>  PRE-CREATION 
>   service/src/java/org/apache/hive/service/cli/operation/Operation.java 
> 0b27608 
>   service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java 
> 0b51591 
> 
> 
> Diff: https://reviews.apache.org/r/59096/diff/3/
> 
> 
> Testing
> ---
> 
> Added new BeeLine test - The original code made the test output different 
> from the Cli test output, since the QueryLog was truncated when the queryId 
> was changed. After the change the BeeLine test output is exactly the same as 
> the Cli output.
> 
> 
> Thanks,
> 
> Peter Vary
> 
>

Re: Review Request 59096: HIVE-16607 ColumnStatsAutoGatherContext regenerates HiveConf.HIVEQUERYID

[Barna Zsombor Klara]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59096/#review174326
---



Thanks for the patch Peter. LGTM, with some minor comments/questions.


ql/src/java/org/apache/hadoop/hive/ql/QueryState.java
Line 32 (original), 31 (patched)
<https://reviews.apache.org/r/59096/#comment247460>

I'm not 100% against the current solution, but if possible I would rather 
see the queryId and maybe the queryString as instance variables of the 
QueryState. Preferably immutable, final ones. Currently we hand out the 
queryConf so it may end up being modified, which we should probably prevent.



ql/src/java/org/apache/hadoop/hive/ql/QueryState.java
Lines 157 (patched)
<https://reviews.apache.org/r/59096/#comment247459>

Is this side effect intended? It probably should be modified on the 
queryConf.


- Barna Zsombor Klara


On May 9, 2017, 3:05 p.m., Peter Vary wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/59096/
> ---
> 
> (Updated May 9, 2017, 3:05 p.m.)
> 
> 
> Review request for hive, Aihua Xu and pengcheng xiong.
> 
> 
> Bugs: HIVE-16607
> https://issues.apache.org/jira/browse/HIVE-16607
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> When creating a QueryState object the caller could specify if new QueryID 
> should be created or the exisiting should be used.
> Created a QueryStateBuilder to make the QueryState object creation more 
> readable.
> New QueryId is only created in two places:
> - Driver constructor
> - Operation constructor
> Otherwise the existing queryId is used
> 
> 
> Diffs
> -
> 
>   
> hcatalog/core/src/test/java/org/apache/hive/hcatalog/mapreduce/TestHCatMultiOutputFormat.java
>  6ff48ee 
>   itests/src/test/resources/testconfiguration.properties 5ab3076 
>   itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java b897ffa 
>   itests/util/src/main/java/org/apache/hive/beeline/QFile.java 3d9ca99 
>   itests/util/src/main/java/org/apache/hive/beeline/QFileBeeLineClient.java 
> 7c50e18 
>   ql/src/java/org/apache/hadoop/hive/ql/Driver.java 29cce9a 
>   ql/src/java/org/apache/hadoop/hive/ql/QueryState.java 6dfaa9f 
>   ql/src/java/org/apache/hadoop/hive/ql/io/rcfile/stats/PartialScanTask.java 
> 77bce97 
>   
> ql/src/java/org/apache/hadoop/hive/ql/metadata/HiveMaterializedViewsRegistry.java
>  b121eea 
>   
> ql/src/java/org/apache/hadoop/hive/ql/parse/ColumnStatsAutoGatherContext.java 
> 3b719af 
>   ql/src/test/org/apache/hadoop/hive/ql/exec/TestExecDriver.java c7266bc 
>   ql/src/test/org/apache/hadoop/hive/ql/parse/TestMacroSemanticAnalyzer.java 
> c734988 
>   ql/src/test/org/apache/hadoop/hive/ql/parse/TestQBCompact.java 201622e 
>   
> ql/src/test/org/apache/hadoop/hive/ql/parse/TestQBJoinTreeApplyPredicate.java 
> e607f10 
>   ql/src/test/org/apache/hadoop/hive/ql/parse/TestQBSubQuery.java 2674835 
>   
> ql/src/test/org/apache/hadoop/hive/ql/parse/TestReplicationSemanticAnalyzer.java
>  80865bd 
>   
> ql/src/test/org/apache/hadoop/hive/ql/parse/TestSemanticAnalyzerFactory.java 
> 5849950 
>   
> ql/src/test/org/apache/hadoop/hive/ql/parse/TestUpdateDeleteSemanticAnalyzer.java
>  a573808 
>   
> ql/src/test/org/apache/hadoop/hive/ql/parse/authorization/TestHiveAuthorizationTaskFactory.java
>  58cb4b4 
>   
> ql/src/test/org/apache/hadoop/hive/ql/parse/authorization/TestPrivilegesV1.java
>  5d01080 
>   
> ql/src/test/org/apache/hadoop/hive/ql/parse/authorization/TestPrivilegesV2.java
>  c552ba7 
>   
> ql/src/test/results/clientpositive/beeline/materialized_view_create_rewrite.q.out
>  PRE-CREATION 
>   service/src/java/org/apache/hive/service/cli/operation/Operation.java 
> 0b27608 
> 
> 
> Diff: https://reviews.apache.org/r/59096/diff/1/
> 
> 
> Testing
> ---
> 
> Added new BeeLine test - The original code made the test output different 
> from the Cli test output, since the QueryLog was truncated when the queryId 
> was changed. After the change the BeeLine test output is exactly the same as 
> the Cli output.
> 
> 
> Thanks,
> 
> Peter Vary
> 
>

Re: Review Request 58501: HIVE-16469: Parquet timestamp table property is not always taken into account

[Barna Zsombor Klara]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/58501/
---

(Updated May 4, 2017, 10:19 a.m.)


Review request for hive, Sergio Pena and Zoltan Ivanfi.


Changes
---

Updated based on the comments. Thanks Sergio Pena, Vihang Karajgaonkar, Cheng 
Xu for the reviews.


Bugs: HIVE-16469
https://issues.apache.org/jira/browse/HIVE-16469


Repository: hive-git


Description
---

HIVE-16469: Parquet timestamp table property is not always taken into account


Diffs (updated)
-

  ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 
757b7fc0eaa39c956014aa446ab1b07fc4abf8d3 
  ql/src/java/org/apache/hadoop/hive/ql/exec/FetchOperator.java 
13750cdc34711d22f2adf2f483a6773ad05fb8d2 
  ql/src/java/org/apache/hadoop/hive/ql/exec/StatsNoJobTask.java 
9c3a664b9aea2d6e050ffe2d7626127827dbc52a 
  ql/src/java/org/apache/hadoop/hive/ql/exec/mr/MapRedTask.java 
1bd4db7805689ae1f91921ffbb5ff7da59f4bf60 
  
ql/src/java/org/apache/hadoop/hive/ql/io/parquet/MapredParquetInputFormat.java 
f4fadbb61bf45f62945700284c0b050f0984b696 
  ql/src/java/org/apache/hadoop/hive/ql/io/parquet/ParquetRecordReaderBase.java 
2954601ce5bb25905cdb29ca0ca4551c2ca12b95 
  ql/src/java/org/apache/hadoop/hive/ql/io/parquet/serde/ParquetHiveSerDe.java 
6413c5add6db2e8c9298285b15dba33ee74379a8 
  ql/src/java/org/apache/hadoop/hive/ql/io/parquet/serde/ParquetTableUtils.java 
b339cc4347eea143dca2f6d98f9aaafdc427 
  ql/src/java/org/apache/hadoop/hive/ql/io/parquet/timestamp/NanoTimeUtils.java 
dbd6fb3d0bc8c753abf86e99b52377617f248b5a 
  
ql/src/test/org/apache/hadoop/hive/ql/io/parquet/AbstractTestParquetDirect.java 
c81499a91c84af3ba33f335506c1c44e7085f13d 
  
ql/src/test/org/apache/hadoop/hive/ql/io/parquet/TestParquetRowGroupFilter.java 
bf363f32a3ac0a4d790e2925d802c6e210adfb4b 
  
ql/src/test/org/apache/hadoop/hive/ql/io/parquet/VectorizedColumnReaderTestBase.java
 f2d79cf9d215e9a6e2a5e88cfc78378be860fd1f 
  
ql/src/test/org/apache/hadoop/hive/ql/io/parquet/timestamp/TestNanoTimeUtils.java
 1e10dbf18742524982606f1e6c6d447d683b2dc3 
  ql/src/test/queries/clientnegative/parquet_int96_alter_invalid_timezone.q 
PRE-CREATION 
  ql/src/test/queries/clientnegative/parquet_int96_create_invalid_timezone.q 
PRE-CREATION 
  ql/src/test/queries/clientpositive/parquet_int96_timestamp.q 
6eadd1b0a3313cbba7a798890b802baae302749e 
  ql/src/test/results/clientnegative/parquet_int96_alter_invalid_timezone.q.out 
PRE-CREATION 
  
ql/src/test/results/clientnegative/parquet_int96_create_invalid_timezone.q.out 
PRE-CREATION 
  ql/src/test/results/clientpositive/parquet_int96_timestamp.q.out 
b9a3664458a83f1856e4bc59eba5d56665df61cc 
  ql/src/test/results/clientpositive/spark/parquet_int96_timestamp.q.out 
PRE-CREATION 


Diff: https://reviews.apache.org/r/58501/diff/5/

Changes: https://reviews.apache.org/r/58501/diff/4-5/


Testing
---

Added qtests for the following cases:
- order by clause
- selfjoin
- calling UDFs with the timestamp values
- where clause with a constant cast as timestamp
- test for HoS
- implicit and explicit timestamp conversions in insert clause

Tested manually but no qtests:
- join between 3 tables all parquet but with different/no timezone property
- subselect in from/where clauses
- exists / union / no exists


Thanks,

Barna Zsombor Klara

Re: Review Request 58501: HIVE-16469: Parquet timestamp table property is not always taken into account

[Barna Zsombor Klara]

> On May 3, 2017, 5 p.m., Vihang Karajgaonkar wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/io/parquet/MapredParquetInputFormat.java
> > Lines 115-120 (patched)
> > <https://reviews.apache.org/r/58501/diff/4/?file=1706946#file1706946line115>
> >
> > Should logs here be warning?

I'm not expecting the exceptions, but good point a warning is probably better.


- Barna Zsombor


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/58501/#review173747
-------


On May 3, 2017, 12:59 p.m., Barna Zsombor Klara wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/58501/
> ---
> 
> (Updated May 3, 2017, 12:59 p.m.)
> 
> 
> Review request for hive, Sergio Pena and Zoltan Ivanfi.
> 
> 
> Bugs: HIVE-16469
> https://issues.apache.org/jira/browse/HIVE-16469
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-16469: Parquet timestamp table property is not always taken into account
> 
> 
> Diffs
> -
> 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 
> 757b7fc0eaa39c956014aa446ab1b07fc4abf8d3 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/FetchOperator.java 
> 13750cdc34711d22f2adf2f483a6773ad05fb8d2 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/StatsNoJobTask.java 
> 9c3a664b9aea2d6e050ffe2d7626127827dbc52a 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/mr/MapRedTask.java 
> 1bd4db7805689ae1f91921ffbb5ff7da59f4bf60 
>   
> ql/src/java/org/apache/hadoop/hive/ql/io/parquet/MapredParquetInputFormat.java
>  f4fadbb61bf45f62945700284c0b050f0984b696 
>   
> ql/src/java/org/apache/hadoop/hive/ql/io/parquet/ParquetRecordReaderBase.java 
> 2954601ce5bb25905cdb29ca0ca4551c2ca12b95 
>   
> ql/src/java/org/apache/hadoop/hive/ql/io/parquet/serde/ParquetHiveSerDe.java 
> 6413c5add6db2e8c9298285b15dba33ee74379a8 
>   
> ql/src/java/org/apache/hadoop/hive/ql/io/parquet/serde/ParquetTableUtils.java 
> b339cc4347eea143dca2f6d98f9aaafdc427 
>   
> ql/src/java/org/apache/hadoop/hive/ql/io/parquet/timestamp/NanoTimeUtils.java 
> dbd6fb3d0bc8c753abf86e99b52377617f248b5a 
>   
> ql/src/test/org/apache/hadoop/hive/ql/io/parquet/AbstractTestParquetDirect.java
>  c81499a91c84af3ba33f335506c1c44e7085f13d 
>   
> ql/src/test/org/apache/hadoop/hive/ql/io/parquet/TestParquetRowGroupFilter.java
>  bf363f32a3ac0a4d790e2925d802c6e210adfb4b 
>   
> ql/src/test/org/apache/hadoop/hive/ql/io/parquet/VectorizedColumnReaderTestBase.java
>  f2d79cf9d215e9a6e2a5e88cfc78378be860fd1f 
>   
> ql/src/test/org/apache/hadoop/hive/ql/io/parquet/timestamp/TestNanoTimeUtils.java
>  1e10dbf18742524982606f1e6c6d447d683b2dc3 
>   ql/src/test/queries/clientnegative/parquet_int96_alter_invalid_timezone.q 
> PRE-CREATION 
>   ql/src/test/queries/clientnegative/parquet_int96_create_invalid_timezone.q 
> PRE-CREATION 
>   ql/src/test/queries/clientpositive/parquet_int96_timestamp.q 
> 6eadd1b0a3313cbba7a798890b802baae302749e 
>   
> ql/src/test/results/clientnegative/parquet_int96_alter_invalid_timezone.q.out 
> PRE-CREATION 
>   
> ql/src/test/results/clientnegative/parquet_int96_create_invalid_timezone.q.out
>  PRE-CREATION 
>   ql/src/test/results/clientpositive/parquet_int96_timestamp.q.out 
> b9a3664458a83f1856e4bc59eba5d56665df61cc 
>   ql/src/test/results/clientpositive/spark/parquet_int96_timestamp.q.out 
> PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/58501/diff/4/
> 
> 
> Testing
> ---
> 
> Added qtests for the following cases:
> - order by clause
> - selfjoin
> - calling UDFs with the timestamp values
> - where clause with a constant cast as timestamp
> - test for HoS
> - implicit and explicit timestamp conversions in insert clause
> 
> Tested manually but no qtests:
> - join between 3 tables all parquet but with different/no timezone property
> - subselect in from/where clauses
> - exists / union / no exists
> 
> 
> Thanks,
> 
> Barna Zsombor Klara
> 
>

Re: Review Request 58501: HIVE-16469: Parquet timestamp table property is not always taken into account

[Barna Zsombor Klara]

> On May 4, 2017, 1:53 a.m., cheng xu wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/exec/FetchOperator.java
> > Lines 372 (patched)
> > <https://reviews.apache.org/r/58501/diff/4/?file=1706943#file1706943line372>
> >
> > Can we check the format type to see whether it's Parquet format?

Ahh very good point, thanks for spotting this. We shouldn't polute other tables 
with our property.


> On May 4, 2017, 1:53 a.m., cheng xu wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/io/parquet/ParquetRecordReaderBase.java
> > Line 181 (original), 181 (patched)
> > <https://reviews.apache.org/r/58501/diff/4/?file=1706947#file1706947line181>
> >
> > Why not passing in the default value here when 
> > PARQUET_INT96_WRITE_ZONE_PROPERTY is not set?

I would prefer not to use the default value here. I want to make sure the 
RecordReader cannot be used without having checked the TimeZone property, so 
the default value is only set in the 
ParquetTableUtils#setParquetTimeZoneIfAbsent where I have the full list of 
table properties. If we use the default value in a case where we forgot to 
check the table property, we may end up reading incorrect timestamp values 
silently from the table.


- Barna Zsombor


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/58501/#review173857
-----------


On May 3, 2017, 12:59 p.m., Barna Zsombor Klara wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/58501/
> ---
> 
> (Updated May 3, 2017, 12:59 p.m.)
> 
> 
> Review request for hive, Sergio Pena and Zoltan Ivanfi.
> 
> 
> Bugs: HIVE-16469
> https://issues.apache.org/jira/browse/HIVE-16469
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-16469: Parquet timestamp table property is not always taken into account
> 
> 
> Diffs
> -
> 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 
> 757b7fc0eaa39c956014aa446ab1b07fc4abf8d3 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/FetchOperator.java 
> 13750cdc34711d22f2adf2f483a6773ad05fb8d2 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/StatsNoJobTask.java 
> 9c3a664b9aea2d6e050ffe2d7626127827dbc52a 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/mr/MapRedTask.java 
> 1bd4db7805689ae1f91921ffbb5ff7da59f4bf60 
>   
> ql/src/java/org/apache/hadoop/hive/ql/io/parquet/MapredParquetInputFormat.java
>  f4fadbb61bf45f62945700284c0b050f0984b696 
>   
> ql/src/java/org/apache/hadoop/hive/ql/io/parquet/ParquetRecordReaderBase.java 
> 2954601ce5bb25905cdb29ca0ca4551c2ca12b95 
>   
> ql/src/java/org/apache/hadoop/hive/ql/io/parquet/serde/ParquetHiveSerDe.java 
> 6413c5add6db2e8c9298285b15dba33ee74379a8 
>   
> ql/src/java/org/apache/hadoop/hive/ql/io/parquet/serde/ParquetTableUtils.java 
> b339cc4347eea143dca2f6d98f9aaafdc427 
>   
> ql/src/java/org/apache/hadoop/hive/ql/io/parquet/timestamp/NanoTimeUtils.java 
> dbd6fb3d0bc8c753abf86e99b52377617f248b5a 
>   
> ql/src/test/org/apache/hadoop/hive/ql/io/parquet/AbstractTestParquetDirect.java
>  c81499a91c84af3ba33f335506c1c44e7085f13d 
>   
> ql/src/test/org/apache/hadoop/hive/ql/io/parquet/TestParquetRowGroupFilter.java
>  bf363f32a3ac0a4d790e2925d802c6e210adfb4b 
>   
> ql/src/test/org/apache/hadoop/hive/ql/io/parquet/VectorizedColumnReaderTestBase.java
>  f2d79cf9d215e9a6e2a5e88cfc78378be860fd1f 
>   
> ql/src/test/org/apache/hadoop/hive/ql/io/parquet/timestamp/TestNanoTimeUtils.java
>  1e10dbf18742524982606f1e6c6d447d683b2dc3 
>   ql/src/test/queries/clientnegative/parquet_int96_alter_invalid_timezone.q 
> PRE-CREATION 
>   ql/src/test/queries/clientnegative/parquet_int96_create_invalid_timezone.q 
> PRE-CREATION 
>   ql/src/test/queries/clientpositive/parquet_int96_timestamp.q 
> 6eadd1b0a3313cbba7a798890b802baae302749e 
>   
> ql/src/test/results/clientnegative/parquet_int96_alter_invalid_timezone.q.out 
> PRE-CREATION 
>   
> ql/src/test/results/clientnegative/parquet_int96_create_invalid_timezone.q.out
>  PRE-CREATION 
>   ql/src/test/results/clientpositive/parquet_int96_timestamp.q.out 
> b9a3664458a83f1856e4bc59eba5d56665df61cc 
>   ql/src/test/results/clientpositive/spark/parquet_int96_timestamp.q.out 
> PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/58501/diff/4/
> 
> 
> Testing
> ---
> 
> Added qtests for the following cases:
> - order by clause
> - selfjoin
> - calling UDFs with the timestamp values
> - where clause with a constant cast as timestamp
> - test for HoS
> - implicit and explicit timestamp conversions in insert clause
> 
> Tested manually but no qtests:
> - join between 3 tables all parquet but with different/no timezone property
> - subselect in from/where clauses
> - exists / union / no exists
> 
> 
> Thanks,
> 
> Barna Zsombor Klara
> 
>

Re: Review Request 58501: HIVE-16469: Parquet timestamp table property is not always taken into account

[Barna Zsombor Klara]

> On May 2, 2017, 5:27 p.m., Sergio Pena wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/exec/mr/MapRedTask.java
> > Line 72 (original), 72 (patched)
> > <https://reviews.apache.org/r/58501/diff/3/?file=1695509#file1695509line72>
> >
> > How does this work? I don't understand this change.

The user.timezone system property is used to set the default timezone of the 
JVM. If this is set on the HS2 instance then we need to propagate it to the 
child VM spawned by a local task or timestamps read by the local task will be 
incorrect.


> On May 2, 2017, 5:27 p.m., Sergio Pena wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/io/parquet/ParquetRecordReaderBase.java
> > Line 181 (original), 181 (patched)
> > <https://reviews.apache.org/r/58501/diff/3/?file=1695511#file1695511line181>
> >
> > Is this compatible with old parquet tables? if the property is not set, 
> > then the validateTimeZonemight fail, right? If so, do we want to fail 
> > reading tables that do not have a property set?
> > 
> > Something else to consider, if a user sets a timezone improperly in a 
> > different tool or something  happened that we got an invalid timezone, 
> > then do we want to fail when reading those files? Just  wondering this 
> > scenario, no need to fix it right away.

At this point the timezone property had to be set by 
ParquetTableUtils#setParquetTimeZoneIfAbsent either from the table properties 
or using the default value TimeZone#getDefault. The core problem is that I 
found it very difficult to make sure that  execution path will check the 
table property.
- The FetchOperator works when we have a local task, but the 
MapRedParquetInputFormat does not (MapWork is null). 
- The FetchOperator will not work with a complex query or an order by clause, 
but the InputFormat should work in this case. 
- For statistics gathering only the StatNoJobTask is executed.
I wanted to make sure that if we have an execution path I forgot about, then we 
should rather fail than to read incorrect timestamp values silently.
Similarly in my opinion if the timezone value is incorrect (because it was set 
by another tool) then we should fail instead of reading illadjusted values.


> On May 2, 2017, 5:27 p.m., Sergio Pena wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/io/parquet/serde/ParquetTableUtils.java
> > Lines 35 (patched)
> > <https://reviews.apache.org/r/58501/diff/3/?file=1695512#file1695512line35>
> >
> > Why is Map used instead of Map<String, String>? Aren't all table 
> > properties key, value string pairs?
> > 
> > Also, the ensureTablePropertySet() name seems not related to what we 
> > want to do. I thought it was going to throw an exception if the property 
> > was not set, but it is setting the value on the JobConf. Should we use a 
> > different name, such as setParquetTimeZoneIfNotSet(),  
> > setParquetTimeZoneIfAbsent() or something like that helps us understand 
> > quickly without looking at the javadoc.

We are calling this method with Properties objects (i.e. from the 
FetchOperator) and using Map<String, String> objects (i.e. from the 
StatsNoJobTask) and the common ancestor for these two is the Map. While it 
is true that the table properties can only be Strings so the Properties should 
only contain String pairs I wanted to avoid the explicit cast.


- Barna Zsombor


-------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/58501/#review173610
---


On May 3, 2017, 12:59 p.m., Barna Zsombor Klara wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/58501/
> ---
> 
> (Updated May 3, 2017, 12:59 p.m.)
> 
> 
> Review request for hive, Sergio Pena and Zoltan Ivanfi.
> 
> 
> Bugs: HIVE-16469
> https://issues.apache.org/jira/browse/HIVE-16469
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-16469: Parquet timestamp table property is not always taken into account
> 
> 
> Diffs
> -
> 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 
> 757b7fc0eaa39c956014aa446ab1b07fc4abf8d3 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/FetchOperator.java 
> 13750cdc34711d22f2adf2f483a6773ad05fb8d2 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/StatsNoJobTask.java 
> 9c3a664b9aea2d6e050ffe2d7626127827dbc52a 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/mr/MapRedTask.java 
> 1bd4db7805689ae1f91921ffbb5ff7da59f4b

Re: Review Request 58501: HIVE-16469: Parquet timestamp table property is not always taken into account

[Barna Zsombor Klara]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/58501/
---

(Updated May 3, 2017, 12:59 p.m.)


Review request for hive, Sergio Pena and Zoltan Ivanfi.


Changes
---

Updated based on comments.


Bugs: HIVE-16469
https://issues.apache.org/jira/browse/HIVE-16469


Repository: hive-git


Description
---

HIVE-16469: Parquet timestamp table property is not always taken into account


Diffs (updated)
-

  ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 
757b7fc0eaa39c956014aa446ab1b07fc4abf8d3 
  ql/src/java/org/apache/hadoop/hive/ql/exec/FetchOperator.java 
13750cdc34711d22f2adf2f483a6773ad05fb8d2 
  ql/src/java/org/apache/hadoop/hive/ql/exec/StatsNoJobTask.java 
9c3a664b9aea2d6e050ffe2d7626127827dbc52a 
  ql/src/java/org/apache/hadoop/hive/ql/exec/mr/MapRedTask.java 
1bd4db7805689ae1f91921ffbb5ff7da59f4bf60 
  
ql/src/java/org/apache/hadoop/hive/ql/io/parquet/MapredParquetInputFormat.java 
f4fadbb61bf45f62945700284c0b050f0984b696 
  ql/src/java/org/apache/hadoop/hive/ql/io/parquet/ParquetRecordReaderBase.java 
2954601ce5bb25905cdb29ca0ca4551c2ca12b95 
  ql/src/java/org/apache/hadoop/hive/ql/io/parquet/serde/ParquetHiveSerDe.java 
6413c5add6db2e8c9298285b15dba33ee74379a8 
  ql/src/java/org/apache/hadoop/hive/ql/io/parquet/serde/ParquetTableUtils.java 
b339cc4347eea143dca2f6d98f9aaafdc427 
  ql/src/java/org/apache/hadoop/hive/ql/io/parquet/timestamp/NanoTimeUtils.java 
dbd6fb3d0bc8c753abf86e99b52377617f248b5a 
  
ql/src/test/org/apache/hadoop/hive/ql/io/parquet/AbstractTestParquetDirect.java 
c81499a91c84af3ba33f335506c1c44e7085f13d 
  
ql/src/test/org/apache/hadoop/hive/ql/io/parquet/TestParquetRowGroupFilter.java 
bf363f32a3ac0a4d790e2925d802c6e210adfb4b 
  
ql/src/test/org/apache/hadoop/hive/ql/io/parquet/VectorizedColumnReaderTestBase.java
 f2d79cf9d215e9a6e2a5e88cfc78378be860fd1f 
  
ql/src/test/org/apache/hadoop/hive/ql/io/parquet/timestamp/TestNanoTimeUtils.java
 1e10dbf18742524982606f1e6c6d447d683b2dc3 
  ql/src/test/queries/clientnegative/parquet_int96_alter_invalid_timezone.q 
PRE-CREATION 
  ql/src/test/queries/clientnegative/parquet_int96_create_invalid_timezone.q 
PRE-CREATION 
  ql/src/test/queries/clientpositive/parquet_int96_timestamp.q 
6eadd1b0a3313cbba7a798890b802baae302749e 
  ql/src/test/results/clientnegative/parquet_int96_alter_invalid_timezone.q.out 
PRE-CREATION 
  
ql/src/test/results/clientnegative/parquet_int96_create_invalid_timezone.q.out 
PRE-CREATION 
  ql/src/test/results/clientpositive/parquet_int96_timestamp.q.out 
b9a3664458a83f1856e4bc59eba5d56665df61cc 
  ql/src/test/results/clientpositive/spark/parquet_int96_timestamp.q.out 
PRE-CREATION 


Diff: https://reviews.apache.org/r/58501/diff/4/

Changes: https://reviews.apache.org/r/58501/diff/3-4/


Testing
---

Added qtests for the following cases:
- order by clause
- selfjoin
- calling UDFs with the timestamp values
- where clause with a constant cast as timestamp
- test for HoS
- implicit and explicit timestamp conversions in insert clause

Tested manually but no qtests:
- join between 3 tables all parquet but with different/no timezone property
- subselect in from/where clauses
- exists / union / no exists


Thanks,

Barna Zsombor Klara

[jira] [Created] (HIVE-16559) Parquet schema evolution for partitioned tables may break if table and partition serdes differ

[Barna Zsombor Klara (JIRA)]

Barna Zsombor Klara created HIVE-16559:
--

 Summary: Parquet schema evolution for partitioned tables may break 
if table and partition serdes differ
 Key: HIVE-16559
 URL: https://issues.apache.org/jira/browse/HIVE-16559
 Project: Hive
  Issue Type: Bug
Reporter: Barna Zsombor Klara
Assignee: Barna Zsombor Klara


Parquet schema evolution should make it possible to have partitions/tables 
 backed by files with different schemas. Hive should match the table columns 
with file columns based on the column name if possible.
However if the serde for a table is missing columns from the serde of a 
partition Hive fails to match the columns together.
Steps to reproduce:
{code}
CREATE TABLE myparquettable_parted
(
  name string,
  favnumber int,
  favcolor string,
  age int,
  favpet string
)
PARTITIONED BY (day string)
STORED AS PARQUET;

INSERT OVERWRITE TABLE myparquettable_parted
PARTITION(day='2017-04-04')
SELECT
   'mary' as name,
   5 AS favnumber,
   'blue' AS favcolor,
   35 AS age,
   'dog' AS favpet;

REPLACE COLUMNS
(
favnumber int,
age int
);   

Re: Review Request 58501: HIVE-16469: Parquet timestamp table property is not always taken into account

[Barna Zsombor Klara]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/58501/
---

(Updated April 20, 2017, 2:11 p.m.)


Review request for hive, Sergio Pena and Zoltan Ivanfi.


Changes
---

Fixed failing tests.


Bugs: HIVE-16469
https://issues.apache.org/jira/browse/HIVE-16469


Repository: hive-git


Description
---

HIVE-16469: Parquet timestamp table property is not always taken into account


Diffs (updated)
-

  ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 
917e565f28b2c9aaea18033ea3b6b20fa41fcd0a 
  ql/src/java/org/apache/hadoop/hive/ql/exec/FetchOperator.java 
004bb2f60299a0635b8f9ca7649ead00b8e16d08 
  ql/src/java/org/apache/hadoop/hive/ql/exec/StatsNoJobTask.java 
9c3a664b9aea2d6e050ffe2d7626127827dbc52a 
  ql/src/java/org/apache/hadoop/hive/ql/exec/mr/MapRedTask.java 
1bd4db7805689ae1f91921ffbb5ff7da59f4bf60 
  
ql/src/java/org/apache/hadoop/hive/ql/io/parquet/MapredParquetInputFormat.java 
f4fadbb61bf45f62945700284c0b050f0984b696 
  ql/src/java/org/apache/hadoop/hive/ql/io/parquet/ParquetRecordReaderBase.java 
2954601ce5bb25905cdb29ca0ca4551c2ca12b95 
  ql/src/java/org/apache/hadoop/hive/ql/io/parquet/serde/ParquetTableUtils.java 
b339cc4347eea143dca2f6d98f9aaafdc427 
  ql/src/java/org/apache/hadoop/hive/ql/io/parquet/timestamp/NanoTimeUtils.java 
dbd6fb3d0bc8c753abf86e99b52377617f248b5a 
  
ql/src/test/org/apache/hadoop/hive/ql/io/parquet/AbstractTestParquetDirect.java 
c81499a91c84af3ba33f335506c1c44e7085f13d 
  
ql/src/test/org/apache/hadoop/hive/ql/io/parquet/TestParquetRowGroupFilter.java 
bf363f32a3ac0a4d790e2925d802c6e210adfb4b 
  
ql/src/test/org/apache/hadoop/hive/ql/io/parquet/VectorizedColumnReaderTestBase.java
 f2d79cf9d215e9a6e2a5e88cfc78378be860fd1f 
  ql/src/test/queries/clientnegative/parquet_int96_alter_invalid_timezone.q 
PRE-CREATION 
  ql/src/test/queries/clientnegative/parquet_int96_create_invalid_timezone.q 
PRE-CREATION 
  ql/src/test/queries/clientpositive/parquet_int96_timestamp.q 
6eadd1b0a3313cbba7a798890b802baae302749e 
  ql/src/test/results/clientnegative/parquet_int96_alter_invalid_timezone.q.out 
PRE-CREATION 
  
ql/src/test/results/clientnegative/parquet_int96_create_invalid_timezone.q.out 
PRE-CREATION 
  ql/src/test/results/clientpositive/parquet_int96_timestamp.q.out 
b9a3664458a83f1856e4bc59eba5d56665df61cc 
  ql/src/test/results/clientpositive/spark/parquet_int96_timestamp.q.out 
PRE-CREATION 


Diff: https://reviews.apache.org/r/58501/diff/3/

Changes: https://reviews.apache.org/r/58501/diff/2-3/


Testing
---

Added qtests for the following cases:
- order by clause
- selfjoin
- calling UDFs with the timestamp values
- where clause with a constant cast as timestamp
- test for HoS
- implicit and explicit timestamp conversions in insert clause

Tested manually but no qtests:
- join between 3 tables all parquet but with different/no timezone property
- subselect in from/where clauses
- exists / union / no exists


Thanks,

Barna Zsombor Klara

Re: Review Request 58501: HIVE-16469: Parquet timestamp table property is not always taken into account

[Barna Zsombor Klara]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/58501/
---

(Updated April 19, 2017, 3:37 p.m.)


Review request for hive, Sergio Pena and Zoltan Ivanfi.


Changes
---

Added another qtest and fixed typos.


Bugs: HIVE-16469
https://issues.apache.org/jira/browse/HIVE-16469


Repository: hive-git


Description
---

HIVE-16469: Parquet timestamp table property is not always taken into account


Diffs (updated)
-

  ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 
917e565f28b2c9aaea18033ea3b6b20fa41fcd0a 
  ql/src/java/org/apache/hadoop/hive/ql/exec/FetchOperator.java 
004bb2f60299a0635b8f9ca7649ead00b8e16d08 
  ql/src/java/org/apache/hadoop/hive/ql/exec/mr/MapRedTask.java 
1bd4db7805689ae1f91921ffbb5ff7da59f4bf60 
  
ql/src/java/org/apache/hadoop/hive/ql/io/parquet/MapredParquetInputFormat.java 
f4fadbb61bf45f62945700284c0b050f0984b696 
  ql/src/java/org/apache/hadoop/hive/ql/io/parquet/ParquetRecordReaderBase.java 
2954601ce5bb25905cdb29ca0ca4551c2ca12b95 
  ql/src/java/org/apache/hadoop/hive/ql/io/parquet/serde/ParquetTableUtils.java 
b339cc4347eea143dca2f6d98f9aaafdc427 
  ql/src/java/org/apache/hadoop/hive/ql/io/parquet/timestamp/NanoTimeUtils.java 
dbd6fb3d0bc8c753abf86e99b52377617f248b5a 
  ql/src/test/queries/clientnegative/parquet_int96_alter_invalid_timezone.q 
PRE-CREATION 
  ql/src/test/queries/clientnegative/parquet_int96_create_invalid_timezone.q 
PRE-CREATION 
  ql/src/test/queries/clientpositive/parquet_int96_timestamp.q 
6eadd1b0a3313cbba7a798890b802baae302749e 
  ql/src/test/results/clientnegative/parquet_int96_alter_invalid_timezone.q.out 
PRE-CREATION 
  
ql/src/test/results/clientnegative/parquet_int96_create_invalid_timezone.q.out 
PRE-CREATION 
  ql/src/test/results/clientpositive/parquet_int96_timestamp.q.out 
b9a3664458a83f1856e4bc59eba5d56665df61cc 
  ql/src/test/results/clientpositive/spark/parquet_int96_timestamp.q.out 
PRE-CREATION 


Diff: https://reviews.apache.org/r/58501/diff/2/

Changes: https://reviews.apache.org/r/58501/diff/1-2/


Testing (updated)
---

Added qtests for the following cases:
- order by clause
- selfjoin
- calling UDFs with the timestamp values
- where clause with a constant cast as timestamp
- test for HoS
- implicit and explicit timestamp conversions in insert clause

Tested manually but no qtests:
- join between 3 tables all parquet but with different/no timezone property
- subselect in from/where clauses
- exists / union / no exists


Thanks,

Barna Zsombor Klara

Review Request 58501: HIVE-16469: Parquet timestamp table property is not always taken into account

[Barna Zsombor Klara]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/58501/
---

Review request for hive and Sergio Pena.


Bugs: HIVE-16469
https://issues.apache.org/jira/browse/HIVE-16469


Repository: hive-git


Description
---

HIVE-16469: Parquet timestamp table property is not always taken into account


Diffs
-

  ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 
917e565f28b2c9aaea18033ea3b6b20fa41fcd0a 
  ql/src/java/org/apache/hadoop/hive/ql/exec/FetchOperator.java 
004bb2f60299a0635b8f9ca7649ead00b8e16d08 
  ql/src/java/org/apache/hadoop/hive/ql/exec/mr/MapRedTask.java 
1bd4db7805689ae1f91921ffbb5ff7da59f4bf60 
  
ql/src/java/org/apache/hadoop/hive/ql/io/parquet/MapredParquetInputFormat.java 
f4fadbb61bf45f62945700284c0b050f0984b696 
  ql/src/java/org/apache/hadoop/hive/ql/io/parquet/ParquetRecordReaderBase.java 
2954601ce5bb25905cdb29ca0ca4551c2ca12b95 
  ql/src/java/org/apache/hadoop/hive/ql/io/parquet/serde/ParquetTableUtils.java 
b339cc4347eea143dca2f6d98f9aaafdc427 
  ql/src/java/org/apache/hadoop/hive/ql/io/parquet/timestamp/NanoTimeUtils.java 
dbd6fb3d0bc8c753abf86e99b52377617f248b5a 
  ql/src/test/queries/clientnegative/parquet_int96_alter_invalid_timezone.q 
PRE-CREATION 
  ql/src/test/queries/clientnegative/parquet_int96_create_invalid_timezone.q 
PRE-CREATION 
  ql/src/test/queries/clientpositive/parquet_int96_timestamp.q 
6eadd1b0a3313cbba7a798890b802baae302749e 
  ql/src/test/results/clientnegative/parquet_int96_alter_invalid_timezone.q.out 
PRE-CREATION 
  
ql/src/test/results/clientnegative/parquet_int96_create_invalid_timezone.q.out 
PRE-CREATION 
  ql/src/test/results/clientpositive/parquet_int96_timestamp.q.out 
b9a3664458a83f1856e4bc59eba5d56665df61cc 
  ql/src/test/results/clientpositive/spark/parquet_int96_timestamp.q.out 
PRE-CREATION 


Diff: https://reviews.apache.org/r/58501/diff/1/


Testing
---

Added qtests for the following cases:
- order by clause
- selfjoin
- calling UDFs with the timestamp values
- where clause with a constant cast as timestamp
- test for HoS

Tested manually but no qtests:
- join between 3 tables all parquet but with different/no timezone property
- subselect in from/where clauses
- exists / union / no exists


Thanks,

Barna Zsombor Klara

[jira] [Created] (HIVE-16469) Parquet timestamp table property is not always taken into account

[Barna Zsombor Klara (JIRA)]

Barna Zsombor Klara created HIVE-16469:
--

 Summary: Parquet timestamp table property is not always taken into 
account
 Key: HIVE-16469
 URL: https://issues.apache.org/jira/browse/HIVE-16469
 Project: Hive
  Issue Type: Bug
Reporter: Barna Zsombor Klara
Assignee: Barna Zsombor Klara


The parquet timestamp timezone property is currently copied over into the 
JobConf in the FetchOperator, but this may be too late for some execution paths.
We should:
1 - copy the property over earlier
2 - set the default value on the JobConf if no property is set, and fail in the 
ParquetRecordReader if the property is missing from the JobConf



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (HIVE-16357) Failed folder creation when creating a new table is reported incorrectly

[Barna Zsombor Klara (JIRA)]

Barna Zsombor Klara created HIVE-16357:
--

 Summary: Failed folder creation when creating a new table is 
reported incorrectly
 Key: HIVE-16357
 URL: https://issues.apache.org/jira/browse/HIVE-16357
 Project: Hive
  Issue Type: Bug
Reporter: Barna Zsombor Klara


If the directory for a Hive table could not be created, them the HMS will throw 
a metaexception:
{code}
 if (tblPath != null) {
  if (!wh.isDir(tblPath)) {
if (!wh.mkdirs(tblPath, true)) {
  throw new MetaException(tblPath
  + " is not a directory or unable to create one");
}
madeDir = true;
  }
}
{code}

However in the finally block we always try to call the DbNotificationListener, 
which in turn will also throw an exception because the directory is missing, 
overwriting the initial exception with a FileNotFoundException.

Actual stacktrace seen by the caller:
{code}
2017-04-03T05:58:00,128 ERROR [pool-7-thread-2] metastore.RetryingHMSHandler: 
MetaException(message:java.lang.RuntimeException: 
java.io.FileNotFoundException: File file:/.../0 does not exist)
at 
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.newMetaException(HiveMetaStore.java:6074)
at 
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_with_environment_context(HiveMetaStore.java:1496)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:148)
at 
org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:107)
at com.sun.proxy.$Proxy28.create_table_with_environment_context(Unknown 
Source)
at 
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$create_table_with_environment_context.getResult(ThriftHiveMetastore.java:11125)
at 
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$create_table_with_environment_context.getResult(ThriftHiveMetastore.java:11109)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
at 
org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:110)
at 
org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:106)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
at 
org.apache.hadoop.hive.metastore.TUGIBasedProcessor.process(TUGIBasedProcessor.java:118)
at 
org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.RuntimeException: java.io.FileNotFoundException: File 
file:/.../0 does not exist
at 
org.apache.hive.hcatalog.listener.DbNotificationListener$FileIterator.(DbNotificationListener.java:203)
at 
org.apache.hive.hcatalog.listener.DbNotificationListener.onCreateTable(DbNotificationListener.java:137)
at 
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_core(HiveMetaStore.java:1463)
at 
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_with_environment_context(HiveMetaStore.java:1482)
... 20 more
Caused by: java.io.FileNotFoundException: File file:/.../0 does not exist
at 
org.apache.hadoop.fs.RawLocalFileSystem.listStatus(RawLocalFileSystem.java:429)
at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1515)
at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1555)
at 
org.apache.hadoop.fs.ChecksumFileSystem.listStatus(ChecksumFileSystem.java:574)
at 
org.apache.hadoop.fs.FilterFileSystem.listStatus(FilterFileSystem.java:243)
at 
org.apache.hadoop.fs.ProxyFileSystem.listStatus(ProxyFileSystem.java:195)
at 
org.apache.hadoop.fs.FilterFileSystem.listStatus(FilterFileSystem.java:243)
at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1515)
at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1555)
at 
org.apache.hive.hcatalog.listener.DbNotificationListener$FileIterator.(DbNotificationListener.java:200)
... 23 more
{code}

The original exception should be thrown back to the caller

[jira] [Created] (HIVE-16312) Flaky test: TestHCatClient.testTransportFailure

[Barna Zsombor Klara (JIRA)]

Barna Zsombor Klara created HIVE-16312:
--

 Summary: Flaky test: TestHCatClient.testTransportFailure
 Key: HIVE-16312
 URL: https://issues.apache.org/jira/browse/HIVE-16312
 Project: Hive
  Issue Type: Sub-task
Reporter: Barna Zsombor Klara


The test has been failing consistently for 10+ builds.
Error message:
{code}
Error Message

The expected exception was never thrown.
{code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: Review Request 57728: HIVE-16231: Parquet timestamp may be stored differently since HIVE-12767

[Barna Zsombor Klara]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/57728/
---

(Updated March 27, 2017, 8 a.m.)


Review request for hive and Sergio Pena.


Changes
---

Removed empty line from DateUtils.


Repository: hive-git


Description
---

HIVE-16231: Parquet timestamp may be stored differently since HIVE-12767


Diffs (updated)
-

  
ql/src/java/org/apache/hadoop/hive/ql/io/parquet/MapredParquetOutputFormat.java 
26f1e75c7d659a634cd4eef3a0cb8e886b22722f 
  ql/src/java/org/apache/hadoop/hive/ql/io/parquet/ParquetRecordReaderBase.java 
8e33b7d437894b33b35f32913a3bc02f2a849ce3 
  ql/src/java/org/apache/hadoop/hive/ql/io/parquet/timestamp/NanoTimeUtils.java 
5dc808800290f3274afbdff12134ac34387a746b 
  
ql/src/test/org/apache/hadoop/hive/ql/io/parquet/timestamp/TestNanoTimeUtils.java
 37cf0e2d74589cfa97fa24c9d2d8d00ea62390ee 
  ql/src/test/queries/clientpositive/parquet_int96_timestamp.q 
5de2c3f1244b8340b97eb0547fe66e52d80fb065 


Diff: https://reviews.apache.org/r/57728/diff/4/

Changes: https://reviews.apache.org/r/57728/diff/3-4/


Testing
---

Tested loading timestamps from a parquet file written by spark.


Thanks,

Barna Zsombor Klara

Re: Review Request 57728: HIVE-16231: Parquet timestamp may be stored differently since HIVE-12767

[Barna Zsombor Klara]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/57728/
---

(Updated March 24, 2017, 9:56 a.m.)


Review request for hive and Sergio Pena.


Changes
---

Move the time zone checking utility method into NanotTimeUtils.


Repository: hive-git


Description
---

HIVE-16231: Parquet timestamp may be stored differently since HIVE-12767


Diffs (updated)
-

  common/src/java/org/apache/hive/common/util/DateUtils.java 
a1068ecce94e9ff1ae78008a0d8c6d67ca4f2690 
  
ql/src/java/org/apache/hadoop/hive/ql/io/parquet/MapredParquetOutputFormat.java 
26f1e75c7d659a634cd4eef3a0cb8e886b22722f 
  ql/src/java/org/apache/hadoop/hive/ql/io/parquet/ParquetRecordReaderBase.java 
8e33b7d437894b33b35f32913a3bc02f2a849ce3 
  ql/src/java/org/apache/hadoop/hive/ql/io/parquet/timestamp/NanoTimeUtils.java 
5dc808800290f3274afbdff12134ac34387a746b 
  
ql/src/test/org/apache/hadoop/hive/ql/io/parquet/timestamp/TestNanoTimeUtils.java
 37cf0e2d74589cfa97fa24c9d2d8d00ea62390ee 
  ql/src/test/queries/clientpositive/parquet_int96_timestamp.q 
5de2c3f1244b8340b97eb0547fe66e52d80fb065 


Diff: https://reviews.apache.org/r/57728/diff/3/

Changes: https://reviews.apache.org/r/57728/diff/2-3/


Testing
---

Tested loading timestamps from a parquet file written by spark.


Thanks,

Barna Zsombor Klara

Re: Review Request 57728: HIVE-16231: Parquet timestamp may be stored differently since HIVE-12767

[Barna Zsombor Klara]

> On March 22, 2017, 6:27 p.m., Sergio Pena wrote:
> > common/src/java/org/apache/hive/common/util/DateUtils.java
> > Lines 84 (patched)
> > <https://reviews.apache.org/r/57728/diff/2/?file=1670971#file1670971line84>
> >
> > Is there another class where to put this method? I don't think 
> > DateUtils is the place where we should keep this.

I couldn't find a much better fit. I looked at HiveUtils and ParquetTableUtils 
but DateUtils seemed better. I can create a TimeZoneUtils class, but I don't 
know if we will ever have a second function in it. Do you have a utility class 
in mind that would be better?


- Barna Zsombor


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/57728/#review169758
-------


On March 21, 2017, 5:28 p.m., Barna Zsombor Klara wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/57728/
> ---
> 
> (Updated March 21, 2017, 5:28 p.m.)
> 
> 
> Review request for hive and Sergio Pena.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-16231: Parquet timestamp may be stored differently since HIVE-12767
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hive/common/util/DateUtils.java 
> a1068ecce94e9ff1ae78008a0d8c6d67ca4f2690 
>   common/src/test/org/apache/hive/common/util/TestDateUtils.java PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/io/parquet/MapredParquetOutputFormat.java
>  26f1e75c7d659a634cd4eef3a0cb8e886b22722f 
>   
> ql/src/java/org/apache/hadoop/hive/ql/io/parquet/ParquetRecordReaderBase.java 
> 8e33b7d437894b33b35f32913a3bc02f2a849ce3 
>   
> ql/src/java/org/apache/hadoop/hive/ql/io/parquet/timestamp/NanoTimeUtils.java 
> 5dc808800290f3274afbdff12134ac34387a746b 
>   ql/src/test/queries/clientpositive/parquet_int96_timestamp.q 
> 5de2c3f1244b8340b97eb0547fe66e52d80fb065 
> 
> 
> Diff: https://reviews.apache.org/r/57728/diff/2/
> 
> 
> Testing
> ---
> 
> Tested loading timestamps from a parquet file written by spark.
> 
> 
> Thanks,
> 
> Barna Zsombor Klara
> 
>

Re: Review Request 57728: HIVE-16231: Parquet timestamp may be stored differently since HIVE-12767

[Barna Zsombor Klara]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/57728/
---

(Updated March 21, 2017, 5:28 p.m.)


Review request for hive and Sergio Pena.


Changes
---

Refactored timezone check into a separate method in DateUtils.


Repository: hive-git


Description
---

HIVE-16231: Parquet timestamp may be stored differently since HIVE-12767


Diffs (updated)
-

  common/src/java/org/apache/hive/common/util/DateUtils.java 
a1068ecce94e9ff1ae78008a0d8c6d67ca4f2690 
  common/src/test/org/apache/hive/common/util/TestDateUtils.java PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/io/parquet/MapredParquetOutputFormat.java 
26f1e75c7d659a634cd4eef3a0cb8e886b22722f 
  ql/src/java/org/apache/hadoop/hive/ql/io/parquet/ParquetRecordReaderBase.java 
8e33b7d437894b33b35f32913a3bc02f2a849ce3 
  ql/src/java/org/apache/hadoop/hive/ql/io/parquet/timestamp/NanoTimeUtils.java 
5dc808800290f3274afbdff12134ac34387a746b 
  ql/src/test/queries/clientpositive/parquet_int96_timestamp.q 
5de2c3f1244b8340b97eb0547fe66e52d80fb065 


Diff: https://reviews.apache.org/r/57728/diff/2/

Changes: https://reviews.apache.org/r/57728/diff/1-2/


Testing
---

Tested loading timestamps from a parquet file written by spark.


Thanks,

Barna Zsombor Klara

[jira] [Created] (HIVE-16256) Flaky test: TestCliDriver.testCliDriver[comments]

[Barna Zsombor Klara (JIRA)]

Barna Zsombor Klara created HIVE-16256:
--

 Summary: Flaky test: TestCliDriver.testCliDriver[comments]
 Key: HIVE-16256
 URL: https://issues.apache.org/jira/browse/HIVE-16256
 Project: Hive
  Issue Type: Sub-task
Reporter: Barna Zsombor Klara


Test has been failing for 6 consecutive runs. Most recent:
https://builds.apache.org/job/PreCommit-HIVE-Build/4245/testReport/

Diff:
147a148
>   COLUMN_STATS_ACCURATE 
> {"BASIC_STATS":"true","COLUMN_STATS":{"key":"true","value":"true"}}
167a169
> COLUMN_STATS_ACCURATE 
> {"BASIC_STATS":"true","COLUMN_STATS":{"key":"true","value":"true"}}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: Review Request 57503: HIVE-16024: MSCK Repair Requires nonstrict hive.mapred.mode

[Barna Zsombor Klara]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/57503/
---

(Updated March 20, 2017, 9:03 a.m.)


Review request for hive, Peter Vary, Sergio Pena, Sahil Takiar, and Vihang 
Karajgaonkar.


Changes
---

Updated qtest to find/repair partitions in strict mode.


Bugs: HIVE-16024
https://issues.apache.org/jira/browse/HIVE-16024


Repository: hive-git


Description
---

HIVE-16024: MSCK Repair Requires nonstrict hive.mapred.mode


Diffs (updated)
-

  ql/src/java/org/apache/hadoop/hive/ql/metadata/HiveMetaStoreChecker.java 
6805c17a116f5ef0febd36c59d454fa631ae0024 
  ql/src/test/queries/clientpositive/msck_repair_0.q 
ce8ef426a2a58845afc8333259d66725db416584 
  ql/src/test/results/clientpositive/msck_repair_0.q.out 
3f2fe75b194f1248bd5c073dd7db6b71b2ffc2ba 


Diff: https://reviews.apache.org/r/57503/diff/3/

Changes: https://reviews.apache.org/r/57503/diff/2-3/


Testing
---

Tested locally and added qtests/unit tests.


Thanks,

Barna Zsombor Klara

Re: Review Request 57503: HIVE-16024: MSCK Repair Requires nonstrict hive.mapred.mode

[Barna Zsombor Klara]

> On March 17, 2017, 8:30 p.m., Sergio Pena wrote:
> > ql/src/test/queries/clientpositive/msck_repair_0.q
> > Lines 19-21 (patched)
> > <https://reviews.apache.org/r/57503/diff/2/?file=1666109#file1666109line19>
> >
> > Is this actually adding new partitios to the metastore? Those are 
> > already discovered above. I think this part is not testing the changes 
> > correctly.

The check was initially failing in strict mode even if there were no partitions 
to be discovered/repaired, but I can change the test to find new partitions 
missing from the HMS.


- Barna Zsombor


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/57503/#review169324
-------


On March 16, 2017, 3:55 p.m., Barna Zsombor Klara wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/57503/
> ---
> 
> (Updated March 16, 2017, 3:55 p.m.)
> 
> 
> Review request for hive, Peter Vary, Sergio Pena, Sahil Takiar, and Vihang 
> Karajgaonkar.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-16024: MSCK Repair Requires nonstrict hive.mapred.mode
> 
> 
> Diffs
> -
> 
>   ql/src/java/org/apache/hadoop/hive/ql/metadata/HiveMetaStoreChecker.java 
> 6805c17a116f5ef0febd36c59d454fa631ae0024 
>   ql/src/test/queries/clientpositive/msck_repair_0.q 
> ce8ef426a2a58845afc8333259d66725db416584 
>   ql/src/test/results/clientpositive/msck_repair_0.q.out 
> 3f2fe75b194f1248bd5c073dd7db6b71b2ffc2ba 
> 
> 
> Diff: https://reviews.apache.org/r/57503/diff/2/
> 
> 
> Testing
> ---
> 
> Tested locally and added qtests/unit tests.
> 
> 
> Thanks,
> 
> Barna Zsombor Klara
> 
>

Review Request 57728: HIVE-16231: Parquet timestamp may be stored differently since HIVE-12767

[Barna Zsombor Klara]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/57728/
---

Review request for hive and Sergio Pena.


Repository: hive-git


Description
---

HIVE-16231: Parquet timestamp may be stored differently since HIVE-12767


Diffs
-

  
ql/src/java/org/apache/hadoop/hive/ql/io/parquet/MapredParquetOutputFormat.java 
26f1e75c7d659a634cd4eef3a0cb8e886b22722f 
  ql/src/java/org/apache/hadoop/hive/ql/io/parquet/ParquetRecordReaderBase.java 
8e33b7d437894b33b35f32913a3bc02f2a849ce3 
  ql/src/java/org/apache/hadoop/hive/ql/io/parquet/timestamp/NanoTimeUtils.java 
5dc808800290f3274afbdff12134ac34387a746b 
  ql/src/test/queries/clientpositive/parquet_int96_timestamp.q 
5de2c3f1244b8340b97eb0547fe66e52d80fb065 


Diff: https://reviews.apache.org/r/57728/diff/1/


Testing
---

Tested loading timestamps from a parquet file written by spark.


Thanks,

Barna Zsombor Klara

Re: Review Request 57503: HIVE-16024: MSCK Repair Requires nonstrict hive.mapred.mode

[Barna Zsombor Klara]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/57503/
---

(Updated March 16, 2017, 3:55 p.m.)


Review request for hive, Peter Vary, Sergio Pena, Sahil Takiar, and Vihang 
Karajgaonkar.


Changes
---

Updated the patch after discussing with Sergio.


Repository: hive-git


Description
---

HIVE-16024: MSCK Repair Requires nonstrict hive.mapred.mode


Diffs (updated)
-

  ql/src/java/org/apache/hadoop/hive/ql/metadata/HiveMetaStoreChecker.java 
6805c17a116f5ef0febd36c59d454fa631ae0024 
  ql/src/test/queries/clientpositive/msck_repair_0.q 
ce8ef426a2a58845afc8333259d66725db416584 
  ql/src/test/results/clientpositive/msck_repair_0.q.out 
3f2fe75b194f1248bd5c073dd7db6b71b2ffc2ba 


Diff: https://reviews.apache.org/r/57503/diff/2/

Changes: https://reviews.apache.org/r/57503/diff/1-2/


Testing
---

Tested locally and added qtests/unit tests.


Thanks,

Barna Zsombor Klara

Re: Review Request 57503: HIVE-16024: MSCK Repair Requires nonstrict hive.mapred.mode

[Barna Zsombor Klara]

> On March 15, 2017, 4:05 a.m., Naveen Gangam wrote:
> > common/src/java/org/apache/hadoop/hive/common/FixedSizeCollection.java
> > Lines 32 (patched)
> > <https://reviews.apache.org/r/57503/diff/1/?file=1661199#file1661199line32>
> >
> > Since we already use common-collections library in hive, would using 
> > something like FixedSizeList or something else address the same purpose 
> > instead of having to add a new class?

Let's wait and see how we decide on the need for this fixed size collection. If 
we are happy with using just the PartitionIterable, then these collections are 
not needed anymore.


- Barna Zsombor


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/57503/#review168969
-----------


On March 10, 2017, 10:36 a.m., Barna Zsombor Klara wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/57503/
> ---
> 
> (Updated March 10, 2017, 10:36 a.m.)
> 
> 
> Review request for hive, Peter Vary, Sergio Pena, Sahil Takiar, and Vihang 
> Karajgaonkar.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-16024: MSCK Repair Requires nonstrict hive.mapred.mode
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/common/FixedSizeCollection.java 
> PRE-CREATION 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 
> a479deb7c0c6b779277f1029009b7dfab6dcb9e3 
>   common/src/test/org/apache/hadoop/hive/common/TestFixedSizeCollection.java 
> PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/metadata/HiveMetaStoreChecker.java 
> 6805c17a116f5ef0febd36c59d454fa631ae0024 
>   ql/src/test/queries/clientnegative/msck_repair_4.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/msck_repair_0.q 
> ce8ef426a2a58845afc8333259d66725db416584 
>   ql/src/test/results/clientnegative/msck_repair_4.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/msck_repair_0.q.out 
> 3f2fe75b194f1248bd5c073dd7db6b71b2ffc2ba 
> 
> 
> Diff: https://reviews.apache.org/r/57503/diff/1/
> 
> 
> Testing
> ---
> 
> Tested locally and added qtests/unit tests.
> 
> 
> Thanks,
> 
> Barna Zsombor Klara
> 
>