Re: [ANNOUNCE] New Hive Committer - Thejas Nair

[Daniel Dai]

Congratulation!


On Tue, Aug 20, 2013 at 4:56 PM, Shreepadma Venugopalan 
shreepa...@cloudera.com wrote:

 Congrats Tejas!


 On Tue, Aug 20, 2013 at 9:32 AM, Eugene Koifman ekoif...@hortonworks.com
 wrote:

  Congrats Thejas!
 
 
  On Tue, Aug 20, 2013 at 3:31 AM, Carl Steinbach c...@apache.org wrote:
 
  The Apache Hive PMC has voted to make Thejas Nair a committer on the
  Apache
  Hive project.
 
  Please join me in congratulating Thejas!
 
 
 
  CONFIDENTIALITY NOTICE
  NOTICE: This message is intended for the use of the individual or entity
  to which it is addressed and may contain information that is
 confidential,
  privileged and exempt from disclosure under applicable law. If the reader
  of this message is not the intended recipient, you are hereby notified
 that
  any printing, copying, dissemination, distribution, disclosure or
  forwarding of this communication is strictly prohibited. If you have
  received this communication in error, please contact the sender
 immediately
  and delete it from your system. Thank You.
 


-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

Re: [ANNOUNCE] New Hive Committer - Yin Huai

[Daniel Dai]

Congratulation!


On Wed, Sep 4, 2013 at 10:39 AM, yongqiang he heyongqiang...@gmail.comwrote:

 Congrats!

 On Wed, Sep 4, 2013 at 10:23 AM, Jason Dere jd...@hortonworks.com wrote:
  Yin, congrats!
 
  Jason
 
  On Sep 4, 2013, at 7:54 AM, Eugene Koifman ekoif...@hortonworks.com
 wrote:
 
  Congrats!
 
 
  On Wed, Sep 4, 2013 at 5:23 AM, Brock Noland br...@cloudera.com
 wrote:
 
  Congrats Yin!!
 
  On Wed, Sep 4, 2013 at 4:14 AM, Lefty Leverenz 
 leftylever...@gmail.com
  wrote:
  Bravo, Yin!
 
  -- Lefty
 
 
  On Wed, Sep 4, 2013 at 4:17 AM, Sushanth Sowmyan khorg...@gmail.com
  wrote:
 
  Congrats, Yin! :)
  On Sep 4, 2013 1:13 AM, Alexander Alten-Lorenz 
 wget.n...@gmail.com
  wrote:
 
  Amazing news, congratz Yin! Well deserved!
 
  On Sep 4, 2013, at 6:49 AM, Carl Steinbach c...@apache.org wrote:
 
  The Apache Hive PMC has voted to make Yin Huai a committer on the
  Apache
  Hive project.
 
  Please join me in congratulating Yin!
 
  Thanks.
 
  Carl
 
  --
  Alexander Alten-Lorenz
  http://mapredit.blogspot.com
  German Hadoop LinkedIn Group: http://goo.gl/N8pCF
 
 
 
 
 
 
  --
  Apache MRUnit - Unit testing MapReduce - http://mrunit.apache.org
 
 
  --
  CONFIDENTIALITY NOTICE
  NOTICE: This message is intended for the use of the individual or
 entity to
  which it is addressed and may contain information that is confidential,
  privileged and exempt from disclosure under applicable law. If the
 reader
  of this message is not the intended recipient, you are hereby notified
 that
  any printing, copying, dissemination, distribution, disclosure or
  forwarding of this communication is strictly prohibited. If you have
  received this communication in error, please contact the sender
 immediately
  and delete it from your system. Thank You.
 
 
  --
  CONFIDENTIALITY NOTICE
  NOTICE: This message is intended for the use of the individual or entity
 to
  which it is addressed and may contain information that is confidential,
  privileged and exempt from disclosure under applicable law. If the reader
  of this message is not the intended recipient, you are hereby notified
 that
  any printing, copying, dissemination, distribution, disclosure or
  forwarding of this communication is strictly prohibited. If you have
  received this communication in error, please contact the sender
 immediately
  and delete it from your system. Thank You.


-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

Review Request 14180: HIVE-4531: [WebHCat] Collecting task logs to hdfs

[Daniel Dai]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/14180/
---

Review request for hive.


Bugs: HIVE-4531
https://issues.apache.org/jira/browse/HIVE-4531


Repository: hive


Description
---

SEE HIVE-4531.


Diffs
-


Diff: https://reviews.apache.org/r/14180/diff/


Testing
---

WebHCat unit tests
e2e tests in HIVE-5078 under both Linux/Windows


Thanks,

Daniel Dai

Re: Review Request 14180: HIVE-4531: [WebHCat] Collecting task logs to hdfs

[Daniel Dai]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/14180/
---

(Updated Sept. 18, 2013, 3:20 p.m.)


Review request for hive.


Changes
---

HIVE-4531-9.patch


Bugs: HIVE-4531
https://issues.apache.org/jira/browse/HIVE-4531


Repository: hive


Description
---

SEE HIVE-4531.


Diffs (updated)
-

  trunk/hcatalog/src/docs/src/documentation/content/xdocs/hive.xml 1524447 
  trunk/hcatalog/src/docs/src/documentation/content/xdocs/mapreducejar.xml 
1524447 
  
trunk/hcatalog/src/docs/src/documentation/content/xdocs/mapreducestreaming.xml 
1524447 
  trunk/hcatalog/src/docs/src/documentation/content/xdocs/pig.xml 1524447 
  
trunk/hcatalog/webhcat/svr/src/main/java/org/apache/hive/hcatalog/templeton/HiveDelegator.java
 1524447 
  
trunk/hcatalog/webhcat/svr/src/main/java/org/apache/hive/hcatalog/templeton/HiveJobIDParser.java
 PRE-CREATION 
  
trunk/hcatalog/webhcat/svr/src/main/java/org/apache/hive/hcatalog/templeton/JarDelegator.java
 1524447 
  
trunk/hcatalog/webhcat/svr/src/main/java/org/apache/hive/hcatalog/templeton/JarJobIDParser.java
 PRE-CREATION 
  
trunk/hcatalog/webhcat/svr/src/main/java/org/apache/hive/hcatalog/templeton/JobIDParser.java
 PRE-CREATION 
  
trunk/hcatalog/webhcat/svr/src/main/java/org/apache/hive/hcatalog/templeton/LauncherDelegator.java
 1524447 
  
trunk/hcatalog/webhcat/svr/src/main/java/org/apache/hive/hcatalog/templeton/LogRetriever.java
 PRE-CREATION 
  
trunk/hcatalog/webhcat/svr/src/main/java/org/apache/hive/hcatalog/templeton/PigDelegator.java
 1524447 
  
trunk/hcatalog/webhcat/svr/src/main/java/org/apache/hive/hcatalog/templeton/PigJobIDParser.java
 PRE-CREATION 
  
trunk/hcatalog/webhcat/svr/src/main/java/org/apache/hive/hcatalog/templeton/Server.java
 1524447 
  
trunk/hcatalog/webhcat/svr/src/main/java/org/apache/hive/hcatalog/templeton/StreamingDelegator.java
 1524447 
  
trunk/hcatalog/webhcat/svr/src/main/java/org/apache/hive/hcatalog/templeton/tool/TempletonControllerJob.java
 1524447 
  
trunk/hcatalog/webhcat/svr/src/main/java/org/apache/hive/hcatalog/templeton/tool/TempletonUtils.java
 1524447 
  trunk/hcatalog/webhcat/svr/src/test/data/status/hive/stderr PRE-CREATION 
  trunk/hcatalog/webhcat/svr/src/test/data/status/jar/stderr PRE-CREATION 
  trunk/hcatalog/webhcat/svr/src/test/data/status/pig/stderr PRE-CREATION 
  trunk/hcatalog/webhcat/svr/src/test/data/status/streaming/stderr PRE-CREATION 
  
trunk/hcatalog/webhcat/svr/src/test/java/org/apache/hive/hcatalog/templeton/TestJobIDParser.java
 PRE-CREATION 
  
trunk/hcatalog/webhcat/svr/src/test/java/org/apache/hive/hcatalog/templeton/tool/TestTempletonUtils.java
 1524447 

Diff: https://reviews.apache.org/r/14180/diff/


Testing
---

WebHCat unit tests
e2e tests in HIVE-5078 under both Linux/Windows


Thanks,

Daniel Dai

Re: Review Request 14180: HIVE-4531: [WebHCat] Collecting task logs to hdfs

[Daniel Dai]

 On Sept. 18, 2013, 11:49 p.m., Eugene Koifman wrote:
  trunk/hcatalog/webhcat/svr/src/main/java/org/apache/hive/hcatalog/templeton/LogRetriever.java,
   line 129
  https://reviews.apache.org/r/14180/diff/1/?file=353367#file353367line129
 
  finally {
  if(listWriter != null ) listWriter.close()
  }
  would be better

Actually I shall not close the listWriter in this case.


 On Sept. 18, 2013, 11:49 p.m., Eugene Koifman wrote:
  trunk/hcatalog/webhcat/svr/src/main/java/org/apache/hive/hcatalog/templeton/LogRetriever.java,
   line 199
  https://reviews.apache.org/r/14180/diff/1/?file=353367#file353367line199
 
  ArrayListString

Cannot create a generic array of ArrayListString


 On Sept. 18, 2013, 11:49 p.m., Eugene Koifman wrote:
  trunk/hcatalog/webhcat/svr/src/main/java/org/apache/hive/hcatalog/templeton/LogRetriever.java,
   line 204
  https://reviews.apache.org/r/14180/diff/1/?file=353367#file353367line204
 
  connection not closed

There is no close method in URLConnection. Close the underlining inputstream 
should be enough according to the docs.


 On Sept. 18, 2013, 11:49 p.m., Eugene Koifman wrote:
  trunk/hcatalog/webhcat/svr/src/main/java/org/apache/hive/hcatalog/templeton/LogRetriever.java,
   line 227
  https://reviews.apache.org/r/14180/diff/1/?file=353367#file353367line227
 
  shouldn't the connection be closed?

See previous comment.


 On Sept. 18, 2013, 11:49 p.m., Eugene Koifman wrote:
  trunk/hcatalog/webhcat/svr/src/main/java/org/apache/hive/hcatalog/templeton/LogRetriever.java,
   line 340
  https://reviews.apache.org/r/14180/diff/1/?file=353367#file353367line340
 
  close connection

See previous comment.


 On Sept. 18, 2013, 11:49 p.m., Eugene Koifman wrote:
  trunk/hcatalog/webhcat/svr/src/main/java/org/apache/hive/hcatalog/templeton/LogRetriever.java,
   line 221
  https://reviews.apache.org/r/14180/diff/1/?file=353367#file353367line221
 
  it seems better that this method use a try/catch(IOException)/finally 
  and handle cleaning up resources here, rather than make every caller do 
  this - all they do is write the stack trace to System.err

This is consistent with other similar methods:getCompletedAttempts, 
getFailedAttempts, etc. One awkward thing I don't like handling it in the 
method is: You will put method body in try/catch block, and in finally, you 
need to close the file within another try/catch block:
finally {
if (writer!=null) {
try {
writer.close();
catch (IOException e) {
}
}


 On Sept. 18, 2013, 11:49 p.m., Eugene Koifman wrote:
  trunk/hcatalog/webhcat/svr/src/main/java/org/apache/hive/hcatalog/templeton/tool/TempletonControllerJob.java,
   line 294
  https://reviews.apache.org/r/14180/diff/1/?file=353372#file353372line294
 
  why is this necessary?  
  There are 2 Watchers created in separate threads in this class.  Both 
  use System.err to log error messages.  If one closes 'err' while the other 
  gets an error right after - it will be a problem.  I think this 
  writer.close() creates a race condition...

I don't realize there are two watcher. I don't remember why I add that, 
probably because one issue I see. I am fine to remove it and keep a close eye 
on that for a while.


- Daniel


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/14180/#review26246
---


On Sept. 18, 2013, 3:20 p.m., Daniel Dai wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/14180/
 ---
 
 (Updated Sept. 18, 2013, 3:20 p.m.)
 
 
 Review request for hive.
 
 
 Bugs: HIVE-4531
 https://issues.apache.org/jira/browse/HIVE-4531
 
 
 Repository: hive
 
 
 Description
 ---
 
 SEE HIVE-4531.
 
 
 Diffs
 -
 
   trunk/hcatalog/src/docs/src/documentation/content/xdocs/hive.xml 1524447 
   trunk/hcatalog/src/docs/src/documentation/content/xdocs/mapreducejar.xml 
 1524447 
   
 trunk/hcatalog/src/docs/src/documentation/content/xdocs/mapreducestreaming.xml
  1524447 
   trunk/hcatalog/src/docs/src/documentation/content/xdocs/pig.xml 1524447 
   
 trunk/hcatalog/webhcat/svr/src/main/java/org/apache/hive/hcatalog/templeton/HiveDelegator.java
  1524447 
   
 trunk/hcatalog/webhcat/svr/src/main/java/org/apache/hive/hcatalog/templeton/HiveJobIDParser.java
  PRE-CREATION 
   
 trunk/hcatalog/webhcat/svr/src/main/java/org/apache/hive/hcatalog/templeton/JarDelegator.java
  1524447 
   
 trunk/hcatalog/webhcat/svr/src/main/java/org/apache/hive/hcatalog/templeton/JarJobIDParser.java
  PRE-CREATION 
   
 trunk/hcatalog/webhcat/svr/src/main/java/org/apache/hive/hcatalog/templeton/JobIDParser.java
  PRE-CREATION 
   
 trunk/hcatalog/webhcat/svr/src/main/java/org/apache/hive/hcatalog/templeton

Re: Where to put hcatalog branches and site code

[Daniel Dai]

Alan,
How do we run HCatalog only unit tests now?

Thanks,
Daniel

On Tue, Mar 26, 2013 at 5:02 PM, Alan Gates ga...@hortonworks.com wrote:
 Right after I moved the hcat code to hive/trunk/hcatalog Owen pointed out 
 that the problem with this is now everyone who checks out Hive pulls _all_ of 
 the hcat code.  This isn't what we want.

 The site code I propose we integrate with Hive's site code.  I'll put up a 
 patch for this shortly.

 The branches we could either move into Hive's branches directory (and move 
 them to hcatalog-branch-0.x) or we could create a /hive/hcatalog-historical 
 and put them there.  I'm fine with either.  Thoughts?

 Alan.

Review Request 24913: HIVE-7222: Support timestamp column statistics in ORC and extend PPD for timestamp

[Daniel Dai]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/24913/
---

Review request for hive and Prasanth_J.


Repository: hive


Description
---

Described in JIRA


Diffs
-

  
trunk/ql/src/gen/protobuf/gen-java/org/apache/hadoop/hive/ql/io/orc/OrcProto.java
 1619218 
  trunk/ql/src/java/org/apache/hadoop/hive/ql/io/orc/ColumnStatisticsImpl.java 
1619218 
  trunk/ql/src/java/org/apache/hadoop/hive/ql/io/orc/RecordReaderImpl.java 
1619218 
  
trunk/ql/src/java/org/apache/hadoop/hive/ql/io/orc/TimestampColumnStatistics.java
 PRE-CREATION 
  trunk/ql/src/java/org/apache/hadoop/hive/ql/io/orc/WriterImpl.java 1619218 
  trunk/ql/src/java/org/apache/hadoop/hive/ql/io/sarg/PredicateLeaf.java 
1619218 
  trunk/ql/src/java/org/apache/hadoop/hive/ql/io/sarg/SearchArgumentImpl.java 
1619218 
  trunk/ql/src/protobuf/org/apache/hadoop/hive/ql/io/orc/orc_proto.proto 
1619218 
  trunk/ql/src/test/queries/clientpositive/orc_ppd_timestamp.q PRE-CREATION 
  trunk/ql/src/test/results/clientpositive/orc_ppd_timestamp.q.out PRE-CREATION 

Diff: https://reviews.apache.org/r/24913/diff/


Testing
---


Thanks,

Daniel Dai

Review Request 32286: HIVE-9767 Fixes in Hive UDF to be usable in Pig

[Daniel Dai]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/32286/
---

Review request for hive and Jason Dere.


Repository: hive


Description
---

HIVE-9767 Fixes in Hive UDF to be usable in Pig


Diffs
-

  
trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFComputeStats.java
 1661840 
  
trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFContextNGrams.java
 1661840 
  trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFCount.java 
1661840 
  
trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFEWAHBitmap.java
 1661840 
  
trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFHistogramNumeric.java
 1661840 
  
trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFMkCollectionEvaluator.java
 1661840 
  
trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFPercentileApprox.java
 1661840 
  
trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFnGrams.java 
1661840 
  trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFCase.java 
1661840 
  trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFIn.java 
1661840 
  
trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFNamedStruct.java
 1661840 
  trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFUtils.java 
1661840 
  trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDTFStack.java 
1661840 
  trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/NGramEstimator.java 
1661840 
  trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/NumericHistogram.java 
1661840 

Diff: https://reviews.apache.org/r/32286/diff/


Testing
---


Thanks,

Daniel Dai

Re: Review Request 32286: HIVE-9767 Fixes in Hive UDF to be usable in Pig

[Daniel Dai]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/32286/
---

(Updated March 24, 2015, 1:23 a.m.)


Review request for hive and Jason Dere.


Repository: hive


Description
---

HIVE-9767 Fixes in Hive UDF to be usable in Pig


Diffs (updated)
-

  
trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFComputeStats.java
 1668765 
  
trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFContextNGrams.java
 1668765 
  trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFCount.java 
1668765 
  
trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFEWAHBitmap.java
 1668765 
  
trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFHistogramNumeric.java
 1668765 
  
trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFMkCollectionEvaluator.java
 1668765 
  
trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFPercentileApprox.java
 1668765 
  
trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFnGrams.java 
1668765 
  trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFCase.java 
1668765 
  trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFIn.java 
1668765 
  
trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFNamedStruct.java
 1668765 
  trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFUtils.java 
1668765 
  trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDTFStack.java 
1668765 
  trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/NGramEstimator.java 
1668765 
  trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/NumericHistogram.java 
1668765 

Diff: https://reviews.apache.org/r/32286/diff/


Testing
---


Thanks,

Daniel Dai

Review Request 38016: HIVE-11711: Merge hbase-metastore branch to trunk

[Daniel Dai]

/metastore/hbase/stats/StringColumnStatsAggregator.java
 PRE-CREATION 
  
metastore/src/java/org/apache/hadoop/hive/metastore/parser/ExpressionTree.java 
781ac63 
  
metastore/src/protobuf/org/apache/hadoop/hive/metastore/hbase/hbase_metastore_proto.proto
 PRE-CREATION 
  
metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreControlledCommit.java
 f184c56 
  
metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreForJdoConnection.java
 83fb4bb 
  metastore/src/test/org/apache/hadoop/hive/metastore/TestObjectStore.java 
a4f9f6c 
  metastore/src/test/org/apache/hadoop/hive/metastore/hbase/MockUtils.java 
PRE-CREATION 
  
metastore/src/test/org/apache/hadoop/hive/metastore/hbase/TestHBaseAggregateStatsCache.java
 PRE-CREATION 
  
metastore/src/test/org/apache/hadoop/hive/metastore/hbase/TestHBaseFilterPlanUtil.java
 PRE-CREATION 
  metastore/src/test/org/apache/hadoop/hive/metastore/hbase/TestHBaseStore.java 
PRE-CREATION 
  
metastore/src/test/org/apache/hadoop/hive/metastore/hbase/TestHBaseStoreCached.java
 PRE-CREATION 
  
metastore/src/test/org/apache/hadoop/hive/metastore/hbase/TestSharedStorageDescriptor.java
 PRE-CREATION 
  pom.xml 15c2805 
  ql/src/java/org/apache/hadoop/hive/ql/Driver.java 4030075 
  ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 734742c 
  ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java 396c070 
  ql/src/java/org/apache/hadoop/hive/ql/metadata/Partition.java 9546191 
  
ql/src/java/org/apache/hadoop/hive/ql/security/authorization/AuthorizationPreEventListener.java
 84f3f76 
  ql/src/java/org/apache/hadoop/hive/ql/stats/StatsUtils.java 55aea0e 
  ql/src/test/org/apache/hadoop/hive/metastore/TestMetastoreExpr.java e226528 
  ql/src/test/queries/clientpositive/dynpart_sort_opt_vectorization.q 8001081 
  ql/src/test/queries/clientpositive/dynpart_sort_optimization.q f842efe 
  ql/src/test/results/clientpositive/dynpart_sort_opt_vectorization.q.out 
216a79c 
  ql/src/test/results/clientpositive/dynpart_sort_optimization.q.out 41049bd 
  ql/src/test/results/clientpositive/tez/dynpart_sort_opt_vectorization.q.out 
4451046 
  ql/src/test/results/clientpositive/tez/dynpart_sort_optimization.q.out 
cb001b9 
  ql/src/test/templates/TestCliDriver.vm ae449c5 
  
serde/src/java/org/apache/hadoop/hive/serde2/binarysortable/BinarySortableSerDeWithEndPrefix.java
 PRE-CREATION 

Diff: https://reviews.apache.org/r/38016/diff/


Testing
---


Thanks,

Daniel Dai

Re: Review Request 38016: HIVE-11711: Merge hbase-metastore branch to trunk

[Daniel Dai]

/src/java/org/apache/hadoop/hive/metastore/hbase/stats/LongColumnStatsAggregator.java
 PRE-CREATION 
  
metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/StringColumnStatsAggregator.java
 PRE-CREATION 
  
metastore/src/java/org/apache/hadoop/hive/metastore/parser/ExpressionTree.java 
781ac63 
  
metastore/src/protobuf/org/apache/hadoop/hive/metastore/hbase/hbase_metastore_proto.proto
 PRE-CREATION 
  
metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreControlledCommit.java
 f184c56 
  
metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreForJdoConnection.java
 83fb4bb 
  metastore/src/test/org/apache/hadoop/hive/metastore/TestObjectStore.java 
a4f9f6c 
  metastore/src/test/org/apache/hadoop/hive/metastore/hbase/MockUtils.java 
PRE-CREATION 
  
metastore/src/test/org/apache/hadoop/hive/metastore/hbase/TestHBaseAggregateStatsCache.java
 PRE-CREATION 
  
metastore/src/test/org/apache/hadoop/hive/metastore/hbase/TestHBaseFilterPlanUtil.java
 PRE-CREATION 
  metastore/src/test/org/apache/hadoop/hive/metastore/hbase/TestHBaseStore.java 
PRE-CREATION 
  
metastore/src/test/org/apache/hadoop/hive/metastore/hbase/TestHBaseStoreCached.java
 PRE-CREATION 
  
metastore/src/test/org/apache/hadoop/hive/metastore/hbase/TestSharedStorageDescriptor.java
 PRE-CREATION 
  pom.xml b55e86a 
  ql/src/java/org/apache/hadoop/hive/ql/Driver.java 4030075 
  ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 734742c 
  ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java c449aee 
  ql/src/java/org/apache/hadoop/hive/ql/metadata/Partition.java 9546191 
  
ql/src/java/org/apache/hadoop/hive/ql/security/authorization/AuthorizationPreEventListener.java
 84f3f76 
  ql/src/java/org/apache/hadoop/hive/ql/stats/StatsUtils.java 55aea0e 
  ql/src/test/org/apache/hadoop/hive/metastore/TestMetastoreExpr.java e226528 
  ql/src/test/queries/clientpositive/dynpart_sort_opt_vectorization.q 8001081 
  ql/src/test/queries/clientpositive/dynpart_sort_optimization.q f842efe 
  ql/src/test/results/clientpositive/dynpart_sort_opt_vectorization.q.out 
216a79c 
  ql/src/test/results/clientpositive/dynpart_sort_optimization.q.out 41049bd 
  ql/src/test/results/clientpositive/tez/dynpart_sort_opt_vectorization.q.out 
4451046 
  ql/src/test/results/clientpositive/tez/dynpart_sort_optimization.q.out 
cb001b9 
  ql/src/test/templates/TestCliDriver.vm ae449c5 
  
serde/src/java/org/apache/hadoop/hive/serde2/binarysortable/BinarySortableSerDeWithEndPrefix.java
 PRE-CREATION 

Diff: https://reviews.apache.org/r/38016/diff/


Testing
---


Thanks,

Daniel Dai

Re: [ANNOUNCE] New Hive PMC Chair - Ashutosh Chauhan

[Daniel Dai]

Congratulations!

On 9/16/15, 2:20 PM, "Szehon Ho"  wrote:

>Congrats to Ashutosh and thanks Carl for the years of service!
>
>On Wed, Sep 16, 2015 at 2:00 PM, Eugene Koifman 
>wrote:
>
>> Congrats!
>>
>> From: Pengcheng Xiong 
>> Reply-To: "u...@hive.apache.org" 
>> Date: Wednesday, September 16, 2015 at 1:23 PM
>> To: "u...@hive.apache.org" 
>>
>> Cc: "dev@hive.apache.org" , Ashutosh Chauhan <
>> hashut...@apache.org>
>> Subject: Re: [ANNOUNCE] New Hive PMC Chair - Ashutosh Chauhan
>>
>> Congratulations Ashutosh!
>>
>> On Wed, Sep 16, 2015 at 1:17 PM, John Pullokkaran <
>> jpullokka...@hortonworks.com> wrote:
>>
>>> Congrats Ashutosh!
>>>
>>> From: Vaibhav Gumashta 
>>> Reply-To: "u...@hive.apache.org" 
>>> Date: Wednesday, September 16, 2015 at 1:01 PM
>>> To: "u...@hive.apache.org" ,
>>>"dev@hive.apache.org"
>>> 
>>> Cc: Ashutosh Chauhan 
>>> Subject: Re: [ANNOUNCE] New Hive PMC Chair - Ashutosh Chauhan
>>>
>>> Congrats Ashutosh!
>>>
>>> ‹Vaibhav
>>>
>>> From: Prasanth Jayachandran 
>>> Reply-To: "u...@hive.apache.org" 
>>> Date: Wednesday, September 16, 2015 at 12:50 PM
>>> To: "dev@hive.apache.org" ,
>>>"u...@hive.apache.org" <
>>> u...@hive.apache.org>
>>> Cc: "dev@hive.apache.org" , Ashutosh Chauhan <
>>> hashut...@apache.org>
>>> Subject: Re: [ANNOUNCE] New Hive PMC Chair - Ashutosh Chauhan
>>>
>>> Congratulations Ashutosh!
>>>
>>>
>>>
>>>
>>>
>>> On Wed, Sep 16, 2015 at 12:48 PM -0700, "Xuefu Zhang" <
>>> xzh...@cloudera.com> wrote:
>>>
>>> Congratulations, Ashutosh!. Well-deserved.
>>>
>>> Thanks to Carl also for the hard work in the past few years!
>>>
>>> --Xuefu
>>>
>>> On Wed, Sep 16, 2015 at 12:39 PM, Carl Steinbach 
>>>wrote:
>>>
>>> > I am very happy to announce that Ashutosh Chauhan is taking over as
>>>the
>>> > new VP of the Apache Hive project. Ashutosh has been a longtime
>>> contributor
>>> > to Hive and has played a pivotal role in many of the major advances
>>>that
>>> > have been made over the past couple of years. Please join me in
>>> > congratulating Ashutosh on his new role!
>>> >
>>>
>>
>>

Re: Review Request 38016: HIVE-11711: Merge hbase-metastore branch to trunk

[Daniel Dai]

/src/java/org/apache/hadoop/hive/metastore/hbase/stats/LongColumnStatsAggregator.java
 PRE-CREATION 
  
metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/StringColumnStatsAggregator.java
 PRE-CREATION 
  
metastore/src/java/org/apache/hadoop/hive/metastore/parser/ExpressionTree.java 
781ac63 
  
metastore/src/protobuf/org/apache/hadoop/hive/metastore/hbase/hbase_metastore_proto.proto
 PRE-CREATION 
  
metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreControlledCommit.java
 f184c56 
  
metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreForJdoConnection.java
 83fb4bb 
  metastore/src/test/org/apache/hadoop/hive/metastore/TestObjectStore.java 
a4f9f6c 
  metastore/src/test/org/apache/hadoop/hive/metastore/hbase/MockUtils.java 
PRE-CREATION 
  
metastore/src/test/org/apache/hadoop/hive/metastore/hbase/TestHBaseAggregateStatsCache.java
 PRE-CREATION 
  
metastore/src/test/org/apache/hadoop/hive/metastore/hbase/TestHBaseFilterPlanUtil.java
 PRE-CREATION 
  metastore/src/test/org/apache/hadoop/hive/metastore/hbase/TestHBaseStore.java 
PRE-CREATION 
  
metastore/src/test/org/apache/hadoop/hive/metastore/hbase/TestHBaseStoreCached.java
 PRE-CREATION 
  
metastore/src/test/org/apache/hadoop/hive/metastore/hbase/TestSharedStorageDescriptor.java
 PRE-CREATION 
  pom.xml b55e86a 
  ql/src/java/org/apache/hadoop/hive/ql/Driver.java 4030075 
  ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 734742c 
  ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java c449aee 
  ql/src/java/org/apache/hadoop/hive/ql/metadata/Partition.java 9546191 
  
ql/src/java/org/apache/hadoop/hive/ql/security/authorization/AuthorizationPreEventListener.java
 84f3f76 
  ql/src/java/org/apache/hadoop/hive/ql/stats/StatsUtils.java 55aea0e 
  ql/src/test/org/apache/hadoop/hive/metastore/TestMetastoreExpr.java e226528 
  ql/src/test/queries/clientpositive/dynpart_sort_opt_vectorization.q 8001081 
  ql/src/test/queries/clientpositive/dynpart_sort_optimization.q f842efe 
  ql/src/test/results/clientpositive/dynpart_sort_opt_vectorization.q.out 
216a79c 
  ql/src/test/results/clientpositive/dynpart_sort_optimization.q.out 41049bd 
  ql/src/test/results/clientpositive/tez/dynpart_sort_opt_vectorization.q.out 
4451046 
  ql/src/test/results/clientpositive/tez/dynpart_sort_optimization.q.out 
cb001b9 
  ql/src/test/templates/TestCliDriver.vm ae449c5 
  
serde/src/java/org/apache/hadoop/hive/serde2/binarysortable/BinarySortableSerDeWithEndPrefix.java
 PRE-CREATION 

Diff: https://reviews.apache.org/r/38016/diff/


Testing
---


Thanks,

Daniel Dai

Re: Invitation for Hive committers to become ORC committers

[Daniel Dai]

I am interested.

Thanks,
Daniel




On 12/15/16, 1:12 PM, "Owen O'Malley"  wrote:

>All,
>   As you are aware, we are in the last stages of removing the forked ORC
>code out of Hive. The goal of moving ORC out of Hive was to increase its
>community and we want to be very deliberately inclusive of the Hive
>development community. Towards that end, the ORC PMC wants to welcome
>anyone who is already a Hive committer to become a committer on ORC.
>
>  Please respond on this thread to let us know if you are interested.
>
>Thanks,
>   Owen on behalf of the ORC PMC

Re: Review Request 54826: HIVE-15448: ChangeManager for replication

[Daniel Dai]

> On Dec. 20, 2016, 8:56 p.m., Thejas Nair wrote:
> > metastore/src/java/org/apache/hadoop/hive/metastore/ReplChangeManager.java, 
> > line 175
> > <https://reviews.apache.org/r/54826/diff/1/?file=1588285#file1588285line175>
> >
> > is there any generic way to find this from FileSystem api ? these 
> > configs are specific to HDFS.
> > But i guess we don't need this if we go with storing only the signature 
> > (see following comment)

I checked HDFS code, doesn't seems another way to do that


> On Dec. 20, 2016, 8:56 p.m., Thejas Nair wrote:
> > itests/hive-unit/src/test/java/org/apache/hadoop/hive/metastore/TestReplChangeManager.java,
> >  line 56
> > <https://reviews.apache.org/r/54826/diff/1/?file=1588283#file1588283line56>
> >
> > with recent changes by Vaibhav to ProxyFileSystem, local files would 
> > also have checksums. Can we switch to local files, so that test speed is 
> > better (local fs might be more reliable than minidfs, as it has less parts 
> > to it!)

setXAttr will need HDFS


- Daniel


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/54826/#review159520
---


On Dec. 16, 2016, 11:14 p.m., Daniel Dai wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/54826/
> ---
> 
> (Updated Dec. 16, 2016, 11:14 p.m.)
> 
> 
> Review request for hive and Thejas Nair.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> See HIVE-15448
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 9064e49 
>   
> itests/hive-unit/src/test/java/org/apache/hadoop/hive/metastore/TestReplChangeManager.java
>  PRE-CREATION 
>   metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java 
> f7b2ed7 
>   metastore/src/java/org/apache/hadoop/hive/metastore/ReplChangeManager.java 
> PRE-CREATION 
>   metastore/src/java/org/apache/hadoop/hive/metastore/Warehouse.java 6aca1b7 
> 
> Diff: https://reviews.apache.org/r/54826/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Daniel Dai
> 
>

Re: Review Request 54826: HIVE-15448: ChangeManager for replication

[Daniel Dai]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/54826/
---

(Updated Dec. 23, 2016, 8:29 a.m.)


Review request for hive and Thejas Nair.


Repository: hive-git


Description
---

See HIVE-15448


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java b4e89b0 
  
itests/hive-unit/src/test/java/org/apache/hadoop/hive/metastore/TestReplChangeManager.java
 PRE-CREATION 
  metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java 
2892da3 
  metastore/src/java/org/apache/hadoop/hive/metastore/ReplChangeManager.java 
PRE-CREATION 
  metastore/src/java/org/apache/hadoop/hive/metastore/Warehouse.java 6aca1b7 

Diff: https://reviews.apache.org/r/54826/diff/


Testing
---


Thanks,

Daniel Dai

Re: Review Request 54826: HIVE-15448: ChangeManager for replication

[Daniel Dai]

> On Dec. 27, 2016, 6:26 a.m., Thejas Nair wrote:
> > metastore/src/java/org/apache/hadoop/hive/metastore/ReplChangeManager.java, 
> > line 55
> > <https://reviews.apache.org/r/54826/diff/1-2/?file=1588285#file1588285line55>
> >
> > seems better as local to the reycle method, as its not used outside.

I mean to cache user/group as it should not be changed. I don't want to 
calculate them everytime, and taking risk current user being changed unexpected.


- Daniel


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/54826/#review160135
---


On Dec. 27, 2016, 6:55 a.m., Daniel Dai wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/54826/
> ---
> 
> (Updated Dec. 27, 2016, 6:55 a.m.)
> 
> 
> Review request for hive and Thejas Nair.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> See HIVE-15448
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java b4e89b0 
>   
> itests/hive-unit/src/test/java/org/apache/hadoop/hive/metastore/TestReplChangeManager.java
>  PRE-CREATION 
>   metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java 
> 2892da3 
>   metastore/src/java/org/apache/hadoop/hive/metastore/ReplChangeManager.java 
> PRE-CREATION 
>   metastore/src/java/org/apache/hadoop/hive/metastore/Warehouse.java 6aca1b7 
> 
> Diff: https://reviews.apache.org/r/54826/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Daniel Dai
> 
>

Re: Review Request 54826: HIVE-15448: ChangeManager for replication

[Daniel Dai]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/54826/
---

(Updated Dec. 27, 2016, 6:55 a.m.)


Review request for hive and Thejas Nair.


Repository: hive-git


Description
---

See HIVE-15448


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java b4e89b0 
  
itests/hive-unit/src/test/java/org/apache/hadoop/hive/metastore/TestReplChangeManager.java
 PRE-CREATION 
  metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java 
2892da3 
  metastore/src/java/org/apache/hadoop/hive/metastore/ReplChangeManager.java 
PRE-CREATION 
  metastore/src/java/org/apache/hadoop/hive/metastore/Warehouse.java 6aca1b7 

Diff: https://reviews.apache.org/r/54826/diff/


Testing
---


Thanks,

Daniel Dai

Re: Review Request 54826: HIVE-15448: ChangeManager for replication

[Daniel Dai]

> On Dec. 27, 2016, 6:26 a.m., Thejas Nair wrote:
> > metastore/src/java/org/apache/hadoop/hive/metastore/ReplChangeManager.java, 
> > line 55
> > <https://reviews.apache.org/r/54826/diff/1-2/?file=1588285#file1588285line55>
> >
> > seems better as local to the reycle method, as its not used outside.
> 
> Daniel Dai wrote:
> I mean to cache user/group as it should not be changed. I don't want to 
> calculate them everytime, and taking risk current user being changed 
> unexpected.

Put in the construct


- Daniel


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/54826/#review160135
-------


On Dec. 27, 2016, 6:55 a.m., Daniel Dai wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/54826/
> ---
> 
> (Updated Dec. 27, 2016, 6:55 a.m.)
> 
> 
> Review request for hive and Thejas Nair.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> See HIVE-15448
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java b4e89b0 
>   
> itests/hive-unit/src/test/java/org/apache/hadoop/hive/metastore/TestReplChangeManager.java
>  PRE-CREATION 
>   metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java 
> 2892da3 
>   metastore/src/java/org/apache/hadoop/hive/metastore/ReplChangeManager.java 
> PRE-CREATION 
>   metastore/src/java/org/apache/hadoop/hive/metastore/Warehouse.java 6aca1b7 
> 
> Diff: https://reviews.apache.org/r/54826/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Daniel Dai
> 
>

Re: Review Request 54826: HIVE-15448: ChangeManager for replication

[Daniel Dai]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/54826/
---

(Updated Dec. 27, 2016, 4:44 p.m.)


Review request for hive and Thejas Nair.


Repository: hive-git


Description
---

See HIVE-15448


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java b4e89b0 
  
itests/hive-unit/src/test/java/org/apache/hadoop/hive/metastore/TestReplChangeManager.java
 PRE-CREATION 
  metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java 
2892da3 
  metastore/src/java/org/apache/hadoop/hive/metastore/ReplChangeManager.java 
PRE-CREATION 
  metastore/src/java/org/apache/hadoop/hive/metastore/Warehouse.java 6aca1b7 

Diff: https://reviews.apache.org/r/54826/diff/


Testing
---


Thanks,

Daniel Dai

Review Request 54826: HIVE-15448: ChangeManager for replication

[Daniel Dai]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/54826/
---

Review request for hive and Thejas Nair.


Repository: hive-git


Description
---

See HIVE-15448


Diffs
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 9064e49 
  
itests/hive-unit/src/test/java/org/apache/hadoop/hive/metastore/TestReplChangeManager.java
 PRE-CREATION 
  metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java 
f7b2ed7 
  metastore/src/java/org/apache/hadoop/hive/metastore/ReplChangeManager.java 
PRE-CREATION 
  metastore/src/java/org/apache/hadoop/hive/metastore/Warehouse.java 6aca1b7 

Diff: https://reviews.apache.org/r/54826/diff/


Testing
---


Thanks,

Daniel Dai

Re: Review Request 59205: HIVE-16579: CachedStore: improvements to partition col stats caching and cache column stats for unpartitioned table

[Daniel Dai]

> On May 14, 2017, 7:38 a.m., Daniel Dai wrote:
> > metastore/src/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java
> > Lines 528 (patched)
> > <https://reviews.apache.org/r/59205/diff/4/?file=1717995#file1717995line554>
> >
> > If using dirty flag, databaseCacheLock can be removed, right?
> 
> Thejas Nair wrote:
> We still need some locking/synchronization to prevent things like this -
> 
> 
> 1. If (isDirtyFlag == false) {
> 2.  // if db gets updated in another thread while this thread is here, we 
> lose the update.
> 3.updateDBCache()
> 2. }
> 
> A read/write lock similar to what is used here appropriate for that. The 
> background thread is the one that holds the write lock.
> However, not sure if we need to busy-wait with trylock though. (I figure, 
> the intent is to give this one higher priority). The update DbCache part 
> should not take that long. (The busy-wait can be dealt with later also).

A simpler solution is move dirtyFlag to SharedCache. 
SharedCache.refreshDatabases will check the flag and 
SharedCache.addDatabaseToCache/alterDatabaseInCache/removeDatabaseFromCache 
will set the flag. There is no synchronization issue in SharedCache since every 
method is synchronized.


- Daniel


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59205/#review174907
---


On May 14, 2017, 5:19 a.m., Vaibhav Gumashta wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/59205/
> -----------
> 
> (Updated May 14, 2017, 5:19 a.m.)
> 
> 
> Review request for hive, Daniel Dai and Thejas Nair.
> 
> 
> Bugs: HIVE-16579
> https://issues.apache.org/jira/browse/HIVE-16579
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> https://issues.apache.org/jira/browse/HIVE-16579
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java d6a80ae 
>   
> itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/listener/DummyRawStoreFailEvent.java
>  91a3a38 
>   itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java b897ffa 
>   metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java 
> b96c27e 
>   metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java 
> 870896c 
>   metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java 
> ed19f42 
>   metastore/src/java/org/apache/hadoop/hive/metastore/RawStore.java c1af690 
>   metastore/src/java/org/apache/hadoop/hive/metastore/cache/CacheUtils.java 
> 668499b 
>   metastore/src/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java 
> 5a187d8 
>   metastore/src/java/org/apache/hadoop/hive/metastore/cache/SharedCache.java 
> 7beee42 
>   metastore/src/java/org/apache/hadoop/hive/metastore/hbase/HBaseStore.java 
> 0c7d8bb 
>   
> metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreControlledCommit.java
>  f613c30 
>   
> metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreForJdoConnection.java
>  1720e37 
>   
> metastore/src/test/org/apache/hadoop/hive/metastore/cache/TestCachedStore.java
>  0ab20d6 
> 
> 
> Diff: https://reviews.apache.org/r/59205/diff/4/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Vaibhav Gumashta
> 
>

Re: Review Request 59205: HIVE-16579: CachedStore: improvements to partition col stats caching and cache column stats for unpartitioned table

[Daniel Dai]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59205/#review175354
---




metastore/src/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java
Lines 827 (patched)
<https://reviews.apache.org/r/59205/#comment248801>

There is no need to alter partition/partition stats/table stats here. 
HiveAlterHandler will invoke alterPartition, updateTableColumnStatistics and 
updatePartitionColumnStatistics respectively.


- Daniel Dai


On May 17, 2017, 6:47 p.m., Vaibhav Gumashta wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/59205/
> ---
> 
> (Updated May 17, 2017, 6:47 p.m.)
> 
> 
> Review request for hive, Daniel Dai and Thejas Nair.
> 
> 
> Bugs: HIVE-16579
> https://issues.apache.org/jira/browse/HIVE-16579
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> https://issues.apache.org/jira/browse/HIVE-16579
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 1c37b6e 
>   
> itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/listener/DummyRawStoreFailEvent.java
>  91a3a38 
>   itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java d296851 
>   metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java 
> b96c27e 
>   metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java 
> 870896c 
>   metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java 
> b28983f 
>   metastore/src/java/org/apache/hadoop/hive/metastore/RawStore.java c1af690 
>   
> metastore/src/java/org/apache/hadoop/hive/metastore/StatObjectConverter.java 
> fcf6f27 
>   metastore/src/java/org/apache/hadoop/hive/metastore/cache/CacheUtils.java 
> 668499b 
>   metastore/src/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java 
> 1cc838f 
>   metastore/src/java/org/apache/hadoop/hive/metastore/cache/SharedCache.java 
> 7beee42 
>   metastore/src/java/org/apache/hadoop/hive/metastore/hbase/HBaseStore.java 
> 0c7d8bb 
>   
> metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/merge/ColumnStatsMergerFactory.java
>  da6cd46 
>   
> metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/merge/DateColumnStatsMerger.java
>  PRE-CREATION 
>   
> metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreControlledCommit.java
>  f613c30 
>   
> metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreForJdoConnection.java
>  1720e37 
>   
> metastore/src/test/org/apache/hadoop/hive/metastore/cache/TestCachedStore.java
>  0ab20d6 
> 
> 
> Diff: https://reviews.apache.org/r/59205/diff/13/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Vaibhav Gumashta
> 
>

Re: Review Request 59205: HIVE-16579: CachedStore: improvements to partition col stats caching and cache column stats for unpartitioned table

[Daniel Dai]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59205/#review174907
---




metastore/src/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java
Line 381 (original), 526 (patched)
<https://reviews.apache.org/r/59205/#comment248177>

I think we only need a dirty flag, it will be reset by the next refresh, 
not end of individual call.



metastore/src/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java
Lines 528 (patched)
<https://reviews.apache.org/r/59205/#comment248178>

If using dirty flag, databaseCacheLock can be removed, right?


- Daniel Dai


On May 14, 2017, 5:19 a.m., Vaibhav Gumashta wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/59205/
> ---
> 
> (Updated May 14, 2017, 5:19 a.m.)
> 
> 
> Review request for hive, Daniel Dai and Thejas Nair.
> 
> 
> Bugs: HIVE-16579
> https://issues.apache.org/jira/browse/HIVE-16579
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> https://issues.apache.org/jira/browse/HIVE-16579
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java d6a80ae 
>   
> itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/listener/DummyRawStoreFailEvent.java
>  91a3a38 
>   itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java b897ffa 
>   metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java 
> b96c27e 
>   metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java 
> 870896c 
>   metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java 
> ed19f42 
>   metastore/src/java/org/apache/hadoop/hive/metastore/RawStore.java c1af690 
>   metastore/src/java/org/apache/hadoop/hive/metastore/cache/CacheUtils.java 
> 668499b 
>   metastore/src/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java 
> 5a187d8 
>   metastore/src/java/org/apache/hadoop/hive/metastore/cache/SharedCache.java 
> 7beee42 
>   metastore/src/java/org/apache/hadoop/hive/metastore/hbase/HBaseStore.java 
> 0c7d8bb 
>   
> metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreControlledCommit.java
>  f613c30 
>   
> metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreForJdoConnection.java
>  1720e37 
>   
> metastore/src/test/org/apache/hadoop/hive/metastore/cache/TestCachedStore.java
>  0ab20d6 
> 
> 
> Diff: https://reviews.apache.org/r/59205/diff/4/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Vaibhav Gumashta
> 
>

Re: Review Request 59205: HIVE-16579: CachedStore: improvements to partition col stats caching and cache column stats for unpartitioned table

[Daniel Dai]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59205/#review175684
---


Ship it!




Ship It!

- Daniel Dai


On May 22, 2017, 7:06 p.m., Vaibhav Gumashta wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/59205/
> ---
> 
> (Updated May 22, 2017, 7:06 p.m.)
> 
> 
> Review request for hive, Daniel Dai and Thejas Nair.
> 
> 
> Bugs: HIVE-16579
> https://issues.apache.org/jira/browse/HIVE-16579
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> https://issues.apache.org/jira/browse/HIVE-16579
> 
> 
> Diffs
> -
> 
>   
> itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/listener/DummyRawStoreFailEvent.java
>  91a3a38 
>   itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java d296851 
>   metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java 
> b96c27e 
>   metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java 
> 870896c 
>   metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java 
> b28983f 
>   metastore/src/java/org/apache/hadoop/hive/metastore/RawStore.java c1af690 
>   
> metastore/src/java/org/apache/hadoop/hive/metastore/StatObjectConverter.java 
> fcf6f27 
>   metastore/src/java/org/apache/hadoop/hive/metastore/cache/CacheUtils.java 
> 668499b 
>   metastore/src/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java 
> 1cc838f 
>   metastore/src/java/org/apache/hadoop/hive/metastore/cache/SharedCache.java 
> 7beee42 
>   metastore/src/java/org/apache/hadoop/hive/metastore/hbase/HBaseStore.java 
> 0c7d8bb 
>   
> metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/merge/ColumnStatsMergerFactory.java
>  da6cd46 
>   
> metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/merge/DateColumnStatsMerger.java
>  PRE-CREATION 
>   
> metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreControlledCommit.java
>  f613c30 
>   
> metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreForJdoConnection.java
>  1720e37 
>   
> metastore/src/test/org/apache/hadoop/hive/metastore/cache/TestCachedStore.java
>  0ab20d6 
> 
> 
> Diff: https://reviews.apache.org/r/59205/diff/19/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Vaibhav Gumashta
> 
>

Review Request 62360: HIVE-16898: Validation of source file after distcp in repl load

[Daniel Dai]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62360/
---

Review request for hive.


Repository: hive-git


Description
---

See HIVE-16898


Diffs
-

  metastore/src/java/org/apache/hadoop/hive/metastore/ReplChangeManager.java 
88d6a7a 
  ql/src/java/org/apache/hadoop/hive/ql/exec/ReplCopyTask.java 54746d3 
  ql/src/java/org/apache/hadoop/hive/ql/parse/repl/CopyUtils.java 28e7bcb 


Diff: https://reviews.apache.org/r/62360/diff/1/


Testing
---

Manually test it with debugger: setup a breakpoint right before copy, and drop 
table in another session.


Thanks,

Daniel Dai

Re: Review Request 62360: HIVE-16898: Validation of source file after distcp in repl load

[Daniel Dai]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62360/
---

(Updated Sept. 19, 2017, 8:06 p.m.)


Review request for hive.


Repository: hive-git


Description
---

See HIVE-16898


Diffs (updated)
-

  metastore/src/java/org/apache/hadoop/hive/metastore/ReplChangeManager.java 
88d6a7a 
  ql/src/java/org/apache/hadoop/hive/ql/exec/ReplCopyTask.java 54746d3 
  ql/src/java/org/apache/hadoop/hive/ql/parse/repl/CopyUtils.java 28e7bcb 


Diff: https://reviews.apache.org/r/62360/diff/5/

Changes: https://reviews.apache.org/r/62360/diff/4-5/


Testing
---

Manually test it with debugger: setup a breakpoint right before copy, and drop 
table in another session.


Thanks,

Daniel Dai

Re: Review Request 62360: HIVE-16898: Validation of source file after distcp in repl load

[Daniel Dai]

> On Sept. 19, 2017, 5:46 a.m., anishek wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/parse/repl/CopyUtils.java
> > Lines 120 (patched)
> > <https://reviews.apache.org/r/62360/diff/3/?file=1828690#file1828690line120>
> >
> > May be i am missing something but even when we are using the CM path in 
> > copyAndVerify towards the last, the file from CM itself might be deleted 
> > via the CM cleaner thread, in that case the doCopy will quietly fail and 
> > move on rather than failing replication.

It is handled by copyAndVerify. If the source file is not there, getting 
checksum of source file will throw exception:
catch (FileNotFoundException e) {
  if (!srcFile.isUseSourcePath()) {
// If we already use CM path, that means the CM path is expired, throw 
exception
// as there is no remedy
throw e;
  }
}

Anyway, this is refactor in the new patch.


- Daniel


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62360/#review185643
-----------


On Sept. 19, 2017, 7:11 a.m., Daniel Dai wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/62360/
> ---
> 
> (Updated Sept. 19, 2017, 7:11 a.m.)
> 
> 
> Review request for hive.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> See HIVE-16898
> 
> 
> Diffs
> -
> 
>   metastore/src/java/org/apache/hadoop/hive/metastore/ReplChangeManager.java 
> 88d6a7a 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/ReplCopyTask.java 54746d3 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/repl/CopyUtils.java 28e7bcb 
> 
> 
> Diff: https://reviews.apache.org/r/62360/diff/4/
> 
> 
> Testing
> ---
> 
> Manually test it with debugger: setup a breakpoint right before copy, and 
> drop table in another session.
> 
> 
> Thanks,
> 
> Daniel Dai
> 
>

Re: Review Request 62360: HIVE-16898: Validation of source file after distcp in repl load

[Daniel Dai]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62360/
---

(Updated Sept. 19, 2017, 7:11 a.m.)


Review request for hive.


Repository: hive-git


Description
---

See HIVE-16898


Diffs (updated)
-

  metastore/src/java/org/apache/hadoop/hive/metastore/ReplChangeManager.java 
88d6a7a 
  ql/src/java/org/apache/hadoop/hive/ql/exec/ReplCopyTask.java 54746d3 
  ql/src/java/org/apache/hadoop/hive/ql/parse/repl/CopyUtils.java 28e7bcb 


Diff: https://reviews.apache.org/r/62360/diff/4/

Changes: https://reviews.apache.org/r/62360/diff/3-4/


Testing
---

Manually test it with debugger: setup a breakpoint right before copy, and drop 
table in another session.


Thanks,

Daniel Dai

Re: Review Request 62360: HIVE-16898: Validation of source file after distcp in repl load

[Daniel Dai]

> On Sept. 19, 2017, 5:48 a.m., anishek wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/parse/repl/CopyUtils.java
> > Lines 153 (patched)
> > <https://reviews.apache.org/r/62360/diff/3/?file=1828690#file1828690line153>
> >
> > use lambda

Already lambda.


- Daniel


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62360/#review185647
---


On Sept. 19, 2017, 7:11 a.m., Daniel Dai wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/62360/
> ---
> 
> (Updated Sept. 19, 2017, 7:11 a.m.)
> 
> 
> Review request for hive.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> See HIVE-16898
> 
> 
> Diffs
> -
> 
>   metastore/src/java/org/apache/hadoop/hive/metastore/ReplChangeManager.java 
> 88d6a7a 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/ReplCopyTask.java 54746d3 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/repl/CopyUtils.java 28e7bcb 
> 
> 
> Diff: https://reviews.apache.org/r/62360/diff/4/
> 
> 
> Testing
> ---
> 
> Manually test it with debugger: setup a breakpoint right before copy, and 
> drop table in another session.
> 
> 
> Thanks,
> 
> Daniel Dai
> 
>

Re: Review Request 62360: HIVE-16898: Validation of source file after distcp in repl load

[Daniel Dai]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62360/
---

(Updated Sept. 20, 2017, 9:41 p.m.)


Review request for hive.


Repository: hive-git


Description
---

See HIVE-16898


Diffs (updated)
-

  ql/src/java/org/apache/hadoop/hive/ql/exec/ReplCopyTask.java 54746d3 
  ql/src/java/org/apache/hadoop/hive/ql/parse/repl/CopyUtils.java 28e7bcb 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/ReplChangeManager.java
 dd9296a 


Diff: https://reviews.apache.org/r/62360/diff/6/

Changes: https://reviews.apache.org/r/62360/diff/5-6/


Testing
---

Manually test it with debugger: setup a breakpoint right before copy, and drop 
table in another session.


Thanks,

Daniel Dai

Re: Review Request 62360: HIVE-16898: Validation of source file after distcp in repl load

[Daniel Dai]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62360/
---

(Updated Sept. 21, 2017, 4:36 p.m.)


Review request for hive.


Repository: hive-git


Description
---

See HIVE-16898


Diffs (updated)
-

  ql/src/java/org/apache/hadoop/hive/ql/exec/ReplCopyTask.java 54746d3 
  ql/src/java/org/apache/hadoop/hive/ql/parse/repl/CopyUtils.java 28e7bcb 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/ReplChangeManager.java
 dd9296a 


Diff: https://reviews.apache.org/r/62360/diff/7/

Changes: https://reviews.apache.org/r/62360/diff/6-7/


Testing
---

Manually test it with debugger: setup a breakpoint right before copy, and drop 
table in another session.


Thanks,

Daniel Dai

Re: Review Request 62360: HIVE-16898: Validation of source file after distcp in repl load

[Daniel Dai]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62360/
---

(Updated Sept. 18, 2017, 10:55 p.m.)


Review request for hive.


Repository: hive-git


Description
---

See HIVE-16898


Diffs (updated)
-

  metastore/src/java/org/apache/hadoop/hive/metastore/ReplChangeManager.java 
88d6a7a 
  ql/src/java/org/apache/hadoop/hive/ql/exec/ReplCopyTask.java 54746d3 
  ql/src/java/org/apache/hadoop/hive/ql/parse/repl/CopyUtils.java 28e7bcb 


Diff: https://reviews.apache.org/r/62360/diff/2/

Changes: https://reviews.apache.org/r/62360/diff/1-2/


Testing
---

Manually test it with debugger: setup a breakpoint right before copy, and drop 
table in another session.


Thanks,

Daniel Dai

Re: Review Request 62360: HIVE-16898: Validation of source file after distcp in repl load

[Daniel Dai]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62360/
---

(Updated Sept. 19, 2017, 12:14 a.m.)


Review request for hive.


Repository: hive-git


Description
---

See HIVE-16898


Diffs (updated)
-

  metastore/src/java/org/apache/hadoop/hive/metastore/ReplChangeManager.java 
88d6a7a 
  ql/src/java/org/apache/hadoop/hive/ql/exec/ReplCopyTask.java 54746d3 
  ql/src/java/org/apache/hadoop/hive/ql/parse/repl/CopyUtils.java 28e7bcb 


Diff: https://reviews.apache.org/r/62360/diff/3/

Changes: https://reviews.apache.org/r/62360/diff/2-3/


Testing
---

Manually test it with debugger: setup a breakpoint right before copy, and drop 
table in another session.


Thanks,

Daniel Dai

Re: Review Request 62360: HIVE-16898: Validation of source file after distcp in repl load

[Daniel Dai]

> On Sept. 18, 2017, 4:49 a.m., anishek wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/parse/repl/CopyUtils.java
> > Lines 73 (patched)
> > <https://reviews.apache.org/r/62360/diff/1/?file=1828081#file1828081line73>
> >
> > Evaluation of doing a regularCopy or distCp can be done in the inner 
> > most function call, this will reduce passing in another variable from the 
> > top which can be evaluated later

I need to cache useRegularCopy and pass it to multiple doCopyRetry, that's why 
I put in the outer function.


> On Sept. 18, 2017, 4:49 a.m., anishek wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/parse/repl/CopyUtils.java
> > Lines 92 (patched)
> > <https://reviews.apache.org/r/62360/diff/1/?file=1828081#file1828081line92>
> >
> > I think eventually we have to move to a model of doing the checksum on 
> > sourceFS vs destinationFS as you have done here, though certain FS 
> > configurations change the value of checksum and unless we can guarantee 
> > that we calculate the checksum on the data by reading the data this might 
> > lead to more failures,
> > 
> > I thought the idea for now was that,
> > 
> > 1>> we get the checksum of the file on sourceFS before copy
> > 2>> we do the copy
> > 3>> we get the checksum on the file on sourceFS again 
> > 4>> we compare the checksum in 1 and 3 and if its not changed then 
> > during our copy the value wouldnt have either. 
> > 
> > until we can figure out the acutal solution to this, the fall back of 
> > doing the check on sourceFS might be the way to go.

Yes, that's right. The checksum of the file is in _files.


> On Sept. 18, 2017, 4:49 a.m., anishek wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/parse/repl/CopyUtils.java
> > Lines 116 (patched)
> > <https://reviews.apache.org/r/62360/diff/1/?file=1828081#file1828081line116>
> >
> > As a part of doing copy if the copy fails due to fileNotFoundException 
> > for a file location to actual location on hdfs then we should retry with 
> > the corresponding CMRoot Path for this file since it was moved while we 
> > were in the porcess of doing the copy.
> > 
> > Also if this happnes for a CM root file then there is an issue in our 
> > configuration such that the CM root FS is cleaned before the copy is done 
> > and we should log this as an error as the cleaner thread for CMroot is not 
> > configured for the right time. i did rather fail repl load, instead of just 
> > logging the error else we might not know how many such instances might 
> > happen before we realize that replication is broken.

Retry with CM path is part of copyAndVerify. doCopyRetry is shared between 
regular import and repl load, it does not deal with CM logic.


- Daniel


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62360/#review185534
---


On Sept. 15, 2017, 6:10 p.m., Daniel Dai wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/62360/
> ---
> 
> (Updated Sept. 15, 2017, 6:10 p.m.)
> 
> 
> Review request for hive.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> See HIVE-16898
> 
> 
> Diffs
> -
> 
>   metastore/src/java/org/apache/hadoop/hive/metastore/ReplChangeManager.java 
> 88d6a7a 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/ReplCopyTask.java 54746d3 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/repl/CopyUtils.java 28e7bcb 
> 
> 
> Diff: https://reviews.apache.org/r/62360/diff/1/
> 
> 
> Testing
> ---
> 
> Manually test it with debugger: setup a breakpoint right before copy, and 
> drop table in another session.
> 
> 
> Thanks,
> 
> Daniel Dai
> 
>

Re: Review Request 62228: HIVE-17495: CachedStore: prewarm improvements, refactoring and caching some aggregate stats

[Daniel Dai]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62228/#review185229
---




metastore/src/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java
Line 258 (original), 264 (patched)
<https://reviews.apache.org/r/62228/#comment261520>

If we want to reduce the num of sql statement, shall we do the same single 
fetch to all table statistics for symmetry?



metastore/src/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java
Lines 295 (patched)
<https://reviews.apache.org/r/62228/#comment261521>

It should be better to get both all/allbutdefault statistics from 
mergeColStatsForPartitions, rather than invoke get_aggr_stats_for twice, that 
would be colstly.


- Daniel Dai


On Sept. 11, 2017, 9:25 p.m., Vaibhav Gumashta wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/62228/
> ---
> 
> (Updated Sept. 11, 2017, 9:25 p.m.)
> 
> 
> Review request for hive, Ashutosh Chauhan and Thejas Nair.
> 
> 
> Bugs: HIVE-17495
> https://issues.apache.org/jira/browse/HIVE-17495
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> https://issues.apache.org/jira/browse/HIVE-17495
> 
> 
> Diffs
> -
> 
>   
> itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/listener/DummyRawStoreFailEvent.java
>  8d861e4 
>   metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java 
> dc1245e 
>   metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java 
> bbe13fd 
>   metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java 
> 3053dcb 
>   metastore/src/java/org/apache/hadoop/hive/metastore/RawStore.java 71982a0 
>   metastore/src/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java 
> 3ba81ce 
>   metastore/src/java/org/apache/hadoop/hive/metastore/cache/SharedCache.java 
> 80b17e0 
>   
> metastore/src/java/org/apache/hadoop/hive/metastore/columnstats/aggr/BinaryColumnStatsAggregator.java
>  e6c836b 
>   
> metastore/src/java/org/apache/hadoop/hive/metastore/columnstats/aggr/BooleanColumnStatsAggregator.java
>  a34bc9f 
>   
> metastore/src/java/org/apache/hadoop/hive/metastore/columnstats/aggr/ColumnStatsAggregator.java
>  a52e5e5 
>   
> metastore/src/java/org/apache/hadoop/hive/metastore/columnstats/aggr/ColumnStatsAggregatorFactory.java
>  dfae708 
>   
> metastore/src/java/org/apache/hadoop/hive/metastore/columnstats/aggr/DateColumnStatsAggregator.java
>  ee95396 
>   
> metastore/src/java/org/apache/hadoop/hive/metastore/columnstats/aggr/DecimalColumnStatsAggregator.java
>  284c12c 
>   
> metastore/src/java/org/apache/hadoop/hive/metastore/columnstats/aggr/DoubleColumnStatsAggregator.java
>  bb4a725 
>   
> metastore/src/java/org/apache/hadoop/hive/metastore/columnstats/aggr/LongColumnStatsAggregator.java
>  5b1145e 
>   
> metastore/src/java/org/apache/hadoop/hive/metastore/columnstats/aggr/StringColumnStatsAggregator.java
>  1b29f92 
>   
> metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreControlledCommit.java
>  4db203d 
>   
> metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreForJdoConnection.java
>  fb16cfc 
> 
> 
> Diff: https://reviews.apache.org/r/62228/diff/1/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Vaibhav Gumashta
> 
>

Re: Review Request 66571: HIVE-19161: Add authorizations to information schema

[Daniel Dai]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66571/
---

(Updated April 28, 2018, 1:09 a.m.)


Review request for hive.


Repository: hive-git


Description
---

See HIVE-19161


Diffs (updated)
-

  
accumulo-handler/src/java/org/apache/hadoop/hive/accumulo/HiveAccumuloHelper.java
 9fccb49 
  accumulo-handler/src/java/org/apache/hadoop/hive/accumulo/Utils.java 3a2facf 
  
accumulo-handler/src/java/org/apache/hadoop/hive/accumulo/serde/CompositeAccumuloRowIdFactory.java
 d8b9aa3 
  
accumulo-handler/src/java/org/apache/hadoop/hive/accumulo/serde/DefaultAccumuloRowIdFactory.java
 bae2930 
  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java f40c606 
  
itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/listener/DummyRawStoreFailEvent.java
 8ecbaad 
  itests/hive-unit/pom.xml 3ae7f2f 
  itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestRestrictedList.java 
79fdb68 
  
itests/hive-unit/src/test/java/org/apache/hive/service/server/TestInformationSchemaWithPrivilege.java
 PRE-CREATION 
  
jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/JdbcStorageHandler.java 
df55272 
  
jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/dao/DatabaseAccessorFactory.java
 6d3c8d9 
  
jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/dao/GenericJdbcDatabaseAccessor.java
 772bc5d 
  
jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/dao/JdbcRecordIterator.java
 638e2b0 
  
jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/dao/MsSqlDatabaseAccessor.java
 PRE-CREATION 
  
jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/dao/OracleDatabaseAccessor.java
 PRE-CREATION 
  
jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/dao/PostgresDatabaseAccessor.java
 PRE-CREATION 
  metastore/scripts/upgrade/hive/hive-schema-3.0.0.hive.sql 339 
  ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java d59bf1f 
  ql/src/java/org/apache/hadoop/hive/ql/metadata/JarUtils.java PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/security/authorization/HiveAuthorizationProvider.java
 60d9dc1 
  
ql/src/java/org/apache/hadoop/hive/ql/security/authorization/PrivilegeSynchonizer.java
 PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/security/authorization/plugin/HiveResourceACLsImpl.java
 PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java 60b63d4 
  
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFCurrentGroups.java 
PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFRestrictInformationSchema.java
 PRE-CREATION 
  ql/src/test/results/clientpositive/llap/resourceplan.q.out 9850276 
  ql/src/test/results/clientpositive/show_functions.q.out 4df555b 
  service/src/java/org/apache/hive/service/server/HiveServer2.java e373628 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
 397a081 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java
 1c8d223 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/IMetaStoreClient.java
 aee416d 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
 184ecb6 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/RawStore.java
 2c9f2e5 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java
 92d000b 
  standalone-metastore/src/main/thrift/hive_metastore.thrift c56a4f9 
  
standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/DummyRawStoreControlledCommit.java
 defc68f 
  
standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/DummyRawStoreForJdoConnection.java
 20c5d8a 
  
standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClientPreCatalog.java
 bf87cfc 


Diff: https://reviews.apache.org/r/66571/diff/6/

Changes: https://reviews.apache.org/r/66571/diff/5-6/


Testing
---


Thanks,

Daniel Dai

Review Request 67102: HIVE-19440: Make StorageBasedAuthorizer work with information schema

[Daniel Dai]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67102/
---

Review request for hive.


Repository: hive-git


Description
---

See HIVE-19440


Diffs
-

  
hcatalog/core/src/main/java/org/apache/hive/hcatalog/storagehandler/DummyHCatAuthProvider.java
 a53028f 
  
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/security/TestHDFSPermissionPolicyProvider.java
 PRE-CREATION 
  
itests/util/src/main/java/org/apache/hadoop/hive/ql/security/DummyHiveMetastoreAuthorizationProvider.java
 31e795c 
  metastore/scripts/upgrade/hive/hive-schema-3.0.0.hive.sql d9606d8 
  ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java 4611ce9 
  
ql/src/java/org/apache/hadoop/hive/ql/security/authorization/HDFSPermissionPolicyProvider.java
 PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/security/authorization/HiveAuthorizationProviderBase.java
 8a7c06d 
  
ql/src/java/org/apache/hadoop/hive/ql/security/authorization/HiveMetastoreAuthorizationProvider.java
 0dab334 
  
ql/src/java/org/apache/hadoop/hive/ql/security/authorization/PolicyProviderContainer.java
 PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/security/authorization/PrivilegeSynchonizer.java
 9b2e6cd 
  
ql/src/java/org/apache/hadoop/hive/ql/security/authorization/StorageBasedAuthorizationProvider.java
 b66d188 
  
ql/src/java/org/apache/hadoop/hive/ql/security/authorization/plugin/HiveV1Authorizer.java
 48798d8 
  
ql/src/java/org/apache/hadoop/hive/ql/security/authorization/plugin/sqlstd/SQLAuthorizationUtils.java
 02ed7aa 
  
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFCurrentAuthorizer.java
 PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFRestrictInformationSchema.java
 3eb0914 
  service/src/java/org/apache/hive/service/server/HiveServer2.java cf3edbf 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
 3978b88 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java
 6af2aa5 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/IMetaStoreClient.java
 09f9bb1 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
 b43334b 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/RawStore.java
 ce7d286 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java
 b223920 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/client/builder/HiveObjectPrivilegeBuilder.java
 d802e1a 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/model/MDBPrivilege.java
 3d8fa21 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/model/MGlobalPrivilege.java
 5b496e0 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/model/MPartitionColumnPrivilege.java
 ab50a92 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/model/MPartitionPrivilege.java
 3193bc1 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/model/MTableColumnPrivilege.java
 ad7322f 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/model/MTablePrivilege.java
 6460400 
  standalone-metastore/src/main/resources/package.jdo 2d2cb19 
  standalone-metastore/src/main/sql/derby/hive-schema-3.0.0.derby.sql 8e09755 
  standalone-metastore/src/main/sql/derby/upgrade-2.3.0-to-3.0.0.derby.sql 
73bef36 
  standalone-metastore/src/main/sql/mssql/hive-schema-3.0.0.mssql.sql 51df92c 
  standalone-metastore/src/main/sql/mssql/upgrade-2.3.0-to-3.0.0.mssql.sql 
a7232dd 
  standalone-metastore/src/main/sql/mysql/hive-schema-3.0.0.mysql.sql 8e55e94 
  standalone-metastore/src/main/sql/mysql/upgrade-2.3.0-to-3.0.0.mysql.sql 
9a48346 
  standalone-metastore/src/main/sql/oracle/hive-schema-3.0.0.oracle.sql 3a12e08 
  standalone-metastore/src/main/sql/oracle/upgrade-2.3.0-to-3.0.0.oracle.sql 
3be7e65 
  standalone-metastore/src/main/sql/postgres/hive-schema-3.0.0.postgres.sql 
0152f48 
  
standalone-metastore/src/main/sql/postgres/upgrade-2.3.0-to-3.0.0.postgres.sql 
fed8a93 
  standalone-metastore/src/main/thrift/hive_metastore.thrift 19d4433 
  
standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/DummyRawStoreControlledCommit.java
 f6899be 
  
standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/DummyRawStoreForJdoConnection.java
 98a85cc 
  
standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClientPreCatalog.java
 7186add 


Diff: https://reviews.apache.org/r/67102/diff/1/


Testing
---


Thanks,

Daniel Dai

Re: Review Request 67102: HIVE-19440: Make StorageBasedAuthorizer work with information schema

[Daniel Dai]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67102/
---

(Updated May 15, 2018, 9:04 p.m.)


Review request for hive.


Repository: hive-git


Description
---

See HIVE-19440


Diffs (updated)
-

  
hcatalog/core/src/main/java/org/apache/hive/hcatalog/storagehandler/DummyHCatAuthProvider.java
 a53028f 
  
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/security/TestHDFSPermissionPolicyProvider.java
 PRE-CREATION 
  
itests/util/src/main/java/org/apache/hadoop/hive/ql/security/DummyHiveMetastoreAuthorizationProvider.java
 31e795c 
  metastore/scripts/upgrade/hive/hive-schema-3.0.0.hive.sql d9606d8 
  ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java a1f549a 
  
ql/src/java/org/apache/hadoop/hive/ql/security/authorization/HDFSPermissionPolicyProvider.java
 PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/security/authorization/HiveAuthorizationProviderBase.java
 8a7c06d 
  
ql/src/java/org/apache/hadoop/hive/ql/security/authorization/HiveMetastoreAuthorizationProvider.java
 0dab334 
  
ql/src/java/org/apache/hadoop/hive/ql/security/authorization/PolicyProviderContainer.java
 PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/security/authorization/PrivilegeSynchonizer.java
 9b2e6cd 
  
ql/src/java/org/apache/hadoop/hive/ql/security/authorization/StorageBasedAuthorizationProvider.java
 b66d188 
  
ql/src/java/org/apache/hadoop/hive/ql/security/authorization/plugin/HiveV1Authorizer.java
 48798d8 
  
ql/src/java/org/apache/hadoop/hive/ql/security/authorization/plugin/sqlstd/SQLAuthorizationUtils.java
 02ed7aa 
  
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFCurrentAuthorizer.java
 PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFRestrictInformationSchema.java
 3eb0914 
  service/src/java/org/apache/hive/service/server/HiveServer2.java 661beb5 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
 92d2e3f 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java
 6af2aa5 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/IMetaStoreClient.java
 09f9bb1 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
 264fdb9 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/RawStore.java
 ce7d286 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java
 b223920 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/client/builder/HiveObjectPrivilegeBuilder.java
 d802e1a 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/model/MDBPrivilege.java
 3d8fa21 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/model/MGlobalPrivilege.java
 5b496e0 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/model/MPartitionColumnPrivilege.java
 ab50a92 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/model/MPartitionPrivilege.java
 3193bc1 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/model/MTableColumnPrivilege.java
 ad7322f 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/model/MTablePrivilege.java
 6460400 
  standalone-metastore/src/main/resources/package.jdo 2d2cb19 
  standalone-metastore/src/main/sql/derby/hive-schema-3.1.0.derby.sql 
PRE-CREATION 
  standalone-metastore/src/main/sql/derby/upgrade-3.0.0-to-3.1.0.derby.sql 
PRE-CREATION 
  standalone-metastore/src/main/sql/mssql/hive-schema-3.1.0.mssql.sql 
PRE-CREATION 
  standalone-metastore/src/main/sql/mssql/upgrade-3.0.0-to-3.1.0.mssql.sql 
PRE-CREATION 
  standalone-metastore/src/main/sql/mysql/hive-schema-3.1.0.mysql.sql 
PRE-CREATION 
  standalone-metastore/src/main/sql/mysql/upgrade-3.0.0-to-3.1.0.mysql.sql 
PRE-CREATION 
  standalone-metastore/src/main/sql/oracle/hive-schema-3.1.0.oracle.sql 
PRE-CREATION 
  standalone-metastore/src/main/sql/oracle/upgrade-3.0.0-to-3.1.0.oracle.sql 
PRE-CREATION 
  standalone-metastore/src/main/sql/postgres/hive-schema-3.1.0.postgres.sql 
PRE-CREATION 
  
standalone-metastore/src/main/sql/postgres/upgrade-3.0.0-to-3.1.0.postgres.sql 
PRE-CREATION 
  standalone-metastore/src/main/thrift/hive_metastore.thrift 19d4433 
  
standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/DummyRawStoreControlledCommit.java
 f6899be 
  
standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/DummyRawStoreForJdoConnection.java
 98a85cc 
  
standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClientPreCatalog.java
 7186add 


Diff: https://reviews.apache.org/r/67102/diff/2/

Changes: https://reviews.apache.org/r/67102/diff/1-2/


Testing
---


Thanks,

Daniel Dai

Re: Question on CachedStore cache update

[Daniel Dai]

Hi, Alan,

If database cache is changed locally, we don’t want to bring remote copy to 
overwrite it as the remote copy doesn’t carry local changes (ideally, we shall 
also apply local changes to the remote copy images we bring in from db, but we 
are not there yet). That’s why we skip the update if there’s local changes, and 
wait for the next iteration to sync with remote. isDatabaseCacheDirty is 
initially set to false unless there’s local update, and will be reset during 
cache swap, thus give a chance for the next iteration to update the cache if 
there’s no local changes.

Thanks,
Daniel

On 2/6/18, 11:57 AM, "Alan Gates"  wrote:

I’m confused by the following code in the CachedStore.  This in in the
CacheUpdateMasterWork thread, in the updateDatabases method (which is
called by update()):

*// Skip background updates if we detect change*

*if *(*isDatabaseCacheDirty*.compareAndSet(*true*, *false*)) {

  *LOG*.debug(*"Skipping database cache update; the database list we have
is dirty."*);

  *return*;

}

Why are we not updating the cache if we’ve dirtied it?  Also, AFAICT no one
ever sets isDatabaseCacheDirty to false, meaning once one database is
created the cache will never be updated.  Am I missing something?

Alan.

Re: Review Request 65634: HIVE-18264: CachedStore: Store cached partitions/col stats within the table cache

[Daniel Dai]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65634/#review197794
---




standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/cache/CacheUtils.java
Lines 65 (patched)
<https://reviews.apache.org/r/65634/#comment278072>

key = String.join(delimit, partVals);



standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java
Line 311 (original), 215 (patched)
<https://reviews.apache.org/r/65634/#comment278114>

This is not introduced in this patch, but getting columns for table and 
apply to partition will not work for schema revolution. We shall get columns 
for every individual partition.



standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java
Line 800 (original), 632 (patched)
<https://reviews.apache.org/r/65634/#comment278175>

I don't remember but why this is get() not getUnsafe()? It sounds the same 
as getAllTables etc. Also apply to getDatabases, alterDatabase, dropDatabase, 
getDatabase and createDatabase


- Daniel Dai


On Feb. 13, 2018, 12:08 p.m., Vaibhav Gumashta wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65634/
> ---
> 
> (Updated Feb. 13, 2018, 12:08 p.m.)
> 
> 
> Review request for hive, Daniel Dai and Thejas Nair.
> 
> 
> Bugs: HIVE-18264
> https://issues.apache.org/jira/browse/HIVE-18264
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> https://issues.apache.org/jira/browse/HIVE-18264
> 
> 
> Diffs
> -
> 
>   
> itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/listener/DummyRawStoreFailEvent.java
>  78b26374f2 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
>  d58ed677f3 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/RawStore.java
>  e4e7d4239d 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/cache/CacheUtils.java
>  f0f650ddcf 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java
>  80aa3bcdb4 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/cache/SharedCache.java
>  32ea17495f 
>   
> standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/DummyRawStoreControlledCommit.java
>  9100c73beb 
>   
> standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/DummyRawStoreForJdoConnection.java
>  86e72d8d76 
>   
> standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/cache/TestCachedStore.java
>  bd61df654a 
> 
> 
> Diff: https://reviews.apache.org/r/65634/diff/1/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Vaibhav Gumashta
> 
>

Re: Review Request 65634: HIVE-18264: CachedStore: Store cached partitions/col stats within the table cache

[Daniel Dai]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65634/#review198206
---


Ship it!




Ship It!

- Daniel Dai


On Feb. 23, 2018, 8:14 p.m., Vaibhav Gumashta wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65634/
> ---
> 
> (Updated Feb. 23, 2018, 8:14 p.m.)
> 
> 
> Review request for hive, Daniel Dai and Thejas Nair.
> 
> 
> Bugs: HIVE-18264
> https://issues.apache.org/jira/browse/HIVE-18264
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> https://issues.apache.org/jira/browse/HIVE-18264
> 
> 
> Diffs
> -
> 
>   
> itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/listener/DummyRawStoreFailEvent.java
>  a3725c5395 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
>  7b44df4128 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/RawStore.java
>  f500d63725 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/cache/CacheUtils.java
>  f0f650ddcf 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java
>  0d132f2074 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/cache/SharedCache.java
>  32ea17495f 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/utils/MetaStoreUtils.java
>  50f873a013 
>   
> standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/DummyRawStoreControlledCommit.java
>  75ea8c4a77 
>   
> standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/DummyRawStoreForJdoConnection.java
>  207d842f94 
>   
> standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/cache/TestCachedStore.java
>  ab6feb6f0b 
> 
> 
> Diff: https://reviews.apache.org/r/65634/diff/2/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Vaibhav Gumashta
> 
>

Re: Review Request 68590: HIVE-20493: Unit test getGroupNames for SessionStateUserAuthenticator

[Daniel Dai]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68590/#review208201
---


Ship it!




- Daniel Dai


On Aug. 31, 2018, 8:24 p.m., Karthik Manamcheri wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/68590/
> ---
> 
> (Updated Aug. 31, 2018, 8:24 p.m.)
> 
> 
> Review request for hive, Alexander Kolbasov and Daniel Dai.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-20493: Unit test getGroupNames for SessionStateUserAuthenticator
> 
> 
> Diffs
> -
> 
>   
> ql/src/test/org/apache/hadoop/hive/ql/parse/authorization/TestSessionUserName.java
>  4b645f390f 
> 
> 
> Diff: https://reviews.apache.org/r/68590/diff/1/
> 
> 
> Testing
> ---
> 
> Ran 'mvn -Dtest=TestSessionUserName test' successfully.
> 
> 
> Thanks,
> 
> Karthik Manamcheri
> 
>

Re: Review Request 68590: HIVE-20493: Unit test getGroupNames for SessionStateUserAuthenticator

[Daniel Dai]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68590/#review208200
---


Ship it!




Ship It!

- Daniel Dai


On Aug. 31, 2018, 8:24 p.m., Karthik Manamcheri wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/68590/
> ---
> 
> (Updated Aug. 31, 2018, 8:24 p.m.)
> 
> 
> Review request for hive, Alexander Kolbasov and Daniel Dai.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-20493: Unit test getGroupNames for SessionStateUserAuthenticator
> 
> 
> Diffs
> -
> 
>   
> ql/src/test/org/apache/hadoop/hive/ql/parse/authorization/TestSessionUserName.java
>  4b645f390f 
> 
> 
> Diff: https://reviews.apache.org/r/68590/diff/1/
> 
> 
> Testing
> ---
> 
> Ran 'mvn -Dtest=TestSessionUserName test' successfully.
> 
> 
> Thanks,
> 
> Karthik Manamcheri
> 
>

Review Request 68630: HIVE-20420: Provide a fallback authorizer when no other authorizer is in use

[Daniel Dai]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68630/
---

Review request for hive.


Repository: hive-git


Description
---

See HIVE-20420


Diffs
-

  ql/pom.xml a55cbe3 
  
ql/src/java/org/apache/hadoop/hive/ql/security/authorization/plugin/SettableConfigUpdater.java
 12be41c 
  
ql/src/java/org/apache/hadoop/hive/ql/security/authorization/plugin/fallback/FallbackHiveAuthorizer.java
 PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/security/authorization/plugin/fallback/FallbackHiveAuthorizerFactory.java
 PRE-CREATION 
  ql/src/test/queries/clientnegative/fallbackauth_addjar.q PRE-CREATION 
  ql/src/test/queries/clientnegative/fallbackauth_compile.q PRE-CREATION 
  ql/src/test/queries/clientnegative/fallbackauth_create_func1.q PRE-CREATION 
  ql/src/test/queries/clientnegative/fallbackauth_create_func2.q PRE-CREATION 
  ql/src/test/queries/clientnegative/fallbackauth_dfs.q PRE-CREATION 
  ql/src/test/queries/clientnegative/fallbackauth_disallow_transform.q 
PRE-CREATION 
  ql/src/test/queries/clientnegative/fallbackauth_load.q PRE-CREATION 
  ql/src/test/queries/clientnegative/fallbackauth_set_invalidconf.q 
PRE-CREATION 
  ql/src/test/results/clientnegative/fallbackauth_addjar.q.out PRE-CREATION 
  ql/src/test/results/clientnegative/fallbackauth_compile.q.out PRE-CREATION 
  ql/src/test/results/clientnegative/fallbackauth_create_func1.q.out 
PRE-CREATION 
  ql/src/test/results/clientnegative/fallbackauth_create_func2.q.out 
PRE-CREATION 
  ql/src/test/results/clientnegative/fallbackauth_dfs.q.out PRE-CREATION 
  ql/src/test/results/clientnegative/fallbackauth_disallow_transform.q.out 
PRE-CREATION 
  ql/src/test/results/clientnegative/fallbackauth_load.q.out PRE-CREATION 
  ql/src/test/results/clientnegative/fallbackauth_set_invalidconf.q.out 
PRE-CREATION 


Diff: https://reviews.apache.org/r/68630/diff/1/


Testing
---


Thanks,

Daniel Dai

Re: Review Request 68630: HIVE-20420: Provide a fallback authorizer when no other authorizer is in use

[Daniel Dai]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68630/
---

(Updated Sept. 11, 2018, 11:19 p.m.)


Review request for hive.


Repository: hive-git


Description
---

See HIVE-20420


Diffs (updated)
-

  ql/pom.xml a55cbe3 
  
ql/src/java/org/apache/hadoop/hive/ql/security/authorization/plugin/SettableConfigUpdater.java
 12be41c 
  
ql/src/java/org/apache/hadoop/hive/ql/security/authorization/plugin/fallback/FallbackHiveAuthorizer.java
 PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/security/authorization/plugin/fallback/FallbackHiveAuthorizerFactory.java
 PRE-CREATION 
  ql/src/test/queries/clientnegative/fallbackauth_addjar.q PRE-CREATION 
  ql/src/test/queries/clientnegative/fallbackauth_compile.q PRE-CREATION 
  ql/src/test/queries/clientnegative/fallbackauth_create_func1.q PRE-CREATION 
  ql/src/test/queries/clientnegative/fallbackauth_create_func2.q PRE-CREATION 
  ql/src/test/queries/clientnegative/fallbackauth_dfs.q PRE-CREATION 
  ql/src/test/queries/clientnegative/fallbackauth_disallow_transform.q 
PRE-CREATION 
  ql/src/test/queries/clientnegative/fallbackauth_load.q PRE-CREATION 
  ql/src/test/queries/clientnegative/fallbackauth_set_invalidconf.q 
PRE-CREATION 
  ql/src/test/results/clientnegative/fallbackauth_addjar.q.out PRE-CREATION 
  ql/src/test/results/clientnegative/fallbackauth_compile.q.out PRE-CREATION 
  ql/src/test/results/clientnegative/fallbackauth_create_func1.q.out 
PRE-CREATION 
  ql/src/test/results/clientnegative/fallbackauth_create_func2.q.out 
PRE-CREATION 
  ql/src/test/results/clientnegative/fallbackauth_dfs.q.out PRE-CREATION 
  ql/src/test/results/clientnegative/fallbackauth_disallow_transform.q.out 
PRE-CREATION 
  ql/src/test/results/clientnegative/fallbackauth_load.q.out PRE-CREATION 
  ql/src/test/results/clientnegative/fallbackauth_set_invalidconf.q.out 
PRE-CREATION 


Diff: https://reviews.apache.org/r/68630/diff/2/

Changes: https://reviews.apache.org/r/68630/diff/1-2/


Testing
---


Thanks,

Daniel Dai

[SECURITY] CVE-2018-1284: Hive UDF series UDFXPathXXXX allow users to pass carefully crafted XML to access arbitrary files

[Daniel Dai]

CVE-2018-1284: Hive UDF series UDFXPath allow users to pass
carefully crafted XML to access arbitrary files

Severity: Important

Vendor: The Apache Software Foundation

Versions Affected: This vulnerability affects all versions from 0.6.0

Description: Malicious user might use any xpath UDFs
(xpath/xpath_string/xpath_boolean/xpath_number/xpath_double/xpath_float/xpath_long/xpath_int/xpath_short)
to expose the content of a file on the machine running HiveServer2
owned by HiveServer2 user (usually hive) if
hive.server2.enable.doAs=false.

Mitigation: Users who use xpath UDFs in HiveServer2 and
hive.server2.enable.doAs=false are recommended to upgrade to 2.3.3, or
update UDFXPathUtil.java to the head of branch-2.3 and rebuild
hive-exec.jar: 
https://git1-us-west.apache.org/repos/asf?p=hive.git;a=blob;f=ql/src/java/org/apache/hadoop/hive/ql/udf/xml/UDFXPathUtil.java;hb=refs/heads/branch-2.3.
If these functions are not being used at present, you can also
disable its use by adding them to the value of the config
hive.server2.builtin.udf.blacklist.

[SECURITY] CVE-2018-1282 JDBC driver is susceptible to SQL injection attack if the input parameters are not properly cleaned

[Daniel Dai]

CVE-2018-1282: JDBC driver is susceptible to SQL injection attack if
the input parameters are not properly cleaned

Severity: Important

Vendor: The Apache Software Foundation

Versions Affected: This vulnerability affects all versions of Hive
JDBC driver from 0.7.1

Description: This vulnerability in Hive allows carefully crafted arguments to be
used to bypass the argument escaping/cleanup that JDBC driver does in
PreparedStatement implementation.

Mitigation: It is recommended to upgrade prior version of Hive JDBC
driver to 2.3.3.
Note Hive JDBC driver is not backward compatible with HiveServer2,
which means newer version of Hive JDBC driver may not talk to older version
of HiveServer2. In particular, Hive JDBC driver 2.3.3 won't talk
to HiveServer2 2.1.1 or prior. If user is using Hive code 2.1.1 or below
they might need to upgrade all the Hive instances to 2.3.3.


Alternative to the upgrade, is to take the follow two actions in your
Hive JDBC client code/application when dealing with user provided
input in PreparedStatement:
1. Avoid passing user input PreparedStatement.setBinaryStream
2. Sanitize the user input for PreparedStatement.setString, by
replacing all occurrences of \' to '

Credit: This issue was discovered by Bear Giles of SnapLogic

[SECURITY] CVE-2018-1315 'COPY FROM FTP' statement in HPL/SQL can write to arbitrary location if the FTP server is compromised

[Daniel Dai]

CVE-2018-1315: 'COPY FROM FTP' statement in HPL/SQL can write to
arbitrary location if the FTP server is compromised

Severity: Moderate

Vendor: The Apache Software Foundation

Versions Affected: Hive 2.1.0 to 2.3.2

Description: When 'COPY FROM FTP' statement is run using HPL/SQL extension to
Hive, a compromised/malicious FTP server can cause the file to be
written to an arbitrary location on the cluster where the command is
run from. This is because FTP client code in HPL/SQL does not verify
the destination
location of the downloaded file. This does not affect hive
cli user and hiveserver2 user as hplsql is a separate command line
script and needs to be invoked differently.

Mitigation: User who use HPL/SQL with Hive 2.1.0 through 2.3.2 should upgrade to
2.3.3 which removes support for "COPY FROM FTP". Alternatively, the
usage of HPL/SQL can be disabled through
other means.

Credit: This issue was discovered by Danny Grander of Snyk

Review Request 66571: HIVE-19161: Add authorizations to information schema

[Daniel Dai]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66571/
---

Review request for hive.


Repository: hive-git


Description
---

See HIVE-19161


Diffs
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 0627c35 
  
itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/listener/DummyRawStoreFailEvent.java
 801de7a 
  itests/hive-unit/pom.xml f473d25 
  
itests/hive-unit/src/test/java/org/apache/hive/service/server/TestInformationSchemaWithPrivilege.java
 PRE-CREATION 
  metastore/scripts/upgrade/hive/hive-schema-3.0.0.hive.sql 339 
  ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java d59bf1f 
  
ql/src/java/org/apache/hadoop/hive/ql/security/authorization/PrivilegeSynchonizer.java
 PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/security/authorization/plugin/HiveResourceACLs.java
 53e221f 
  ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java 6003ced 
  
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFCurrentGroups.java 
PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFRestrictInformationSchema.java
 PRE-CREATION 
  service/src/java/org/apache/hive/service/server/HiveServer2.java 6308c5c 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
 450da4f 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java
 ebbf465 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/IMetaStoreClient.java
 b2c40c2 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
 2056930 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/RawStore.java
 f6c46ee 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java
 c47856d 
  standalone-metastore/src/main/thrift/hive_metastore.thrift 7450439 
  
standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/DummyRawStoreControlledCommit.java
 304f567 
  
standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/DummyRawStoreForJdoConnection.java
 85c6727 
  
standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClientPreCatalog.java
 7d37262 


Diff: https://reviews.apache.org/r/66571/diff/1/


Testing
---


Thanks,

Daniel Dai

Re: Review Request 66571: HIVE-19161: Add authorizations to information schema

[Daniel Dai]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66571/
---

(Updated April 14, 2018, 7:06 a.m.)


Review request for hive.


Repository: hive-git


Description
---

See HIVE-19161


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java e533ee6 
  
itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/listener/DummyRawStoreFailEvent.java
 801de7a 
  itests/hive-unit/pom.xml 05c362e 
  
itests/hive-unit/src/test/java/org/apache/hive/service/server/TestInformationSchemaWithPrivilege.java
 PRE-CREATION 
  
jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/dao/DatabaseAccessorFactory.java
 7dc690f 
  
jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/dao/GenericJdbcDatabaseAccessor.java
 178c97d 
  
jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/dao/JdbcRecordIterator.java
 638e2b0 
  
jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/dao/MsSqlDatabaseAccessor.java
 PRE-CREATION 
  
jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/dao/OracleDatabaseAccessor.java
 PRE-CREATION 
  
jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/dao/PostgresDatabaseAccessor.java
 PRE-CREATION 
  metastore/scripts/upgrade/hive/hive-schema-3.0.0.hive.sql 339 
  ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java d59bf1f 
  
ql/src/java/org/apache/hadoop/hive/ql/security/authorization/HiveAuthorizationProvider.java
 60d9dc1 
  
ql/src/java/org/apache/hadoop/hive/ql/security/authorization/PrivilegeSynchonizer.java
 PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/security/authorization/plugin/HiveResourceACLsImpl.java
 PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java 6003ced 
  
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFCurrentGroups.java 
PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFRestrictInformationSchema.java
 PRE-CREATION 
  service/src/java/org/apache/hive/service/server/HiveServer2.java 47f84b5 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
 565549a 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java
 9a43b2c 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/IMetaStoreClient.java
 98674cf 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
 c5da7b5 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/RawStore.java
 f6c46ee 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java
 1ce86bb 
  standalone-metastore/src/main/thrift/hive_metastore.thrift 612afe1 
  
standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/DummyRawStoreControlledCommit.java
 304f567 
  
standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/DummyRawStoreForJdoConnection.java
 85c6727 
  
standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClientPreCatalog.java
 ecddc7b 


Diff: https://reviews.apache.org/r/66571/diff/5/

Changes: https://reviews.apache.org/r/66571/diff/4-5/


Testing
---


Thanks,

Daniel Dai

Re: Review Request 66571: HIVE-19161: Add authorizations to information schema

[Daniel Dai]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66571/
---

(Updated April 13, 2018, 8:49 p.m.)


Review request for hive.


Repository: hive-git


Description
---

See HIVE-19161


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 0627c35 
  
itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/listener/DummyRawStoreFailEvent.java
 801de7a 
  itests/hive-unit/pom.xml f473d25 
  
itests/hive-unit/src/test/java/org/apache/hive/service/server/TestInformationSchemaWithPrivilege.java
 PRE-CREATION 
  
jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/dao/DatabaseAccessorFactory.java
 7dc690f 
  
jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/dao/GenericJdbcDatabaseAccessor.java
 178c97d 
  
jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/dao/JdbcRecordIterator.java
 638e2b0 
  
jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/dao/MsSqlDatabaseAccessor.java
 PRE-CREATION 
  
jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/dao/OracleDatabaseAccessor.java
 PRE-CREATION 
  
jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/dao/PostgresDatabaseAccessor.java
 PRE-CREATION 
  metastore/scripts/upgrade/hive/hive-schema-3.0.0.hive.sql 339 
  ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java d59bf1f 
  
ql/src/java/org/apache/hadoop/hive/ql/security/authorization/HiveAuthorizationProvider.java
 60d9dc1 
  
ql/src/java/org/apache/hadoop/hive/ql/security/authorization/PrivilegeSynchonizer.java
 PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/security/authorization/plugin/HiveResourceACLsImpl.java
 PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java 6003ced 
  
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFCurrentGroups.java 
PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFRestrictInformationSchema.java
 PRE-CREATION 
  service/src/java/org/apache/hive/service/server/HiveServer2.java 6308c5c 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
 450da4f 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java
 ebbf465 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/IMetaStoreClient.java
 b2c40c2 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
 2056930 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/RawStore.java
 f6c46ee 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java
 c47856d 
  standalone-metastore/src/main/thrift/hive_metastore.thrift 7450439 
  
standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/DummyRawStoreControlledCommit.java
 304f567 
  
standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/DummyRawStoreForJdoConnection.java
 85c6727 
  
standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClientPreCatalog.java
 7d37262 


Diff: https://reviews.apache.org/r/66571/diff/3/

Changes: https://reviews.apache.org/r/66571/diff/2-3/


Testing
---


Thanks,

Daniel Dai

Re: Review Request 66571: HIVE-19161: Add authorizations to information schema

[Daniel Dai]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66571/
---

(Updated April 13, 2018, 8:48 p.m.)


Review request for hive.


Repository: hive-git


Description
---

See HIVE-19161


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 0627c35 
  
itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/listener/DummyRawStoreFailEvent.java
 801de7a 
  itests/hive-unit/pom.xml f473d25 
  
itests/hive-unit/src/test/java/org/apache/hive/service/server/TestInformationSchemaWithPrivilege.java
 PRE-CREATION 
  
jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/dao/DatabaseAccessorFactory.java
 7dc690f 
  
jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/dao/GenericJdbcDatabaseAccessor.java
 178c97d 
  
jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/dao/JdbcRecordIterator.java
 638e2b0 
  
jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/dao/MsSqlDatabaseAccessor.java
 PRE-CREATION 
  
jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/dao/OracleDatabaseAccessor.java
 PRE-CREATION 
  
jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/dao/PostgresDatabaseAccessor.java
 PRE-CREATION 
  metastore/scripts/upgrade/hive/hive-schema-3.0.0.hive.sql 339 
  ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java d59bf1f 
  
ql/src/java/org/apache/hadoop/hive/ql/security/authorization/HiveAuthorizationProvider.java
 60d9dc1 
  
ql/src/java/org/apache/hadoop/hive/ql/security/authorization/PrivilegeSynchonizer.java
 PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/security/authorization/plugin/HiveResourceACLsImpl.java
 PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java 6003ced 
  
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFCurrentGroups.java 
PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFRestrictInformationSchema.java
 PRE-CREATION 
  service/src/java/org/apache/hive/service/server/HiveServer2.java 6308c5c 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
 450da4f 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java
 ebbf465 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/IMetaStoreClient.java
 b2c40c2 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
 2056930 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/RawStore.java
 f6c46ee 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java
 c47856d 
  standalone-metastore/src/main/thrift/hive_metastore.thrift 7450439 
  
standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/DummyRawStoreControlledCommit.java
 304f567 
  
standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/DummyRawStoreForJdoConnection.java
 85c6727 
  
standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClientPreCatalog.java
 7d37262 


Diff: https://reviews.apache.org/r/66571/diff/2/

Changes: https://reviews.apache.org/r/66571/diff/1-2/


Testing
---


Thanks,

Daniel Dai

Re: Review Request 66571: HIVE-19161: Add authorizations to information schema

[Daniel Dai]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66571/
---

(Updated April 13, 2018, 10:30 p.m.)


Review request for hive.


Repository: hive-git


Description
---

See HIVE-19161


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 0627c35 
  
itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/listener/DummyRawStoreFailEvent.java
 801de7a 
  itests/hive-unit/pom.xml f473d25 
  
itests/hive-unit/src/test/java/org/apache/hive/service/server/TestInformationSchemaWithPrivilege.java
 PRE-CREATION 
  
jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/dao/DatabaseAccessorFactory.java
 7dc690f 
  
jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/dao/GenericJdbcDatabaseAccessor.java
 178c97d 
  
jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/dao/JdbcRecordIterator.java
 638e2b0 
  
jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/dao/MsSqlDatabaseAccessor.java
 PRE-CREATION 
  
jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/dao/OracleDatabaseAccessor.java
 PRE-CREATION 
  
jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/dao/PostgresDatabaseAccessor.java
 PRE-CREATION 
  metastore/scripts/upgrade/hive/hive-schema-3.0.0.hive.sql 339 
  ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java d59bf1f 
  
ql/src/java/org/apache/hadoop/hive/ql/security/authorization/HiveAuthorizationProvider.java
 60d9dc1 
  
ql/src/java/org/apache/hadoop/hive/ql/security/authorization/PrivilegeSynchonizer.java
 PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/security/authorization/plugin/HiveResourceACLsImpl.java
 PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java 6003ced 
  
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFCurrentGroups.java 
PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFRestrictInformationSchema.java
 PRE-CREATION 
  service/src/java/org/apache/hive/service/server/HiveServer2.java 6308c5c 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
 450da4f 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java
 ebbf465 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/IMetaStoreClient.java
 b2c40c2 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
 2056930 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/RawStore.java
 f6c46ee 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java
 c47856d 
  standalone-metastore/src/main/thrift/hive_metastore.thrift 7450439 
  
standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/DummyRawStoreControlledCommit.java
 304f567 
  
standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/DummyRawStoreForJdoConnection.java
 85c6727 
  
standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClientPreCatalog.java
 7d37262 


Diff: https://reviews.apache.org/r/66571/diff/4/

Changes: https://reviews.apache.org/r/66571/diff/3-4/


Testing
---


Thanks,

Daniel Dai

[ANNOUNCE] Apache Hive 2.3.3 Released

[Daniel Dai]

The Apache Hive team is proud to announce the release of Apache Hive
version 2.3.3.

The Apache Hive (TM) data warehouse software facilitates querying and
managing large datasets residing in distributed storage. Built on top
of Apache Hadoop (TM), it provides, among others:

* Tools to enable easy data extract/transform/load (ETL)

* A mechanism to impose structure on a variety of data formats

* Access to files stored either directly in Apache HDFS (TM) or in other
  data storage systems such as Apache HBase (TM)

* Query execution via Apache Hadoop MapReduce, Apache Tez and Apache Spark
frameworks.

For Hive release details and downloads, please visit:
https://hive.apache.org/downloads.html

Hive 2.3.3 Release Notes are available here:
https://issues.apache.org/jira/secure/ReleaseNote.jspa?version=12342162=Text=12310843

We would like to thank the many contributors who made this release
possible.

Regards,

The Apache Hive Team

Re: Review Request 66503: HIVE-19126: CachedStore: Use memory estimation to limit cache size during prewarm

[Daniel Dai]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66503/#review201706
---




standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/cache/SharedCache.java
Lines 52 (patched)
<https://reviews.apache.org/r/66503/#comment283258>

standalone-metastore shall not depend on ql.


- Daniel Dai


On April 16, 2018, 9:42 p.m., Vaibhav Gumashta wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66503/
> ---
> 
> (Updated April 16, 2018, 9:42 p.m.)
> 
> 
> Review request for hive and Thejas Nair.
> 
> 
> Bugs: HIVE-19126
> https://issues.apache.org/jira/browse/HIVE-19126
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> https://issues.apache.org/jira/browse/HIVE-19126
> 
> 
> Diffs
> -
> 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/IncrementalObjectSizeEstimator.java
>  6f4ec6f1ea 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/io/metadata/OrcFileEstimateErrors.java
>  2f7fa24558 
>   
> llap-server/src/test/org/apache/hadoop/hive/llap/cache/TestIncrementalObjectSizeEstimator.java
>  0bbaf7e459 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java
>  1ce86bbdba 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/cache/SharedCache.java
>  89b400697b 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java
>  f007261daf 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/conf/SizeValidator.java
>  PRE-CREATION 
>   
> standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/cache/TestCachedStore.java
>  d451f966b0 
> 
> 
> Diff: https://reviews.apache.org/r/66503/diff/6/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Vaibhav Gumashta
> 
>

[VOTE] Apache Hive 2.3.3 Release Candidate 0

[Daniel Dai]

Apache Hive 2.3.3 Release Candidate 0 is available here:

http://people.apache.org/~daijy/apache-hive-2.3.3-rc-0/

Maven artifacts are available here:

https://repository.apache.org/content/repositories/orgapachehive-1084/

Source tag for RCN is at:
https://github.com/apache/hive/tree/release-2.3.3-rc0

Voting will conclude in 72 hours.

Hive PMC Members: Please test and vote.

Thanks.

[RESULT][VOTE] Apache Hive 2.3.3 Release Candidate 0

[Daniel Dai]

With three +1’s (Alan, Thejas, Daniel) and no -1’s the vote passes.

I will wrap up the release and send announcement once done.

Re: Review Request 65634: HIVE-18264: CachedStore: Store cached partitions/col stats within the table cache

[Daniel Dai]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65634/#review199444
---


Ship it!




+1, let's get this in. We can open new tickets for any remaining review 
Alexander has.

- Daniel Dai


On March 9, 2018, 9 p.m., Vaibhav Gumashta wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65634/
> ---
> 
> (Updated March 9, 2018, 9 p.m.)
> 
> 
> Review request for hive, Alexander Kolbasov, Daniel Dai, and Thejas Nair.
> 
> 
> Bugs: HIVE-18264
> https://issues.apache.org/jira/browse/HIVE-18264
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> https://issues.apache.org/jira/browse/HIVE-18264
> 
> 
> Diffs
> -
> 
>   
> itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/listener/DummyRawStoreFailEvent.java
>  a3725c5395 
>   service/src/java/org/apache/hive/service/server/HiveServer2.java 86c9c2b33c 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
>  ac71d0882f 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
>  7b44df4128 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/RawStore.java
>  f500d63725 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/cache/CacheUtils.java
>  f0f650ddcf 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java
>  0d132f2074 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/cache/SharedCache.java
>  32ea17495f 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/utils/MetaStoreUtils.java
>  50f873a013 
>   
> standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/DummyRawStoreControlledCommit.java
>  75ea8c4a77 
>   
> standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/DummyRawStoreForJdoConnection.java
>  207d842f94 
>   
> standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/cache/TestCachedStore.java
>  ab6feb6f0b 
>   standalone-metastore/src/test/resources/log4j2.properties 365687e1c9 
> 
> 
> Diff: https://reviews.apache.org/r/65634/diff/5/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Vaibhav Gumashta
> 
>

Re: Review Request 65634: HIVE-18264: CachedStore: Store cached partitions/col stats within the table cache

[Daniel Dai]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65634/#review198471
---


Ship it!




Ship It!

- Daniel Dai


On March 1, 2018, 11:09 a.m., Vaibhav Gumashta wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65634/
> ---
> 
> (Updated March 1, 2018, 11:09 a.m.)
> 
> 
> Review request for hive, Daniel Dai and Thejas Nair.
> 
> 
> Bugs: HIVE-18264
> https://issues.apache.org/jira/browse/HIVE-18264
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> https://issues.apache.org/jira/browse/HIVE-18264
> 
> 
> Diffs
> -
> 
>   
> itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/listener/DummyRawStoreFailEvent.java
>  a3725c5395 
>   service/src/java/org/apache/hive/service/server/HiveServer2.java 86c9c2b33c 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
>  ac71d0882f 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
>  7b44df4128 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/RawStore.java
>  f500d63725 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/cache/CacheUtils.java
>  f0f650ddcf 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java
>  0d132f2074 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/cache/SharedCache.java
>  32ea17495f 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/utils/MetaStoreUtils.java
>  50f873a013 
>   
> standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/DummyRawStoreControlledCommit.java
>  75ea8c4a77 
>   
> standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/DummyRawStoreForJdoConnection.java
>  207d842f94 
>   
> standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/cache/TestCachedStore.java
>  ab6feb6f0b 
>   standalone-metastore/src/test/resources/log4j2.properties 365687e1c9 
> 
> 
> Diff: https://reviews.apache.org/r/65634/diff/4/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Vaibhav Gumashta
> 
>

Re: Review Request 65634: HIVE-18264: CachedStore: Store cached partitions/col stats within the table cache

[Daniel Dai]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65634/#review198378
---


Ship it!




Ship It!

- Daniel Dai


On Feb. 26, 2018, 9:47 p.m., Vaibhav Gumashta wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65634/
> ---
> 
> (Updated Feb. 26, 2018, 9:47 p.m.)
> 
> 
> Review request for hive, Daniel Dai and Thejas Nair.
> 
> 
> Bugs: HIVE-18264
> https://issues.apache.org/jira/browse/HIVE-18264
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> https://issues.apache.org/jira/browse/HIVE-18264
> 
> 
> Diffs
> -
> 
>   
> itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/listener/DummyRawStoreFailEvent.java
>  a3725c5395 
>   service/src/java/org/apache/hive/service/server/HiveServer2.java 6c1a0b98cc 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
>  c6e34a8a22 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
>  7b44df4128 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/RawStore.java
>  f500d63725 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/cache/CacheUtils.java
>  f0f650ddcf 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java
>  0d132f2074 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/cache/SharedCache.java
>  32ea17495f 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/utils/MetaStoreUtils.java
>  50f873a013 
>   
> standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/DummyRawStoreControlledCommit.java
>  75ea8c4a77 
>   
> standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/DummyRawStoreForJdoConnection.java
>  207d842f94 
>   
> standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/cache/TestCachedStore.java
>  ab6feb6f0b 
>   standalone-metastore/src/test/resources/log4j2.properties 365687e1c9 
> 
> 
> Diff: https://reviews.apache.org/r/65634/diff/3/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Vaibhav Gumashta
> 
>

Re: Review Request 65634: HIVE-18264: CachedStore: Store cached partitions/col stats within the table cache

[Daniel Dai]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65634/#review198377
---


Fix it, then Ship it!




Ship It!


standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java
Line 583 (original), 568 (patched)
<https://reviews.apache.org/r/65634/#comment278513>

Blindly update cache may overwrite newer content. But will leave this to 
notification based update.


- Daniel Dai


On Feb. 26, 2018, 9:47 p.m., Vaibhav Gumashta wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65634/
> ---
> 
> (Updated Feb. 26, 2018, 9:47 p.m.)
> 
> 
> Review request for hive, Daniel Dai and Thejas Nair.
> 
> 
> Bugs: HIVE-18264
> https://issues.apache.org/jira/browse/HIVE-18264
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> https://issues.apache.org/jira/browse/HIVE-18264
> 
> 
> Diffs
> -
> 
>   
> itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/listener/DummyRawStoreFailEvent.java
>  a3725c5395 
>   service/src/java/org/apache/hive/service/server/HiveServer2.java 6c1a0b98cc 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
>  c6e34a8a22 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
>  7b44df4128 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/RawStore.java
>  f500d63725 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/cache/CacheUtils.java
>  f0f650ddcf 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java
>  0d132f2074 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/cache/SharedCache.java
>  32ea17495f 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/utils/MetaStoreUtils.java
>  50f873a013 
>   
> standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/DummyRawStoreControlledCommit.java
>  75ea8c4a77 
>   
> standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/DummyRawStoreForJdoConnection.java
>  207d842f94 
>   
> standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/cache/TestCachedStore.java
>  ab6feb6f0b 
>   standalone-metastore/src/test/resources/log4j2.properties 365687e1c9 
> 
> 
> Diff: https://reviews.apache.org/r/65634/diff/3/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Vaibhav Gumashta
> 
>

Review Request 69050: HIVE-20720: Add partition column option to JDBC handler

[Daniel Dai]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69050/
---

Review request for hive.


Repository: hive-git


Description
---

See HIVE-20720.


Diffs
-

  common/src/java/org/apache/hadoop/hive/conf/Constants.java 1190679 
  itests/src/test/resources/testconfiguration.properties 9a87464 
  jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/JdbcInputFormat.java 
74999db 
  jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/JdbcInputSplit.java 
3a6ada8 
  jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/JdbcRecordReader.java 
1da6213 
  jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/JdbcSerDe.java 
5947628 
  
jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/conf/JdbcStorageConfigManager.java
 18e2397 
  
jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/dao/DatabaseAccessor.java
 fdaa794 
  
jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/dao/GenericJdbcDatabaseAccessor.java
 abdc5f0 
  
jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/dao/JdbcRecordIterator.java
 a95aca2 
  
jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/dao/JethroDatabaseAccessor.java
 db0454e 
  
jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/dao/MySqlDatabaseAccessor.java
 86fde7c 
  
jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/spitter/DateIntervalSplitter.java
 PRE-CREATION 
  
jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/spitter/DecimalIntervalSplitter.java
 PRE-CREATION 
  
jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/spitter/DoubleIntervalSplitter.java
 PRE-CREATION 
  
jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/spitter/IntervalSplitter.java
 PRE-CREATION 
  
jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/spitter/IntervalSplitterFactory.java
 PRE-CREATION 
  
jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/spitter/LongIntervalSpitter.java
 PRE-CREATION 
  
jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/spitter/TimestampIntervalSplitter.java
 PRE-CREATION 
  
jdbc-handler/src/test/java/org/apache/hive/storage/jdbc/TestJdbcInputFormat.java
 b146633 
  
jdbc-handler/src/test/java/org/apache/hive/storage/jdbc/dao/TestGenericJdbcDatabaseAccessor.java
 34f061e 
  ql/src/test/queries/clientpositive/external_jdbc_table_partition.q 
PRE-CREATION 
  ql/src/test/queries/clientpositive/external_jdbc_table_typeconversion.q 
PRE-CREATION 
  ql/src/test/results/clientpositive/llap/external_jdbc_table_partition.q.out 
PRE-CREATION 
  
ql/src/test/results/clientpositive/llap/external_jdbc_table_typeconversion.q.out
 PRE-CREATION 


Diff: https://reviews.apache.org/r/69050/diff/1/


Testing
---


Thanks,

Daniel Dai

Re: Review Request 69050: HIVE-20720: Add partition column option to JDBC handler

[Daniel Dai]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69050/
---

(Updated Oct. 16, 2018, 5:08 p.m.)


Review request for hive.


Repository: hive-git


Description
---

See HIVE-20720.


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/Constants.java 1190679 
  itests/src/test/resources/testconfiguration.properties 9a87464 
  jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/JdbcInputFormat.java 
74999db 
  jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/JdbcInputSplit.java 
3a6ada8 
  jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/JdbcRecordReader.java 
1da6213 
  jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/JdbcSerDe.java 
5947628 
  
jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/conf/JdbcStorageConfigManager.java
 18e2397 
  
jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/dao/DatabaseAccessor.java
 fdaa794 
  
jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/dao/GenericJdbcDatabaseAccessor.java
 abdc5f0 
  
jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/dao/JdbcRecordIterator.java
 a95aca2 
  
jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/dao/JethroDatabaseAccessor.java
 db0454e 
  
jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/dao/MySqlDatabaseAccessor.java
 86fde7c 
  
jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/spitter/DateIntervalSplitter.java
 PRE-CREATION 
  
jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/spitter/DecimalIntervalSplitter.java
 PRE-CREATION 
  
jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/spitter/DoubleIntervalSplitter.java
 PRE-CREATION 
  
jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/spitter/IntervalSplitter.java
 PRE-CREATION 
  
jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/spitter/IntervalSplitterFactory.java
 PRE-CREATION 
  
jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/spitter/LongIntervalSpitter.java
 PRE-CREATION 
  
jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/spitter/TimestampIntervalSplitter.java
 PRE-CREATION 
  
jdbc-handler/src/test/java/org/apache/hive/storage/jdbc/TestJdbcInputFormat.java
 b146633 
  
jdbc-handler/src/test/java/org/apache/hive/storage/jdbc/dao/TestGenericJdbcDatabaseAccessor.java
 34f061e 
  ql/src/test/queries/clientpositive/external_jdbc_table_partition.q 
PRE-CREATION 
  ql/src/test/queries/clientpositive/external_jdbc_table_typeconversion.q 
PRE-CREATION 
  ql/src/test/results/clientpositive/llap/external_jdbc_table_partition.q.out 
PRE-CREATION 
  
ql/src/test/results/clientpositive/llap/external_jdbc_table_typeconversion.q.out
 PRE-CREATION 


Diff: https://reviews.apache.org/r/69050/diff/2/

Changes: https://reviews.apache.org/r/69050/diff/1-2/


Testing
---


Thanks,

Daniel Dai

Re: Incorrect Release Notes for Hive-2.3.3

[Daniel Dai]

Yes, I cleared fixed version from the Jiras. It should be fixed.

Thanks,
Daniel

From: Oleksiy S 
Reply-To: "u...@hive.apache.org" 
Date: Friday, October 5, 2018 at 5:02 AM
To: "dev@hive.apache.org" , "u...@hive.apache.org" 

Subject: Re: Incorrect Release Notes for Hive-2.3.3

Guys any updates?

On Thu, Oct 4, 2018 at 11:15 AM Oleksiy S 
mailto:osayankin.superu...@gmail.com>> wrote:
Current release notes for Hive-2.3.3 are: 
RNs

Release Notes - Hive - Version 2.3.3

** Bug
* [HIVE-16939] - metastore error: 'export: -Dproc_metastore : not a valid 
identifier'
* [HIVE-18767] - Some alterPartitions invocations throw 
'NumberFormatException: null'
* [HIVE-18788] - Clean up inputs in JDBC PreparedStatement
* [HIVE-18815] - Remove unused feature in HPL/SQL
* [HIVE-18879] - Disallow embedded element in UDFXPathUtil needs to work if 
xercesImpl.jar in classpath
* [HIVE-18885] - DbNotificationListener has a deadlock between Java and DB 
locks (2.x line)
* [HIVE-20304] - When hive.optimize.skewjoin and hive.auto.convert.join are 
both set to true, and the execution engine is mr, same stage may launch twice 
due to the wrong generated plan
* [HIVE-20441] - NPE in ExprNodeGenericFuncDesc  when 
hive.allow.udf.load.on.demand is set to true

** Improvement
* [HIVE-19900] - HiveCLI HoS Performs Invalid Impersonation If User Name 
Truncated
* [HIVE-20284] - In strict mode, if constant propagation is enable, the 
partition filter may be folded before partition pruner lead to error "No 
partition predicate for Alias"


See table for short summary.
Issue

Comment

HIVE-18767

Issue is not in Hive-2.3.3

HIVE-20304

Issue is unresolved in Apache Hive JIRA

HIVE-20441

Issue is unresolved in Apache Hive JIRA

HIVE-19900

Resolved: Workaround. No commit to see

HIVE-20284

Issue is unresolved in Apache Hive JIRA


Please remove incorrect JIRAs from RNs.

--
Oleksiy


--
Oleksiy

Re: Review Request 69050: HIVE-20720: Add partition column option to JDBC handler

[Daniel Dai]

> On Oct. 17, 2018, 8:29 p.m., Jesús Camacho Rodríguez wrote:
> > jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/dao/GenericJdbcDatabaseAccessor.java
> > Lines 233 (patched)
> > <https://reviews.apache.org/r/69050/diff/2/?file=2098532#file2098532line298>
> >
> > Could we use a case insensitive regex that would match something like 
> > 'FROM\s+tableString'? I believe it would be safer. If you do not tackle it 
> > in this issue, let's create a follow-up.

Yes, it is better as it won't match anything before the first FROM. However, it 
could be quite complex to get it absolutely right. We also need to make sure it 
will search after "JOIN" and won't search beyond "WHERE/GROUP" etc. And can 
there be multiple "FROM"? I adopt the "from" part in the patch, but still leave 
the TODO tag and will think about a better way in the future.


- Daniel


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69050/#review209714
-------


On Oct. 16, 2018, 5:08 p.m., Daniel Dai wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/69050/
> ---
> 
> (Updated Oct. 16, 2018, 5:08 p.m.)
> 
> 
> Review request for hive.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> See HIVE-20720.
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/Constants.java 1190679 
>   itests/src/test/resources/testconfiguration.properties 9a87464 
>   
> jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/JdbcInputFormat.java 
> 74999db 
>   jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/JdbcInputSplit.java 
> 3a6ada8 
>   
> jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/JdbcRecordReader.java 
> 1da6213 
>   jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/JdbcSerDe.java 
> 5947628 
>   
> jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/conf/JdbcStorageConfigManager.java
>  18e2397 
>   
> jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/dao/DatabaseAccessor.java
>  fdaa794 
>   
> jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/dao/GenericJdbcDatabaseAccessor.java
>  abdc5f0 
>   
> jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/dao/JdbcRecordIterator.java
>  a95aca2 
>   
> jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/dao/JethroDatabaseAccessor.java
>  db0454e 
>   
> jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/dao/MySqlDatabaseAccessor.java
>  86fde7c 
>   
> jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/spitter/DateIntervalSplitter.java
>  PRE-CREATION 
>   
> jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/spitter/DecimalIntervalSplitter.java
>  PRE-CREATION 
>   
> jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/spitter/DoubleIntervalSplitter.java
>  PRE-CREATION 
>   
> jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/spitter/IntervalSplitter.java
>  PRE-CREATION 
>   
> jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/spitter/IntervalSplitterFactory.java
>  PRE-CREATION 
>   
> jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/spitter/LongIntervalSpitter.java
>  PRE-CREATION 
>   
> jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/spitter/TimestampIntervalSplitter.java
>  PRE-CREATION 
>   
> jdbc-handler/src/test/java/org/apache/hive/storage/jdbc/TestJdbcInputFormat.java
>  b146633 
>   
> jdbc-handler/src/test/java/org/apache/hive/storage/jdbc/dao/TestGenericJdbcDatabaseAccessor.java
>  34f061e 
>   ql/src/test/queries/clientpositive/external_jdbc_table_partition.q 
> PRE-CREATION 
>   ql/src/test/queries/clientpositive/external_jdbc_table_typeconversion.q 
> PRE-CREATION 
>   ql/src/test/results/clientpositive/llap/external_jdbc_table_partition.q.out 
> PRE-CREATION 
>   
> ql/src/test/results/clientpositive/llap/external_jdbc_table_typeconversion.q.out
>  PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/69050/diff/2/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Daniel Dai
> 
>

Re: Review Request 69143: CachedStore: Add more UT coverage (outside of .q files)

[Daniel Dai]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69143/#review210011
---




standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java
Lines 94 (patched)
<https://reviews.apache.org/r/69143/#comment294650>

Can you add a comment what's the difference between isCachePrewarmed?



standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java
Lines 227 (patched)
<https://reviews.apache.org/r/69143/#comment294647>

Log full stack



standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java
Lines 491 (patched)
<https://reviews.apache.org/r/69143/#comment294646>

Can you log the full stack trace (maybe using 
org.apache.hadoop.util.StringUtils.stringifyException)? There's another 
instance below.



standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java
Lines 522 (patched)
<https://reviews.apache.org/r/69143/#comment294648>

Make maxTries a constant (and reuse it below).



standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java
Lines 544 (patched)
<https://reviews.apache.org/r/69143/#comment294649>

maxTries should be constant.


- Daniel Dai


On Oct. 24, 2018, 5:04 p.m., Vaibhav Gumashta wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/69143/
> ---
> 
> (Updated Oct. 24, 2018, 5:04 p.m.)
> 
> 
> Review request for hive, Daniel Dai and Thejas Nair.
> 
> 
> Bugs: HIVE-20613
> https://issues.apache.org/jira/browse/HIVE-20613
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> https://issues.apache.org/jira/browse/HIVE-20613
> 
> 
> Diffs
> -
> 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/cache/CacheUtils.java
>  944c81313a 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java
>  70490f09e7 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/cache/SharedCache.java
>  c24e7160ac 
>   
> standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/cache/TestCachedStore.java
>  bb20d9f42a 
>   standalone-metastore/metastore-server/src/test/resources/log4j2.properties 
> 365687e1c9 
> 
> 
> Diff: https://reviews.apache.org/r/69143/diff/1/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Vaibhav Gumashta
> 
>

[VOTE] Apache Hive 3.1.1 Release Candidate 0

[Daniel Dai]

Apache Hive 3.1.1 Release Candidate 0 is available here:

http://people.apache.org/~daijy/apache-hive-3.1.1-rc-0/

Maven artifacts are available here:

https://repository.apache.org/content/repositories/orgapachehive-1092

Source tag for RCN is at:

https://github.com/apache/hive/tree/release-3.1.1-rc0

Voting will conclude in 72 hours.

Hive PMC Members: Please test and vote.

Thanks.

[RESULT][VOTE] Apache Hive 3.1.1 Release Candidate 0

[Daniel Dai]

[RESULT][VOTE] Apache Hive 3.1.1 Release Candidate 0

With three +1’s (Thejas, Zoltan, Daniel) and no -1’s the vote passes.

I will wrap up the release and send announcement once done.

[VOTE] Apache Hive 2.3.4 Release Candidate 0

[Daniel Dai]

Apache Hive 2.3.4 Release Candidate 0 is available here:

http://people.apache.org/~daijy/apache-hive-2.3.4-rc-0/

Maven artifacts are available here:

https://repository.apache.org/content/repositories/orgapachehive-1093

Source tag for RCN is at:

https://github.com/apache/hive/tree/release-2.3.4-rc0

Voting will conclude in 72 hours.

Hive PMC Members: Please test and vote.

Thanks.

[ANNOUNCE] Apache Hive 3.1.1 Released

[Daniel Dai]

The Apache Hive team is proud to announce the release of Apache Hive
version 3.1.1.

The Apache Hive (TM) data warehouse software facilitates querying and
managing large datasets residing in distributed storage. Built on top
of Apache Hadoop (TM), it provides, among others:

* Tools to enable easy data extract/transform/load (ETL)

* A mechanism to impose structure on a variety of data formats

* Access to files stored either directly in Apache HDFS (TM) or in other
  data storage systems such as Apache HBase (TM)

* Query execution via Apache Hadoop MapReduce, Apache Tez and Apache Spark
frameworks.

For Hive release details and downloads, please visit:
https://hive.apache.org/downloads.html

Hive 3.1.1 Release Notes are available here:
https://issues.apache.org/jira/secure/ReleaseNote.jspa?version=12344240=Text=12310843

We would like to thank the many contributors who made this release
possible.

Regards,

The Apache Hive Team

Re: Review Request 69143: CachedStore: Add more UT coverage (outside of .q files)

[Daniel Dai]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69143/#review210272
---


Ship it!




Ship It!

- Daniel Dai


On Nov. 2, 2018, 12:46 a.m., Vaibhav Gumashta wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/69143/
> ---
> 
> (Updated Nov. 2, 2018, 12:46 a.m.)
> 
> 
> Review request for hive, Daniel Dai and Thejas Nair.
> 
> 
> Bugs: HIVE-20613
> https://issues.apache.org/jira/browse/HIVE-20613
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> https://issues.apache.org/jira/browse/HIVE-20613
> 
> 
> Diffs
> -
> 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/cache/CacheUtils.java
>  944c81313a 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java
>  70490f09e7 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/cache/SharedCache.java
>  c24e7160ac 
>   
> standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/cache/TestCachedStore.java
>  bb20d9f42a 
>   standalone-metastore/metastore-server/src/test/resources/log4j2.properties 
> 365687e1c9 
> 
> 
> Diff: https://reviews.apache.org/r/69143/diff/2/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Vaibhav Gumashta
> 
>

[RESULT][VOTE] Apache Hive 2.3.4 Release Candidate 0

[Daniel Dai]

With three +1’s (Alan, Thejas, Daniel) and no -1’s the vote passes.

I will wrap up the release and send announcement once done.
 

[SECURITY] CVE-2018-11777: Blocking local resource access in HiveServer2

[Daniel Dai]

CVE-2018-11777: Blocking local resource access in HiveServer2

Severity: Important

Vendor: The Apache Software Foundation

Versions Affected: This vulnerability affects all versions of Hive,
including 2.3.3, 3.1.0 and earlier

Description: Local resources on HiveServer2 machines are not properly
protected against malicious user if ranger, sentry or sql standard
authorizer is not in use.

Mitigation: It is recommended to upgrade to 2.3.4 or 3.1.1 or later if
HiveServer2 is used, and ranger, sentry or sql standard authorizer
is not in use. Admin needs to specify the following entries in
hiveserver2-site.xml:


  hive.security.authorization.enabled
  true


  hive.security.authorization.manager
  
org.apache.hadoop.hive.ql.security.authorization.plugin.fallback.FallbackHiveAuthorizerFactory


FallbackHiveAuthorizerFactory will do the following to mitigate above
mentioned threat:
1. Disallow local file location in sql statements except for admin
2. Allow "set" only selected whitelist parameters
3. Disallow dfs commands except for admin
4. Disallow "ADD JAR" statement
5. Disallow "COMPILE" statement
6. Disallow "TRANSFORM" statement

Credit: This issue was discovered by Mithun Radhakrishnan of Oath Inc

[ANNOUNCE] Apache Hive 2.3.4 Released

[Daniel Dai]

The Apache Hive team is proud to announce the release of Apache Hive
version 2.3.4.

The Apache Hive (TM) data warehouse software facilitates querying and
managing large datasets residing in distributed storage. Built on top
of Apache Hadoop (TM), it provides, among others:

* Tools to enable easy data extract/transform/load (ETL)

* A mechanism to impose structure on a variety of data formats

* Access to files stored either directly in Apache HDFS (TM) or in other
  data storage systems such as Apache HBase (TM)

* Query execution via Apache Hadoop MapReduce, Apache Tez and Apache Spark
frameworks.

For Hive release details and downloads, please visit:
https://hive.apache.org/downloads.html

Hive 2.3.4 Release Notes are available here:
https://issues.apache.org/jira/secure/ReleaseNote.jspa?version=12344319=Text=12310843

We would like to thank the many contributors who made this release
possible.

Regards,

The Apache Hive Team

[SECURITY] CVE-2018-1314: Hive explain query not being authorized

[Daniel Dai]

CVE-2018-1314: Hive explain query not being authorized

Severity: Important

Vendor: The Apache Software Foundation

Versions Affected: This vulnerability affects all versions of Hive,
including 2.3.3, 3.1.0 and earlier

Description: Hive "EXPLAIN" operation does not check for necessary
authorization of involved entities in a query. An unauthorized user
can do "EXPLAIN" on arbitrary table or view and expose table metadata
and statistics.

Mitigation: all Hive users shall upgrade to 2.3.4 or 3.1.1 or later

Review Request 68917: HIVE-20651: JdbcStorageHandler password should be encrypted

[Daniel Dai]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68917/
---

Review request for hive and Sankar Hariappan.


Repository: hive-git


Description
---

See HIVE-20651


Diffs
-

  itests/src/test/resources/testconfiguration.properties fdd8ecc 
  jdbc-handler/pom.xml f90892f 
  jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/JdbcInputFormat.java 
caa823f 
  
jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/conf/JdbcStorageConfigManager.java
 55fc0ea 
  
jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/dao/GenericJdbcDatabaseAccessor.java
 b2ad9a6 
  
jdbc-handler/src/test/java/org/apache/hive/storage/jdbc/TestJdbcInputFormat.java
 e904774 
  pom.xml 5008923 
  ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 74fb1ba 
  ql/src/java/org/apache/hadoop/hive/ql/exec/mr/ExecMapper.java 99b33a3 
  ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkMapRecordHandler.java 
7cd853f 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/MapRecordProcessor.java 
ac43917 
  ql/src/java/org/apache/hadoop/hive/ql/io/HiveInputFormat.java a86656c 
  ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java 82c3ca9 
  ql/src/java/org/apache/hadoop/hive/ql/plan/PlanUtils.java 250a085 
  ql/src/java/org/apache/hadoop/hive/ql/plan/TableDesc.java b73faa5 
  ql/src/test/queries/clientpositive/external_jdbc_auth.q PRE-CREATION 
  ql/src/test/results/clientpositive/llap/external_jdbc_auth.q.out PRE-CREATION 


Diff: https://reviews.apache.org/r/68917/diff/1/


Testing
---


Thanks,

Daniel Dai

Re: Review Request 68917: HIVE-20651: JdbcStorageHandler password should be encrypted

[Daniel Dai]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68917/
---

(Updated Oct. 5, 2018, 5:24 a.m.)


Review request for hive and Sankar Hariappan.


Repository: hive-git


Description
---

See HIVE-20651


Diffs (updated)
-

  itests/src/test/resources/testconfiguration.properties fdd8ecc 
  jdbc-handler/pom.xml f90892f 
  jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/JdbcInputFormat.java 
caa823f 
  
jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/conf/JdbcStorageConfigManager.java
 55fc0ea 
  
jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/dao/GenericJdbcDatabaseAccessor.java
 b2ad9a6 
  
jdbc-handler/src/test/java/org/apache/hive/storage/jdbc/TestJdbcInputFormat.java
 e904774 
  pom.xml 5008923 
  ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 74fb1ba 
  ql/src/java/org/apache/hadoop/hive/ql/exec/mr/ExecMapper.java 99b33a3 
  ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkMapRecordHandler.java 
7cd853f 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/MapRecordProcessor.java 
ac43917 
  ql/src/java/org/apache/hadoop/hive/ql/io/HiveInputFormat.java a86656c 
  ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java 82c3ca9 
  ql/src/java/org/apache/hadoop/hive/ql/plan/PlanUtils.java 250a085 
  ql/src/java/org/apache/hadoop/hive/ql/plan/TableDesc.java b73faa5 


Diff: https://reviews.apache.org/r/68917/diff/2/

Changes: https://reviews.apache.org/r/68917/diff/1-2/


Testing
---


Thanks,

Daniel Dai

Review Request 68733: HIVE-20549: Allow user set query tag, and kill query with tag

[Daniel Dai]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68733/
---

Review request for hive.


Repository: hive-git


Description
---

See HIVE-20549


Diffs
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java aa58d74 
  
itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestJdbcWithMiniLlapArrow.java
 4942ed9 
  ql/src/java/org/apache/hadoop/hive/ql/Driver.java dad2035 
  ql/src/java/org/apache/hadoop/hive/ql/QueryState.java 028dd60 
  ql/src/java/org/apache/hadoop/hive/ql/parse/ReplicationSemanticAnalyzer.java 
e4186c4 
  ql/src/test/queries/clientnegative/authorization_kill_query.q 5379f87 
  service/src/java/org/apache/hive/service/cli/operation/OperationManager.java 
8db6a29 
  service/src/java/org/apache/hive/service/server/KillQueryImpl.java 490a04d 


Diff: https://reviews.apache.org/r/68733/diff/1/


Testing
---


Thanks,

Daniel Dai

[jira] [Created] (HIVE-2800) NPE in create index without comment clause in external metastore

[Daniel Dai (Created) (JIRA)]

NPE in create index without comment clause in external metastore
--

 Key: HIVE-2800
 URL: https://issues.apache.org/jira/browse/HIVE-2800
 Project: Hive
  Issue Type: Bug
  Components: Metastore
Affects Versions: 0.8.1, 0.9.0
Reporter: Daniel Dai
Priority: Minor


This happens only when using external metastore (with --hiveconf 
hive.metastore.uris=thrift://localhost:8088 --hiveconf 
hive.metastore.local=false). Also if I gave a comment in the statement, this 
exception go away.

Here is the statement:
create index test111 on table hcat_test(name) as 'compact' with deferred 
rebuild;

Here is the stack:
2012-02-10 17:07:42,612 ERROR exec.Task (SessionState.java:printError(380)) - 
FAILED: Error in metadata: java.lang.NullPointerException
org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.NullPointerException
at org.apache.hadoop.hive.ql.metadata.Hive.createIndex(Hive.java:725)
at org.apache.hadoop.hive.ql.exec.DDLTask.createIndex(DDLTask.java:822)
at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:231)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:134)
at 
org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:57)
at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1291)
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1082)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:933)
at 
org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:255)
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:212)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:403)
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:671)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:554)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
Caused by: java.lang.NullPointerException
at 
org.apache.thrift.protocol.TBinaryProtocol.writeString(TBinaryProtocol.java:185)
at org.apache.hadoop.hive.metastore.api.Index.write(Index.java:1032)
at 
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$add_index_args.write(ThriftHiveMetastore.java:47518)
at org.apache.thrift.TServiceClient.sendBase(TServiceClient.java:63)
at 
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.send_add_index(ThriftHiveMetastore.java:1675)
at 
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.add_index(ThriftHiveMetastore.java:1666)
at 
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.createIndex(HiveMetaStoreClient.java:853)
at org.apache.hadoop.hive.ql.metadata.Hive.createIndex(Hive.java:722)
... 17 more

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HIVE-4578) Changes to Pig's test harness broke HCat e2e tests

[Daniel Dai (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-4578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13662406#comment-13662406
 ] 

Daniel Dai commented on HIVE-4578:
--

+1

 Changes to Pig's test harness broke HCat e2e tests
 --

 Key: HIVE-4578
 URL: https://issues.apache.org/jira/browse/HIVE-4578
 Project: Hive
  Issue Type: Bug
  Components: Tests
Affects Versions: 0.12.0
Reporter: Alan Gates
Assignee: Alan Gates
 Fix For: 0.12.0

 Attachments: HIVE-4578.patch


 HCatalog externs the test harness from Pig.  Pig recently made some changes 
 to the test harness to work better across Unix and Windows.  These changes 
 require new OS specific files.  HCatalog will also need these files in order 
 to work with the test harness.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HIVE-4586) [HCatalog] WebHCat should return 400 error for undefined resource

[Daniel Dai (JIRA)]

Daniel Dai created HIVE-4586:


 Summary: [HCatalog] WebHCat should return 400 error for undefined 
resource
 Key: HIVE-4586
 URL: https://issues.apache.org/jira/browse/HIVE-4586
 Project: Hive
  Issue Type: Bug
Reporter: Daniel Dai




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HIVE-4586) [HCatalog] WebHCat should return 400 error for undefined resource

[Daniel Dai (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-4586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13663611#comment-13663611
 ] 

Daniel Dai commented on HIVE-4586:
--

If user request a resource which is not exist, webhcat returns a 500 error:
eg, http://localhost:50111/templeton/v1/ddl/databae/abc?user.name=hcat, 
misspell database

This should be 400 instead.

 [HCatalog] WebHCat should return 400 error for undefined resource
 -

 Key: HIVE-4586
 URL: https://issues.apache.org/jira/browse/HIVE-4586
 Project: Hive
  Issue Type: Bug
Reporter: Daniel Dai



--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HIVE-4586) [HCatalog] WebHCat should return 400 error for undefined resource

[Daniel Dai (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-4586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Dai updated HIVE-4586:
-

Attachment: HIVE-4586-1.patch

Here is what WebHCat does after patch:
1. If misspell on the resource name, get 400
2. If database schema name is wrong, get 404
3. If some internal error (such as HCatException), get 500

 [HCatalog] WebHCat should return 400 error for undefined resource
 -

 Key: HIVE-4586
 URL: https://issues.apache.org/jira/browse/HIVE-4586
 Project: Hive
  Issue Type: Bug
Reporter: Daniel Dai
 Attachments: HIVE-4586-1.patch




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HIVE-4586) [HCatalog] WebHCat should return 400 error for undefined resource

[Daniel Dai (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-4586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Dai updated HIVE-4586:
-

Attachment: (was: HIVE-4586-1.patch)

 [HCatalog] WebHCat should return 400 error for undefined resource
 -

 Key: HIVE-4586
 URL: https://issues.apache.org/jira/browse/HIVE-4586
 Project: Hive
  Issue Type: Bug
Reporter: Daniel Dai
 Attachments: HIVE-4586-1.patch




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HIVE-4586) [HCatalog] WebHCat should return 400 error for undefined resource

[Daniel Dai (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-4586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Dai updated HIVE-4586:
-

Attachment: HIVE-4586-1.patch

 [HCatalog] WebHCat should return 400 error for undefined resource
 -

 Key: HIVE-4586
 URL: https://issues.apache.org/jira/browse/HIVE-4586
 Project: Hive
  Issue Type: Bug
Reporter: Daniel Dai
 Attachments: HIVE-4586-1.patch




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HIVE-4586) [HCatalog] WebHCat should return 400 error for undefined resource

[Daniel Dai (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-4586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13663625#comment-13663625
 ] 

Daniel Dai commented on HIVE-4586:
--

Change case 1 to 404 as well.

 [HCatalog] WebHCat should return 400 error for undefined resource
 -

 Key: HIVE-4586
 URL: https://issues.apache.org/jira/browse/HIVE-4586
 Project: Hive
  Issue Type: Bug
Reporter: Daniel Dai
 Attachments: HIVE-4586-1.patch




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HIVE-4586) [HCatalog] WebHCat should return 404 error for undefined resource

[Daniel Dai (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-4586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Dai updated HIVE-4586:
-

Summary: [HCatalog] WebHCat should return 404 error for undefined resource  
(was: [HCatalog] WebHCat should return 400 error for undefined resource)

 [HCatalog] WebHCat should return 404 error for undefined resource
 -

 Key: HIVE-4586
 URL: https://issues.apache.org/jira/browse/HIVE-4586
 Project: Hive
  Issue Type: Bug
Reporter: Daniel Dai
 Attachments: HIVE-4586-1.patch




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HIVE-4531) [WebHCat] Collecting task logs to hdfs

[Daniel Dai (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-4531?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Dai updated HIVE-4531:
-

Attachment: samplestatusdirwithlist.tar.gz
HIVE-4531-5.patch

Adding a list file to the logs. Attached samplestatusdirwithlist.tar.gz for a 
sample status directory. Here is a sample list file (list.txt):

job: job_201305221327_0068(name=PigLatin:73.pig,status=SUCCEEDED)
  
attempt:attempt_201305221327_0068_m_00_0(type=map,status=completed,starttime=22-May-2013
 17:10:26,endtime=22-May-2013 17:10:32)
  
attempt:attempt_201305221327_0068_m_02_0(type=setup,status=completed,starttime=22-May-2013
 17:10:17,endtime=22-May-2013 17:10:26)
  
attempt:attempt_201305221327_0068_m_01_0(type=cleanup,status=completed,starttime=22-May-2013
 17:10:32,endtime=22-May-2013 17:10:38)

job: job_201305221327_0069(name=PigLatin:73.pig,status=SUCCEEDED)
  
attempt:attempt_201305221327_0069_m_00_0(type=map,status=completed,starttime=22-May-2013
 17:10:53,endtime=22-May-2013 17:10:59)
  
attempt:attempt_201305221327_0069_r_00_0(type=reduce,status=completed,starttime=22-May-2013
 17:10:59,endtime=22-May-2013 17:11:11)
  
attempt:attempt_201305221327_0069_m_02_0(type=setup,status=completed,starttime=22-May-2013
 17:10:44,endtime=22-May-2013 17:10:53)
  
attempt:attempt_201305221327_0069_m_01_0(type=cleanup,status=completed,starttime=22-May-2013
 17:11:11,endtime=22-May-2013 17:11:17)

job: job_201305221327_0070(name=PigLatin:73.pig,status=SUCCEEDED)
  
attempt:attempt_201305221327_0070_m_00_0(type=map,status=completed,starttime=22-May-2013
 17:11:32,endtime=22-May-2013 17:11:38)
  
attempt:attempt_201305221327_0070_r_00_0(type=reduce,status=completed,starttime=22-May-2013
 17:11:38,endtime=22-May-2013 17:11:50)
  
attempt:attempt_201305221327_0070_m_02_0(type=setup,status=completed,starttime=22-May-2013
 17:11:23,endtime=22-May-2013 17:11:32)
  
attempt:attempt_201305221327_0070_m_01_0(type=cleanup,status=completed,starttime=22-May-2013
 17:11:50,endtime=22-May-2013 17:11:56)

job: job_201305221327_0071(name=PigLatin:73.pig,status=FAILED)
  
attempt:attempt_201305221327_0071_m_00_0(type=map,status=completed,starttime=22-May-2013
 17:12:11,endtime=22-May-2013 17:12:17)
  
attempt:attempt_201305221327_0071_m_01_0(type=map,status=completed,starttime=22-May-2013
 17:12:17,endtime=22-May-2013 17:12:23)
  
attempt:attempt_201305221327_0071_m_03_0(type=setup,status=completed,starttime=22-May-2013
 17:12:02,endtime=22-May-2013 17:12:11)
  
attempt:attempt_201305221327_0071_m_02_0(type=cleanup,status=completed,starttime=22-May-2013
 17:13:11,endtime=22-May-2013 17:13:17)
  
attempt:attempt_201305221327_0071_r_00_0(type=reduce,status=failed,starttime=22-May-2013
 17:12:17,endtime=22-May-2013 17:12:29)
  
attempt:attempt_201305221327_0071_r_00_1(type=reduce,status=failed,starttime=22-May-2013
 17:12:35,endtime=22-May-2013 17:12:33)
  
attempt:attempt_201305221327_0071_r_00_2(type=reduce,status=failed,starttime=22-May-2013
 17:12:47,endtime=22-May-2013 17:12:43)
  
attempt:attempt_201305221327_0071_r_00_3(type=reduce,status=failed,starttime=22-May-2013
 17:12:59,endtime=22-May-2013 17:12:46)


 [WebHCat] Collecting task logs to hdfs
 --

 Key: HIVE-4531
 URL: https://issues.apache.org/jira/browse/HIVE-4531
 Project: Hive
  Issue Type: New Feature
  Components: HCatalog
Reporter: Daniel Dai
 Attachments: HIVE-4531-1.patch, HIVE-4531-2.patch, HIVE-4531-3.patch, 
 HIVE-4531-4.patch, HIVE-4531-5.patch, samplestatusdirwithlist.tar.gz


 It would be nice we collect task logs after job finish. This is similar to 
 what Amazon EMR does.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HIVE-2670) A cluster test utility for Hive

[Daniel Dai (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-2670?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13672204#comment-13672204
 ] 

Daniel Dai commented on HIVE-2670:
--

Checkin_2 fail on my Mac. Change hive_nightly.conf:41, from:
  'sortArgs' = ['-t', '   ', '+1', '-2'],
to
  'sortArgs' = ['-t', '', '-k', '2,2n'],
Solve the problem.

Otherwise +1.

 A cluster test utility for Hive
 ---

 Key: HIVE-2670
 URL: https://issues.apache.org/jira/browse/HIVE-2670
 Project: Hive
  Issue Type: New Feature
  Components: Testing Infrastructure
Reporter: Alan Gates
Assignee: Johnny Zhang
 Attachments: harness.tar, HIVE-2670_5.patch, 
 hive_cluster_test_2.patch, hive_cluster_test_3.patch, 
 hive_cluster_test_4.patch, hive_cluster_test.patch


 Hive has an extensive set of unit tests, but it does not have an 
 infrastructure for testing in a cluster environment.  Pig and HCatalog have 
 been using a test harness for cluster testing for some time.  We have written 
 Hive drivers and tests to run in this harness.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HIVE-4677) [HCatalog] WebHCat e2e tests fail on Hadoop 2

[Daniel Dai (JIRA)]

Daniel Dai created HIVE-4677:


 Summary: [HCatalog] WebHCat e2e tests fail on Hadoop 2
 Key: HIVE-4677
 URL: https://issues.apache.org/jira/browse/HIVE-4677
 Project: Hive
  Issue Type: Bug
Reporter: Daniel Dai
 Attachments: HIVE-4677-1.patch

curl 
http://hor5n26.gq1.ygridcore.net:50111/templeton/v1/queue/job_1370377838831_0012?user.name=hrt_qa
{error:Does not contain a valid host:port authority: local}

Here is the detailed stacktrace from the server:
{code}
WARN  | 04 Jun 2013 22:21:52,204 | org.apache.hadoop.conf.Configuration | 
mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
ERROR | 04 Jun 2013 22:21:52,204 | 
org.apache.hcatalog.templeton.CatchallExceptionMapper | Does not contain a 
valid host:port authority: local
java.lang.IllegalArgumentException: Does not contain a valid host:port 
authority: local
at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:211)
at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:163)
at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:152)
at 
org.apache.hcatalog.templeton.TempletonDelegator.getAddress(TempletonDelegator.java:41)
at 
org.apache.hcatalog.templeton.StatusDelegator.run(StatusDelegator.java:47)
at org.apache.hcatalog.templeton.Server.showQueueId(Server.java:688)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)

{code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HIVE-4677) [HCatalog] WebHCat e2e tests fail on Hadoop 2

[Daniel Dai (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-4677?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Dai updated HIVE-4677:
-

Attachment: HIVE-4677-1.patch

 [HCatalog] WebHCat e2e tests fail on Hadoop 2
 -

 Key: HIVE-4677
 URL: https://issues.apache.org/jira/browse/HIVE-4677
 Project: Hive
  Issue Type: Bug
Reporter: Daniel Dai
 Attachments: HIVE-4677-1.patch


 curl 
 http://hor5n26.gq1.ygridcore.net:50111/templeton/v1/queue/job_1370377838831_0012?user.name=hrt_qa
 {error:Does not contain a valid host:port authority: local}
 Here is the detailed stacktrace from the server:
 {code}
 WARN  | 04 Jun 2013 22:21:52,204 | org.apache.hadoop.conf.Configuration | 
 mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
 ERROR | 04 Jun 2013 22:21:52,204 | 
 org.apache.hcatalog.templeton.CatchallExceptionMapper | Does not contain a 
 valid host:port authority: local
 java.lang.IllegalArgumentException: Does not contain a valid host:port 
 authority: local
 at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:211)
 at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:163)
 at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:152)
 at 
 org.apache.hcatalog.templeton.TempletonDelegator.getAddress(TempletonDelegator.java:41)
 at 
 org.apache.hcatalog.templeton.StatusDelegator.run(StatusDelegator.java:47)
 at org.apache.hcatalog.templeton.Server.showQueueId(Server.java:688)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at 
 sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
 at 
 sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
 at java.lang.reflect.Method.invoke(Method.java:597)
 
 {code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HIVE-4784) ant testreport doesn't include any HCatalog tests

[Daniel Dai (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-4784?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13695584#comment-13695584
 ] 

Daniel Dai commented on HIVE-4784:
--

+1

 ant testreport doesn't include any HCatalog tests
 -

 Key: HIVE-4784
 URL: https://issues.apache.org/jira/browse/HIVE-4784
 Project: Hive
  Issue Type: Bug
  Components: HCatalog
Affects Versions: 0.11.0
Reporter: Eugene Koifman
Assignee: Eugene Koifman
 Fix For: 0.12.0

 Attachments: HIVE-4784.patch


 run 
 1. ant test -Dmodule=hcatalog
 2. ant testreport
 no .html file is generated.  In particular, Apache builds don't show anything 
 about HCat test (so it's not even obvious if it's running them)

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HIVE-4784) ant testreport doesn't include any HCatalog tests

[Daniel Dai (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-4784?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13695597#comment-13695597
 ] 

Daniel Dai commented on HIVE-4784:
--

Patch committed to trunk.

 ant testreport doesn't include any HCatalog tests
 -

 Key: HIVE-4784
 URL: https://issues.apache.org/jira/browse/HIVE-4784
 Project: Hive
  Issue Type: Bug
  Components: HCatalog
Affects Versions: 0.11.0
Reporter: Eugene Koifman
Assignee: Eugene Koifman
 Fix For: 0.12.0

 Attachments: HIVE-4784.patch


 run 
 1. ant test -Dmodule=hcatalog
 2. ant testreport
 no .html file is generated.  In particular, Apache builds don't show anything 
 about HCat test (so it's not even obvious if it's running them)

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HIVE-4591) Making changes to webhcat-site.xml have no effect

[Daniel Dai (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-4591?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13695599#comment-13695599
 ] 

Daniel Dai commented on HIVE-4591:
--

Patch committed to trunk.

 Making changes to webhcat-site.xml have no effect
 -

 Key: HIVE-4591
 URL: https://issues.apache.org/jira/browse/HIVE-4591
 Project: Hive
  Issue Type: Bug
  Components: HCatalog
Affects Versions: 0.11.0
Reporter: Eugene Koifman
Assignee: Eugene Koifman
 Fix For: 0.12.0

 Attachments: HIVE-4591.patch

   Original Estimate: 24h
  Time Spent: 4h
  Remaining Estimate: 20h

 Looks like WebHCat configuration is read as follows:
 Configuration: core-default.xml, core-site.xml, mapred-default.xml, 
 mapred-site.xml, 
 jar:file:/Users/ekoifman/dev/hive/build/dist/hcatalog/share/webhcat/svr/webhcat-0.12.0-SNAPSHOT.jar!/webhcat-default.xml
 creating 
 /Users/ekoifman/dev/hive/build/dist/hcatalog/etc/webhcat/webhcat-site.xml and 
 setting templeton.exec.timeout has no effect as can be seen in ExecServiceImpl
 Probably the webhcat_server.sh script is missing something

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HIVE-4591) Making changes to webhcat-site.xml have no effect

[Daniel Dai (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-4591?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Dai updated HIVE-4591:
-

  Resolution: Fixed
Hadoop Flags: Reviewed
  Status: Resolved  (was: Patch Available)

 Making changes to webhcat-site.xml have no effect
 -

 Key: HIVE-4591
 URL: https://issues.apache.org/jira/browse/HIVE-4591
 Project: Hive
  Issue Type: Bug
  Components: HCatalog
Affects Versions: 0.11.0
Reporter: Eugene Koifman
Assignee: Eugene Koifman
 Fix For: 0.12.0

 Attachments: HIVE-4591.patch

   Original Estimate: 24h
  Time Spent: 4h
  Remaining Estimate: 20h

 Looks like WebHCat configuration is read as follows:
 Configuration: core-default.xml, core-site.xml, mapred-default.xml, 
 mapred-site.xml, 
 jar:file:/Users/ekoifman/dev/hive/build/dist/hcatalog/share/webhcat/svr/webhcat-0.12.0-SNAPSHOT.jar!/webhcat-default.xml
 creating 
 /Users/ekoifman/dev/hive/build/dist/hcatalog/etc/webhcat/webhcat-site.xml and 
 setting templeton.exec.timeout has no effect as can be seen in ExecServiceImpl
 Probably the webhcat_server.sh script is missing something

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HIVE-4784) ant testreport doesn't include any HCatalog tests

[Daniel Dai (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-4784?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Dai updated HIVE-4784:
-

  Resolution: Fixed
Hadoop Flags: Reviewed
  Status: Resolved  (was: Patch Available)

 ant testreport doesn't include any HCatalog tests
 -

 Key: HIVE-4784
 URL: https://issues.apache.org/jira/browse/HIVE-4784
 Project: Hive
  Issue Type: Bug
  Components: HCatalog
Affects Versions: 0.11.0
Reporter: Eugene Koifman
Assignee: Eugene Koifman
 Fix For: 0.12.0

 Attachments: HIVE-4784.patch


 run 
 1. ant test -Dmodule=hcatalog
 2. ant testreport
 no .html file is generated.  In particular, Apache builds don't show anything 
 about HCat test (so it's not even obvious if it's running them)

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HIVE-4820) webhcat_config.sh should set default values for HIVE_HOME and HCAT_PREFIX that work with default build tree structure

[Daniel Dai (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-4820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Dai updated HIVE-4820:
-

  Resolution: Fixed
Hadoop Flags: Reviewed
  Status: Resolved  (was: Patch Available)

Patch committed to trunk. Thanks Eugene!

 webhcat_config.sh should set default values for HIVE_HOME and HCAT_PREFIX 
 that work with default build tree structure
 -

 Key: HIVE-4820
 URL: https://issues.apache.org/jira/browse/HIVE-4820
 Project: Hive
  Issue Type: Bug
  Components: HCatalog
Affects Versions: 0.12.0
Reporter: Eugene Koifman
Assignee: Eugene Koifman
 Fix For: 0.12.0

 Attachments: hive4820.2.patch, HIVE4820.patch


 Currently they are expected to be set by the user which makes development 
 inconvenient.
 It makes writing unit tests for WebHcat more difficult as well.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HIVE-4893) [WebHCat] HTTP 500 errors should be mapped to 400 for bad request

[Daniel Dai (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-4893?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Dai updated HIVE-4893:
-

Attachment: HIVE-4893-1.patch

 [WebHCat] HTTP 500 errors should be mapped to 400 for bad request
 -

 Key: HIVE-4893
 URL: https://issues.apache.org/jira/browse/HIVE-4893
 Project: Hive
  Issue Type: Bug
  Components: HCatalog
Reporter: Daniel Dai
Assignee: Daniel Dai
 Fix For: 0.12.0

 Attachments: HIVE-4893-1.patch




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HIVE-4893) [WebHCat] HTTP 500 errors should be mapped to 400 for bad request

[Daniel Dai (JIRA)]

Daniel Dai created HIVE-4893:


 Summary: [WebHCat] HTTP 500 errors should be mapped to 400 for bad 
request
 Key: HIVE-4893
 URL: https://issues.apache.org/jira/browse/HIVE-4893
 Project: Hive
  Issue Type: Bug
  Components: HCatalog
Reporter: Daniel Dai
Assignee: Daniel Dai
 Fix For: 0.12.0
 Attachments: HIVE-4893-1.patch



--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira