[jira] [Created] (HIVE-15096) hplsql registerUDF conflicts with pom.xml

2016-10-28 Thread Fei Hui (JIRA) 
Fei Hui created HIVE-15096:
--

 Summary: hplsql registerUDF conflicts with pom.xml
 Key: HIVE-15096
 URL: https://issues.apache.org/jira/browse/HIVE-15096
 Project: Hive
  Issue Type: Bug
  Components: hpl/sql
Affects Versions: 2.0.1, 2.1.0, 2.0.0
Reporter: Fei Hui
 Fix For: 2.2.0


in hplsql code, registerUDF code is

sql.add("ADD JAR " + dir + "hplsql.jar");
sql.add("ADD JAR " + dir + "antlr-runtime-4.5.jar");
sql.add("ADD FILE " + dir + Conf.SITE_XML);

but pom configufation is

  
org.apache.hive
hive
2.2.0-SNAPSHOT
../pom.xml
  

  hive-hplsql
  jar
  Hive HPL/SQL


   org.antlr
   antlr4-runtime
   4.5


when run hplsql , errors occur as below

 Error while processing statement: 
/opt/apps/apache-hive-2.0.0-bin/lib/hplsql.jar does not exist



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-15095) TestHiveDruidQueryBasedInputFormat.testCreateSplitsIntervals fails

2016-10-28 Thread Sergey Shelukhin (JIRA) 
Sergey Shelukhin created HIVE-15095:
---

 Summary: 
TestHiveDruidQueryBasedInputFormat.testCreateSplitsIntervals fails
 Key: HIVE-15095
 URL: https://issues.apache.org/jira/browse/HIVE-15095
 Project: Hive
  Issue Type: Bug
Reporter: Sergey Shelukhin


{noformat}
junit.framework.AssertionFailedError: 
expected:<[[2009-12-31T16:00:00.000-08:00/2010-04-01T23:00:00.000-07:00], 
[2010-04-01T23:00:00.000-07:00/2010-07-02T05:00:00.000-07:00], 
[2010-07-02T05:00:00.000-07:00/2010-10-01T11:00:00.000-07:00], 
[2010-10-01T11:00:00.000-07:00/2010-12-31T16:00:00.000-08:00]]> but 
was:<[[2010-01-01T00:00:00.000Z/2010-04-02T06:00:00.000Z], 
[2010-04-02T06:00:00.000Z/2010-07-02T12:00:00.000Z], 
[2010-07-02T12:00:00.000Z/2010-10-01T18:00:00.000Z], 
[2010-10-01T18:00:00.000Z/2011-01-01T00:00:00.000Z]]>at 
org.apache.hadoop.hive.druid.TestHiveDruidQueryBasedInputFormat.testCreateSplitsIntervals(TestHiveDruidQueryBasedInputFormat.java:54)
{noformat}
Seems offset by 7-8 hours.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-15094) Fix test failures for 2.1.1 regarding schema evolution with DECIMAL types

2016-10-28 Thread JIRA 
Sergio Peña created HIVE-15094:
--

 Summary: Fix test failures for 2.1.1 regarding schema evolution 
with DECIMAL types
 Key: HIVE-15094
 URL: https://issues.apache.org/jira/browse/HIVE-15094
 Project: Hive
  Issue Type: Sub-task
  Components: Hive
Affects Versions: 2.1.0
Reporter: Sergio Peña


Several tests failures related to schema evolution are happening on branch-2.1 
due to a patch reverted in the past.

{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_text_vec_mapwork_part
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_orc_nonvec_mapwork_part
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_orc_nonvec_mapwork_table
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_text_vecrow_mapwork_table
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_text_nonvec_mapwork_part
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_orc_acidvec_mapwork_part
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_orc_vec_mapwork_table
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_text_vec_mapwork_table
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_orc_acid_mapwork_table
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_text_nonvec_mapwork_table
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_text_vecrow_mapwork_part
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_orc_nonvec_fetchwork_table
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_orc_acid_mapwork_part
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_orc_vec_mapwork_part
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_orc_acidvec_mapwork_table
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_orc_nonvec_fetchwork_part
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_orc_acid_mapwork_table
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_text_vecrow_mapwork_part
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_orc_vec_mapwork_table
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_orc_nonvec_mapwork_part
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_text_vec_mapwork_table
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_text_nonvec_mapwork_table
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_orc_nonvec_fetchwork_table
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_orc_acid_mapwork_part
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_orc_vec_mapwork_part
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_orc_acidvec_mapwork_table
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_orc_nonvec_fetchwork_part
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_text_vec_mapwork_part
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_orc_nonvec_mapwork_table
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_text_vecrow_mapwork_table
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_text_nonvec_mapwork_part
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_orc_acidvec_mapwork_part
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_schema_evol_text_vecrow_mapwork_table
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_schema_evol_text_vec_mapwork_part
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_schema_evol_orc_acid_mapwork_table
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_schema_evol_orc_vec_mapwork_part
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_schema_evol_orc_acidvec_mapwork_table
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_schema_evol_orc_nonvec_mapwork_table
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_schema_evol_text_vecrow_mapwork_part
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_schema_evol_orc_nonvec_fetchwork_part
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_schema_evol_text_vec_mapwork_table
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_schema_evol_orc_acidvec_mapwork_part
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_schema_evol_orc_nonvec_mapwork_part
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_schema_evol_text_nonvec_mapwork_part
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_schema_evol_text_nonvec_mapwork_table
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_schema_evol_orc_nonvec_fetchwork_table

Re: build bot on master

2016-10-28 Thread Siddharth Seth 
Restarted the ptest server. That normally fixes this error.

On Fri, Oct 28, 2016 at 10:33 AM, Eugene Koifman 
wrote:

> Hi,
> all test runs are failing with the error below.   Does anyone know how to
> fix it?
>
> Thanks,
> Eugene
>
> from https://builds.apache.org/view/H-L/view/Hive/job/
> PreCommit-HIVE-Build/1866/console
>
>
>
>
> Build/hive/build/hive/testutils/ptest2/target/lib/
> guice-assistedinject-3.0.jar
> [INFO] Copying velocity-1.7.jar to /home/jenkins/jenkins-slave/
> workspace/PreCommit-HIVE-Build/hive/build/hive/
> testutils/ptest2/target/lib/velocity-1.7.jar
> [INFO] 
> 
> [INFO] BUILD SUCCESS
> [INFO] 
> 
> [INFO] Total time: 10.833 s
> [INFO] Finished at: 2016-10-28T15:57:26+00:00
> [INFO] Final Memory: 29M/723M
> [INFO] 
> 
> + local 'PTEST_CLASSPATH=/home/jenkins/jenkins-slave/
> workspace/PreCommit-HIVE-Build/hive/build/hive/
> testutils/ptest2/target/hive-ptest-1.0-classes.jar:/home/
> jenkins/jenkins-slave/workspace/PreCommit-HIVE-Build/hive/build/hive/
> testutils/ptest2/target/lib/*'
> + java -cp '/home/jenkins/jenkins-slave/workspace/PreCommit-HIVE-
> Build/hive/build/hive/testutils/ptest2/target/hive-
> ptest-1.0-classes.jar:/home/jenkins/jenkins-slave/
> workspace/PreCommit-HIVE-Build/hive/build/hive/testutils/ptest2/target/lib/*'
> org.apache.hive.ptest.api.client.PTestClient --command testStart
> --outputDir /home/jenkins/jenkins-slave/workspace/PreCommit-HIVE-
> Build/hive/build/hive/testutils/ptest2/target --password ''
> --testHandle PreCommit-HIVE-Build-1866 --endpoint
> http://104.198.109.242:8080/hive-ptest-1.0 --logsEndpoint
> http://104.198.109.242/logs/ --profile master-mr2 --patch
> https://issues.apache.org/jira/secure/attachment/
> 12835837/HIVE-15093.1.patch --jira HIVE-15093
>
>
> Logs are located: http://104.198.109.242/logs/PreCommit-HIVE-Build-1866
>
>
> Exception in thread "main" java.lang.RuntimeException: 404 Not Found
> at org.apache.hive.ptest.api.client.PTestClient.
> downloadTestResults(PTestClient.java:189)
> at org.apache.hive.ptest.api.client.PTestClient.testStart(
> PTestClient.java:137)
> at org.apache.hive.ptest.api.client.PTestClient.main(
> PTestClient.java:320)
> + ret=1
> + unpack_test_results
> + '[' -z /home/jenkins/jenkins-slave/workspace/PreCommit-HIVE-Build/hive/build
> ']'
> + cd /home/jenkins/jenkins-slave/workspace/PreCommit-HIVE-
> Build/hive/build/hive/testutils/ptest2/target
> + [[ -f test-results.tar.gz ]]
> + exit 1
> + rm -f /tmp/tmp.r59JTiDlN1
> Build step 'Execute shell' marked build as failure
> Recording test results
> ERROR: Step ?Publish JUnit test result report? failed: Test reports were
> found but none of them are new. Did tests run?
> For example, /home/jenkins/jenkins-slave/workspace/PreCommit-HIVE-
> Build/hive/build/test-results/TEST-1-TestCliDriver-udf_
> upper.q-ctas_date.q-schema_evol_orc_acidvec_table_update.
> q-and-27-more-TEST-org.apache.hadoop.hive.cli.TestCliDriver.xml is 2 days
> 19 hr old
>
> [description-setter] Description set ci.org/search?query=[description-setter]%20Description%20set>:
> HIVE-15093  /   master-mr2
> Finished: FAILURE
>

build bot on master

2016-10-28 Thread Eugene Koifman 
Hi,
all test runs are failing with the error below.   Does anyone know how to fix 
it?

Thanks,
Eugene

from 
https://builds.apache.org/view/H-L/view/Hive/job/PreCommit-HIVE-Build/1866/console




Build/hive/build/hive/testutils/ptest2/target/lib/guice-assistedinject-3.0.jar
[INFO] Copying velocity-1.7.jar to 
/home/jenkins/jenkins-slave/workspace/PreCommit-HIVE-Build/hive/build/hive/testutils/ptest2/target/lib/velocity-1.7.jar
[INFO] 
[INFO] BUILD SUCCESS
[INFO] 
[INFO] Total time: 10.833 s
[INFO] Finished at: 2016-10-28T15:57:26+00:00
[INFO] Final Memory: 29M/723M
[INFO] 
+ local 
'PTEST_CLASSPATH=/home/jenkins/jenkins-slave/workspace/PreCommit-HIVE-Build/hive/build/hive/testutils/ptest2/target/hive-ptest-1.0-classes.jar:/home/jenkins/jenkins-slave/workspace/PreCommit-HIVE-Build/hive/build/hive/testutils/ptest2/target/lib/*'
+ java -cp 
'/home/jenkins/jenkins-slave/workspace/PreCommit-HIVE-Build/hive/build/hive/testutils/ptest2/target/hive-ptest-1.0-classes.jar:/home/jenkins/jenkins-slave/workspace/PreCommit-HIVE-Build/hive/build/hive/testutils/ptest2/target/lib/*'
 org.apache.hive.ptest.api.client.PTestClient --command testStart --outputDir 
/home/jenkins/jenkins-slave/workspace/PreCommit-HIVE-Build/hive/build/hive/testutils/ptest2/target
 --password '' --testHandle PreCommit-HIVE-Build-1866 --endpoint 
http://104.198.109.242:8080/hive-ptest-1.0 --logsEndpoint 
http://104.198.109.242/logs/ --profile master-mr2 --patch 
https://issues.apache.org/jira/secure/attachment/12835837/HIVE-15093.1.patch 
--jira HIVE-15093


Logs are located: http://104.198.109.242/logs/PreCommit-HIVE-Build-1866


Exception in thread "main" java.lang.RuntimeException: 404 Not Found
at 
org.apache.hive.ptest.api.client.PTestClient.downloadTestResults(PTestClient.java:189)
at 
org.apache.hive.ptest.api.client.PTestClient.testStart(PTestClient.java:137)
at 
org.apache.hive.ptest.api.client.PTestClient.main(PTestClient.java:320)
+ ret=1
+ unpack_test_results
+ '[' -z /home/jenkins/jenkins-slave/workspace/PreCommit-HIVE-Build/hive/build 
']'
+ cd 
/home/jenkins/jenkins-slave/workspace/PreCommit-HIVE-Build/hive/build/hive/testutils/ptest2/target
+ [[ -f test-results.tar.gz ]]
+ exit 1
+ rm -f /tmp/tmp.r59JTiDlN1
Build step 'Execute shell' marked build as failure
Recording test results
ERROR: Step ?Publish JUnit test result report? failed: Test reports were found 
but none of them are new. Did tests run?
For example, 
/home/jenkins/jenkins-slave/workspace/PreCommit-HIVE-Build/hive/build/test-results/TEST-1-TestCliDriver-udf_upper.q-ctas_date.q-schema_evol_orc_acidvec_table_update.q-and-27-more-TEST-org.apache.hadoop.hive.cli.TestCliDriver.xml
 is 2 days 19 hr old

[description-setter] Description 
set:
 HIVE-15093  /   master-mr2
Finished: FAILURE

[jira] [Created] (HIVE-15093) For S3-to-S3 renames, files should be moved individually rather than at a directory level

2016-10-28 Thread Sahil Takiar (JIRA) 
Sahil Takiar created HIVE-15093:
---

 Summary: For S3-to-S3 renames, files should be moved individually 
rather than at a directory level
 Key: HIVE-15093
 URL: https://issues.apache.org/jira/browse/HIVE-15093
 Project: Hive
  Issue Type: Sub-task
  Components: Hive
Affects Versions: 2.1.0
Reporter: Sahil Takiar
Assignee: Sahil Takiar


Hive's MoveTask uses the Hive.moveFile method to move data within a distributed 
filesystem as well as blobstore filesystems.

If the move is done within the same filesystem:

1: If the source path is a subdirectory of the destination path, files will be 
moved one by one using a threapool of workers

2: If the source path is not a subdirectory of the destination path, a single 
rename operation is used to move the entire directory

The second option may not work well on blobstores such as S3. Renames are not 
metadata operations and require copying all the data. Client connectors to 
blobstores may not efficiently rename directories. Worst case, the connector 
will copy each file one by one, sequentially rather than using a threadpool of 
workers to copy the data (e.g. HADOOP-13600).

Hive already has code to rename files using a threadpool of workers, but this 
only occurs in case number 1.

This JIRA aims to modify the code so that case 1 is triggered when copying 
within a blobstore. The focus is on copies within a blobstore because 
needToCopy will return true if the src and target filesystems are different, in 
which case a different code path is triggered.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 53166: HIVE-14910: Flaky test: TestSparkClient.testJobSubmission

2016-10-28 Thread Xuefu Zhang 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/53166/#review154132
---




spark-client/src/main/java/org/apache/hive/spark/client/JobHandleImpl.java 
(line 165)


Nit: could we avoid using single letter variable name for things other than 
integer iterators? Same below.


- Xuefu Zhang


On Oct. 26, 2016, 1:48 p.m., Barna Zsombor Klara wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/53166/
> ---
> 
> (Updated Oct. 26, 2016, 1:48 p.m.)
> 
> 
> Review request for hive, Mohit Sabharwal, Siddharth Seth, and Xuefu Zhang.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-14910: Flaky test: TestSparkClient.testJobSubmission
> I ran into this problem today while investigating a flaky test. I think the 
> failure is coming from this race condition: the listener can be added to the 
> JobHandle only after the job has been submitted. So there is no guarantee 
> that every method of the listener will be invoked, some state changes may 
> have happened before the caller received the handler back.
> I propose a slight change in the API. We should add the listeners as an 
> argument of the submit method, so we can set them on the Handler before the 
> job itself is submitted. This way any status change should be signalled to 
> the listener.
> 
> 
> Diffs
> -
> 
>   spark-client/src/main/java/org/apache/hive/spark/client/JobHandle.java 
> 44aa255a8271894ed3e787c3e7d1323628db63c4 
>   spark-client/src/main/java/org/apache/hive/spark/client/JobHandleImpl.java 
> 17c8f40edd472682d5604f41980d06e60cc92893 
>   spark-client/src/main/java/org/apache/hive/spark/client/SparkClient.java 
> 3e921a5d9b77966d368684ee7b6f1c861ac60e08 
>   
> spark-client/src/main/java/org/apache/hive/spark/client/SparkClientImpl.java 
> e2a30a76e0f7fe95d8a453f502311baa08abcbe2 
>   spark-client/src/test/java/org/apache/hive/spark/client/TestJobHandle.java 
> e8f352dce9f618573c2d79e9b8c59e19fad7298a 
>   
> spark-client/src/test/java/org/apache/hive/spark/client/TestSparkClient.java 
> b95cd7a05d44224b53bf2cef9170146b8b2eb4a8 
> 
> Diff: https://reviews.apache.org/r/53166/diff/
> 
> 
> Testing
> ---
> 
> Unit tests modified and tested.
> I also ran a simple query with HoS as the execution engine.
> 
> 
> Thanks,
> 
> Barna Zsombor Klara
> 
>

Re: Review Request 53261: HIVE-15090 Temporary DB failure can stop ExpiredTokenRemover thread

2016-10-28 Thread Aihua Xu 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/53261/#review154126
---


Ship it!




Ship It!

- Aihua Xu


On Oct. 28, 2016, 2:31 p.m., Peter Vary wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/53261/
> ---
> 
> (Updated Oct. 28, 2016, 2:31 p.m.)
> 
> 
> Review request for hive, Aihua Xu, Piotr Wikie?, and Thejas Nair.
> 
> 
> Bugs: HIVE-15090
> https://issues.apache.org/jira/browse/HIVE-15090
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Moved the try/catch into the while cycle, so the thread will keep running 
> after the exception.
> Added an extra sleep when there is an exception so we do not end up busy 
> waiting for the solution of the problem which caused the exception
> 
> 
> Diffs
> -
> 
>   
> shims/common/src/main/java/org/apache/hadoop/hive/thrift/TokenStoreDelegationTokenSecretManager.java
>  abe8cc2 
> 
> Diff: https://reviews.apache.org/r/53261/diff/
> 
> 
> Testing
> ---
> 
> I have tested with the following configuration:
> 
> 
>   
>   hive.cluster.delegation.token.store.class
>   org.apache.hadoop.hive.thrift.DBTokenStore
> 
> 
>
>hive.cluster.delegation.token.gc-interval
>10
> 
> 
> Started the Metastore, and after a while I have stopped the database. When 
> the database was turned off the metastore logged the ExpiredTokenRemover 
> thread received unexpected exception message, after the database restarted 
> the messages are stopped, and everything become normal. In the debugger I 
> have verified the thread is running.
> 
> 
> Thanks,
> 
> Peter Vary
> 
>

Re: Review Request 53261: HIVE-15090 Temporary DB failure can stop ExpiredTokenRemover thread

2016-10-28 Thread Peter Vary 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/53261/
---

(Updated Oct. 28, 2016, 2:31 p.m.)


Review request for hive, Aihua Xu, Piotr Wikie?, and Thejas Nair.


Changes
---

Spelling issues pointed out by Zsombor, and fixed


Bugs: HIVE-15090
https://issues.apache.org/jira/browse/HIVE-15090


Repository: hive-git


Description
---

Moved the try/catch into the while cycle, so the thread will keep running after 
the exception.
Added an extra sleep when there is an exception so we do not end up busy 
waiting for the solution of the problem which caused the exception


Diffs (updated)
-

  
shims/common/src/main/java/org/apache/hadoop/hive/thrift/TokenStoreDelegationTokenSecretManager.java
 abe8cc2 

Diff: https://reviews.apache.org/r/53261/diff/


Testing
---

I have tested with the following configuration:


  
  hive.cluster.delegation.token.store.class
  org.apache.hadoop.hive.thrift.DBTokenStore


   
   hive.cluster.delegation.token.gc-interval
   10


Started the Metastore, and after a while I have stopped the database. When the 
database was turned off the metastore logged the ExpiredTokenRemover thread 
received unexpected exception message, after the database restarted the 
messages are stopped, and everything become normal. In the debugger I have 
verified the thread is running.


Thanks,

Peter Vary

Re: Review Request 53261: HIVE-15090 Temporary DB failure can stop ExpiredTokenRemover thread

2016-10-28 Thread Barna Zsombor Klara 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/53261/#review154125
---


Ship it!




LGTM.
Thanks for the patch.

- Barna Zsombor Klara


On Oct. 28, 2016, 1:56 p.m., Peter Vary wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/53261/
> ---
> 
> (Updated Oct. 28, 2016, 1:56 p.m.)
> 
> 
> Review request for hive, Aihua Xu, Piotr Wikie?, and Thejas Nair.
> 
> 
> Bugs: HIVE-15090
> https://issues.apache.org/jira/browse/HIVE-15090
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Moved the try/catch into the while cycle, so the thread will keep running 
> after the exception.
> Added an extra sleep when there is an exception so we do not end up busy 
> waiting for the solution of the problem which caused the exception
> 
> 
> Diffs
> -
> 
>   
> shims/common/src/main/java/org/apache/hadoop/hive/thrift/TokenStoreDelegationTokenSecretManager.java
>  abe8cc2 
> 
> Diff: https://reviews.apache.org/r/53261/diff/
> 
> 
> Testing
> ---
> 
> I have tested with the following configuration:
> 
> 
>   
>   hive.cluster.delegation.token.store.class
>   org.apache.hadoop.hive.thrift.DBTokenStore
> 
> 
>
>hive.cluster.delegation.token.gc-interval
>10
> 
> 
> Started the Metastore, and after a while I have stopped the database. When 
> the database was turned off the metastore logged the ExpiredTokenRemover 
> thread received unexpected exception message, after the database restarted 
> the messages are stopped, and everything become normal. In the debugger I 
> have verified the thread is running.
> 
> 
> Thanks,
> 
> Peter Vary
> 
>

Review Request 53261: HIVE-15090 Temporary DB failure can stop ExpiredTokenRemover thread

2016-10-28 Thread Peter Vary 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/53261/
---

Review request for hive, Aihua Xu, Piotr Wikie?, and Thejas Nair.


Bugs: HIVE-15090
https://issues.apache.org/jira/browse/HIVE-15090


Repository: hive-git


Description
---

Moved the try/catch into the while cycle, so the thread will keep running after 
the exception.
Added an extra sleep when there is an exception so we do not end up busy 
waiting for the solution of the problem which caused the exception


Diffs
-

  
shims/common/src/main/java/org/apache/hadoop/hive/thrift/TokenStoreDelegationTokenSecretManager.java
 abe8cc2 

Diff: https://reviews.apache.org/r/53261/diff/


Testing
---

I have tested with the following configuration:


  
  hive.cluster.delegation.token.store.class
  org.apache.hadoop.hive.thrift.DBTokenStore


   
   hive.cluster.delegation.token.gc-interval
   10


Started the Metastore, and after a while I have stopped the database. When the 
database was turned off the metastore logged the ExpiredTokenRemover thread 
received unexpected exception message, after the database restarted the 
messages are stopped, and everything become normal. In the debugger I have 
verified the thread is running.


Thanks,

Peter Vary

[jira] [Created] (HIVE-15092) WebHCat doesn't support RM HA

2016-10-28 Thread Alexandre Linte (JIRA) 
Alexandre Linte created HIVE-15092:
--

 Summary: WebHCat doesn't support RM HA
 Key: HIVE-15092
 URL: https://issues.apache.org/jira/browse/HIVE-15092
 Project: Hive
  Issue Type: Bug
  Components: WebHCat
Affects Versions: 2.1.0
 Environment: Hadoop 2.7.2, Hive 2.1.0
Reporter: Alexandre Linte


We have two resource managers in our environment. For testing purpose, the 
first resourcemanager has been turned off. In that case, launching a job 
through webhcat will not work. The webhcat server returns the stacktrace below:
{noformat}
Oct 27 14:32:19 webhcat01.bigdata.fr webhcat INFO - 
org.apache.hadoop.io.retry.RetryInvocationHandlerException while invoking 
getDelegationToken of class ApplicationClientProtocolPBClientImpl over rm1 
after 8 fail over attempts. Trying to fail over after sleeping for 23697ms.
java.net.ConnectException: Call From webhcat01.bigdata.fr/192.168.200.214 to 
resourcemanager01.bigdata.fr:8032 failed on connection exception: 
java.net.ConnectException: Connection refused; For more details see:  
http://wiki.apache.org/hadoop/ConnectionRefused
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native 
Method) ~[?:1.7.0_101]
at 
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
 ~[?:1.7.0_101]
at 
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
 ~[?:1.7.0_101]
at java.lang.reflect.Constructor.newInstance(Constructor.java:526) 
~[?:1.7.0_101]
at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:792) 
~[hadoop-common-2.7.2.jar:?]
at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:732) 
~[hadoop-common-2.7.2.jar:?]
at org.apache.hadoop.ipc.Client.call(Client.java:1479) 
~[hadoop-common-2.7.2.jar:?]
at org.apache.hadoop.ipc.Client.call(Client.java:1412) 
~[hadoop-common-2.7.2.jar:?]
at 
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229)
 ~[hadoop-common-2.7.2.jar:?]
at com.sun.proxy.$Proxy79.getDelegationToken(Unknown Source) ~[?:?]
at 
org.apache.hadoop.yarn.api.impl.pb.client.ApplicationClientProtocolPBClientImpl.getDelegationToken(ApplicationClientProtocolPBClientImpl.java:310)
 ~[hadoop-yarn-common-2.7.2.jar:?]
at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) ~[?:?]
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 ~[?:1.7.0_101]
at java.lang.reflect.Method.invoke(Method.java:606) ~[?:1.7.0_101]
at 
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:191)
 ~[hadoop-common-2.7.2.jar:?]
at 
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
 [hadoop-common-2.7.2.jar:?]
at com.sun.proxy.$Proxy80.getDelegationToken(Unknown Source) [?:?]
at 
org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getRMDelegationToken(YarnClientImpl.java:513)
 [hadoop-yarn-client-2.7.2.jar:?]
at 
org.apache.hadoop.mapred.ResourceMgrDelegate.getDelegationToken(ResourceMgrDelegate.java:175)
 [spark-assembly-1.4.1-hadoop2.7.1.jar:?]
at 
org.apache.hadoop.mapred.YARNRunner.getDelegationToken(YARNRunner.java:221) 
[spark-assembly-1.4.1-hadoop2.7.1.jar:?]
at 
org.apache.hadoop.mapreduce.Cluster.getDelegationToken(Cluster.java:401) 
[hadoop-mapreduce-client-core-2.7.2.jar:?]
at org.apache.hadoop.mapred.JobClient$16.run(JobClient.java:1232) 
[hadoop-mapreduce-client-core-2.7.2.jar:?]
at org.apache.hadoop.mapred.JobClient$16.run(JobClient.java:1229) 
[hadoop-mapreduce-client-core-2.7.2.jar:?]
at java.security.AccessController.doPrivileged(Native Method) 
~[?:1.7.0_101]
at javax.security.auth.Subject.doAs(Subject.java:415) [?:1.7.0_101]
at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
 [hadoop-common-2.7.2.jar:?]
at 
org.apache.hadoop.mapred.JobClient.getDelegationToken(JobClient.java:1228) 
[hadoop-mapreduce-client-core-2.7.2.jar:?]
at 
org.apache.hive.hcatalog.templeton.tool.TempletonControllerJob.run(TempletonControllerJob.java:137)
 [hive-webhcat-2.1.0.jar:2.1.0]
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) 
[hadoop-common-2.7.2.jar:?]
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84) 
[hadoop-common-2.7.2.jar:?]
at 
org.apache.hive.hcatalog.templeton.LauncherDelegator$1.run(LauncherDelegator.java:111)
 [hive-webhcat-2.1.0.jar:2.1.0]
at 
org.apache.hive.hcatalog.templeton.LauncherDelegator$1.run(LauncherDelegator.java:107)
 [hive-webhcat-2.1.0.jar:2.1.0]
at java.security.AccessController.doPrivileged(Native Method) 
~[?:1.7.0_101]
at

[jira] [Created] (HIVE-15091) Master: Update errata.txt for the missing JIRA number in HIVE-14909 commit msg

2016-10-28 Thread Chaoyu Tang (JIRA) 
Chaoyu Tang created HIVE-15091:
--

 Summary: Master: Update errata.txt for the missing JIRA number in 
HIVE-14909 commit msg
 Key: HIVE-15091
 URL: https://issues.apache.org/jira/browse/HIVE-15091
 Project: Hive
  Issue Type: Bug
  Components: Hive
Affects Versions: 2.2.0
Reporter: Chaoyu Tang
Assignee: Chaoyu Tang
Priority: Trivial


Missing the JIRA number in commit msg for master branch, see 
https://issues.apache.org/jira/browse/HIVE-14909?focusedCommentId=15614056=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15614056



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 53204: HIVE-15076 Improve scalability of LDAP authentication provider group filter

2016-10-28 Thread Illya Yalovyy 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/53204/
---

(Updated Oct. 28, 2016, 12:59 p.m.)


Review request for hive, Aihua Xu, Ashutosh Chauhan, Chaoyu Tang, and Szehon Ho.


Changes
---

1. Removed trailing spaces.


Repository: hive-git


Description
---

HIVE-15076 Improve scalability of LDAP authentication provider group filter

https://issues.apache.org/jira/browse/HIVE-15076


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 5ea9751 
  service/src/java/org/apache/hive/service/auth/ldap/DirSearch.java 33b6088 
  service/src/java/org/apache/hive/service/auth/ldap/GroupFilterFactory.java 
152c4b2 
  service/src/java/org/apache/hive/service/auth/ldap/LdapSearch.java 65076ea 
  service/src/java/org/apache/hive/service/auth/ldap/Query.java b8bf938 
  service/src/java/org/apache/hive/service/auth/ldap/QueryFactory.java e9172d3 
  
service/src/test/org/apache/hive/service/auth/TestLdapAtnProviderWithMiniDS.java
 cd62935 
  
service/src/test/org/apache/hive/service/auth/TestLdapAuthenticationProviderImpl.java
 4fad755 
  
service/src/test/org/apache/hive/service/auth/ldap/LdapAuthenticationTestCase.java
 acde8c1 
  service/src/test/org/apache/hive/service/auth/ldap/TestGroupFilter.java 
0cc2ead 
  service/src/test/org/apache/hive/service/auth/ldap/TestLdapSearch.java 
499b624 
  service/src/test/org/apache/hive/service/auth/ldap/TestQueryFactory.java 
3054e33 
  service/src/test/resources/ldap/ad.example.com.ldif PRE-CREATION 
  service/src/test/resources/ldap/microsoft.schema.ldif PRE-CREATION 

Diff: https://reviews.apache.org/r/53204/diff/


Testing
---

Build succeeded.

Test results:

Tests run: 149, Failures: 0, Errors: 0, Skipped: 0

[INFO] 
[INFO] BUILD SUCCESS
[INFO] 
[INFO] Total time: 03:14 min
[INFO] Finished at: 2016-10-26T13:53:15-07:00
[INFO] Final Memory: 36M/1091M
[INFO] 


Thanks,

Illya Yalovyy

[jira] [Created] (HIVE-15090) Temporary DB failure can stop ExpiredTokenRemover thread

2016-10-28 Thread Peter Vary (JIRA) 
Peter Vary created HIVE-15090:
-

 Summary: Temporary DB failure can stop ExpiredTokenRemover thread
 Key: HIVE-15090
 URL: https://issues.apache.org/jira/browse/HIVE-15090
 Project: Hive
  Issue Type: Bug
  Components: Metastore
Affects Versions: 2.2.0
Reporter: Peter Vary
Assignee: Peter Vary


In HIVE-13090 we decided that we should not close the metastore if there is an 
unexpected exception during the expired token removal process, but that fix 
leaves a running metastore without ExpiredTokenRemover thread.

To fix this I will move the catch inside the running loop, and hope the thread 
could recover from the exception



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 53257: HIVE-14960: Improve the stability of TestNotificationListener

2016-10-28 Thread Barna Zsombor Klara 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/53257/#review154114
---


Ship it!




+1.
Thanks for the patch.


hcatalog/server-extensions/src/test/java/org/apache/hive/hcatalog/listener/TestNotificationListener.java
 (line 112)


nit: I would raise this up the intance level and change the countdownlatch 
constr argument to the size of the list.



hcatalog/server-extensions/src/test/java/org/apache/hive/hcatalog/listener/TestNotificationListener.java
 (line 253)


I would put this into a finally block. I know that the timeout will prevent 
the test from hanging indefinitely, but let's not wait 30s unless necessary.


- Barna Zsombor Klara


On Oct. 28, 2016, 10:08 a.m., Marta Kuczora wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/53257/
> ---
> 
> (Updated Oct. 28, 2016, 10:08 a.m.)
> 
> 
> Review request for hive, Aihua Xu, Chaoyu Tang, Peter Vary, and Barna Zsombor 
> Klara.
> 
> 
> Bugs: HIVE-14960
> https://issues.apache.org/jira/browse/HIVE-14960
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> The TestNotificationListener test fails occasionally. It happens if the 
> testAMQListener method is completed and the recevied messages are checked 
> before the last "DROP_TABLE" message got processed and put to the 
> "actualMessages" list by the onMessage method. 
> As a solution I used a CountDownLatch which count is decreased by 1 when a 
> message is processed. And the "testAMQListener" method will wait for the 
> Latch to reach zero or a maximum time limit before complete.
> 
> 
> Diffs
> -
> 
>   
> hcatalog/server-extensions/src/test/java/org/apache/hive/hcatalog/listener/TestNotificationListener.java
>  9e03da4 
> 
> Diff: https://reviews.apache.org/r/53257/diff/
> 
> 
> Testing
> ---
> 
> The change effects only a unit test.
> Ran the test many times locally.
> Added random sleeps to simulate the delay of the message processing.
> 
> 
> Thanks,
> 
> Marta Kuczora
> 
>

Review Request 53257: HIVE-14960: Improve the stability of TestNotificationListener

2016-10-28 Thread Marta Kuczora 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/53257/
---

Review request for hive, Aihua Xu, Chaoyu Tang, Peter Vary, and Barna Zsombor 
Klara.


Bugs: HIVE-14960
https://issues.apache.org/jira/browse/HIVE-14960


Repository: hive-git


Description
---

The TestNotificationListener test fails occasionally. It happens if the 
testAMQListener method is completed and the recevied messages are checked 
before the last "DROP_TABLE" message got processed and put to the 
"actualMessages" list by the onMessage method. 
As a solution I used a CountDownLatch which count is decreased by 1 when a 
message is processed. And the "testAMQListener" method will wait for the Latch 
to reach zero or a maximum time limit before complete.


Diffs
-

  
hcatalog/server-extensions/src/test/java/org/apache/hive/hcatalog/listener/TestNotificationListener.java
 9e03da4 

Diff: https://reviews.apache.org/r/53257/diff/


Testing
---

The change effects only a unit test.
Ran the test many times locally.
Added random sleeps to simulate the delay of the message processing.


Thanks,

Marta Kuczora

Re: Review Request 52923: HIVE-14979 Removing stale Zookeeper locks at HiveServer2 initialization

2016-10-28 Thread Lefty Leverenz 


> On Oct. 20, 2016, 7:19 a.m., Lefty Leverenz wrote:
> > common/src/java/org/apache/hadoop/hive/conf/HiveConf.java, lines 1766-1772
> > 
> >
> > Looks good, thanks.  Just one more nit (on 2 lines):  Zookeeper -> 
> > ZooKeeper.
> > 
> > See https://zookeeper.apache.org if you don't believe me.  ;)
> 
> Peter Vary wrote:
> Thanks, this escaped my attention. Sorry!

+1  Looks good now, thanks Peter.


- Lefty


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/52923/#review153357
---


On Oct. 26, 2016, 3:24 p.m., Peter Vary wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/52923/
> ---
> 
> (Updated Oct. 26, 2016, 3:24 p.m.)
> 
> 
> Review request for hive, Ashutosh Chauhan, Marta Kuczora, Miklos Csanady, 
> namit jain, Sergio Pena, and Barna Zsombor Klara.
> 
> 
> Bugs: HIVE-14979
> https://issues.apache.org/jira/browse/HIVE-14979
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Adding a new configuration option to HiveConf to signal whether stale lock 
> removal is requested on startup.
> Adding a new method to ZooKeeperHiveLockManager to remove stale locks
> Modifying the HiveServer2 to instantiate a lock manager and call the new 
> method if defined by the configuration.
> 
> Please take extra care when reviewing these:
> - Modifying the lock fetching method to use the clientIp from the lock, and 
> not update with the current ip - Not sure why it was done before
> - Instantiation of the lock manager - I might not chose the best method for it
> 
> Open for any suggestions :)
> 
> Thanks,
> Peter
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 8ffae3b 
>   
> ql/src/java/org/apache/hadoop/hive/ql/lockmgr/zookeeper/ZooKeeperHiveLockManager.java
>  14d0ef4 
>   
> ql/src/test/org/apache/hadoop/hive/ql/lockmgr/zookeeper/TestZookeeperLockManager.java
>  3f9926e 
>   service/src/java/org/apache/hive/service/server/HiveServer2.java 590b1f3 
> 
> Diff: https://reviews.apache.org/r/52923/diff/
> 
> 
> Testing
> ---
> 
> Created 2 unit test cases:
> - Removing own locks
> - Not removing other server's locks
> 
> Manually tested the Lock manager instantiation method on HiveServer2
> 
> 
> Thanks,
> 
> Peter Vary
> 
>