[jira] [Assigned] (HIVE-11225) Running all Hive UTs or itests executes only small subset of tests
[
https://issues.apache.org/jira/browse/HIVE-11225?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ferdinand Xu reassigned HIVE-11225:
---
Assignee: Ferdinand Xu
Running all Hive UTs or itests executes only small subset of tests
--
Key: HIVE-11225
URL: https://issues.apache.org/jira/browse/HIVE-11225
Project: Hive
Issue Type: Bug
Components: Tests
Reporter: Thomas Friedrich
Assignee: Ferdinand Xu
Trying to run all unit tests runs only a small sub-set of the tests:
For example:
mvn test -Phadoop-2
only ran 272 tests.
Others ran into similar issues when running the UTs from a particular package
like ql.
This is potentially related to HIVE-10941 where an additional option to skip
the Spark UTs was added. When I take out
exclude%regex[${skip.spark.files}]/exclude
from the exclude list of tests, all tests are executed again. This is not a
fix though, instead the spark-test profile should be fixed.
I see the problem in both the 1.2 branch as well as master.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-11225) Running all Hive UTs or itests executes only small subset of tests
[
https://issues.apache.org/jira/browse/HIVE-11225?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ferdinand Xu updated HIVE-11225:
Attachment: HIVE-11225.patch
The regexp is not correct. It will skip lots of test.class. [~hsubramaniyan],
could you review this patch?
Running all Hive UTs or itests executes only small subset of tests
--
Key: HIVE-11225
URL: https://issues.apache.org/jira/browse/HIVE-11225
Project: Hive
Issue Type: Bug
Components: Tests
Reporter: Thomas Friedrich
Assignee: Ferdinand Xu
Attachments: HIVE-11225.patch
Trying to run all unit tests runs only a small sub-set of the tests:
For example:
mvn test -Phadoop-2
only ran 272 tests.
Others ran into similar issues when running the UTs from a particular package
like ql.
This is potentially related to HIVE-10941 where an additional option to skip
the Spark UTs was added. When I take out
exclude%regex[${skip.spark.files}]/exclude
from the exclude list of tests, all tests are executed again. This is not a
fix though, instead the spark-test profile should be fixed.
I see the problem in both the 1.2 branch as well as master.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-11225) Running all Hive UTs or itests executes only small subset of tests
[
https://issues.apache.org/jira/browse/HIVE-11225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14621817#comment-14621817
]
Hari Sankar Sivarama Subramaniyan commented on HIVE-11225:
--
+1 pending tests. Can you please set the status to Patch available so that the
precommit tests kick in.
Thanks
Hari
Running all Hive UTs or itests executes only small subset of tests
--
Key: HIVE-11225
URL: https://issues.apache.org/jira/browse/HIVE-11225
Project: Hive
Issue Type: Bug
Components: Tests
Reporter: Thomas Friedrich
Assignee: Ferdinand Xu
Attachments: HIVE-11225.patch
Trying to run all unit tests runs only a small sub-set of the tests:
For example:
mvn test -Phadoop-2
only ran 272 tests.
Others ran into similar issues when running the UTs from a particular package
like ql.
This is potentially related to HIVE-10941 where an additional option to skip
the Spark UTs was added. When I take out
exclude%regex[${skip.spark.files}]/exclude
from the exclude list of tests, all tests are executed again. This is not a
fix though, instead the spark-test profile should be fixed.
I see the problem in both the 1.2 branch as well as master.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-11223) CBO (Calcite Return Path): MapJoin and SMBJoin conversion not triggered
[
https://issues.apache.org/jira/browse/HIVE-11223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14622001#comment-14622001
]
Hive QA commented on HIVE-11223:
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12744615/HIVE-11223.patch
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 9149 tests executed
*Failed tests:*
{noformat}
org.apache.hive.spark.client.TestSparkClient.testRemoteClient
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/4564/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/4564/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-4564/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12744615 - PreCommit-HIVE-TRUNK-Build
CBO (Calcite Return Path): MapJoin and SMBJoin conversion not triggered
---
Key: HIVE-11223
URL: https://issues.apache.org/jira/browse/HIVE-11223
Project: Hive
Issue Type: Sub-task
Components: CBO
Reporter: Jesus Camacho Rodriguez
Assignee: Jesus Camacho Rodriguez
Attachments: HIVE-11223.patch
Information in aux data structures is not complete, thus MapJoin and SMBJoin
conversion are not triggered.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-10937) LLAP: make ObjectCache for plans work properly in the daemon
[
https://issues.apache.org/jira/browse/HIVE-10937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14622004#comment-14622004
]
Hive QA commented on HIVE-10937:
{color:red}Overall{color}: -1 no tests executed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12743826/HIVE-10937.02.patch
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/4565/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/4565/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-4565/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hive-ptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ [[ -n /usr/java/jdk1.7.0_45-cloudera ]]
+ export JAVA_HOME=/usr/java/jdk1.7.0_45-cloudera
+ JAVA_HOME=/usr/java/jdk1.7.0_45-cloudera
+ export
PATH=/usr/java/jdk1.7.0_45-cloudera/bin/:/usr/local/apache-maven-3.0.5/bin:/usr/java/jdk1.7.0_45-cloudera/bin:/usr/local/apache-ant-1.9.1/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hiveptest/bin
+
PATH=/usr/java/jdk1.7.0_45-cloudera/bin/:/usr/local/apache-maven-3.0.5/bin:/usr/java/jdk1.7.0_45-cloudera/bin:/usr/local/apache-ant-1.9.1/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hiveptest/bin
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'M2_OPTS=-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost
-Dhttp.proxyPort=3128'
+ M2_OPTS='-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost
-Dhttp.proxyPort=3128'
+ cd /data/hive-ptest/working/
+ tee /data/hive-ptest/logs/PreCommit-HIVE-TRUNK-Build-4565/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ cd apache-github-source-source
+ git fetch origin
From https://github.com/apache/hive
e9b6863..2d49e5a branch-1 - origin/branch-1
d89a7d1..b61e6b5 master - origin/master
+ git reset --hard HEAD
HEAD is now at d89a7d1 HIVE-11170 : port parts of HIVE-11015 to master for ease
of future merging (Sergey Shelukhin, reviewed by Vikram Dixit K)
+ git clean -f -d
+ git checkout master
Already on 'master'
Your branch is behind 'origin/master' by 1 commit, and can be fast-forwarded.
+ git reset --hard origin/master
HEAD is now at b61e6b5 HIVE-11221: In Tez mode, alter table concatenate orc
files can intermittently fail with NPE (Prasanth Jayachandran reviewed by
Vikram Dixit)
+ git merge --ff-only origin/master
Already up-to-date.
+ git gc
+ patchCommandPath=/data/hive-ptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hive-ptest/working/scratch/build.patch
+ [[ -f /data/hive-ptest/working/scratch/build.patch ]]
+ chmod +x /data/hive-ptest/working/scratch/smart-apply-patch.sh
+ /data/hive-ptest/working/scratch/smart-apply-patch.sh
/data/hive-ptest/working/scratch/build.patch
The patch does not appear to apply with p0, p1, or p2
+ exit 1
'
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12743826 - PreCommit-HIVE-TRUNK-Build
LLAP: make ObjectCache for plans work properly in the daemon
Key: HIVE-10937
URL: https://issues.apache.org/jira/browse/HIVE-10937
Project: Hive
Issue Type: Sub-task
Reporter: Sergey Shelukhin
Assignee: Sergey Shelukhin
Fix For: llap
Attachments: HIVE-10937.01.patch, HIVE-10937.02.patch,
HIVE-10937.patch
There's perf hit otherwise, esp. when stupid planner creates 1009 reducers of
4Mb each.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-11200) LLAP: Cache BuddyAllocator throws NPE
[
https://issues.apache.org/jira/browse/HIVE-11200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14622009#comment-14622009
]
Hive QA commented on HIVE-11200:
{color:red}Overall{color}: -1 no tests executed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12744372/HIVE-11200.patch
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/4567/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/4567/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-4567/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hive-ptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ [[ -n /usr/java/jdk1.7.0_45-cloudera ]]
+ export JAVA_HOME=/usr/java/jdk1.7.0_45-cloudera
+ JAVA_HOME=/usr/java/jdk1.7.0_45-cloudera
+ export
PATH=/usr/java/jdk1.7.0_45-cloudera/bin/:/usr/local/apache-maven-3.0.5/bin:/usr/java/jdk1.7.0_45-cloudera/bin:/usr/local/apache-ant-1.9.1/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hiveptest/bin
+
PATH=/usr/java/jdk1.7.0_45-cloudera/bin/:/usr/local/apache-maven-3.0.5/bin:/usr/java/jdk1.7.0_45-cloudera/bin:/usr/local/apache-ant-1.9.1/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hiveptest/bin
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'M2_OPTS=-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost
-Dhttp.proxyPort=3128'
+ M2_OPTS='-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost
-Dhttp.proxyPort=3128'
+ cd /data/hive-ptest/working/
+ tee /data/hive-ptest/logs/PreCommit-HIVE-TRUNK-Build-4567/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at b61e6b5 HIVE-11221: In Tez mode, alter table concatenate orc
files can intermittently fail with NPE (Prasanth Jayachandran reviewed by
Vikram Dixit)
+ git clean -f -d
+ git checkout master
Already on 'master'
+ git reset --hard origin/master
HEAD is now at b61e6b5 HIVE-11221: In Tez mode, alter table concatenate orc
files can intermittently fail with NPE (Prasanth Jayachandran reviewed by
Vikram Dixit)
+ git merge --ff-only origin/master
Already up-to-date.
+ git gc
+ patchCommandPath=/data/hive-ptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hive-ptest/working/scratch/build.patch
+ [[ -f /data/hive-ptest/working/scratch/build.patch ]]
+ chmod +x /data/hive-ptest/working/scratch/smart-apply-patch.sh
+ /data/hive-ptest/working/scratch/smart-apply-patch.sh
/data/hive-ptest/working/scratch/build.patch
The patch does not appear to apply with p0, p1, or p2
+ exit 1
'
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12744372 - PreCommit-HIVE-TRUNK-Build
LLAP: Cache BuddyAllocator throws NPE
-
Key: HIVE-11200
URL: https://issues.apache.org/jira/browse/HIVE-11200
Project: Hive
Issue Type: Sub-task
Affects Versions: llap
Environment: large perf cluster - with 64Gb cache sizes
Reporter: Gopal V
Assignee: Sergey Shelukhin
Priority: Minor
Fix For: llap
Attachments: HIVE-11200.patch
Built off da1e0cf21aeff0a9501c5e220a6f66ba61f6da94 merge point
{code}
Caused by: java.lang.NullPointerException
at
org.apache.hadoop.hive.llap.cache.BuddyAllocator$Arena.allocateWithSplit(BuddyAllocator.java:331)
at
org.apache.hadoop.hive.llap.cache.BuddyAllocator$Arena.allocateWithExpand(BuddyAllocator.java:399)
at
org.apache.hadoop.hive.llap.cache.BuddyAllocator$Arena.access$300(BuddyAllocator.java:228)
at
org.apache.hadoop.hive.llap.cache.BuddyAllocator.allocateMultiple(BuddyAllocator.java:156)
at
org.apache.hadoop.hive.ql.io.orc.InStream.readEncodedStream(InStream.java:761)
at
org.apache.hadoop.hive.ql.io.orc.EncodedReaderImpl.readEncodedColumns(EncodedReaderImpl.java:462)
at
org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader.callInternal(OrcEncodedDataReader.java:342)
at
org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader.callInternal(OrcEncodedDataReader.java:59)
at
org.apache.hadoop.hive.common.CallableWithNdc.call(CallableWithNdc.java:37)
... 4 more
2015-07-08 01:17:42,798
[jira] [Commented] (HIVE-11222) LLAP: occasional NPE in parallel queries in ORC reader
[
https://issues.apache.org/jira/browse/HIVE-11222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14622007#comment-14622007
]
Hive QA commented on HIVE-11222:
{color:red}Overall{color}: -1 no tests executed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12744608/HIVE-11222.patch
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/4566/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/4566/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-4566/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hive-ptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ [[ -n /usr/java/jdk1.7.0_45-cloudera ]]
+ export JAVA_HOME=/usr/java/jdk1.7.0_45-cloudera
+ JAVA_HOME=/usr/java/jdk1.7.0_45-cloudera
+ export
PATH=/usr/java/jdk1.7.0_45-cloudera/bin/:/usr/local/apache-maven-3.0.5/bin:/usr/java/jdk1.7.0_45-cloudera/bin:/usr/local/apache-ant-1.9.1/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hiveptest/bin
+
PATH=/usr/java/jdk1.7.0_45-cloudera/bin/:/usr/local/apache-maven-3.0.5/bin:/usr/java/jdk1.7.0_45-cloudera/bin:/usr/local/apache-ant-1.9.1/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hiveptest/bin
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'M2_OPTS=-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost
-Dhttp.proxyPort=3128'
+ M2_OPTS='-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost
-Dhttp.proxyPort=3128'
+ cd /data/hive-ptest/working/
+ tee /data/hive-ptest/logs/PreCommit-HIVE-TRUNK-Build-4566/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at b61e6b5 HIVE-11221: In Tez mode, alter table concatenate orc
files can intermittently fail with NPE (Prasanth Jayachandran reviewed by
Vikram Dixit)
+ git clean -f -d
+ git checkout master
Already on 'master'
+ git reset --hard origin/master
HEAD is now at b61e6b5 HIVE-11221: In Tez mode, alter table concatenate orc
files can intermittently fail with NPE (Prasanth Jayachandran reviewed by
Vikram Dixit)
+ git merge --ff-only origin/master
Already up-to-date.
+ git gc
+ patchCommandPath=/data/hive-ptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hive-ptest/working/scratch/build.patch
+ [[ -f /data/hive-ptest/working/scratch/build.patch ]]
+ chmod +x /data/hive-ptest/working/scratch/smart-apply-patch.sh
+ /data/hive-ptest/working/scratch/smart-apply-patch.sh
/data/hive-ptest/working/scratch/build.patch
The patch does not appear to apply with p0, p1, or p2
+ exit 1
'
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12744608 - PreCommit-HIVE-TRUNK-Build
LLAP: occasional NPE in parallel queries in ORC reader
--
Key: HIVE-11222
URL: https://issues.apache.org/jira/browse/HIVE-11222
Project: Hive
Issue Type: Sub-task
Reporter: Sergey Shelukhin
Assignee: Sergey Shelukhin
Fix For: llap
Attachments: HIVE-11222.patch
{noformat}
Caused by: java.io.IOException: java.lang.NullPointerException
at
org.apache.hadoop.hive.llap.io.api.impl.LlapInputFormat$LlapRecordReader.rethrowErrorIfAny(LlapInputFormat.java:275)
at
org.apache.hadoop.hive.llap.io.api.impl.LlapInputFormat$LlapRecordReader.nextCvb(LlapInputFormat.java:227)
at
org.apache.hadoop.hive.llap.io.api.impl.LlapInputFormat$LlapRecordReader.next(LlapInputFormat.java:155)
at
org.apache.hadoop.hive.llap.io.api.impl.LlapInputFormat$LlapRecordReader.next(LlapInputFormat.java:101)
at
org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.doNext(HiveContextAwareRecordReader.java:350)
... 22 more
Caused by: java.lang.NullPointerException
at
org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl$SargApplier.pickRowGroups(RecordReaderImpl.java:709)
at
org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader.determineRgsToRead(OrcEncodedDataReader.java:618)
at
org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader.callInternal(OrcEncodedDataReader.java:195)
at
org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader.callInternal(OrcEncodedDataReader.java:59)
at
[jira] [Updated] (HIVE-11225) Running all Hive UTs or itests executes only small subset of tests
[
https://issues.apache.org/jira/browse/HIVE-11225?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ferdinand Xu updated HIVE-11225:
Attachment: HIVE-11225.1.patch
Seems the substitution is not working in the regex block. Use the full name in
the new patch.
Running all Hive UTs or itests executes only small subset of tests
--
Key: HIVE-11225
URL: https://issues.apache.org/jira/browse/HIVE-11225
Project: Hive
Issue Type: Bug
Components: Tests
Reporter: Thomas Friedrich
Assignee: Ferdinand Xu
Attachments: HIVE-11225.1.patch, HIVE-11225.patch
Trying to run all unit tests runs only a small sub-set of the tests:
For example:
mvn test -Phadoop-2
only ran 272 tests.
Others ran into similar issues when running the UTs from a particular package
like ql.
This is potentially related to HIVE-10941 where an additional option to skip
the Spark UTs was added. When I take out
exclude%regex[${skip.spark.files}]/exclude
from the exclude list of tests, all tests are executed again. This is not a
fix though, instead the spark-test profile should be fixed.
I see the problem in both the 1.2 branch as well as master.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-11225) Running all Hive UTs or itests executes only small subset of tests
[
https://issues.apache.org/jira/browse/HIVE-11225?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ferdinand Xu updated HIVE-11225:
Attachment: HIVE-11225.1.patch
Running all Hive UTs or itests executes only small subset of tests
--
Key: HIVE-11225
URL: https://issues.apache.org/jira/browse/HIVE-11225
Project: Hive
Issue Type: Bug
Components: Tests
Reporter: Thomas Friedrich
Assignee: Ferdinand Xu
Attachments: HIVE-11225.1.patch, HIVE-11225.patch
Trying to run all unit tests runs only a small sub-set of the tests:
For example:
mvn test -Phadoop-2
only ran 272 tests.
Others ran into similar issues when running the UTs from a particular package
like ql.
This is potentially related to HIVE-10941 where an additional option to skip
the Spark UTs was added. When I take out
exclude%regex[${skip.spark.files}]/exclude
from the exclude list of tests, all tests are executed again. This is not a
fix though, instead the spark-test profile should be fixed.
I see the problem in both the 1.2 branch as well as master.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-11225) Running all Hive UTs or itests executes only small subset of tests
[
https://issues.apache.org/jira/browse/HIVE-11225?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ferdinand Xu updated HIVE-11225:
Attachment: (was: HIVE-11225.1.patch)
Running all Hive UTs or itests executes only small subset of tests
--
Key: HIVE-11225
URL: https://issues.apache.org/jira/browse/HIVE-11225
Project: Hive
Issue Type: Bug
Components: Tests
Reporter: Thomas Friedrich
Assignee: Ferdinand Xu
Attachments: HIVE-11225.1.patch, HIVE-11225.patch
Trying to run all unit tests runs only a small sub-set of the tests:
For example:
mvn test -Phadoop-2
only ran 272 tests.
Others ran into similar issues when running the UTs from a particular package
like ql.
This is potentially related to HIVE-10941 where an additional option to skip
the Spark UTs was added. When I take out
exclude%regex[${skip.spark.files}]/exclude
from the exclude list of tests, all tests are executed again. This is not a
fix though, instead the spark-test profile should be fixed.
I see the problem in both the 1.2 branch as well as master.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-11216) UDF GenericUDFMapKeys throws NPE when a null map value is passed in
[
https://issues.apache.org/jira/browse/HIVE-11216?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14621857#comment-14621857
]
Hive QA commented on HIVE-11216:
{color:green}Overall{color}: +1 all checks pass
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12744588/HIVE-11216.1.patch
{color:green}SUCCESS:{color} +1 9149 tests passed
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/4563/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/4563/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-4563/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12744588 - PreCommit-HIVE-TRUNK-Build
UDF GenericUDFMapKeys throws NPE when a null map value is passed in
---
Key: HIVE-11216
URL: https://issues.apache.org/jira/browse/HIVE-11216
Project: Hive
Issue Type: Bug
Components: UDF
Affects Versions: 1.2.0
Reporter: Yibing Shi
Assignee: Yibing Shi
Attachments: HIVE-11216.1.patch, HIVE-11216.patch
We can reproduce the problem as below:
{noformat}
hive show create table map_txt;
OK
CREATE TABLE `map_txt`(
`id` int,
`content` mapint,string)
ROW FORMAT SERDE
'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe'
STORED AS INPUTFORMAT
'org.apache.hadoop.mapred.TextInputFormat'
OUTPUTFORMAT
'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat'
...
Time taken: 0.233 seconds, Fetched: 18 row(s)
hive select * from map_txt;
OK
1 NULL
Time taken: 0.679 seconds, Fetched: 1 row(s)
hive select id, map_keys(content) from map_txt;
Error during job, obtaining debugging information...
Examining task ID: task_1435534231122_0025_m_00 (and more) from job
job_1435534231122_0025
Task with the most failures(4):
-
Task ID:
task_1435534231122_0025_m_00
URL:
http://host-10-17-80-40.coe.cloudera.com:8088/taskdetails.jsp?jobid=job_1435534231122_0025tipid=task_1435534231122_0025_m_00
-
Diagnostic Messages for this Task:
Error: java.lang.RuntimeException:
org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while
processing row {id:1,content:null}
at
org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:198)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:450)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1642)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime
Error while processing row {id:1,content:null}
at
org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:559)
at
org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:180)
... 8 more
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Error evaluating
map_keys(content)
at
org.apache.hadoop.hive.ql.exec.SelectOperator.processOp(SelectOperator.java:84)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:796)
at
org.apache.hadoop.hive.ql.exec.TableScanOperator.processOp(TableScanOperator.java:92)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:796)
at
org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:549)
... 9 more
Caused by: java.lang.NullPointerException
at
org.apache.hadoop.hive.ql.udf.generic.GenericUDFMapKeys.evaluate(GenericUDFMapKeys.java:64)
at
org.apache.hadoop.hive.ql.exec.ExprNodeGenericFuncEvaluator._evaluate(ExprNodeGenericFuncEvaluator.java:166)
at
org.apache.hadoop.hive.ql.exec.ExprNodeEvaluator.evaluate(ExprNodeEvaluator.java:77)
at
org.apache.hadoop.hive.ql.exec.ExprNodeEvaluator.evaluate(ExprNodeEvaluator.java:65)
at
org.apache.hadoop.hive.ql.exec.SelectOperator.processOp(SelectOperator.java:79)
... 13 more
FAILED: Execution Error, return code 2 from
org.apache.hadoop.hive.ql.exec.mr.MapRedTask
MapReduce Jobs Launched:
[jira] [Updated] (HIVE-11227) Kryo exception during table creation in Hive
[ https://issues.apache.org/jira/browse/HIVE-11227?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akamai updated HIVE-11227: -- Attachment: Kryo Exception.txt error log Kryo exception during table creation in Hive Key: HIVE-11227 URL: https://issues.apache.org/jira/browse/HIVE-11227 Project: Hive Issue Type: Bug Components: Configuration, Database/Schema, Hive, HiveServer2, Indexing, Locking Affects Versions: 0.13.1 Environment: CentOS 6.5, jdk 1.7, cpu: 2x1.9 GHz 6-core Xeon (24 cores), Ram: 64GB-128GB Reporter: Akamai Fix For: 0.14.1 Attachments: Kryo Exception.txt Exception is getting thorwn during table creation in Hive Error: java.lang.RuntimeException: org.apache.hive.com/esotericsoftware.kryo.KryoException: Encountered unregistered class ID: 380 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11190) No prompting info or warning provided when METASTORE_FILTER_HOOK in authorization V2 is overridden
[
https://issues.apache.org/jira/browse/HIVE-11190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14622237#comment-14622237
]
Hive QA commented on HIVE-11190:
{color:green}Overall{color}: +1 all checks pass
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12744639/HIVE-11190.003.patch
{color:green}SUCCESS:{color} +1 9150 tests passed
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/4569/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/4569/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-4569/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12744639 - PreCommit-HIVE-TRUNK-Build
No prompting info or warning provided when METASTORE_FILTER_HOOK in
authorization V2 is overridden
--
Key: HIVE-11190
URL: https://issues.apache.org/jira/browse/HIVE-11190
Project: Hive
Issue Type: Bug
Reporter: Dapeng Sun
Assignee: Dapeng Sun
Attachments: HIVE-11190.001.patch, HIVE-11190.002.patch,
HIVE-11190.003.patch
ConfVars.METASTORE_FILTER_HOOK in authorization V2 is will be override
without prompting info or warning.
it will cause user failed to customize the METASTORE_FILTER_HOOK. We should
log information such as this value is ignored when override happens.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-11189) Add 'IGNORE NULLS' to FIRST_VALUE/LAST_VALUE
[ https://issues.apache.org/jira/browse/HIVE-11189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14622162#comment-14622162 ] ratnakar commented on HIVE-11189: - When used with last_value(col, true), I am getting all values as NULL, its not working correctly. Add 'IGNORE NULLS' to FIRST_VALUE/LAST_VALUE Key: HIVE-11189 URL: https://issues.apache.org/jira/browse/HIVE-11189 Project: Hive Issue Type: Improvement Components: PTF-Windowing Reporter: Prateek Rungta Other RDBMS support the specification of 'IGNORE NULLS' over a paritition to skip NULL values for Analytic Functions. Example - Oracle's docs: http://docs.oracle.com/cd/B19306_01/server.102/b14200/functions057.htm Please consider adding this to Hive. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11224) AggregateStatsCache triggers java.util.ConcurrentModificationException under some conditions
[
https://issues.apache.org/jira/browse/HIVE-11224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14622108#comment-14622108
]
Hive QA commented on HIVE-11224:
{color:green}Overall{color}: +1 all checks pass
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12744635/HIVE-11224.01.patch
{color:green}SUCCESS:{color} +1 9150 tests passed
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/4568/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/4568/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-4568/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12744635 - PreCommit-HIVE-TRUNK-Build
AggregateStatsCache triggers java.util.ConcurrentModificationException under
some conditions
Key: HIVE-11224
URL: https://issues.apache.org/jira/browse/HIVE-11224
Project: Hive
Issue Type: Bug
Reporter: Pengcheng Xiong
Assignee: Pengcheng Xiong
Attachments: HIVE-11224.01.patch
Stack
{code}
java.util.ConcurrentModificationException
at java.util.HashMap$HashIterator.nextEntry(HashMap.java:922)
at java.util.HashMap$EntryIterator.next(HashMap.java:962)
at java.util.HashMap$EntryIterator.next(HashMap.java:960)
at
org.apache.hadoop.hive.metastore.AggregateStatsCache.findBestMatch(AggregateStatsCache.java:244)
at
org.apache.hadoop.hive.metastore.AggregateStatsCache.get(AggregateStatsCache.java:186)
at
org.apache.hadoop.hive.metastore.MetaStoreDirectSql.aggrColStatsForPartitions(MetaStoreDirectSql.java:1131)
at
org.apache.hadoop.hive.metastore.ObjectStore$9.getSqlResult(ObjectStore.java:6174)
at
org.apache.hadoop.hive.metastore.ObjectStore$9.getSqlResult(ObjectStore.java:6170)
at
org.apache.hadoop.hive.metastore.ObjectStore$GetHelper.run(ObjectStore.java:2405)
at
org.apache.hadoop.hive.metastore.ObjectStore.get_aggr_stats_for(ObjectStore.java:6170)
at sun.reflect.GeneratedMethodAccessor103.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:114)
at com.sun.proxy.$Proxy6.get_aggr_stats_for(Unknown Source)
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_aggr_stats_for(HiveMetaStore.java:5707)
at sun.reflect.GeneratedMethodAccessor102.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:107)
at com.sun.proxy.$Proxy8.get_aggr_stats_for(Unknown Source)
at
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getAggrColStatsFor(HiveMetaStoreClient.java:2067)
at sun.reflect.GeneratedMethodAccessor101.invoke(Unknown Source)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-10882) CBO: Calcite Operator To Hive Operator (Calcite Return Path) empty filtersMap of join operator causes wrong results
[ https://issues.apache.org/jira/browse/HIVE-10882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14622176#comment-14622176 ] Jesus Camacho Rodriguez commented on HIVE-10882: Let's wait till HIVE-11197 and HIVE-11206 go in; I'll rebase the patch and we can trigger a new QA run. About your concern, I think what I replied in HIVE-11206 applies here too. I agree with you we shouldn't create duplicate code; the problem in this case is that the filtersMap is generated from the QBJoinTree in SemanticAnalyzer, while it is generated from the predicates when the return path is on. So it is difficult to actually factorize that code as there are not so many common parts. CBO: Calcite Operator To Hive Operator (Calcite Return Path) empty filtersMap of join operator causes wrong results --- Key: HIVE-10882 URL: https://issues.apache.org/jira/browse/HIVE-10882 Project: Hive Issue Type: Sub-task Components: CBO Reporter: Pengcheng Xiong Assignee: Jesus Camacho Rodriguez Attachments: HIVE-10882.01.patch CBO return path creates join operator with empty filtersMap. This causes outer joins to produce wrong results. To reproduce, run louter_join_ppr.q with return path turned on. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11197) While extracting join conditions follow Hive rules for type conversion instead of Calcite
[ https://issues.apache.org/jira/browse/HIVE-11197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14622138#comment-14622138 ] Jesus Camacho Rodriguez commented on HIVE-11197: [~ashutoshc], I checked RB and last version of the patch addresses my concerns about when to bail out of CBO or not. +1 While extracting join conditions follow Hive rules for type conversion instead of Calcite - Key: HIVE-11197 URL: https://issues.apache.org/jira/browse/HIVE-11197 Project: Hive Issue Type: Bug Components: CBO Reporter: Ashutosh Chauhan Assignee: Ashutosh Chauhan Attachments: HIVE-11197.2.patch, HIVE-11197.2.patch, HIVE-11197.3.patch, HIVE-11197.patch, HIVE-11197.patch Calcite strict type system throws exception in those cases, which are legal in Hive. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11222) LLAP: occasional NPE in parallel queries in ORC reader
[
https://issues.apache.org/jira/browse/HIVE-11222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14623061#comment-14623061
]
Sergey Shelukhin commented on HIVE-11222:
-
After the fix, yes
LLAP: occasional NPE in parallel queries in ORC reader
--
Key: HIVE-11222
URL: https://issues.apache.org/jira/browse/HIVE-11222
Project: Hive
Issue Type: Sub-task
Reporter: Sergey Shelukhin
Assignee: Sergey Shelukhin
Fix For: llap
Attachments: HIVE-11222.patch
{noformat}
Caused by: java.io.IOException: java.lang.NullPointerException
at
org.apache.hadoop.hive.llap.io.api.impl.LlapInputFormat$LlapRecordReader.rethrowErrorIfAny(LlapInputFormat.java:275)
at
org.apache.hadoop.hive.llap.io.api.impl.LlapInputFormat$LlapRecordReader.nextCvb(LlapInputFormat.java:227)
at
org.apache.hadoop.hive.llap.io.api.impl.LlapInputFormat$LlapRecordReader.next(LlapInputFormat.java:155)
at
org.apache.hadoop.hive.llap.io.api.impl.LlapInputFormat$LlapRecordReader.next(LlapInputFormat.java:101)
at
org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.doNext(HiveContextAwareRecordReader.java:350)
... 22 more
Caused by: java.lang.NullPointerException
at
org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl$SargApplier.pickRowGroups(RecordReaderImpl.java:709)
at
org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader.determineRgsToRead(OrcEncodedDataReader.java:618)
at
org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader.callInternal(OrcEncodedDataReader.java:195)
at
org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader.callInternal(OrcEncodedDataReader.java:59)
at
org.apache.hadoop.hive.common.CallableWithNdc.call(CallableWithNdc.java:37)
... 4 more
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-11222) LLAP: occasional NPE in parallel queries in ORC reader
[
https://issues.apache.org/jira/browse/HIVE-11222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14623064#comment-14623064
]
Prasanth Jayachandran commented on HIVE-11222:
--
Got it! Make sense. LGTM, +1
LLAP: occasional NPE in parallel queries in ORC reader
--
Key: HIVE-11222
URL: https://issues.apache.org/jira/browse/HIVE-11222
Project: Hive
Issue Type: Sub-task
Reporter: Sergey Shelukhin
Assignee: Sergey Shelukhin
Fix For: llap
Attachments: HIVE-11222.patch
{noformat}
Caused by: java.io.IOException: java.lang.NullPointerException
at
org.apache.hadoop.hive.llap.io.api.impl.LlapInputFormat$LlapRecordReader.rethrowErrorIfAny(LlapInputFormat.java:275)
at
org.apache.hadoop.hive.llap.io.api.impl.LlapInputFormat$LlapRecordReader.nextCvb(LlapInputFormat.java:227)
at
org.apache.hadoop.hive.llap.io.api.impl.LlapInputFormat$LlapRecordReader.next(LlapInputFormat.java:155)
at
org.apache.hadoop.hive.llap.io.api.impl.LlapInputFormat$LlapRecordReader.next(LlapInputFormat.java:101)
at
org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.doNext(HiveContextAwareRecordReader.java:350)
... 22 more
Caused by: java.lang.NullPointerException
at
org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl$SargApplier.pickRowGroups(RecordReaderImpl.java:709)
at
org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader.determineRgsToRead(OrcEncodedDataReader.java:618)
at
org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader.callInternal(OrcEncodedDataReader.java:195)
at
org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader.callInternal(OrcEncodedDataReader.java:59)
at
org.apache.hadoop.hive.common.CallableWithNdc.call(CallableWithNdc.java:37)
... 4 more
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-10838) Allow the Hive metastore client to bind to a specific address when connecting to the server
[
https://issues.apache.org/jira/browse/HIVE-10838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14623076#comment-14623076
]
Hive QA commented on HIVE-10838:
{color:green}Overall{color}: +1 all checks pass
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12741948/HIVE-10838.patch
{color:green}SUCCESS:{color} +1 9151 tests passed
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/4576/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/4576/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-4576/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12741948 - PreCommit-HIVE-TRUNK-Build
Allow the Hive metastore client to bind to a specific address when connecting
to the server
---
Key: HIVE-10838
URL: https://issues.apache.org/jira/browse/HIVE-10838
Project: Hive
Issue Type: Bug
Affects Versions: 1.2.0
Reporter: HeeSoo Kim
Assignee: HeeSoo Kim
Attachments: HIVE-10838.patch
+*In a cluster with Kerberos authentication*+
When a Hive metastore client (e.g. HS2, oozie) has been configured with a
logical hostname (e.g. hiveserver/hiveserver_logical_hostn...@example.com),
it still uses its physical hostname to try to connect to the hive metastore.
For example, we specifiy, in hive-site.xml:
{noformat}
property
namehive.server2.authentication.kerberos.principal/name
valuehiveserver/hiveserver_logical_hostn...@example.com/value
/property
{noformat}
When the client tried to get a delegation token from the metastore, an
exception occurred:
{noformat}
2015-05-21 23:17:59,554 ERROR metadata.Hive
(Hive.java:getDelegationToken(2638)) - MetaException(message:Unauthorized
connection for super-user: hiveserver/hiveserver_logical_hostn...@example.com
from IP 10.250.16.43)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_delegation_token_result$get_delegation_token_resultStandardScheme.read(ThriftHiveMetastore.java)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_delegation_token_result$get_delegation_token_resultStandardScheme.read(ThriftHiveMetastore.java)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_delegation_token_result.read(ThriftHiveMetastore.java)
at
org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:78)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_delegation_token(ThriftHiveMetastore.java:3293)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_delegation_token(ThriftHiveMetastore.java:3279)
at
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getDelegationToken(HiveMetaStoreClient.java:1559)
{noformat}
We need to set the bind address when Hive metastore client tries to connect
Hive metastore based on logical hostname of Kerberos.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-9420) LLAP: separate decoding thread from read/uncompress thread
[ https://issues.apache.org/jira/browse/HIVE-9420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-9420: --- Attachment: HIVE-9420.WIP.patch Something like this... need to finish it LLAP: separate decoding thread from read/uncompress thread -- Key: HIVE-9420 URL: https://issues.apache.org/jira/browse/HIVE-9420 Project: Hive Issue Type: Sub-task Reporter: Sergey Shelukhin Assignee: Sergey Shelukhin Attachments: HIVE-9420.WIP.patch Right now reader is a callable that is dumped into threadpool, and all the data loading for one request happens on one thread (in parallel with operator pipeline). We might want to separate reading from HDFS/decompressing data, from decoding data and low-level row-level SARGs (that, and high level cache which is AWOL now, were the reasons for encoded data production separation from final VRB production) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11030) Enhance storage layer to create one delta file per write
[ https://issues.apache.org/jira/browse/HIVE-11030?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-11030: -- Attachment: HIVE-11030.8.patch This would only happen due to a bug. removed the stat. Enhance storage layer to create one delta file per write Key: HIVE-11030 URL: https://issues.apache.org/jira/browse/HIVE-11030 Project: Hive Issue Type: Sub-task Components: Transactions Affects Versions: 1.2.0 Reporter: Eugene Koifman Assignee: Eugene Koifman Attachments: HIVE-11030.2.patch, HIVE-11030.3.patch, HIVE-11030.4.patch, HIVE-11030.5.patch, HIVE-11030.6.patch, HIVE-11030.7.patch, HIVE-11030.8.patch Currently each txn using ACID insert/update/delete will generate a delta directory like delta_100_101. In order to support multi-statement transactions we must generate one delta per operation within the transaction so the deltas would be named like delta_100_101_0001, etc. Support for MERGE (HIVE-10924) would need the same. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11194) Exchange partition on external tables should fail with error message when target folder already exists
[
https://issues.apache.org/jira/browse/HIVE-11194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14623200#comment-14623200
]
Hive QA commented on HIVE-11194:
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12744821/HIVE-11194.patch
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 9136 tests executed
*Failed tests:*
{noformat}
TestCliDriver-udf_notequal.q-nullscript.q-filter_join_breaktask.q-and-12-more -
did not produce a TEST-*.xml file
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/4578/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/4578/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-4578/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12744821 - PreCommit-HIVE-TRUNK-Build
Exchange partition on external tables should fail with error message when
target folder already exists
--
Key: HIVE-11194
URL: https://issues.apache.org/jira/browse/HIVE-11194
Project: Hive
Issue Type: Bug
Components: Hive
Affects Versions: 2.0.0
Reporter: Aihua Xu
Assignee: Aihua Xu
Attachments: HIVE-11194.patch
Steps to repro:
{noformat}
Create /data/a1/pkey=1 directory with some data in it.
Create /data/a2/pkey=1 directory with some data in it.
create external table a1 (value string) partitioned by (pkey int) location
'/data/a1';
create external table a2 (value string) partitioned by (pkey int) location
'/data/a2';
alter table a2 add partition (pkey=1);
alter table a1 exchange partition (pkey=1) with table a2;
select * from a1 should now fail.
{noformat}
pkey=1 is not a partition of a1 but the folder exists. We should give an
error message for that.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-10944) Fix HS2 for Metrics
[
https://issues.apache.org/jira/browse/HIVE-10944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14623208#comment-14623208
]
Lefty Leverenz commented on HIVE-10944:
---
No doc needed: Although this changes the default value of the configuration
parameter *hive.service.metrics.file.location* from
{{file:///tmp/my-logging.properties}} to {{file:///tmp/report.json}},
HIVE-10927 changes it again to {{/tmp/report.json}}.
Fix HS2 for Metrics
---
Key: HIVE-10944
URL: https://issues.apache.org/jira/browse/HIVE-10944
Project: Hive
Issue Type: Bug
Reporter: Szehon Ho
Assignee: Szehon Ho
Fix For: 1.3.0, 2.0.0
Attachments: HIVE-10944.2.patch, HIVE-10944.3.patch,
HIVE-10944.4.patch, HIVE-10944.patch
Some issues about initializing the new HS2 metrics
1. Metrics is not working properly in HS2 due to wrong init checks
2. If not enabled, JVMPauseMonitor logs trash to HS2 logs as it wasnt
checking if metrics was enabled.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-10101) LLAP: enable yourkit profiling of tasks
[
https://issues.apache.org/jira/browse/HIVE-10101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14623227#comment-14623227
]
Hive QA commented on HIVE-10101:
{color:red}Overall{color}: -1 no tests executed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12733969/HIVE-10101.03.patch
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/4580/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/4580/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-4580/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hive-ptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ [[ -n /usr/java/jdk1.7.0_45-cloudera ]]
+ export JAVA_HOME=/usr/java/jdk1.7.0_45-cloudera
+ JAVA_HOME=/usr/java/jdk1.7.0_45-cloudera
+ export
PATH=/usr/java/jdk1.7.0_45-cloudera/bin/:/usr/local/apache-maven-3.0.5/bin:/usr/java/jdk1.7.0_45-cloudera/bin:/usr/local/apache-ant-1.9.1/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hiveptest/bin
+
PATH=/usr/java/jdk1.7.0_45-cloudera/bin/:/usr/local/apache-maven-3.0.5/bin:/usr/java/jdk1.7.0_45-cloudera/bin:/usr/local/apache-ant-1.9.1/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hiveptest/bin
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'M2_OPTS=-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost
-Dhttp.proxyPort=3128'
+ M2_OPTS='-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost
-Dhttp.proxyPort=3128'
+ cd /data/hive-ptest/working/
+ tee /data/hive-ptest/logs/PreCommit-HIVE-TRUNK-Build-4580/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at e6ea691 HIVE-11206 : CBO (Calcite Return Path): Join translation
should update all ExprNode recursively (Jesus Camacho Rodriguez via Ashutosh
Chauhan)
+ git clean -f -d
+ git checkout master
Already on 'master'
+ git reset --hard origin/master
HEAD is now at e6ea691 HIVE-11206 : CBO (Calcite Return Path): Join translation
should update all ExprNode recursively (Jesus Camacho Rodriguez via Ashutosh
Chauhan)
+ git merge --ff-only origin/master
Already up-to-date.
+ git gc
+ patchCommandPath=/data/hive-ptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hive-ptest/working/scratch/build.patch
+ [[ -f /data/hive-ptest/working/scratch/build.patch ]]
+ chmod +x /data/hive-ptest/working/scratch/smart-apply-patch.sh
+ /data/hive-ptest/working/scratch/smart-apply-patch.sh
/data/hive-ptest/working/scratch/build.patch
The patch does not appear to apply with p0, p1, or p2
+ exit 1
'
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12733969 - PreCommit-HIVE-TRUNK-Build
LLAP: enable yourkit profiling of tasks
---
Key: HIVE-10101
URL: https://issues.apache.org/jira/browse/HIVE-10101
Project: Hive
Issue Type: Sub-task
Reporter: Sergey Shelukhin
Assignee: Sergey Shelukhin
Attachments: HIVE-10101.02.patch, HIVE-10101.03.patch,
HIVE-10101.patch
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (HIVE-9420) LLAP: separate decoding thread from read/uncompress thread
[ https://issues.apache.org/jira/browse/HIVE-9420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin reassigned HIVE-9420: -- Assignee: Sergey Shelukhin LLAP: separate decoding thread from read/uncompress thread -- Key: HIVE-9420 URL: https://issues.apache.org/jira/browse/HIVE-9420 Project: Hive Issue Type: Sub-task Reporter: Sergey Shelukhin Assignee: Sergey Shelukhin Right now reader is a callable that is dumped into threadpool, and all the data loading for one request happens on one thread (in parallel with operator pipeline). We might want to separate reading from HDFS/decompressing data, from decoding data and low-level row-level SARGs (that, and high level cache which is AWOL now, were the reasons for encoded data production separation from final VRB production) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9420) LLAP: separate decoding thread from read/uncompress thread
[ https://issues.apache.org/jira/browse/HIVE-9420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14623121#comment-14623121 ] Sergey Shelukhin commented on HIVE-9420: There is loss of ordering if this is done in a straightforward way (wrapper for consumer that offloads consumeData, the whole patch would be 30 lines)... setDone will potentially be called before some of the async consumeData-s on the consumer that is consuming from intermediate MTT consumer. LLAP: separate decoding thread from read/uncompress thread -- Key: HIVE-9420 URL: https://issues.apache.org/jira/browse/HIVE-9420 Project: Hive Issue Type: Sub-task Reporter: Sergey Shelukhin Assignee: Sergey Shelukhin Right now reader is a callable that is dumped into threadpool, and all the data loading for one request happens on one thread (in parallel with operator pipeline). We might want to separate reading from HDFS/decompressing data, from decoding data and low-level row-level SARGs (that, and high level cache which is AWOL now, were the reasons for encoded data production separation from final VRB production) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10292) Add support for HS2 to use custom authentication class with kerberos environment
[
https://issues.apache.org/jira/browse/HIVE-10292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14623166#comment-14623166
]
Hive QA commented on HIVE-10292:
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12743981/HIVE-10292.patch
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 9150 tests executed
*Failed tests:*
{noformat}
org.apache.hive.hcatalog.hbase.TestPigHBaseStorageHandler.org.apache.hive.hcatalog.hbase.TestPigHBaseStorageHandler
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/4577/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/4577/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-4577/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12743981 - PreCommit-HIVE-TRUNK-Build
Add support for HS2 to use custom authentication class with kerberos
environment
Key: HIVE-10292
URL: https://issues.apache.org/jira/browse/HIVE-10292
Project: Hive
Issue Type: New Feature
Components: HiveServer2
Affects Versions: 1.2.0
Reporter: Heesoo Kim
Assignee: HeeSoo Kim
Attachments: HIVE-10292.patch
In the kerberos environment, Hiveserver2 only supports GSSAPI and DIGEST-MD5
authentication mechanism.
We would like to add the ability to use custom authentication class in
conjunction with Kerberos.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-10761) Create codahale-based metrics system for Hive
[ https://issues.apache.org/jira/browse/HIVE-10761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14623209#comment-14623209 ] Lefty Leverenz commented on HIVE-10761: --- FYI, HIVE-10944 also changed the default of *hive.service.metrics.file.location* in both 2.0.0 and 1.3.0, but then HIVE-10927 changed it again. * HIVE-10761 (this issue): original default = file:///tmp/my-logging.properties * HIVE-10944 (first change): file:///tmp/report.json * HIVE-10927 (next change): /tmp/report.json Create codahale-based metrics system for Hive - Key: HIVE-10761 URL: https://issues.apache.org/jira/browse/HIVE-10761 Project: Hive Issue Type: New Feature Components: Diagnosability Reporter: Szehon Ho Assignee: Szehon Ho Labels: TODOC1.3, TODOC2.0 Fix For: 1.3.0, 2.0.0 Attachments: HIVE-10761.2.patch, HIVE-10761.3.patch, HIVE-10761.4.patch, HIVE-10761.5.patch, HIVE-10761.6.patch, HIVE-10761.patch, hms-metrics.json There is a current Hive metrics system that hooks up to a JMX reporting, but all its measurements, models are custom. This is to make another metrics system that will be based on Codahale (ie yammer, dropwizard), which has the following advantage: * Well-defined metric model for frequently-needed metrics (ie JVM metrics) * Well-defined measurements for all metrics (ie max, mean, stddev, mean_rate, etc), * Built-in reporting frameworks like JMX, Console, Log, JSON webserver It is used for many projects, including several Apache projects like Oozie. Overall, monitoring tools should find it easier to understand these common metric, measurement, reporting models. The existing metric subsystem will be kept and can be enabled if backward compatibility is desired. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11209) Clean up dependencies in HiveDecimalWritable
[
https://issues.apache.org/jira/browse/HIVE-11209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14623229#comment-14623229
]
Hive QA commented on HIVE-11209:
{color:red}Overall{color}: -1 no tests executed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12744806/HIVE-11209.patch
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/4581/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/4581/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-4581/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hive-ptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ [[ -n /usr/java/jdk1.7.0_45-cloudera ]]
+ export JAVA_HOME=/usr/java/jdk1.7.0_45-cloudera
+ JAVA_HOME=/usr/java/jdk1.7.0_45-cloudera
+ export
PATH=/usr/java/jdk1.7.0_45-cloudera/bin/:/usr/local/apache-maven-3.0.5/bin:/usr/java/jdk1.7.0_45-cloudera/bin:/usr/local/apache-ant-1.9.1/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hiveptest/bin
+
PATH=/usr/java/jdk1.7.0_45-cloudera/bin/:/usr/local/apache-maven-3.0.5/bin:/usr/java/jdk1.7.0_45-cloudera/bin:/usr/local/apache-ant-1.9.1/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hiveptest/bin
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'M2_OPTS=-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost
-Dhttp.proxyPort=3128'
+ M2_OPTS='-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost
-Dhttp.proxyPort=3128'
+ cd /data/hive-ptest/working/
+ tee /data/hive-ptest/logs/PreCommit-HIVE-TRUNK-Build-4581/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at e6ea691 HIVE-11206 : CBO (Calcite Return Path): Join translation
should update all ExprNode recursively (Jesus Camacho Rodriguez via Ashutosh
Chauhan)
+ git clean -f -d
+ git checkout master
Already on 'master'
+ git reset --hard origin/master
HEAD is now at e6ea691 HIVE-11206 : CBO (Calcite Return Path): Join translation
should update all ExprNode recursively (Jesus Camacho Rodriguez via Ashutosh
Chauhan)
+ git merge --ff-only origin/master
Already up-to-date.
+ git gc
+ patchCommandPath=/data/hive-ptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hive-ptest/working/scratch/build.patch
+ [[ -f /data/hive-ptest/working/scratch/build.patch ]]
+ chmod +x /data/hive-ptest/working/scratch/smart-apply-patch.sh
+ /data/hive-ptest/working/scratch/smart-apply-patch.sh
/data/hive-ptest/working/scratch/build.patch
The patch does not appear to apply with p0, p1, or p2
+ exit 1
'
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12744806 - PreCommit-HIVE-TRUNK-Build
Clean up dependencies in HiveDecimalWritable
Key: HIVE-11209
URL: https://issues.apache.org/jira/browse/HIVE-11209
Project: Hive
Issue Type: Sub-task
Reporter: Owen O'Malley
Assignee: Owen O'Malley
Attachments: HIVE-11209.patch, HIVE-11209.patch
Currently HiveDecimalWritable depends on:
* org.apache.hadoop.hive.serde2.ByteStream
* org.apache.hadoop.hive.serde2.lazybinary.LazyBinaryUtils
* org.apache.hadoop.hive.serde2.typeinfo.HiveDecimalUtils
since we need HiveDecimalWritable for the decimal VectorizedColumnBatch,
breaking these dependencies will improve things.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-10884) Enable some beeline tests and turn on HIVE-4239 by default
[ https://issues.apache.org/jira/browse/HIVE-10884?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-10884: Attachment: HIVE-10884.06.patch I've ran a few tests one by one. Let's see if the stuck-test problem is only on my machine, or also on HiveQA Enable some beeline tests and turn on HIVE-4239 by default -- Key: HIVE-10884 URL: https://issues.apache.org/jira/browse/HIVE-10884 Project: Hive Issue Type: Bug Reporter: Sergey Shelukhin Assignee: Sergey Shelukhin Attachments: HIVE-10884.01.patch, HIVE-10884.02.patch, HIVE-10884.03.patch, HIVE-10884.04.patch, HIVE-10884.05.patch, HIVE-10884.06.patch, HIVE-10884.patch See comments in HIVE-4239. Beeline tests with parallelism need to be enabled to turn compilation parallelism on by default. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11231) CBO: Calcite Operator To Hive Operator (Calcite Return Path): make the output of ba_table_union.q more stable
[
https://issues.apache.org/jira/browse/HIVE-11231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14623225#comment-14623225
]
Hive QA commented on HIVE-11231:
{color:green}Overall{color}: +1 all checks pass
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12744773/HIVE-11231.01.patch
{color:green}SUCCESS:{color} +1 9150 tests passed
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/4579/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/4579/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-4579/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12744773 - PreCommit-HIVE-TRUNK-Build
CBO: Calcite Operator To Hive Operator (Calcite Return Path): make the output
of ba_table_union.q more stable
-
Key: HIVE-11231
URL: https://issues.apache.org/jira/browse/HIVE-11231
Project: Hive
Issue Type: Sub-task
Components: CBO
Reporter: Pengcheng Xiong
Assignee: Pengcheng Xiong
Priority: Minor
Attachments: HIVE-11231.01.patch
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-11230) IMetaStoreClient getPartition method arguments incorrectly named
[
https://issues.apache.org/jira/browse/HIVE-11230?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Elliot West updated HIVE-11230:
---
Attachment: HIVE-11230.1.patch
IMetaStoreClient getPartition method arguments incorrectly named
Key: HIVE-11230
URL: https://issues.apache.org/jira/browse/HIVE-11230
Project: Hive
Issue Type: Bug
Affects Versions: 2.0.0
Reporter: Elliot West
Assignee: Elliot West
Priority: Minor
Labels: metastore
Attachments: HIVE-11230.1.patch
The interface {{org.apache.hadoop.hive.metastore.IMetaStoreClient}} declares
a method {{getPartition}} as:
{code}
Partition getPartition(String tblName, String dbName, ListString partVals)
throws NoSuchObjectException, MetaException, TException;
{code}
This is implemented in
{{org.apache.hadoop.hive.metastore.HiveMetaStoreClient}} as:
{code}
public Partition getPartition(String db_name, String tbl_name, ListString
part_vals) throws NoSuchObjectException, MetaException, TException {
return deepCopy(filterHook.filterPartition(client.get_partition(db_name,
tbl_name, part_vals)));
}
{code}
Note that in the above code the param named {{db_name}} occupies the
{{tbl_name}} equivalent in the {{IMetaStoreClient}} method and vice-versa.
The method called in {{HiveMetaStoreClient}} also seemingly expects
{{db_name}} to appear first (see {{ThriftHiveMetastore.Iface.get_partition}}):
{code}
public Partition get_partition(String db_name, String tbl_name,
ListString part_vals) throws MetaException, NoSuchObjectException,
org.apache.thrift.TException;
{code}
And following this all the way through to the implementation of
{{org.apache.hadoop.hive.metastore.ObjectStore.getMTable(String, String)}}
shows that the database name is ultimately taken from the first argument.
Therefore I propose that the order needs to be corrected on
{{IMetaStoreClient}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-11160) Auto-gather column stats
[
https://issues.apache.org/jira/browse/HIVE-11160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14622508#comment-14622508
]
Hive QA commented on HIVE-11160:
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12744642/HIVE-11160.02.patch
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 9151 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_schemeAuthority
org.apache.hive.hcatalog.streaming.TestStreaming.testTransactionBatchEmptyCommit
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/4571/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/4571/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-4571/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12744642 - PreCommit-HIVE-TRUNK-Build
Auto-gather column stats
Key: HIVE-11160
URL: https://issues.apache.org/jira/browse/HIVE-11160
Project: Hive
Issue Type: New Feature
Reporter: Pengcheng Xiong
Assignee: Pengcheng Xiong
Attachments: HIVE-11160.01.patch, HIVE-11160.02.patch
Hive will collect table stats when set hive.stats.autogather=true during the
INSERT OVERWRITE command. And then the users need to collect the column stats
themselves using Analyze command. In this patch, the column stats will also
be collected automatically.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-11206) CBO (Calcite Return Path): Join translation should update all ExprNode recursively
[ https://issues.apache.org/jira/browse/HIVE-11206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14622484#comment-14622484 ] Ashutosh Chauhan commented on HIVE-11206: - ok.. +1 CBO (Calcite Return Path): Join translation should update all ExprNode recursively -- Key: HIVE-11206 URL: https://issues.apache.org/jira/browse/HIVE-11206 Project: Hive Issue Type: Sub-task Components: CBO Reporter: Jesus Camacho Rodriguez Assignee: Jesus Camacho Rodriguez Attachments: HIVE-11206.01.patch, HIVE-11206.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11206) CBO (Calcite Return Path): Join translation should update all ExprNode recursively
[ https://issues.apache.org/jira/browse/HIVE-11206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-11206: --- Attachment: HIVE-11206.01.patch CBO (Calcite Return Path): Join translation should update all ExprNode recursively -- Key: HIVE-11206 URL: https://issues.apache.org/jira/browse/HIVE-11206 Project: Hive Issue Type: Sub-task Components: CBO Reporter: Jesus Camacho Rodriguez Assignee: Jesus Camacho Rodriguez Attachments: HIVE-11206.01.patch, HIVE-11206.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11206) CBO (Calcite Return Path): Join translation should update all ExprNode recursively
[ https://issues.apache.org/jira/browse/HIVE-11206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14622171#comment-14622171 ] Jesus Camacho Rodriguez commented on HIVE-11206: [~ashutoshc], I added some additional comments to the patch. About the pieces from SemanticAnalyzer, the problem is that here we are doing something very specific of the return path: the predicates in the join come expressed in terms of the input/output columns, but the filters which are not on joining columns need to be expressed wrt to the output of the RS. In SemanticAnalyzer, we do not need to worry about this. Plus in addition, some of the data structures are generated from the QBJoinTree (e.g. filterMap), while here we need to generate them from the Calcite plan. Please, let me know if you had something specific in mind. CBO (Calcite Return Path): Join translation should update all ExprNode recursively -- Key: HIVE-11206 URL: https://issues.apache.org/jira/browse/HIVE-11206 Project: Hive Issue Type: Sub-task Components: CBO Reporter: Jesus Camacho Rodriguez Assignee: Jesus Camacho Rodriguez Attachments: HIVE-11206.01.patch, HIVE-11206.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11159) Integrate hplsql.Conf with HiveConf
[
https://issues.apache.org/jira/browse/HIVE-11159?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14622187#comment-14622187
]
Dmitry Tolpeko commented on HIVE-11159:
---
For examle, dual table name (this name is automatically appended when user uses
SELECT without FROM):
{code}
hplsql.dual.table
{code}
Various connection options (tool allows you to save Hive results to RDBMS,
write log records to RDBMS and so on).
{code}
hplsql.conn.db2conn
hplsql.conn.tdconn
hplsql.conn.mysqlconn
...
{code}
I am ok to read them from hive-site.xml if you think this will not confuse Hive
CLI users.
Integrate hplsql.Conf with HiveConf
---
Key: HIVE-11159
URL: https://issues.apache.org/jira/browse/HIVE-11159
Project: Hive
Issue Type: Task
Components: hpl/sql
Affects Versions: 2.0.0
Reporter: Alan Gates
Assignee: Dmitry Tolpeko
HPL/SQL has it's own Conf object. It should re-use HiveConf.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-11229) Mutation API: Coordinator communication with meta store should be optional
[
https://issues.apache.org/jira/browse/HIVE-11229?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Elliot West updated HIVE-11229:
---
Attachment: HIVE-11229.1.patch
Mutation API: Coordinator communication with meta store should be optional
--
Key: HIVE-11229
URL: https://issues.apache.org/jira/browse/HIVE-11229
Project: Hive
Issue Type: Bug
Components: HCatalog
Affects Versions: 2.0.0
Reporter: Elliot West
Assignee: Elliot West
Labels: streaming_api
Attachments: HIVE-11229.1.patch
[~ekoifman] raised a theoretical issue with the streaming mutation API
(HIVE-10165) where worker nodes operating in a distributed cluster might
overwhelm a meta store while trying to obtain partition locks. Although this
does not happen in practice (see HIVE-11228), the API does communicate with
the meta store in this manner to obtain partition paths and create new
partitions. Therefore the issue described does in fact exist in the current
implementation, albeit in a different code path. I’d like to make such
communication optional like so:
* When the user chooses not to create partitions on demand, no meta store
connection will be created in the {{MutationCoordinators}}. Additionally,
partition paths will be resolved using
{{org.apache.hadoop.hive.metastore.Warehouse.getPartitionPath(Path,
LinkedHashMapString, String)}} which should be suitable so long as standard
Hive partition layouts are followed.
* If the user does choose to create partitions on demand then the system will
operate as is does currently; using the meta store to both issue
{{add_partition}} events and look up partition meta data.
* The documentation will be updated to describe these behaviours and outline
alternative approaches to collecting affected partition names and creating
partitions in a less intensive manner.
Side note for follow up: The parameter names {{tblName}} and {{dbName}} seem
to be the wrong way around on the method
{{org.apache.hadoop.hive.metastore.IMetaStoreClient.getPartition(String,
String, ListString)}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-11225) Running all Hive UTs or itests executes only small subset of tests
[
https://issues.apache.org/jira/browse/HIVE-11225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14622619#comment-14622619
]
Hive QA commented on HIVE-11225:
{color:green}Overall{color}: +1 all checks pass
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12744667/HIVE-11225.1.patch
{color:green}SUCCESS:{color} +1 9150 tests passed
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/4572/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/4572/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-4572/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12744667 - PreCommit-HIVE-TRUNK-Build
Running all Hive UTs or itests executes only small subset of tests
--
Key: HIVE-11225
URL: https://issues.apache.org/jira/browse/HIVE-11225
Project: Hive
Issue Type: Bug
Components: Tests
Reporter: Thomas Friedrich
Assignee: Ferdinand Xu
Attachments: HIVE-11225.1.patch, HIVE-11225.patch
Trying to run all unit tests runs only a small sub-set of the tests:
For example:
mvn test -Phadoop-2
only ran 272 tests.
Others ran into similar issues when running the UTs from a particular package
like ql.
This is potentially related to HIVE-10941 where an additional option to skip
the Spark UTs was added. When I take out
exclude%regex[${skip.spark.files}]/exclude
from the exclude list of tests, all tests are executed again. This is not a
fix though, instead the spark-test profile should be fixed.
I see the problem in both the 1.2 branch as well as master.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-11129) Issue a warning when copied from UTF-8 to ISO 8859-1
[
https://issues.apache.org/jira/browse/HIVE-11129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14622680#comment-14622680
]
Szehon Ho commented on HIVE-11129:
--
+1
Issue a warning when copied from UTF-8 to ISO 8859-1
Key: HIVE-11129
URL: https://issues.apache.org/jira/browse/HIVE-11129
Project: Hive
Issue Type: Bug
Components: File Formats
Reporter: Aihua Xu
Assignee: Aihua Xu
Fix For: 2.0.0
Attachments: HIVE-11129.patch
Copying data from a table using UTF-8 encoding to one using ISO 8859-1
encoding causes data corruption without warning.
{noformat}
CREATE TABLE person_utf8 (name STRING)
ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe'
WITH SERDEPROPERTIES ('serialization.encoding'='UTF8');
{noformat}
Put the following data in the table:
Müller,Thomas
Jørgensen,Jørgen
Vega,Andrés
中村,浩人
אביה,נועם
{noformat}
CREATE TABLE person_2 ROW FORMAT SERDE
'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe'
WITH SERDEPROPERTIES ('serialization.encoding'='ISO8859_1')
AS select * from person_utf8;
{noformat}
expected to get mangled data but we should give a warning.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-11190) No prompting info or warning provided when METASTORE_FILTER_HOOK in authorization V2 is overridden
[ https://issues.apache.org/jira/browse/HIVE-11190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14622589#comment-14622589 ] Thejas M Nair commented on HIVE-11190: -- +1 No prompting info or warning provided when METASTORE_FILTER_HOOK in authorization V2 is overridden -- Key: HIVE-11190 URL: https://issues.apache.org/jira/browse/HIVE-11190 Project: Hive Issue Type: Bug Reporter: Dapeng Sun Assignee: Dapeng Sun Attachments: HIVE-11190.001.patch, HIVE-11190.002.patch, HIVE-11190.003.patch ConfVars.METASTORE_FILTER_HOOK in authorization V2 is will be override without prompting info or warning. it will cause user failed to customize the METASTORE_FILTER_HOOK. We should log information such as this value is ignored when override happens. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11231) CBO: Calcite Operator To Hive Operator (Calcite Return Path): make the output of ba_table_union.q more stable
[ https://issues.apache.org/jira/browse/HIVE-11231?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-11231: --- Attachment: HIVE-11231.01.patch CBO: Calcite Operator To Hive Operator (Calcite Return Path): make the output of ba_table_union.q more stable - Key: HIVE-11231 URL: https://issues.apache.org/jira/browse/HIVE-11231 Project: Hive Issue Type: Sub-task Components: CBO Reporter: Pengcheng Xiong Assignee: Pengcheng Xiong Priority: Minor Attachments: HIVE-11231.01.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-10882) CBO: Calcite Operator To Hive Operator (Calcite Return Path) empty filtersMap of join operator causes wrong results
[ https://issues.apache.org/jira/browse/HIVE-10882?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-10882: --- Attachment: HIVE-10882.02.patch Rebasing patch to trigger another QA run. CBO: Calcite Operator To Hive Operator (Calcite Return Path) empty filtersMap of join operator causes wrong results --- Key: HIVE-10882 URL: https://issues.apache.org/jira/browse/HIVE-10882 Project: Hive Issue Type: Sub-task Components: CBO Reporter: Pengcheng Xiong Assignee: Jesus Camacho Rodriguez Attachments: HIVE-10882.01.patch, HIVE-10882.02.patch CBO return path creates join operator with empty filtersMap. This causes outer joins to produce wrong results. To reproduce, run louter_join_ppr.q with return path turned on. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10165) Improve hive-hcatalog-streaming extensibility and support updates and deletes.
[
https://issues.apache.org/jira/browse/HIVE-10165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14622409#comment-14622409
]
Elliot West commented on HIVE-10165:
This has been very helpful, thank you. Firstly, I've attempted to address the
inadequate locking using the approach you outlined. This is tracked by
HIVE-11228. However, in addition to this, your concerns regarding the workers
accessing the meta store led me to reevaluate the manner in which partition
locations are currently resolved and created by workers. It so happens that
here also the workers connect to the meta store from the grid. I've modified
the implementation to provide some alternatives and make this behaviour
optional, see HIVE-11229.
Improve hive-hcatalog-streaming extensibility and support updates and deletes.
--
Key: HIVE-10165
URL: https://issues.apache.org/jira/browse/HIVE-10165
Project: Hive
Issue Type: Improvement
Components: HCatalog
Affects Versions: 1.2.0
Reporter: Elliot West
Assignee: Elliot West
Labels: TODOC2.0, streaming_api
Fix For: 2.0.0
Attachments: HIVE-10165.0.patch, HIVE-10165.10.patch,
HIVE-10165.4.patch, HIVE-10165.5.patch, HIVE-10165.6.patch,
HIVE-10165.7.patch, HIVE-10165.9.patch, mutate-system-overview.png
h3. Overview
I'd like to extend the
[hive-hcatalog-streaming|https://cwiki.apache.org/confluence/display/Hive/Streaming+Data+Ingest]
API so that it also supports the writing of record updates and deletes in
addition to the already supported inserts.
h3. Motivation
We have many Hadoop processes outside of Hive that merge changed facts into
existing datasets. Traditionally we achieve this by: reading in a
ground-truth dataset and a modified dataset, grouping by a key, sorting by a
sequence and then applying a function to determine inserted, updated, and
deleted rows. However, in our current scheme we must rewrite all partitions
that may potentially contain changes. In practice the number of mutated
records is very small when compared with the records contained in a
partition. This approach results in a number of operational issues:
* Excessive amount of write activity required for small data changes.
* Downstream applications cannot robustly read these datasets while they are
being updated.
* Due to scale of the updates (hundreds or partitions) the scope for
contention is high.
I believe we can address this problem by instead writing only the changed
records to a Hive transactional table. This should drastically reduce the
amount of data that we need to write and also provide a means for managing
concurrent access to the data. Our existing merge processes can read and
retain each record's {{ROW_ID}}/{{RecordIdentifier}} and pass this through to
an updated form of the hive-hcatalog-streaming API which will then have the
required data to perform an update or insert in a transactional manner.
h3. Benefits
* Enables the creation of large-scale dataset merge processes
* Opens up Hive transactional functionality in an accessible manner to
processes that operate outside of Hive.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-11068) Hive throws OOM in client side
[
https://issues.apache.org/jira/browse/HIVE-11068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14622914#comment-14622914
]
Mostafa Mokhtar commented on HIVE-11068:
[~jpullokkaran]
This is the one I was referring to.
Fixed at this point.
Hive throws OOM in client side
--
Key: HIVE-11068
URL: https://issues.apache.org/jira/browse/HIVE-11068
Project: Hive
Issue Type: Bug
Reporter: Rajesh Balamohan
Assignee: Prasanth Jayachandran
Attachments: Yourkit_String.png, Yourkit_TablScanDesc.png,
hive_cli_debug.log.gz
Hive build: (Latest on Jun 21. commit
142426394cfdc8a1fea51f7642c63f43f36b0333).
Query: Query 64 TPC-DS
(https://github.com/cartershanklin/hive-testbench/blob/master/sample-queries-tpcds/query64.sql)
Hive throws following OOM in client side.
{noformat}
Exception in thread main java.lang.OutOfMemoryError: Java heap space
at java.lang.StringCoding$StringDecoder.decode(StringCoding.java:149)
at java.lang.StringCoding.decode(StringCoding.java:193)
at java.lang.String.init(String.java:414)
at java.lang.String.init(String.java:479)
at
org.apache.hadoop.hive.ql.exec.Utilities.serializeExpression(Utilities.java:799)
at
org.apache.hadoop.hive.ql.plan.TableScanDesc.setFilterExpr(TableScanDesc.java:153)
at
org.apache.hadoop.hive.ql.ppd.OpProcFactory.pushFilterToStorageHandler(OpProcFactory.java:901)
at
org.apache.hadoop.hive.ql.ppd.OpProcFactory.createFilter(OpProcFactory.java:818)
at
org.apache.hadoop.hive.ql.ppd.OpProcFactory.createFilter(OpProcFactory.java:788)
at
org.apache.hadoop.hive.ql.ppd.OpProcFactory$TableScanPPD.process(OpProcFactory.java:388)
at
org.apache.hadoop.hive.ql.lib.DefaultRuleDispatcher.dispatch(DefaultRuleDispatcher.java:90)
at
org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatchAndReturn(DefaultGraphWalker.java:95)
at
org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatch(DefaultGraphWalker.java:79)
at
org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.walk(DefaultGraphWalker.java:133)
at
org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.startWalking(DefaultGraphWalker.java:110)
at
org.apache.hadoop.hive.ql.ppd.PredicatePushDown.transform(PredicatePushDown.java:135)
at
org.apache.hadoop.hive.ql.optimizer.Optimizer.optimize(Optimizer.java:192)
at
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:10171)
at
org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:207)
at
org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:227)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:424)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:308)
at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1124)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1172)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1061)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1051)
at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:213)
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:165)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:376)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:311)
at org.apache.hadoop.hive.cli.CliDriver.processReader(CliDriver.java:409)
at org.apache.hadoop.hive.cli.CliDriver.processFile(CliDriver.java:425)
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-11194) Exchange partition on external tables should fail with error message when target folder already exists
[
https://issues.apache.org/jira/browse/HIVE-11194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aihua Xu updated HIVE-11194:
Attachment: HIVE-11194.patch
Exchange partition on external tables should fail with error message when
target folder already exists
--
Key: HIVE-11194
URL: https://issues.apache.org/jira/browse/HIVE-11194
Project: Hive
Issue Type: Bug
Components: Hive
Affects Versions: 2.0.0
Reporter: Aihua Xu
Assignee: Aihua Xu
Attachments: HIVE-11194.patch
Steps to repro:
{noformat}
Create /data/a1/pkey=1 directory with some data in it.
Create /data/a2/pkey=1 directory with some data in it.
create external table a1 (value string) partitioned by (pkey int) location
'/data/a1';
create external table a2 (value string) partitioned by (pkey int) location
'/data/a2';
alter table a2 add partition (pkey=1);
alter table a1 exchange partition (pkey=1) with table a2;
select * from a1 should now fail.
{noformat}
pkey=1 is not a partition of a1 but the folder exists. We should give an
error message for that.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-11194) Exchange partition on external tables should fail with error message when target folder already exists
[
https://issues.apache.org/jira/browse/HIVE-11194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aihua Xu updated HIVE-11194:
Attachment: (was: HIVE-11194.patch)
Exchange partition on external tables should fail with error message when
target folder already exists
--
Key: HIVE-11194
URL: https://issues.apache.org/jira/browse/HIVE-11194
Project: Hive
Issue Type: Bug
Components: Hive
Affects Versions: 2.0.0
Reporter: Aihua Xu
Assignee: Aihua Xu
Attachments: HIVE-11194.patch
Steps to repro:
{noformat}
Create /data/a1/pkey=1 directory with some data in it.
Create /data/a2/pkey=1 directory with some data in it.
create external table a1 (value string) partitioned by (pkey int) location
'/data/a1';
create external table a2 (value string) partitioned by (pkey int) location
'/data/a2';
alter table a2 add partition (pkey=1);
alter table a1 exchange partition (pkey=1) with table a2;
select * from a1 should now fail.
{noformat}
pkey=1 is not a partition of a1 but the folder exists. We should give an
error message for that.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-10048) JDBC - Support SSL encryption regardless of Authentication mechanism
[ https://issues.apache.org/jira/browse/HIVE-10048?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14622935#comment-14622935 ] Mike Yoder commented on HIVE-10048: --- LGTM, +1 JDBC - Support SSL encryption regardless of Authentication mechanism Key: HIVE-10048 URL: https://issues.apache.org/jira/browse/HIVE-10048 Project: Hive Issue Type: Improvement Components: JDBC Affects Versions: 1.0.0 Reporter: Mubashir Kazia Assignee: Mubashir Kazia Labels: newbie, patch Attachments: HIVE-10048.1.patch JDBC driver currently only supports SSL Transport if the Authentication mechanism is SASL Plain with username and password. SSL transport should be decoupled from Authentication mechanism. If the customer chooses to do Kerberos Authentication and SSL encryption over the wire it should be supported. The Server side already supports this but the driver does not. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11232) CBO: Calcite Operator To Hive Operator (Calcite Return Path): fix the output of select_same_col.q
[ https://issues.apache.org/jira/browse/HIVE-11232?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-11232: --- Attachment: HIVE-11232.01.patch CBO: Calcite Operator To Hive Operator (Calcite Return Path): fix the output of select_same_col.q - Key: HIVE-11232 URL: https://issues.apache.org/jira/browse/HIVE-11232 Project: Hive Issue Type: Sub-task Components: CBO Reporter: Pengcheng Xiong Assignee: Pengcheng Xiong Attachments: HIVE-11232.01.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11228) Mutation API should use semi-shared locks.
[
https://issues.apache.org/jira/browse/HIVE-11228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14622860#comment-14622860
]
Hive QA commented on HIVE-11228:
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12744720/HIVE-11228.1.patch
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 9151 tests executed
*Failed tests:*
{noformat}
TestMarkPartition - did not produce a TEST-*.xml file
org.apache.hive.hcatalog.streaming.TestStreaming.testTransactionBatchCommit_Json
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/4574/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/4574/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-4574/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12744720 - PreCommit-HIVE-TRUNK-Build
Mutation API should use semi-shared locks.
--
Key: HIVE-11228
URL: https://issues.apache.org/jira/browse/HIVE-11228
Project: Hive
Issue Type: Bug
Components: HCatalog
Affects Versions: 2.0.0
Reporter: Elliot West
Assignee: Elliot West
Labels: streaming_api, transaction, transactions
Attachments: HIVE-11228.1.patch
An [issue was
identified|https://issues.apache.org/jira/browse/HIVE-10165?focusedCommentId=14619640page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14619640]
by [~ekoifman] in the streaming mutation API (HIVE-10165) where an
insufficiently restrictive lock was being used when issuing updates and
deletes to ACID tables and partitions. A shared lock was being used where in
fact a semi-shared lock is required. Additionally, the current lock scope
targets the entire table, whereas in theory if the table is partitioned, then
only the affected partitions are required to participate in the semi-shared
lock. However, there are a couple of technical challenges that prevent the
locks currently being applied on a per-partition basis:
* It is expected that the affected partitions are not known in advance so
individual partition locks would need to be acquired as needed.
* The API is expected to execute in a clustered environment and so acquiring
these locks as on an ‘as needed’ basis presents a risk that the meta store
may become overwhelmed. This is expected to be less of an problem when an
HBase based meta store is introduced (HIVE-9452).
* My understanding is that multiple fine grained lock acquisitions for a
single transaction are not possible at present. When they are available
they’ll introduce the possibility that deadlocks can occur. This should be
better handled when HIVE-9675 is complete.
Therefore, as advised, at this time the system will obtain a semi-shared lock
on participating tables. Although this will prevent other concurrent writes,
it will preserve snapshot isolation when reading.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-10101) LLAP: enable yourkit profiling of tasks
[ https://issues.apache.org/jira/browse/HIVE-10101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14622866#comment-14622866 ] Sergey Shelukhin commented on HIVE-10101: - [~gopalv] this jira :) LLAP: enable yourkit profiling of tasks --- Key: HIVE-10101 URL: https://issues.apache.org/jira/browse/HIVE-10101 Project: Hive Issue Type: Sub-task Reporter: Sergey Shelukhin Assignee: Sergey Shelukhin Attachments: HIVE-10101.02.patch, HIVE-10101.03.patch, HIVE-10101.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-10799) Refactor the SearchArgumentFactory to remove the dependence on ExprNodeGenericFuncDesc
[ https://issues.apache.org/jira/browse/HIVE-10799?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Owen O'Malley updated HIVE-10799: - Attachment: HIVE-10799.patch Resubmit for jenkins. Refactor the SearchArgumentFactory to remove the dependence on ExprNodeGenericFuncDesc -- Key: HIVE-10799 URL: https://issues.apache.org/jira/browse/HIVE-10799 Project: Hive Issue Type: Sub-task Reporter: Owen O'Malley Assignee: Owen O'Malley Attachments: HIVE-10799.patch, HIVE-10799.patch SearchArgumentFactory and SearchArgumentImpl are high level and shouldn't depend on the internals of Hive's AST model. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11137) In DateWritable remove the use of LazyBinaryUtils
[ https://issues.apache.org/jira/browse/HIVE-11137?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Owen O'Malley updated HIVE-11137: - Attachment: HIVE-11137.patch rebased patch for jenkins. In DateWritable remove the use of LazyBinaryUtils - Key: HIVE-11137 URL: https://issues.apache.org/jira/browse/HIVE-11137 Project: Hive Issue Type: Sub-task Reporter: Owen O'Malley Assignee: Owen O'Malley Attachments: HIVE-11137.patch, HIVE-11137.patch Currently the DateWritable class uses LazyBinaryUtils, which has a lot of dependencies. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11210) Remove dependency on HiveConf from Orc reader writer
[ https://issues.apache.org/jira/browse/HIVE-11210?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Owen O'Malley updated HIVE-11210: - Attachment: HIVE-11210.patch reload for jenkins. Remove dependency on HiveConf from Orc reader writer -- Key: HIVE-11210 URL: https://issues.apache.org/jira/browse/HIVE-11210 Project: Hive Issue Type: Sub-task Reporter: Owen O'Malley Assignee: Owen O'Malley Attachments: HIVE-11210.patch, HIVE-11210.patch Currently the ORC reader and writer get their default values from HiveConf. I propose that we make the reader and writer have their own programatic defaults and the OrcInputFormat and OrcOutputFormat can use the version in HiveConf. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9152) Dynamic Partition Pruning [Spark Branch]
[ https://issues.apache.org/jira/browse/HIVE-9152?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Sun updated HIVE-9152: --- Attachment: HIVE-9152.10-spark.patch Dynamic Partition Pruning [Spark Branch] Key: HIVE-9152 URL: https://issues.apache.org/jira/browse/HIVE-9152 Project: Hive Issue Type: Sub-task Components: Spark Affects Versions: spark-branch Reporter: Brock Noland Assignee: Chao Sun Attachments: HIVE-9152.1-spark.patch, HIVE-9152.10-spark.patch, HIVE-9152.2-spark.patch, HIVE-9152.3-spark.patch, HIVE-9152.4-spark.patch, HIVE-9152.5-spark.patch, HIVE-9152.6-spark.patch, HIVE-9152.8-spark.patch, HIVE-9152.9-spark.patch Tez implemented dynamic partition pruning in HIVE-7826. This is a nice optimization and we should implement the same in HOS. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9152) Dynamic Partition Pruning [Spark Branch]
[ https://issues.apache.org/jira/browse/HIVE-9152?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Sun updated HIVE-9152: --- Attachment: (was: HIVE-9152.10-spark.patch) Dynamic Partition Pruning [Spark Branch] Key: HIVE-9152 URL: https://issues.apache.org/jira/browse/HIVE-9152 Project: Hive Issue Type: Sub-task Components: Spark Affects Versions: spark-branch Reporter: Brock Noland Assignee: Chao Sun Attachments: HIVE-9152.1-spark.patch, HIVE-9152.10-spark.patch, HIVE-9152.2-spark.patch, HIVE-9152.3-spark.patch, HIVE-9152.4-spark.patch, HIVE-9152.5-spark.patch, HIVE-9152.6-spark.patch, HIVE-9152.8-spark.patch, HIVE-9152.9-spark.patch Tez implemented dynamic partition pruning in HIVE-7826. This is a nice optimization and we should implement the same in HOS. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11124) Move OrcRecordUpdater.getAcidEventFields to RecordReaderFactory
[ https://issues.apache.org/jira/browse/HIVE-11124?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Owen O'Malley updated HIVE-11124: - Attachment: HIVE-11124.patch Reupload patch for jenkins. Move OrcRecordUpdater.getAcidEventFields to RecordReaderFactory --- Key: HIVE-11124 URL: https://issues.apache.org/jira/browse/HIVE-11124 Project: Hive Issue Type: Sub-task Reporter: Owen O'Malley Assignee: Owen O'Malley Attachments: HIVE-11124.patch, HIVE-11124.patch Move OrcRecordUpdater.getAcidEventFields to RecordReaderFactory to avoid the extra dependence. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10884) Enable some beeline tests and turn on HIVE-4239 by default
[
https://issues.apache.org/jira/browse/HIVE-10884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14622885#comment-14622885
]
Sergey Shelukhin commented on HIVE-10884:
-
After some more changes that I will post shortly just for kicks, it passes one
test and then gets stuck on a linux box (cannot make it work on Mac, screw Mac).
The only errors I can see are {noformat}
2015-07-10 13:42:45,170 ERROR [ProcessThread(sid:0 cport:-1):]:
server.NIOServerCnxnFactory (NIOServerCnxnFactory.java:uncaughtException(44)) -
Thread Thread[ProcessThread(sid:0 cport:-1):,5,main] died
java.lang.NoClassDefFoundError: org/apache/zookeeper/txn/ErrorTxn
at
org.apache.zookeeper.server.PrepRequestProcessor.pRequest(PrepRequestProcessor.java:643)
at
org.apache.zookeeper.server.PrepRequestProcessor.run(PrepRequestProcessor.java:131)
Caused by: java.lang.ClassNotFoundException: org.apache.zookeeper.txn.ErrorTxn
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
... 2 more
{noformat}
Enable some beeline tests and turn on HIVE-4239 by default
--
Key: HIVE-10884
URL: https://issues.apache.org/jira/browse/HIVE-10884
Project: Hive
Issue Type: Bug
Reporter: Sergey Shelukhin
Assignee: Sergey Shelukhin
Attachments: HIVE-10884.01.patch, HIVE-10884.02.patch,
HIVE-10884.03.patch, HIVE-10884.04.patch, HIVE-10884.05.patch,
HIVE-10884.patch
See comments in HIVE-4239.
Beeline tests with parallelism need to be enabled to turn compilation
parallelism on by default.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-11194) Exchange partition on external tables should fail with error message when target folder already exists
[
https://issues.apache.org/jira/browse/HIVE-11194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aihua Xu updated HIVE-11194:
Attachment: HIVE-11194.patch
Exchange partition on external tables should fail with error message when
target folder already exists
--
Key: HIVE-11194
URL: https://issues.apache.org/jira/browse/HIVE-11194
Project: Hive
Issue Type: Bug
Components: Hive
Affects Versions: 2.0.0
Reporter: Aihua Xu
Assignee: Aihua Xu
Attachments: HIVE-11194.patch
Steps to repro:
{noformat}
Create /data/a1/pkey=1 directory with some data in it.
Create /data/a2/pkey=1 directory with some data in it.
create external table a1 (value string) partitioned by (pkey int) location
'/data/a1';
create external table a2 (value string) partitioned by (pkey int) location
'/data/a2';
alter table a2 add partition (pkey=1);
alter table a1 exchange partition (pkey=1) with table a2;
select * from a1 should now fail.
{noformat}
pkey=1 is not a partition of a1 but the folder exists. We should give an
error message for that.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-10673) Dynamically partitioned hash join for Tez
[ https://issues.apache.org/jira/browse/HIVE-10673?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Dere updated HIVE-10673: -- Attachment: HIVE-10673.10.patch Patch v10: - Rebase with trunk, looks like some methods in GenTezUtils were converted to static - When selecting distributed hash join, the join operator should get OpTraits/stats set - For the issue regarding the flattened expressions in the vectorized rowObjectInspector, change the workaround to un-flatten the object inspector during JoinUtil.getObjectInspectorsFromEvaluators(). This is still a bit of a workaround, but only requires a change in 1 place, rather than the 2 changes needed in the previous solution (having to modify the column names during vectorized MapJoinOperator, as well as when generating the vectorized rowObjectInspector in VectorizedBatchUtil) - In the reducer, only the big table's input source should be vectorized Dynamically partitioned hash join for Tez - Key: HIVE-10673 URL: https://issues.apache.org/jira/browse/HIVE-10673 Project: Hive Issue Type: New Feature Components: Query Planning, Query Processor Reporter: Jason Dere Assignee: Jason Dere Attachments: HIVE-10673.1.patch, HIVE-10673.10.patch, HIVE-10673.2.patch, HIVE-10673.3.patch, HIVE-10673.4.patch, HIVE-10673.5.patch, HIVE-10673.6.patch, HIVE-10673.7.patch, HIVE-10673.8.patch, HIVE-10673.9.patch Some analysis of shuffle join queries by [~mmokhtar]/[~gopalv] found about 2/3 of the CPU was spent during sorting/merging. While this does not work for MR, for other execution engines (such as Tez), it is possible to create a reduce-side join that uses unsorted inputs in order to eliminate the sorting, which may be faster than a shuffle join. To join on unsorted inputs, we can use the hash join algorithm to perform the join in the reducer. This will require the small tables in the join to fit in the reducer/hash table for this to work. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-10927) Add number of HMS/HS2 connection metrics
[ https://issues.apache.org/jira/browse/HIVE-10927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szehon Ho updated HIVE-10927: - Labels: TODOC1.3 TODOC2.0 (was: TODOC2.0) Fix Version/s: 1.3.0 Cherry-picked to branch-1. Add number of HMS/HS2 connection metrics Key: HIVE-10927 URL: https://issues.apache.org/jira/browse/HIVE-10927 Project: Hive Issue Type: Sub-task Components: Diagnosability Reporter: Szehon Ho Assignee: Szehon Ho Labels: TODOC1.3, TODOC2.0 Fix For: 1.3.0, 2.0.0 Attachments: HIVE-10927.2.patch, HIVE-10927.2.patch, HIVE-10927.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-10761) Create codahale-based metrics system for Hive
[ https://issues.apache.org/jira/browse/HIVE-10761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szehon Ho updated HIVE-10761: - Labels: TODOC1.3 TODOC2.0 (was: TODOC1.3) Cherry-picked to branch-1 Create codahale-based metrics system for Hive - Key: HIVE-10761 URL: https://issues.apache.org/jira/browse/HIVE-10761 Project: Hive Issue Type: New Feature Components: Diagnosability Reporter: Szehon Ho Assignee: Szehon Ho Labels: TODOC1.3, TODOC2.0 Fix For: 1.3.0, 2.0.0 Attachments: HIVE-10761.2.patch, HIVE-10761.3.patch, HIVE-10761.4.patch, HIVE-10761.5.patch, HIVE-10761.6.patch, HIVE-10761.patch, hms-metrics.json There is a current Hive metrics system that hooks up to a JMX reporting, but all its measurements, models are custom. This is to make another metrics system that will be based on Codahale (ie yammer, dropwizard), which has the following advantage: * Well-defined metric model for frequently-needed metrics (ie JVM metrics) * Well-defined measurements for all metrics (ie max, mean, stddev, mean_rate, etc), * Built-in reporting frameworks like JMX, Console, Log, JSON webserver It is used for many projects, including several Apache projects like Oozie. Overall, monitoring tools should find it easier to understand these common metric, measurement, reporting models. The existing metric subsystem will be kept and can be enabled if backward compatibility is desired. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-10944) Fix HS2 for Metrics
[ https://issues.apache.org/jira/browse/HIVE-10944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szehon Ho updated HIVE-10944: - Fix Version/s: 2.0.0 1.3.0 Cherry-picked to branch-1 Fix HS2 for Metrics --- Key: HIVE-10944 URL: https://issues.apache.org/jira/browse/HIVE-10944 Project: Hive Issue Type: Bug Reporter: Szehon Ho Assignee: Szehon Ho Fix For: 1.3.0, 2.0.0 Attachments: HIVE-10944.2.patch, HIVE-10944.3.patch, HIVE-10944.4.patch, HIVE-10944.patch Some issues about initializing the new HS2 metrics 1. Metrics is not working properly in HS2 due to wrong init checks 2. If not enabled, JVMPauseMonitor logs trash to HS2 logs as it wasnt checking if metrics was enabled. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11222) LLAP: occasional NPE in parallel queries in ORC reader
[
https://issues.apache.org/jira/browse/HIVE-11222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14623022#comment-14623022
]
Prasanth Jayachandran commented on HIVE-11222:
--
[~sershe] Under what condition be the indexes not populated and pickRowGroups()
being called? Why do we need to throw an AssertionError? We should not call
pickRowGroups() without any indexes, right?
LLAP: occasional NPE in parallel queries in ORC reader
--
Key: HIVE-11222
URL: https://issues.apache.org/jira/browse/HIVE-11222
Project: Hive
Issue Type: Sub-task
Reporter: Sergey Shelukhin
Assignee: Sergey Shelukhin
Fix For: llap
Attachments: HIVE-11222.patch
{noformat}
Caused by: java.io.IOException: java.lang.NullPointerException
at
org.apache.hadoop.hive.llap.io.api.impl.LlapInputFormat$LlapRecordReader.rethrowErrorIfAny(LlapInputFormat.java:275)
at
org.apache.hadoop.hive.llap.io.api.impl.LlapInputFormat$LlapRecordReader.nextCvb(LlapInputFormat.java:227)
at
org.apache.hadoop.hive.llap.io.api.impl.LlapInputFormat$LlapRecordReader.next(LlapInputFormat.java:155)
at
org.apache.hadoop.hive.llap.io.api.impl.LlapInputFormat$LlapRecordReader.next(LlapInputFormat.java:101)
at
org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.doNext(HiveContextAwareRecordReader.java:350)
... 22 more
Caused by: java.lang.NullPointerException
at
org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl$SargApplier.pickRowGroups(RecordReaderImpl.java:709)
at
org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader.determineRgsToRead(OrcEncodedDataReader.java:618)
at
org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader.callInternal(OrcEncodedDataReader.java:195)
at
org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader.callInternal(OrcEncodedDataReader.java:59)
at
org.apache.hadoop.hive.common.CallableWithNdc.call(CallableWithNdc.java:37)
... 4 more
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-11222) LLAP: occasional NPE in parallel queries in ORC reader
[
https://issues.apache.org/jira/browse/HIVE-11222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14623027#comment-14623027
]
Sergey Shelukhin commented on HIVE-11222:
-
Assertion error was just to get a better error if the fix doesn't work. Right
now there's blah.getBlah().getBlah() line there and it's not clear what is null.
The race was that we cache metadata with necessary indexes only, not with all
of them, and add additional ones to cached object as needed.
So if query X is reading cols 0,1 and query Y cols 0,2, from the same stripe,
at the same time on the same machine, they both can see nothing in cache; read
in parallel; X would cache with 0,1, then Y would try to cache with 0,2, and
get an old object from X from the cache instead. Then the index would be null
for column 2 when it's needed.
The condition is rare enough that it's simpler, code-wise, to double check and
reread than it is to reconcile.
LLAP: occasional NPE in parallel queries in ORC reader
--
Key: HIVE-11222
URL: https://issues.apache.org/jira/browse/HIVE-11222
Project: Hive
Issue Type: Sub-task
Reporter: Sergey Shelukhin
Assignee: Sergey Shelukhin
Fix For: llap
Attachments: HIVE-11222.patch
{noformat}
Caused by: java.io.IOException: java.lang.NullPointerException
at
org.apache.hadoop.hive.llap.io.api.impl.LlapInputFormat$LlapRecordReader.rethrowErrorIfAny(LlapInputFormat.java:275)
at
org.apache.hadoop.hive.llap.io.api.impl.LlapInputFormat$LlapRecordReader.nextCvb(LlapInputFormat.java:227)
at
org.apache.hadoop.hive.llap.io.api.impl.LlapInputFormat$LlapRecordReader.next(LlapInputFormat.java:155)
at
org.apache.hadoop.hive.llap.io.api.impl.LlapInputFormat$LlapRecordReader.next(LlapInputFormat.java:101)
at
org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.doNext(HiveContextAwareRecordReader.java:350)
... 22 more
Caused by: java.lang.NullPointerException
at
org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl$SargApplier.pickRowGroups(RecordReaderImpl.java:709)
at
org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader.determineRgsToRead(OrcEncodedDataReader.java:618)
at
org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader.callInternal(OrcEncodedDataReader.java:195)
at
org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader.callInternal(OrcEncodedDataReader.java:59)
at
org.apache.hadoop.hive.common.CallableWithNdc.call(CallableWithNdc.java:37)
... 4 more
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-10048) JDBC - Support SSL encryption regardless of Authentication mechanism
[ https://issues.apache.org/jira/browse/HIVE-10048?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14623030#comment-14623030 ] Lenni Kuff commented on HIVE-10048: --- [~prasadm] - Can you take a look? JDBC - Support SSL encryption regardless of Authentication mechanism Key: HIVE-10048 URL: https://issues.apache.org/jira/browse/HIVE-10048 Project: Hive Issue Type: Improvement Components: JDBC Affects Versions: 1.0.0 Reporter: Mubashir Kazia Assignee: Mubashir Kazia Labels: newbie, patch Attachments: HIVE-10048.1.patch JDBC driver currently only supports SSL Transport if the Authentication mechanism is SASL Plain with username and password. SSL transport should be decoupled from Authentication mechanism. If the customer chooses to do Kerberos Authentication and SSL encryption over the wire it should be supported. The Server side already supports this but the driver does not. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9152) Dynamic Partition Pruning [Spark Branch]
[
https://issues.apache.org/jira/browse/HIVE-9152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14623046#comment-14623046
]
Hive QA commented on HIVE-9152:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12744811/HIVE-9152.10-spark.patch
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 7609 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.initializationError
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_spark_dynamic_partition_pruning_2
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/928/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/928/console
Test logs:
http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-SPARK-Build-928/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12744811 - PreCommit-HIVE-SPARK-Build
Dynamic Partition Pruning [Spark Branch]
Key: HIVE-9152
URL: https://issues.apache.org/jira/browse/HIVE-9152
Project: Hive
Issue Type: Sub-task
Components: Spark
Affects Versions: spark-branch
Reporter: Brock Noland
Assignee: Chao Sun
Attachments: HIVE-9152.1-spark.patch, HIVE-9152.10-spark.patch,
HIVE-9152.2-spark.patch, HIVE-9152.3-spark.patch, HIVE-9152.4-spark.patch,
HIVE-9152.5-spark.patch, HIVE-9152.6-spark.patch, HIVE-9152.8-spark.patch,
HIVE-9152.9-spark.patch
Tez implemented dynamic partition pruning in HIVE-7826. This is a nice
optimization and we should implement the same in HOS.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-11228) Mutation API should use semi-shared locks.
[ https://issues.apache.org/jira/browse/HIVE-11228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14623052#comment-14623052 ] Eugene Koifman commented on HIVE-11228: --- +1 Mutation API should use semi-shared locks. -- Key: HIVE-11228 URL: https://issues.apache.org/jira/browse/HIVE-11228 Project: Hive Issue Type: Bug Components: HCatalog Affects Versions: 2.0.0 Reporter: Elliot West Assignee: Elliot West Labels: streaming_api, transaction, transactions Attachments: HIVE-11228.1.patch An [issue was identified|https://issues.apache.org/jira/browse/HIVE-10165?focusedCommentId=14619640page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14619640] by [~ekoifman] in the streaming mutation API (HIVE-10165) where an insufficiently restrictive lock was being used when issuing updates and deletes to ACID tables and partitions. A shared lock was being used where in fact a semi-shared lock is required. Additionally, the current lock scope targets the entire table, whereas in theory if the table is partitioned, then only the affected partitions are required to participate in the semi-shared lock. However, there are a couple of technical challenges that prevent the locks currently being applied on a per-partition basis: * It is expected that the affected partitions are not known in advance so individual partition locks would need to be acquired as needed. * The API is expected to execute in a clustered environment and so acquiring these locks as on an ‘as needed’ basis presents a risk that the meta store may become overwhelmed. This is expected to be less of an problem when an HBase based meta store is introduced (HIVE-9452). * My understanding is that multiple fine grained lock acquisitions for a single transaction are not possible at present. When they are available they’ll introduce the possibility that deadlocks can occur. This should be better handled when HIVE-9675 is complete. Therefore, as advised, at this time the system will obtain a semi-shared lock on participating tables. Although this will prevent other concurrent writes, it will preserve snapshot isolation when reading. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11222) LLAP: occasional NPE in parallel queries in ORC reader
[
https://issues.apache.org/jira/browse/HIVE-11222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14623054#comment-14623054
]
Prasanth Jayachandran commented on HIVE-11222:
--
Y gets back the Index loaded by X but the missing the column should be filled
right?
Assuming we just have 3 columns, when X puts the index into the cache, the
cache will return indexes with [1, 1, -1] which represent 2nd column index is
not loaded.
Now when Y puts it's index [1, -1, 1] into the cache, the cache should return
[1, 1, 1] meaning all column indexes are present. We should first get, merge
and then put. Right?
LLAP: occasional NPE in parallel queries in ORC reader
--
Key: HIVE-11222
URL: https://issues.apache.org/jira/browse/HIVE-11222
Project: Hive
Issue Type: Sub-task
Reporter: Sergey Shelukhin
Assignee: Sergey Shelukhin
Fix For: llap
Attachments: HIVE-11222.patch
{noformat}
Caused by: java.io.IOException: java.lang.NullPointerException
at
org.apache.hadoop.hive.llap.io.api.impl.LlapInputFormat$LlapRecordReader.rethrowErrorIfAny(LlapInputFormat.java:275)
at
org.apache.hadoop.hive.llap.io.api.impl.LlapInputFormat$LlapRecordReader.nextCvb(LlapInputFormat.java:227)
at
org.apache.hadoop.hive.llap.io.api.impl.LlapInputFormat$LlapRecordReader.next(LlapInputFormat.java:155)
at
org.apache.hadoop.hive.llap.io.api.impl.LlapInputFormat$LlapRecordReader.next(LlapInputFormat.java:101)
at
org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.doNext(HiveContextAwareRecordReader.java:350)
... 22 more
Caused by: java.lang.NullPointerException
at
org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl$SargApplier.pickRowGroups(RecordReaderImpl.java:709)
at
org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader.determineRgsToRead(OrcEncodedDataReader.java:618)
at
org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader.callInternal(OrcEncodedDataReader.java:195)
at
org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader.callInternal(OrcEncodedDataReader.java:59)
at
org.apache.hadoop.hive.common.CallableWithNdc.call(CallableWithNdc.java:37)
... 4 more
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
