date:20180408

[jira] [Commented] (HIVE-18857) Store default value text instead of default value expression in metastore

2018-04-08 Thread Hive QA (JIRA)


[ 
https://issues.apache.org/jira/browse/HIVE-18857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16430113#comment-16430113
 ] 

Hive QA commented on HIVE-18857:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12917949/HIVE-18857.2.patch

{color:red}ERROR:{color} -1 due to build exiting with an error

Test results: 
https://builds.apache.org/job/PreCommit-HIVE-Build/10087/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/10087/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-10087/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit 
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2018-04-09 05:49:46.617
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export 
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ 
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-10087/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2018-04-09 05:49:46.620
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at a263f08 HIVE-18878 : Lower MoveTask Lock Logging to Debug 
(Beluga Behr, Igor Kryvenko via Sahil Takiar)
+ git clean -f -d
+ git checkout master
Already on 'master'
Your branch is up-to-date with 'origin/master'.
+ git reset --hard origin/master
HEAD is now at a263f08 HIVE-18878 : Lower MoveTask Lock Logging to Debug 
(Beluga Behr, Igor Kryvenko via Sahil Takiar)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2018-04-09 05:49:50.167
+ rm -rf ../yetus_PreCommit-HIVE-Build-10087
+ mkdir ../yetus_PreCommit-HIVE-Build-10087
+ git gc
+ cp -R . ../yetus_PreCommit-HIVE-Build-10087
+ mkdir /data/hiveptest/logs/PreCommit-HIVE-Build-10087/yetus
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh 
/data/hiveptest/working/scratch/build.patch
error: a/ql/src/java/org/apache/hadoop/hive/ql/parse/BaseSemanticAnalyzer.java: 
does not exist in index
Going to apply patch with: git apply -p1
+ [[ maven == \m\a\v\e\n ]]
+ rm -rf /data/hiveptest/working/maven/org/apache/hive
+ mvn -B clean install -DskipTests -T 4 -q 
-Dmaven.repo.local=/data/hiveptest/working/maven
protoc-jar: executing: [/tmp/protoc3050485303383953766.exe, --version]
libprotoc 2.5.0
protoc-jar: executing: [/tmp/protoc3050485303383953766.exe, 
-I/data/hiveptest/working/apache-github-source-source/standalone-metastore/src/main/protobuf/org/apache/hadoop/hive/metastore,
 
--java_out=/data/hiveptest/working/apache-github-source-source/standalone-metastore/target/generated-sources,
 
/data/hiveptest/working/apache-github-source-source/standalone-metastore/src/main/protobuf/org/apache/hadoop/hive/metastore/metastore.proto]
ANTLR Parser Generator  Version 3.5.2
Output file 
/data/hiveptest/working/apache-github-source-source/standalone-metastore/target/generated-sources/org/apache/hadoop/hive/metastore/parser/FilterParser.java
 does not exist: must build 
/data/hiveptest/working/apache-github-source-source/standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/parser/Filter.g
org/apache/hadoop/hive/metastore/parser/Filter.g
log4j:WARN No appenders could be found for logger (DataNucleus.Persistence).
log4j:WARN Please initialize the log4j system properly.
DataNucleus Enhancer (version 4.1.17) for API "JDO"
DataNucleus Enhancer completed with success for 39 classes.
ANTLR Parser Generator  Version 3.5.2
Output file 
/data/hiveptest/working/apache-github-source-source/ql/target/generated-sources/antlr3/org/apache/hadoop/hive/ql/parse/HiveLexer.java
 does not exist: must build 
/data/hiveptest/working/apache-github-source-source/ql/src/java/org/apache/hadoop/hive/ql/parse/HiveLexer.g
org/apache/hadoop/hive/ql/parse/HiveLexer.g
Output file

[jira] [Commented] (HIVE-13532) MapredLocalTask should use the same security settings as remote task

2018-04-08 Thread Hive QA (JIRA)


[ 
https://issues.apache.org/jira/browse/HIVE-13532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16430108#comment-16430108
 ] 

Hive QA commented on HIVE-13532:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12918014/HIVE-13532.2.patch

{color:red}ERROR:{color} -1 due to no test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 150 failed/errored test(s), 13584 tests 
executed
*Failed tests:*
{noformat}
TestBeeLineDriver - did not produce a TEST-*.xml file (likely timed out) 
(batchId=252)
TestCopyUtils - did not produce a TEST-*.xml file (likely timed out) 
(batchId=230)
TestDbNotificationListener - did not produce a TEST-*.xml file (likely timed 
out) (batchId=246)
TestDummy - did not produce a TEST-*.xml file (likely timed out) (batchId=252)
TestExportImport - did not produce a TEST-*.xml file (likely timed out) 
(batchId=230)
TestHCatHiveCompatibility - did not produce a TEST-*.xml file (likely timed 
out) (batchId=246)
TestMiniDruidCliDriver - did not produce a TEST-*.xml file (likely timed out) 
(batchId=252)
TestMiniDruidKafkaCliDriver - did not produce a TEST-*.xml file (likely timed 
out) (batchId=252)
TestNegativeCliDriver - did not produce a TEST-*.xml file (likely timed out) 
(batchId=95)

[jira] [Commented] (HIVE-13532) MapredLocalTask should use the same security settings as remote task

2018-04-08 Thread Hive QA (JIRA)


[ 
https://issues.apache.org/jira/browse/HIVE-13532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16430016#comment-16430016
 ] 

Hive QA commented on HIVE-13532:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m  
1s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  8m 
41s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
17s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
45s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
10s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  1m 
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m  
3s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red}  0m 
16s{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 17m 15s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-10086/dev-support/hive-personality.sh
 |
| git revision | master / a263f08 |
| Default Java | 1.8.0_111 |
| asflicense | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-10086/yetus/patch-asflicense-problems.txt
 |
| modules | C: ql U: ql |
| Console output | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-10086/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |


This message was automatically generated.



> MapredLocalTask should use the same security settings as remote task
> 
>
> Key: HIVE-13532
> URL: https://issues.apache.org/jira/browse/HIVE-13532
> Project: Hive
>  Issue Type: Bug
>  Components: Security
>Affects Versions: 1.1.0
> Environment: HADOOP_PROXY_USER is set.
>Reporter: Zhiwen Sun
>Assignee: Changshu Liu
>Priority: Major
> Fix For: 1.3.0, 2.4.0
>
> Attachments: HIVE-13532.1.patch, HIVE-13532.2.patch
>
>
> Map join set HADOOP_USER_NAME should be realuser's username.
> Current, hive set HADOOP_USER_NAME env for mapjoin local process according:
> {quote}
>String endUserName = Utils.getUGI().getShortUserName();
> {quote}
> suppose set HADOOP_PROXY_USER=abc in shell.
> map join local job will have following env:
> {quote}
> HADOOP_USER_NAME=abc
> HADOOP_PROXY_NAME=abc
> {quote}
> this will cause such exception:
> {quote}
> java.io.IOException: 
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.authorize.AuthorizationException):
>  User: abc is not allowed to impersonate 
> {quote}
> I think we should set HADOOP_USER_NAME to realuser:
> {quote}
>String endUserName = Utils.getUGI().getRealUser().getShortUserName();
> {quote}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-19128) Update golden files for spark perf tests

2018-04-08 Thread Hive QA (JIRA)


[ 
https://issues.apache.org/jira/browse/HIVE-19128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16430010#comment-16430010
 ] 

Hive QA commented on HIVE-19128:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12917925/HIVE-19128.patch

{color:red}ERROR:{color} -1 due to no test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 45 failed/errored test(s), 13187 tests 
executed
*Failed tests:*
{noformat}
TestBeeLineDriver - did not produce a TEST-*.xml file (likely timed out) 
(batchId=252)
TestCopyUtils - did not produce a TEST-*.xml file (likely timed out) 
(batchId=230)
TestDbNotificationListener - did not produce a TEST-*.xml file (likely timed 
out) (batchId=246)
TestDummy - did not produce a TEST-*.xml file (likely timed out) (batchId=252)
TestExportImport - did not produce a TEST-*.xml file (likely timed out) 
(batchId=230)
TestHCatHiveCompatibility - did not produce a TEST-*.xml file (likely timed 
out) (batchId=246)
TestMiniDruidCliDriver - did not produce a TEST-*.xml file (likely timed out) 
(batchId=252)
TestMiniDruidKafkaCliDriver - did not produce a TEST-*.xml file (likely timed 
out) (batchId=252)
TestNegativeCliDriver - did not produce a TEST-*.xml file (likely timed out) 
(batchId=95)

[jira] [Commented] (HIVE-13532) MapredLocalTask should use the same security settings as remote task

2018-04-08 Thread Changshu Liu (JIRA)


[ 
https://issues.apache.org/jira/browse/HIVE-13532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16429979#comment-16429979
 ] 

Changshu Liu commented on HIVE-13532:
-

RB: https://reviews.apache.org/r/66496/

> MapredLocalTask should use the same security settings as remote task
> 
>
> Key: HIVE-13532
> URL: https://issues.apache.org/jira/browse/HIVE-13532
> Project: Hive
>  Issue Type: Bug
>  Components: Security
>Affects Versions: 1.1.0
> Environment: HADOOP_PROXY_USER is set.
>Reporter: Zhiwen Sun
>Assignee: Changshu Liu
>Priority: Major
> Fix For: 1.3.0, 2.4.0
>
> Attachments: HIVE-13532.1.patch, HIVE-13532.2.patch
>
>
> Map join set HADOOP_USER_NAME should be realuser's username.
> Current, hive set HADOOP_USER_NAME env for mapjoin local process according:
> {quote}
>String endUserName = Utils.getUGI().getShortUserName();
> {quote}
> suppose set HADOOP_PROXY_USER=abc in shell.
> map join local job will have following env:
> {quote}
> HADOOP_USER_NAME=abc
> HADOOP_PROXY_NAME=abc
> {quote}
> this will cause such exception:
> {quote}
> java.io.IOException: 
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.authorize.AuthorizationException):
>  User: abc is not allowed to impersonate 
> {quote}
> I think we should set HADOOP_USER_NAME to realuser:
> {quote}
>String endUserName = Utils.getUGI().getRealUser().getShortUserName();
> {quote}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-13532) MapredLocalTask should use the same security settings as remote task

2018-04-08 Thread Changshu Liu (JIRA)


 [ 
https://issues.apache.org/jira/browse/HIVE-13532?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Changshu Liu updated HIVE-13532:

Fix Version/s: 2.4.0
   1.3.0

> MapredLocalTask should use the same security settings as remote task
> 
>
> Key: HIVE-13532
> URL: https://issues.apache.org/jira/browse/HIVE-13532
> Project: Hive
>  Issue Type: Bug
>  Components: Security
>Affects Versions: 1.1.0
> Environment: HADOOP_PROXY_USER is set.
>Reporter: Zhiwen Sun
>Assignee: Changshu Liu
>Priority: Major
> Fix For: 1.3.0, 2.4.0
>
> Attachments: HIVE-13532.1.patch, HIVE-13532.2.patch
>
>
> Map join set HADOOP_USER_NAME should be realuser's username.
> Current, hive set HADOOP_USER_NAME env for mapjoin local process according:
> {quote}
>String endUserName = Utils.getUGI().getShortUserName();
> {quote}
> suppose set HADOOP_PROXY_USER=abc in shell.
> map join local job will have following env:
> {quote}
> HADOOP_USER_NAME=abc
> HADOOP_PROXY_NAME=abc
> {quote}
> this will cause such exception:
> {quote}
> java.io.IOException: 
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.authorize.AuthorizationException):
>  User: abc is not allowed to impersonate 
> {quote}
> I think we should set HADOOP_USER_NAME to realuser:
> {quote}
>String endUserName = Utils.getUGI().getRealUser().getShortUserName();
> {quote}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-13532) MapredLocalTask should use the same security settings as remote task

2018-04-08 Thread Changshu Liu (JIRA)


 [ 
https://issues.apache.org/jira/browse/HIVE-13532?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Changshu Liu updated HIVE-13532:

Component/s: Security

> MapredLocalTask should use the same security settings as remote task
> 
>
> Key: HIVE-13532
> URL: https://issues.apache.org/jira/browse/HIVE-13532
> Project: Hive
>  Issue Type: Bug
>  Components: Security
>Affects Versions: 1.1.0
> Environment: HADOOP_PROXY_USER is set.
>Reporter: Zhiwen Sun
>Assignee: Changshu Liu
>Priority: Major
> Attachments: HIVE-13532.1.patch, HIVE-13532.2.patch
>
>
> Map join set HADOOP_USER_NAME should be realuser's username.
> Current, hive set HADOOP_USER_NAME env for mapjoin local process according:
> {quote}
>String endUserName = Utils.getUGI().getShortUserName();
> {quote}
> suppose set HADOOP_PROXY_USER=abc in shell.
> map join local job will have following env:
> {quote}
> HADOOP_USER_NAME=abc
> HADOOP_PROXY_NAME=abc
> {quote}
> this will cause such exception:
> {quote}
> java.io.IOException: 
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.authorize.AuthorizationException):
>  User: abc is not allowed to impersonate 
> {quote}
> I think we should set HADOOP_USER_NAME to realuser:
> {quote}
>String endUserName = Utils.getUGI().getRealUser().getShortUserName();
> {quote}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Issue Comment Deleted] (HIVE-13532) MapredLocalTask should use the same security settings as remote task

2018-04-08 Thread Changshu Liu (JIRA)


 [ 
https://issues.apache.org/jira/browse/HIVE-13532?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Changshu Liu updated HIVE-13532:

Comment: was deleted

(was: attached a patch, it tries to pass both HADOOP_USER_NAME and 
HADOOP_PROXY_USER env to spawn JVM, please review )

> MapredLocalTask should use the same security settings as remote task
> 
>
> Key: HIVE-13532
> URL: https://issues.apache.org/jira/browse/HIVE-13532
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 1.1.0
> Environment: HADOOP_PROXY_USER is set.
>Reporter: Zhiwen Sun
>Assignee: Changshu Liu
>Priority: Major
> Attachments: HIVE-13532.1.patch, HIVE-13532.2.patch
>
>
> Map join set HADOOP_USER_NAME should be realuser's username.
> Current, hive set HADOOP_USER_NAME env for mapjoin local process according:
> {quote}
>String endUserName = Utils.getUGI().getShortUserName();
> {quote}
> suppose set HADOOP_PROXY_USER=abc in shell.
> map join local job will have following env:
> {quote}
> HADOOP_USER_NAME=abc
> HADOOP_PROXY_NAME=abc
> {quote}
> this will cause such exception:
> {quote}
> java.io.IOException: 
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.authorize.AuthorizationException):
>  User: abc is not allowed to impersonate 
> {quote}
> I think we should set HADOOP_USER_NAME to realuser:
> {quote}
>String endUserName = Utils.getUGI().getRealUser().getShortUserName();
> {quote}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-13532) MapredLocalTask should use the same security settings as remote task

2018-04-08 Thread Changshu Liu (JIRA)


 [ 
https://issues.apache.org/jira/browse/HIVE-13532?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Changshu Liu updated HIVE-13532:

Status: Patch Available  (was: In Progress)

attached a patch, it tries to pass both HADOOP_USER_NAME and HADOOP_PROXY_USER 
env to spawn JVM, please review

> MapredLocalTask should use the same security settings as remote task
> 
>
> Key: HIVE-13532
> URL: https://issues.apache.org/jira/browse/HIVE-13532
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 1.1.0
> Environment: HADOOP_PROXY_USER is set.
>Reporter: Zhiwen Sun
>Assignee: Changshu Liu
>Priority: Major
> Attachments: HIVE-13532.1.patch, HIVE-13532.2.patch
>
>
> Map join set HADOOP_USER_NAME should be realuser's username.
> Current, hive set HADOOP_USER_NAME env for mapjoin local process according:
> {quote}
>String endUserName = Utils.getUGI().getShortUserName();
> {quote}
> suppose set HADOOP_PROXY_USER=abc in shell.
> map join local job will have following env:
> {quote}
> HADOOP_USER_NAME=abc
> HADOOP_PROXY_NAME=abc
> {quote}
> this will cause such exception:
> {quote}
> java.io.IOException: 
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.authorize.AuthorizationException):
>  User: abc is not allowed to impersonate 
> {quote}
> I think we should set HADOOP_USER_NAME to realuser:
> {quote}
>String endUserName = Utils.getUGI().getRealUser().getShortUserName();
> {quote}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-13532) MapredLocalTask should use the same security settings as remote task

2018-04-08 Thread Changshu Liu (JIRA)


[ 
https://issues.apache.org/jira/browse/HIVE-13532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16429970#comment-16429970
 ] 

Changshu Liu commented on HIVE-13532:
-

attached a patch, it tries to pass both HADOOP_USER_NAME and HADOOP_PROXY_USER 
env to spawn JVM, please review 

> MapredLocalTask should use the same security settings as remote task
> 
>
> Key: HIVE-13532
> URL: https://issues.apache.org/jira/browse/HIVE-13532
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 1.1.0
> Environment: HADOOP_PROXY_USER is set.
>Reporter: Zhiwen Sun
>Assignee: Changshu Liu
>Priority: Major
> Attachments: HIVE-13532.1.patch, HIVE-13532.2.patch
>
>
> Map join set HADOOP_USER_NAME should be realuser's username.
> Current, hive set HADOOP_USER_NAME env for mapjoin local process according:
> {quote}
>String endUserName = Utils.getUGI().getShortUserName();
> {quote}
> suppose set HADOOP_PROXY_USER=abc in shell.
> map join local job will have following env:
> {quote}
> HADOOP_USER_NAME=abc
> HADOOP_PROXY_NAME=abc
> {quote}
> this will cause such exception:
> {quote}
> java.io.IOException: 
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.authorize.AuthorizationException):
>  User: abc is not allowed to impersonate 
> {quote}
> I think we should set HADOOP_USER_NAME to realuser:
> {quote}
>String endUserName = Utils.getUGI().getRealUser().getShortUserName();
> {quote}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-13532) MapredLocalTask should use the same security settings as remote task

2018-04-08 Thread Changshu Liu (JIRA)


 [ 
https://issues.apache.org/jira/browse/HIVE-13532?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Changshu Liu updated HIVE-13532:

Attachment: HIVE-13532.2.patch

> MapredLocalTask should use the same security settings as remote task
> 
>
> Key: HIVE-13532
> URL: https://issues.apache.org/jira/browse/HIVE-13532
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 1.1.0
> Environment: HADOOP_PROXY_USER is set.
>Reporter: Zhiwen Sun
>Assignee: Changshu Liu
>Priority: Major
> Attachments: HIVE-13532.1.patch, HIVE-13532.2.patch
>
>
> Map join set HADOOP_USER_NAME should be realuser's username.
> Current, hive set HADOOP_USER_NAME env for mapjoin local process according:
> {quote}
>String endUserName = Utils.getUGI().getShortUserName();
> {quote}
> suppose set HADOOP_PROXY_USER=abc in shell.
> map join local job will have following env:
> {quote}
> HADOOP_USER_NAME=abc
> HADOOP_PROXY_NAME=abc
> {quote}
> this will cause such exception:
> {quote}
> java.io.IOException: 
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.authorize.AuthorizationException):
>  User: abc is not allowed to impersonate 
> {quote}
> I think we should set HADOOP_USER_NAME to realuser:
> {quote}
>String endUserName = Utils.getUGI().getRealUser().getShortUserName();
> {quote}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Comment Edited] (HIVE-13532) MapredLocalTask should use the same security settings as remote task

2018-04-08 Thread Changshu Liu (JIRA)


[ 
https://issues.apache.org/jira/browse/HIVE-13532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16429881#comment-16429881
 ] 

Changshu Liu edited comment on HIVE-13532 at 4/9/18 2:44 AM:
-

We encountered this issue using Hive 1.2 branch, the problem is that the 
MapredLocalTask dose not respect the Hive (Hive CLI and HiveServer2) security 
settings.

We just need to pass the correct Hadoop permission information down to local 
task via env variable.

here are the full call stack if this issue happened:
2018-03-06 19:48:00,343 INFO [main]: exec.HashTableSinkOperator 
(Operator.java:initialize(373)) - Initialization Done 1 HASHTABLESINK done is 
reset.
2018-03-06 19:48:00,343 INFO [main]: exec.HashTableSinkOperator 
(Operator.java:initializeChildren(430)) - Operator 1 HASHTABLESINK initialized
2018-03-06 19:48:00,344 INFO [main]: mr.MapredLocalTask 
(MapredLocalTask.java:initializeOperators(465)) - fetchoperator for $INTNAME 
initialized
2018-03-06 19:48:02,281 ERROR [main]: mr.MapredLocalTask 
(MapredLocalTask.java:executeInProcess(361)) - Hive Runtime Error: Map local 
work failed
java.io.IOException: 
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.authorize.AuthorizationException):
 User: abc is not allowed to impersonate abc
at 
org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:508)
at 
org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask.startForward(MapredLocalTask.java:409)
at 
org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask.startForward(MapredLocalTask.java:384)
at 
org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask.executeInProcess(MapredLocalTask.java:350)
at org.apache.hadoop.hive.ql.exec.mr.ExecDriver.main(ExecDriver.java:746)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
Caused by: 
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.authorize.AuthorizationException):
 User: monarch is not allowed to impersonate monarch
at org.apache.hadoop.ipc.Client.call(Client.java:1476)
at org.apache.hadoop.ipc.Client.call(Client.java:1407)
at 
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229)
at com.sun.proxy.$Proxy11.getFileInfo(Unknown Source)
at 
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:771)




was (Author: csliu):
We encountered this issue using Hive 1.2 branch, the problem is that the 
MapredLocalTask dose not respect the Hive (Hive CLI and HiveServer2) security 
settings, here are the full call stack:

2018-03-06 19:48:00,343 INFO [main]: exec.HashTableSinkOperator 
(Operator.java:initialize(373)) - Initialization Done 1 HASHTABLESINK done is 
reset.
2018-03-06 19:48:00,343 INFO [main]: exec.HashTableSinkOperator 
(Operator.java:initializeChildren(430)) - Operator 1 HASHTABLESINK initialized
2018-03-06 19:48:00,344 INFO [main]: mr.MapredLocalTask 
(MapredLocalTask.java:initializeOperators(465)) - fetchoperator for $INTNAME 
initialized
2018-03-06 19:48:02,281 ERROR [main]: mr.MapredLocalTask 
(MapredLocalTask.java:executeInProcess(361)) - Hive Runtime Error: Map local 
work failed
java.io.IOException: 
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.authorize.AuthorizationException):
 User: abc is not allowed to impersonate abc
at 
org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:508)
at 
org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask.startForward(MapredLocalTask.java:409)
at 
org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask.startForward(MapredLocalTask.java:384)
at 
org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask.executeInProcess(MapredLocalTask.java:350)
at org.apache.hadoop.hive.ql.exec.mr.ExecDriver.main(ExecDriver.java:746)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
Caused by: 
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.authorize.AuthorizationException):
 User: monarch is not allowed to impersonate monarch
at org.apache.hadoop.ipc.Client.call(Client.java:1476)
at org.apache.hadoop.ipc.Client.call(Client.java:1407)
at 
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229)
at com.sun.proxy.$Proxy11.getFileInfo(Unknown Source)
at

[jira] [Commented] (HIVE-19128) Update golden files for spark perf tests

2018-04-08 Thread Hive QA (JIRA)


[ 
https://issues.apache.org/jira/browse/HIVE-19128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16429962#comment-16429962
 ] 

Hive QA commented on HIVE-19128:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
1s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 1s{color} | {color:green} The patch has no whitespace issues. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red}  0m 
12s{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black}  2m 29s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-10085/dev-support/hive-personality.sh
 |
| git revision | master / a263f08 |
| asflicense | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-10085/yetus/patch-asflicense-problems.txt
 |
| modules | C: ql U: ql |
| Console output | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-10085/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |


This message was automatically generated.



> Update golden files for spark perf tests
> 
>
> Key: HIVE-19128
> URL: https://issues.apache.org/jira/browse/HIVE-19128
> Project: Hive
>  Issue Type: Test
>  Components: Spark
>Reporter: Ashutosh Chauhan
>Priority: Major
> Attachments: HIVE-19128.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-18739) Add support for Export from Acid table

2018-04-08 Thread Hive QA (JIRA)


[ 
https://issues.apache.org/jira/browse/HIVE-18739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16429957#comment-16429957
 ] 

Hive QA commented on HIVE-18739:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12918005/HIVE-18739.17.patch

{color:green}SUCCESS:{color} +1 due to 4 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 155 failed/errored test(s), 13599 tests 
executed
*Failed tests:*
{noformat}
TestBeeLineDriver - did not produce a TEST-*.xml file (likely timed out) 
(batchId=252)
TestCopyUtils - did not produce a TEST-*.xml file (likely timed out) 
(batchId=230)
TestDbNotificationListener - did not produce a TEST-*.xml file (likely timed 
out) (batchId=246)
TestDummy - did not produce a TEST-*.xml file (likely timed out) (batchId=252)
TestExportImport - did not produce a TEST-*.xml file (likely timed out) 
(batchId=230)
TestHCatHiveCompatibility - did not produce a TEST-*.xml file (likely timed 
out) (batchId=246)
TestMiniDruidCliDriver - did not produce a TEST-*.xml file (likely timed out) 
(batchId=252)
TestMiniDruidKafkaCliDriver - did not produce a TEST-*.xml file (likely timed 
out) (batchId=252)
TestNegativeCliDriver - did not produce a TEST-*.xml file (likely timed out) 
(batchId=96)

[jira] [Updated] (HIVE-18878) Lower MoveTask Lock Logging to Debug

2018-04-08 Thread Ashutosh Chauhan (JIRA)


 [ 
https://issues.apache.org/jira/browse/HIVE-18878?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashutosh Chauhan updated HIVE-18878:

   Resolution: Fixed
Fix Version/s: 3.0.0
   Status: Resolved  (was: Patch Available)

Pushed to master. Thanks, Igor, Beluga!

> Lower MoveTask Lock Logging to Debug
> 
>
> Key: HIVE-18878
> URL: https://issues.apache.org/jira/browse/HIVE-18878
> Project: Hive
>  Issue Type: Improvement
>  Components: Locking
>Affects Versions: 3.0.0
>Reporter: BELUGA BEHR
>Assignee: Igor Kryvenko
>Priority: Minor
>  Labels: noob
> Fix For: 3.0.0
>
> Attachments: HIVE-18878.1.patch, HIVE-18878.2.patch, HIVE-18878.patch
>
>
> https://github.com/apache/hive/blob/cbb9233a3b39ab8489d777fc76f0758c49b69bef/ql/src/java/org/apache/hadoop/hive/ql/exec/MoveTask.java#L233-L234
> {code}
> LOG.info("about to release lock for output: {} lock: {}", output,
>   lock.getHiveLockObject().getName());
> try {
>   lockMgr.unlock(lock);
> } catch (LockException le) {
> {code}
> Change this to _debug_ level logging.  The lock manager's 'unlock' method 
> should have additional logging available.  Knowing that the unlock is about 
> to take place is only helpful for debugging, not for normal operation.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-18878) Lower MoveTask Lock Logging to Debug

2018-04-08 Thread Ashutosh Chauhan (JIRA)


[ 
https://issues.apache.org/jira/browse/HIVE-18878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16429949#comment-16429949
 ] 

Ashutosh Chauhan commented on HIVE-18878:
-

+1

> Lower MoveTask Lock Logging to Debug
> 
>
> Key: HIVE-18878
> URL: https://issues.apache.org/jira/browse/HIVE-18878
> Project: Hive
>  Issue Type: Improvement
>  Components: Locking
>Affects Versions: 3.0.0
>Reporter: BELUGA BEHR
>Assignee: Igor Kryvenko
>Priority: Minor
>  Labels: noob
> Attachments: HIVE-18878.1.patch, HIVE-18878.2.patch, HIVE-18878.patch
>
>
> https://github.com/apache/hive/blob/cbb9233a3b39ab8489d777fc76f0758c49b69bef/ql/src/java/org/apache/hadoop/hive/ql/exec/MoveTask.java#L233-L234
> {code}
> LOG.info("about to release lock for output: {} lock: {}", output,
>   lock.getHiveLockObject().getName());
> try {
>   lockMgr.unlock(lock);
> } catch (LockException le) {
> {code}
> Change this to _debug_ level logging.  The lock manager's 'unlock' method 
> should have additional logging available.  Knowing that the unlock is about 
> to take place is only helpful for debugging, not for normal operation.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-18886) ACID: NPE on unexplained mysql exceptions

2018-04-08 Thread Ashutosh Chauhan (JIRA)


 [ 
https://issues.apache.org/jira/browse/HIVE-18886?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashutosh Chauhan updated HIVE-18886:

   Resolution: Fixed
Fix Version/s: 3.0.0
   Status: Resolved  (was: Patch Available)

> ACID: NPE on unexplained mysql exceptions 
> --
>
> Key: HIVE-18886
> URL: https://issues.apache.org/jira/browse/HIVE-18886
> Project: Hive
>  Issue Type: Bug
>  Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Gopal V
>Assignee: Gopal V
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HIVE-18886.1.patch
>
>
> At 200+ sessions on a single HS2, the DbLock impl fails to propagate mysql 
> exceptions
> {code}
> 2018-03-06T22:55:16,197 ERROR [HiveServer2-Background-Pool: Thread-12867]: 
> ql.Driver (:()) - FAILED: Error in acquiring locks: null
> java.lang.NullPointerException
> at 
> org.apache.hadoop.hive.metastore.DatabaseProduct.isDeadlock(DatabaseProduct.java:56)
> at 
> org.apache.hadoop.hive.metastore.txn.TxnHandler.checkRetryable(TxnHandler.java:2459)
> at 
> org.apache.hadoop.hive.metastore.txn.TxnHandler.getOpenTxns(TxnHandler.java:499)
> {code}
> {code}
> return e instanceof SQLTransactionRollbackException
> || ((dbProduct == MYSQL || dbProduct == POSTGRES || dbProduct == 
> SQLSERVER)
> && e.getSQLState().equals("40001"))
> || (dbProduct == POSTGRES && e.getSQLState().equals("40P01"))
> || (dbProduct == ORACLE && (e.getMessage().contains("deadlock 
> detected")
> || e.getMessage().contains("can't serialize access for this 
> transaction")));
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-18739) Add support for Export from Acid table

2018-04-08 Thread Hive QA (JIRA)


[ 
https://issues.apache.org/jira/browse/HIVE-18739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16429939#comment-16429939
 ] 

Hive QA commented on HIVE-18739:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m  
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
37s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  8m 
 9s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
34s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
 8s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
18s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m  
9s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  1m 
34s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 
53s{color} | {color:red} ql: The patch generated 69 new + 1097 unchanged - 19 
fixed = 1166 total (was 1116) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
19s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red}  0m 
16s{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 19m 37s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-10084/dev-support/hive-personality.sh
 |
| git revision | master / a2eb16d |
| Default Java | 1.8.0_111 |
| checkstyle | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-10084/yetus/diff-checkstyle-ql.txt
 |
| asflicense | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-10084/yetus/patch-asflicense-problems.txt
 |
| modules | C: common ql U: . |
| Console output | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-10084/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |


This message was automatically generated.



> Add support for Export from Acid table
> --
>
> Key: HIVE-18739
> URL: https://issues.apache.org/jira/browse/HIVE-18739
> Project: Hive
>  Issue Type: Sub-task
>  Components: Transactions
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>Priority: Major
> Attachments: HIVE-18739.01.patch, HIVE-18739.04.patch, 
> HIVE-18739.04.patch, HIVE-18739.06.patch, HIVE-18739.08.patch, 
> HIVE-18739.09.patch, HIVE-18739.10.patch, HIVE-18739.11.patch, 
> HIVE-18739.12.patch, HIVE-18739.13.patch, HIVE-18739.14.patch, 
> HIVE-18739.15.patch, HIVE-18739.16.patch, HIVE-18739.17.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-18739) Add support for Export from Acid table

2018-04-08 Thread Eugene Koifman (JIRA)


 [ 
https://issues.apache.org/jira/browse/HIVE-18739?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eugene Koifman updated HIVE-18739:
--
Attachment: HIVE-18739.17.patch

> Add support for Export from Acid table
> --
>
> Key: HIVE-18739
> URL: https://issues.apache.org/jira/browse/HIVE-18739
> Project: Hive
>  Issue Type: Sub-task
>  Components: Transactions
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>Priority: Major
> Attachments: HIVE-18739.01.patch, HIVE-18739.04.patch, 
> HIVE-18739.04.patch, HIVE-18739.06.patch, HIVE-18739.08.patch, 
> HIVE-18739.09.patch, HIVE-18739.10.patch, HIVE-18739.11.patch, 
> HIVE-18739.12.patch, HIVE-18739.13.patch, HIVE-18739.14.patch, 
> HIVE-18739.15.patch, HIVE-18739.16.patch, HIVE-18739.17.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-18910) Migrate to Murmur hash for shuffle and bucketing

2018-04-08 Thread Hive QA (JIRA)


[ 
https://issues.apache.org/jira/browse/HIVE-18910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16429918#comment-16429918
 ] 

Hive QA commented on HIVE-18910:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12918002/HIVE-18910.26.patch

{color:green}SUCCESS:{color} +1 due to 15 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 108 failed/errored test(s), 13183 tests 
executed
*Failed tests:*
{noformat}
TestBeeLineDriver - did not produce a TEST-*.xml file (likely timed out) 
(batchId=251)
TestCopyUtils - did not produce a TEST-*.xml file (likely timed out) 
(batchId=229)
TestDbNotificationListener - did not produce a TEST-*.xml file (likely timed 
out) (batchId=245)
TestDummy - did not produce a TEST-*.xml file (likely timed out) (batchId=251)
TestExportImport - did not produce a TEST-*.xml file (likely timed out) 
(batchId=229)
TestHCatHiveCompatibility - did not produce a TEST-*.xml file (likely timed 
out) (batchId=245)
TestMiniDruidCliDriver - did not produce a TEST-*.xml file (likely timed out) 
(batchId=251)
TestMiniDruidKafkaCliDriver - did not produce a TEST-*.xml file (likely timed 
out) (batchId=251)
TestNegativeCliDriver - did not produce a TEST-*.xml file (likely timed out) 
(batchId=94)

[jira] [Commented] (HIVE-18910) Migrate to Murmur hash for shuffle and bucketing

2018-04-08 Thread Hive QA (JIRA)


[ 
https://issues.apache.org/jira/browse/HIVE-18910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16429909#comment-16429909
 ] 

Hive QA commented on HIVE-18910:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m  
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
22s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  7m 
52s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  4m 
53s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  3m 
16s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  4m 
29s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
10s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red}  0m 
19s{color} | {color:red} streaming in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red}  0m 
58s{color} | {color:red} ql in the patch failed. {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  4m 
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  4m 
56s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 
14s{color} | {color:red} storage-api: The patch generated 3 new + 97 unchanged 
- 3 fixed = 100 total (was 100) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 
21s{color} | {color:red} serde: The patch generated 150 new + 214 unchanged - 3 
fixed = 364 total (was 217) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 
17s{color} | {color:red} hcatalog/streaming: The patch generated 1 new + 257 
unchanged - 0 fixed = 258 total (was 257) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 
14s{color} | {color:red} hcatalog/webhcat/java-client: The patch generated 1 
new + 108 unchanged - 0 fixed = 109 total (was 108) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  1m  
6s{color} | {color:red} ql: The patch generated 26 new + 1586 unchanged - 3 
fixed = 1612 total (was 1589) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red}  0m  
0s{color} | {color:red} The patch 248 line(s) with tabs. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  4m 
50s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red}  0m 
16s{color} | {color:red} The patch generated 2 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 43m 18s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-10083/dev-support/hive-personality.sh
 |
| git revision | master / 6ba33b8 |
| Default Java | 1.8.0_111 |
| mvninstall | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-10083/yetus/patch-mvninstall-hcatalog_streaming.txt
 |
| mvninstall | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-10083/yetus/patch-mvninstall-ql.txt
 |
| checkstyle | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-10083/yetus/diff-checkstyle-storage-api.txt
 |
| checkstyle | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-10083/yetus/diff-checkstyle-serde.txt
 |
| checkstyle | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-10083/yetus/diff-checkstyle-hcatalog_streaming.txt
 |
| checkstyle | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-10083/yetus/diff-checkstyle-hcatalog_webhcat_java-client.txt
 |
| checkstyle | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-10083/yetus/diff-checkstyle-ql.txt
 |
| whitespace |

[jira] [Updated] (HIVE-19033) Provide an option to purge LLAP IO cache

2018-04-08 Thread Prasanth Jayachandran (JIRA)


 [ 
https://issues.apache.org/jira/browse/HIVE-19033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Prasanth Jayachandran updated HIVE-19033:
-
Resolution: Fixed
Status: Resolved  (was: Patch Available)

Recent test failures are unrelated. Committed to master. 

> Provide an option to purge LLAP IO cache
> 
>
> Key: HIVE-19033
> URL: https://issues.apache.org/jira/browse/HIVE-19033
> Project: Hive
>  Issue Type: Improvement
>  Components: llap
>Affects Versions: 3.0.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HIVE-19033.1.patch, HIVE-19033.2.patch, 
> HIVE-19033.3.patch, HIVE-19033.4.patch, HIVE-19033.5.patch, 
> HIVE-19033.6.patch, HIVE-19033.7.patch, HIVE-19033.8.patch, HIVE-19033.8.patch
>
>
> Provide an API endpoint that will trigger purging of LLAP IO cache. Also CLI 
> tool to invoke the endpoint of all LLAP daemons. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-18910) Migrate to Murmur hash for shuffle and bucketing

2018-04-08 Thread Deepak Jaiswal (JIRA)


 [ 
https://issues.apache.org/jira/browse/HIVE-18910?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Deepak Jaiswal updated HIVE-18910:
--
Attachment: HIVE-18910.26.patch

> Migrate to Murmur hash for shuffle and bucketing
> 
>
> Key: HIVE-18910
> URL: https://issues.apache.org/jira/browse/HIVE-18910
> Project: Hive
>  Issue Type: Task
>Reporter: Deepak Jaiswal
>Assignee: Deepak Jaiswal
>Priority: Major
> Attachments: HIVE-18910.1.patch, HIVE-18910.10.patch, 
> HIVE-18910.11.patch, HIVE-18910.12.patch, HIVE-18910.13.patch, 
> HIVE-18910.14.patch, HIVE-18910.15.patch, HIVE-18910.16.patch, 
> HIVE-18910.17.patch, HIVE-18910.18.patch, HIVE-18910.19.patch, 
> HIVE-18910.2.patch, HIVE-18910.20.patch, HIVE-18910.21.patch, 
> HIVE-18910.22.patch, HIVE-18910.23.patch, HIVE-18910.24.patch, 
> HIVE-18910.25.patch, HIVE-18910.26.patch, HIVE-18910.3.patch, 
> HIVE-18910.4.patch, HIVE-18910.5.patch, HIVE-18910.6.patch, 
> HIVE-18910.7.patch, HIVE-18910.8.patch, HIVE-18910.9.patch
>
>
> Hive uses JAVA hash which is not as good as murmur for better distribution 
> and efficiency in bucketing a table.
> Migrate to murmur hash but still keep backward compatibility for existing 
> users so that they dont have to reload the existing tables.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work started] (HIVE-13532) MapredLocalTask should use the same security settings as remote task

2018-04-08 Thread Changshu Liu (JIRA)


 [ 
https://issues.apache.org/jira/browse/HIVE-13532?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Work on HIVE-13532 started by Changshu Liu.
---
> MapredLocalTask should use the same security settings as remote task
> 
>
> Key: HIVE-13532
> URL: https://issues.apache.org/jira/browse/HIVE-13532
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 1.1.0
> Environment: HADOOP_PROXY_USER is set.
>Reporter: Zhiwen Sun
>Assignee: Changshu Liu
>Priority: Major
> Attachments: HIVE-13532.1.patch
>
>
> Map join set HADOOP_USER_NAME should be realuser's username.
> Current, hive set HADOOP_USER_NAME env for mapjoin local process according:
> {quote}
>String endUserName = Utils.getUGI().getShortUserName();
> {quote}
> suppose set HADOOP_PROXY_USER=abc in shell.
> map join local job will have following env:
> {quote}
> HADOOP_USER_NAME=abc
> HADOOP_PROXY_NAME=abc
> {quote}
> this will cause such exception:
> {quote}
> java.io.IOException: 
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.authorize.AuthorizationException):
>  User: abc is not allowed to impersonate 
> {quote}
> I think we should set HADOOP_USER_NAME to realuser:
> {quote}
>String endUserName = Utils.getUGI().getRealUser().getShortUserName();
> {quote}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-13532) MapredLocalTask should use the same security settings as remote task

2018-04-08 Thread Changshu Liu (JIRA)


 [ 
https://issues.apache.org/jira/browse/HIVE-13532?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Changshu Liu updated HIVE-13532:

Attachment: HIVE-13532.1.patch

> MapredLocalTask should use the same security settings as remote task
> 
>
> Key: HIVE-13532
> URL: https://issues.apache.org/jira/browse/HIVE-13532
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 1.1.0
> Environment: HADOOP_PROXY_USER is set.
>Reporter: Zhiwen Sun
>Assignee: Changshu Liu
>Priority: Major
> Attachments: HIVE-13532.1.patch
>
>
> Map join set HADOOP_USER_NAME should be realuser's username.
> Current, hive set HADOOP_USER_NAME env for mapjoin local process according:
> {quote}
>String endUserName = Utils.getUGI().getShortUserName();
> {quote}
> suppose set HADOOP_PROXY_USER=abc in shell.
> map join local job will have following env:
> {quote}
> HADOOP_USER_NAME=abc
> HADOOP_PROXY_NAME=abc
> {quote}
> this will cause such exception:
> {quote}
> java.io.IOException: 
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.authorize.AuthorizationException):
>  User: abc is not allowed to impersonate 
> {quote}
> I think we should set HADOOP_USER_NAME to realuser:
> {quote}
>String endUserName = Utils.getUGI().getRealUser().getShortUserName();
> {quote}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-13532) MapredLocalTask should use the same security settings as remote task

2018-04-08 Thread Changshu Liu (JIRA)


[ 
https://issues.apache.org/jira/browse/HIVE-13532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16429881#comment-16429881
 ] 

Changshu Liu commented on HIVE-13532:
-

We encountered this issue using Hive 1.2 branch, the problem is that the 
MapredLocalTask dose not respect the Hive (Hive CLI and HiveServer2) security 
settings, here are the full call stack:

2018-03-06 19:48:00,343 INFO [main]: exec.HashTableSinkOperator 
(Operator.java:initialize(373)) - Initialization Done 1 HASHTABLESINK done is 
reset.
2018-03-06 19:48:00,343 INFO [main]: exec.HashTableSinkOperator 
(Operator.java:initializeChildren(430)) - Operator 1 HASHTABLESINK initialized
2018-03-06 19:48:00,344 INFO [main]: mr.MapredLocalTask 
(MapredLocalTask.java:initializeOperators(465)) - fetchoperator for $INTNAME 
initialized
2018-03-06 19:48:02,281 ERROR [main]: mr.MapredLocalTask 
(MapredLocalTask.java:executeInProcess(361)) - Hive Runtime Error: Map local 
work failed
java.io.IOException: 
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.authorize.AuthorizationException):
 User: abc is not allowed to impersonate abc
at 
org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:508)
at 
org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask.startForward(MapredLocalTask.java:409)
at 
org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask.startForward(MapredLocalTask.java:384)
at 
org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask.executeInProcess(MapredLocalTask.java:350)
at org.apache.hadoop.hive.ql.exec.mr.ExecDriver.main(ExecDriver.java:746)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
Caused by: 
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.authorize.AuthorizationException):
 User: monarch is not allowed to impersonate monarch
at org.apache.hadoop.ipc.Client.call(Client.java:1476)
at org.apache.hadoop.ipc.Client.call(Client.java:1407)
at 
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229)
at com.sun.proxy.$Proxy11.getFileInfo(Unknown Source)
at 
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:771)



> MapredLocalTask should use the same security settings as remote task
> 
>
> Key: HIVE-13532
> URL: https://issues.apache.org/jira/browse/HIVE-13532
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 1.1.0
> Environment: HADOOP_PROXY_USER is set.
>Reporter: Zhiwen Sun
>Assignee: Changshu Liu
>Priority: Major
>
> Map join set HADOOP_USER_NAME should be realuser's username.
> Current, hive set HADOOP_USER_NAME env for mapjoin local process according:
> {quote}
>String endUserName = Utils.getUGI().getShortUserName();
> {quote}
> suppose set HADOOP_PROXY_USER=abc in shell.
> map join local job will have following env:
> {quote}
> HADOOP_USER_NAME=abc
> HADOOP_PROXY_NAME=abc
> {quote}
> this will cause such exception:
> {quote}
> java.io.IOException: 
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.authorize.AuthorizationException):
>  User: abc is not allowed to impersonate 
> {quote}
> I think we should set HADOOP_USER_NAME to realuser:
> {quote}
>String endUserName = Utils.getUGI().getRealUser().getShortUserName();
> {quote}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (HIVE-13532) MapredLocalTask should use the same security settings as remote task

2018-04-08 Thread Changshu Liu (JIRA)


 [ 
https://issues.apache.org/jira/browse/HIVE-13532?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Changshu Liu reassigned HIVE-13532:
---

Assignee: Changshu Liu  (was: feiwei)

> MapredLocalTask should use the same security settings as remote task
> 
>
> Key: HIVE-13532
> URL: https://issues.apache.org/jira/browse/HIVE-13532
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 1.1.0
> Environment: HADOOP_PROXY_USER is set.
>Reporter: Zhiwen Sun
>Assignee: Changshu Liu
>Priority: Major
>
> Map join set HADOOP_USER_NAME should be realuser's username.
> Current, hive set HADOOP_USER_NAME env for mapjoin local process according:
> {quote}
>String endUserName = Utils.getUGI().getShortUserName();
> {quote}
> suppose set HADOOP_PROXY_USER=abc in shell.
> map join local job will have following env:
> {quote}
> HADOOP_USER_NAME=abc
> HADOOP_PROXY_NAME=abc
> {quote}
> this will cause such exception:
> {quote}
> java.io.IOException: 
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.authorize.AuthorizationException):
>  User: abc is not allowed to impersonate 
> {quote}
> I think we should set HADOOP_USER_NAME to realuser:
> {quote}
>String endUserName = Utils.getUGI().getRealUser().getShortUserName();
> {quote}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-13532) MapredLocalTask should use the same security settings as remote task

2018-04-08 Thread Changshu Liu (JIRA)


 [ 
https://issues.apache.org/jira/browse/HIVE-13532?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Changshu Liu updated HIVE-13532:

Summary: MapredLocalTask should use the same security settings as remote 
task  (was: Mapjoin should set realuser's username)

> MapredLocalTask should use the same security settings as remote task
> 
>
> Key: HIVE-13532
> URL: https://issues.apache.org/jira/browse/HIVE-13532
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 1.1.0
> Environment: HADOOP_PROXY_USER is set.
>Reporter: Zhiwen Sun
>Assignee: feiwei
>Priority: Major
>
> Map join set HADOOP_USER_NAME should be realuser's username.
> Current, hive set HADOOP_USER_NAME env for mapjoin local process according:
> {quote}
>String endUserName = Utils.getUGI().getShortUserName();
> {quote}
> suppose set HADOOP_PROXY_USER=abc in shell.
> map join local job will have following env:
> {quote}
> HADOOP_USER_NAME=abc
> HADOOP_PROXY_NAME=abc
> {quote}
> this will cause such exception:
> {quote}
> java.io.IOException: 
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.authorize.AuthorizationException):
>  User: abc is not allowed to impersonate 
> {quote}
> I think we should set HADOOP_USER_NAME to realuser:
> {quote}
>String endUserName = Utils.getUGI().getRealUser().getShortUserName();
> {quote}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-18988) Support bootstrap replication of ACID tables

2018-04-08 Thread Hive QA (JIRA)


[ 
https://issues.apache.org/jira/browse/HIVE-18988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16429871#comment-16429871
 ] 

Hive QA commented on HIVE-18988:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12917996/HIVE-18988.02.patch

{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 96 failed/errored test(s), 13254 tests 
executed
*Failed tests:*
{noformat}
TestBeeLineDriver - did not produce a TEST-*.xml file (likely timed out) 
(batchId=252)
TestDbNotificationListener - did not produce a TEST-*.xml file (likely timed 
out) (batchId=246)
TestDummy - did not produce a TEST-*.xml file (likely timed out) (batchId=252)
TestHCatHiveCompatibility - did not produce a TEST-*.xml file (likely timed 
out) (batchId=246)
TestMiniDruidCliDriver - did not produce a TEST-*.xml file (likely timed out) 
(batchId=252)
TestMiniDruidKafkaCliDriver - did not produce a TEST-*.xml file (likely timed 
out) (batchId=252)
TestNegativeCliDriver - did not produce a TEST-*.xml file (likely timed out) 
(batchId=95)

[jira] [Commented] (HIVE-18988) Support bootstrap replication of ACID tables

2018-04-08 Thread Hive QA (JIRA)


[ 
https://issues.apache.org/jira/browse/HIVE-18988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16429852#comment-16429852
 ] 

Hive QA commented on HIVE-18988:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m  
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
42s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  8m 
15s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  3m 
25s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  2m 
 1s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  3m 
30s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
10s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red}  0m 
21s{color} | {color:red} server-extensions in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red}  1m  
1s{color} | {color:red} ql in the patch failed. {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  3m 
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  3m 
19s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 
53s{color} | {color:red} ql: The patch generated 10 new + 506 unchanged - 2 
fixed = 516 total (was 508) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 
36s{color} | {color:red} standalone-metastore: The patch generated 1 new + 1542 
unchanged - 7 fixed = 1543 total (was 1549) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red}  0m  
0s{color} | {color:red} The patch has 23 line(s) that end in whitespace. Use 
git apply --whitespace=fix <>. Refer 
https://git-scm.com/docs/git-apply {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  3m 
11s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red}  0m 
16s{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 31m 28s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-10082/dev-support/hive-personality.sh
 |
| git revision | master / 6ba33b8 |
| Default Java | 1.8.0_111 |
| mvninstall | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-10082/yetus/patch-mvninstall-hcatalog_server-extensions.txt
 |
| mvninstall | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-10082/yetus/patch-mvninstall-ql.txt
 |
| checkstyle | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-10082/yetus/diff-checkstyle-ql.txt
 |
| checkstyle | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-10082/yetus/diff-checkstyle-standalone-metastore.txt
 |
| whitespace | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-10082/yetus/whitespace-eol.txt
 |
| asflicense | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-10082/yetus/patch-asflicense-problems.txt
 |
| modules | C: hcatalog/server-extensions itests/hive-unit ql 
standalone-metastore U: . |
| Console output | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-10082/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |


This message was automatically generated.



> Support bootstrap replication of ACID tables
> 
>
> Key: HIVE-18988
> URL: https://issues.apache.org/jira/browse/HIVE-18988
> Project: Hive
>  Issue Type: Sub-task
>  Components: HiveServer2, repl
>Affects Versions: 3.0.0
>Reporter: Sankar Hariappan
>

[jira] [Updated] (HIVE-18988) Support bootstrap replication of ACID tables

2018-04-08 Thread Sankar Hariappan (JIRA)


 [ 
https://issues.apache.org/jira/browse/HIVE-18988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sankar Hariappan updated HIVE-18988:

Status: Patch Available  (was: Open)

> Support bootstrap replication of ACID tables
> 
>
> Key: HIVE-18988
> URL: https://issues.apache.org/jira/browse/HIVE-18988
> Project: Hive
>  Issue Type: Sub-task
>  Components: HiveServer2, repl
>Affects Versions: 3.0.0
>Reporter: Sankar Hariappan
>Assignee: Sankar Hariappan
>Priority: Major
>  Labels: ACID, DR, pull-request-available, replication
> Fix For: 3.0.0
>
> Attachments: HIVE-18988.01.patch, HIVE-18988.02.patch
>
>
> Bootstrapping of ACID tables, need special handling to replicate a stable 
> state of data.
>  - If ACID feature enables, then perform bootstrap dump for ACID tables with 
> in read txn.
>  -> Dump table/partition metadata.
>  -> Get the list of valid data files for a table using same logic as read txn 
> do.
>  -> Dump latest ValidWriteIdList as per current read txn.
>  - Find the valid last replication state such that it points to event ID of 
> open_txn event of oldest on-going txn.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-18988) Support bootstrap replication of ACID tables

2018-04-08 Thread Sankar Hariappan (JIRA)


[ 
https://issues.apache.org/jira/browse/HIVE-18988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16429839#comment-16429839
 ] 

Sankar Hariappan commented on HIVE-18988:
-

Attached 02.patch with
 * Fixes for check style issues.
 * Fixed an assert failure when partition/table exist without any data file.
 * Updated test case to include empty partition and ACID table converted from 
non-empty non-ACID table.

 

> Support bootstrap replication of ACID tables
> 
>
> Key: HIVE-18988
> URL: https://issues.apache.org/jira/browse/HIVE-18988
> Project: Hive
>  Issue Type: Sub-task
>  Components: HiveServer2, repl
>Affects Versions: 3.0.0
>Reporter: Sankar Hariappan
>Assignee: Sankar Hariappan
>Priority: Major
>  Labels: ACID, DR, pull-request-available, replication
> Fix For: 3.0.0
>
> Attachments: HIVE-18988.01.patch, HIVE-18988.02.patch
>
>
> Bootstrapping of ACID tables, need special handling to replicate a stable 
> state of data.
>  - If ACID feature enables, then perform bootstrap dump for ACID tables with 
> in read txn.
>  -> Dump table/partition metadata.
>  -> Get the list of valid data files for a table using same logic as read txn 
> do.
>  -> Dump latest ValidWriteIdList as per current read txn.
>  - Find the valid last replication state such that it points to event ID of 
> open_txn event of oldest on-going txn.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-18988) Support bootstrap replication of ACID tables

2018-04-08 Thread Sankar Hariappan (JIRA)


 [ 
https://issues.apache.org/jira/browse/HIVE-18988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sankar Hariappan updated HIVE-18988:

Attachment: HIVE-18988.02.patch

> Support bootstrap replication of ACID tables
> 
>
> Key: HIVE-18988
> URL: https://issues.apache.org/jira/browse/HIVE-18988
> Project: Hive
>  Issue Type: Sub-task
>  Components: HiveServer2, repl
>Affects Versions: 3.0.0
>Reporter: Sankar Hariappan
>Assignee: Sankar Hariappan
>Priority: Major
>  Labels: ACID, DR, pull-request-available, replication
> Fix For: 3.0.0
>
> Attachments: HIVE-18988.01.patch, HIVE-18988.02.patch
>
>
> Bootstrapping of ACID tables, need special handling to replicate a stable 
> state of data.
>  - If ACID feature enables, then perform bootstrap dump for ACID tables with 
> in read txn.
>  -> Dump table/partition metadata.
>  -> Get the list of valid data files for a table using same logic as read txn 
> do.
>  -> Dump latest ValidWriteIdList as per current read txn.
>  - Find the valid last replication state such that it points to event ID of 
> open_txn event of oldest on-going txn.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-18739) Add support for Export from Acid table

2018-04-08 Thread Hive QA (JIRA)


[ 
https://issues.apache.org/jira/browse/HIVE-18739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16429836#comment-16429836
 ] 

Hive QA commented on HIVE-18739:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12917992/HIVE-18739.16.patch

{color:green}SUCCESS:{color} +1 due to 3 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 100 failed/errored test(s), 13187 tests 
executed
*Failed tests:*
{noformat}
TestBeeLineDriver - did not produce a TEST-*.xml file (likely timed out) 
(batchId=252)
TestCopyUtils - did not produce a TEST-*.xml file (likely timed out) 
(batchId=230)
TestDbNotificationListener - did not produce a TEST-*.xml file (likely timed 
out) (batchId=246)
TestDummy - did not produce a TEST-*.xml file (likely timed out) (batchId=252)
TestExportImport - did not produce a TEST-*.xml file (likely timed out) 
(batchId=230)
TestHCatHiveCompatibility - did not produce a TEST-*.xml file (likely timed 
out) (batchId=246)
TestMiniDruidCliDriver - did not produce a TEST-*.xml file (likely timed out) 
(batchId=252)
TestMiniDruidKafkaCliDriver - did not produce a TEST-*.xml file (likely timed 
out) (batchId=252)
TestNegativeCliDriver - did not produce a TEST-*.xml file (likely timed out) 
(batchId=95)

[jira] [Updated] (HIVE-18988) Support bootstrap replication of ACID tables

2018-04-08 Thread Sankar Hariappan (JIRA)


 [ 
https://issues.apache.org/jira/browse/HIVE-18988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sankar Hariappan updated HIVE-18988:

Status: Open  (was: Patch Available)

> Support bootstrap replication of ACID tables
> 
>
> Key: HIVE-18988
> URL: https://issues.apache.org/jira/browse/HIVE-18988
> Project: Hive
>  Issue Type: Sub-task
>  Components: HiveServer2, repl
>Affects Versions: 3.0.0
>Reporter: Sankar Hariappan
>Assignee: Sankar Hariappan
>Priority: Major
>  Labels: ACID, DR, pull-request-available, replication
> Fix For: 3.0.0
>
> Attachments: HIVE-18988.01.patch
>
>
> Bootstrapping of ACID tables, need special handling to replicate a stable 
> state of data.
>  - If ACID feature enables, then perform bootstrap dump for ACID tables with 
> in read txn.
>  -> Dump table/partition metadata.
>  -> Get the list of valid data files for a table using same logic as read txn 
> do.
>  -> Dump latest ValidWriteIdList as per current read txn.
>  - Find the valid last replication state such that it points to event ID of 
> open_txn event of oldest on-going txn.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-18739) Add support for Export from Acid table

2018-04-08 Thread Hive QA (JIRA)


[ 
https://issues.apache.org/jira/browse/HIVE-18739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16429811#comment-16429811
 ] 

Hive QA commented on HIVE-18739:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m  
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
45s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  7m 
56s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
34s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
11s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
21s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m  
9s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  2m 
 4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  1m 
43s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 
54s{color} | {color:red} ql: The patch generated 35 new + 1097 unchanged - 19 
fixed = 1132 total (was 1116) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
25s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red}  0m 
16s{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 19m 58s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-10081/dev-support/hive-personality.sh
 |
| git revision | master / 6ba33b8 |
| Default Java | 1.8.0_111 |
| checkstyle | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-10081/yetus/diff-checkstyle-ql.txt
 |
| asflicense | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-10081/yetus/patch-asflicense-problems.txt
 |
| modules | C: common ql U: . |
| Console output | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-10081/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |


This message was automatically generated.



> Add support for Export from Acid table
> --
>
> Key: HIVE-18739
> URL: https://issues.apache.org/jira/browse/HIVE-18739
> Project: Hive
>  Issue Type: Sub-task
>  Components: Transactions
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>Priority: Major
> Attachments: HIVE-18739.01.patch, HIVE-18739.04.patch, 
> HIVE-18739.04.patch, HIVE-18739.06.patch, HIVE-18739.08.patch, 
> HIVE-18739.09.patch, HIVE-18739.10.patch, HIVE-18739.11.patch, 
> HIVE-18739.12.patch, HIVE-18739.13.patch, HIVE-18739.14.patch, 
> HIVE-18739.15.patch, HIVE-18739.16.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-18988) Support bootstrap replication of ACID tables

2018-04-08 Thread Hive QA (JIRA)


[ 
https://issues.apache.org/jira/browse/HIVE-18988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16429802#comment-16429802
 ] 

Hive QA commented on HIVE-18988:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12917990/HIVE-18988.01.patch

{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 99 failed/errored test(s), 13254 tests 
executed
*Failed tests:*
{noformat}
TestBeeLineDriver - did not produce a TEST-*.xml file (likely timed out) 
(batchId=252)
TestDbNotificationListener - did not produce a TEST-*.xml file (likely timed 
out) (batchId=246)
TestDummy - did not produce a TEST-*.xml file (likely timed out) (batchId=252)
TestHCatHiveCompatibility - did not produce a TEST-*.xml file (likely timed 
out) (batchId=246)
TestMiniDruidCliDriver - did not produce a TEST-*.xml file (likely timed out) 
(batchId=252)
TestMiniDruidKafkaCliDriver - did not produce a TEST-*.xml file (likely timed 
out) (batchId=252)
TestNegativeCliDriver - did not produce a TEST-*.xml file (likely timed out) 
(batchId=95)

[jira] [Updated] (HIVE-18739) Add support for Export from Acid table

2018-04-08 Thread Eugene Koifman (JIRA)


 [ 
https://issues.apache.org/jira/browse/HIVE-18739?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eugene Koifman updated HIVE-18739:
--
Status: Patch Available  (was: Open)

> Add support for Export from Acid table
> --
>
> Key: HIVE-18739
> URL: https://issues.apache.org/jira/browse/HIVE-18739
> Project: Hive
>  Issue Type: Sub-task
>  Components: Transactions
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>Priority: Major
> Attachments: HIVE-18739.01.patch, HIVE-18739.04.patch, 
> HIVE-18739.04.patch, HIVE-18739.06.patch, HIVE-18739.08.patch, 
> HIVE-18739.09.patch, HIVE-18739.10.patch, HIVE-18739.11.patch, 
> HIVE-18739.12.patch, HIVE-18739.13.patch, HIVE-18739.14.patch, 
> HIVE-18739.15.patch, HIVE-18739.16.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-18739) Add support for Export from Acid table

2018-04-08 Thread Eugene Koifman (JIRA)


 [ 
https://issues.apache.org/jira/browse/HIVE-18739?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eugene Koifman updated HIVE-18739:
--
Status: Open  (was: Patch Available)

> Add support for Export from Acid table
> --
>
> Key: HIVE-18739
> URL: https://issues.apache.org/jira/browse/HIVE-18739
> Project: Hive
>  Issue Type: Sub-task
>  Components: Transactions
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>Priority: Major
> Attachments: HIVE-18739.01.patch, HIVE-18739.04.patch, 
> HIVE-18739.04.patch, HIVE-18739.06.patch, HIVE-18739.08.patch, 
> HIVE-18739.09.patch, HIVE-18739.10.patch, HIVE-18739.11.patch, 
> HIVE-18739.12.patch, HIVE-18739.13.patch, HIVE-18739.14.patch, 
> HIVE-18739.15.patch, HIVE-18739.16.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-18739) Add support for Export from Acid table

2018-04-08 Thread Eugene Koifman (JIRA)


 [ 
https://issues.apache.org/jira/browse/HIVE-18739?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eugene Koifman updated HIVE-18739:
--
Attachment: HIVE-18739.16.patch

> Add support for Export from Acid table
> --
>
> Key: HIVE-18739
> URL: https://issues.apache.org/jira/browse/HIVE-18739
> Project: Hive
>  Issue Type: Sub-task
>  Components: Transactions
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>Priority: Major
> Attachments: HIVE-18739.01.patch, HIVE-18739.04.patch, 
> HIVE-18739.04.patch, HIVE-18739.06.patch, HIVE-18739.08.patch, 
> HIVE-18739.09.patch, HIVE-18739.10.patch, HIVE-18739.11.patch, 
> HIVE-18739.12.patch, HIVE-18739.13.patch, HIVE-18739.14.patch, 
> HIVE-18739.15.patch, HIVE-18739.16.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-18988) Support bootstrap replication of ACID tables

2018-04-08 Thread Hive QA (JIRA)


[ 
https://issues.apache.org/jira/browse/HIVE-18988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16429788#comment-16429788
 ] 

Hive QA commented on HIVE-18988:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m  
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
48s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  7m 
36s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  3m 
16s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
55s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  3m  
1s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
10s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red}  0m 
22s{color} | {color:red} server-extensions in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red}  0m 
57s{color} | {color:red} ql in the patch failed. {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  3m 
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  3m 
13s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 
18s{color} | {color:red} itests/hive-unit: The patch generated 6 new + 9 
unchanged - 0 fixed = 15 total (was 9) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 
49s{color} | {color:red} ql: The patch generated 19 new + 506 unchanged - 2 
fixed = 525 total (was 508) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 
33s{color} | {color:red} standalone-metastore: The patch generated 4 new + 1547 
unchanged - 2 fixed = 1551 total (was 1549) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red}  0m  
1s{color} | {color:red} The patch has 23 line(s) that end in whitespace. Use 
git apply --whitespace=fix <>. Refer 
https://git-scm.com/docs/git-apply {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  2m 
59s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red}  0m 
15s{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 29m 32s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-10080/dev-support/hive-personality.sh
 |
| git revision | master / 6ba33b8 |
| Default Java | 1.8.0_111 |
| mvninstall | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-10080/yetus/patch-mvninstall-hcatalog_server-extensions.txt
 |
| mvninstall | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-10080/yetus/patch-mvninstall-ql.txt
 |
| checkstyle | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-10080/yetus/diff-checkstyle-itests_hive-unit.txt
 |
| checkstyle | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-10080/yetus/diff-checkstyle-ql.txt
 |
| checkstyle | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-10080/yetus/diff-checkstyle-standalone-metastore.txt
 |
| whitespace | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-10080/yetus/whitespace-eol.txt
 |
| asflicense | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-10080/yetus/patch-asflicense-problems.txt
 |
| modules | C: hcatalog/server-extensions itests/hive-unit ql 
standalone-metastore U: . |
| Console output | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-10080/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |


This message was automatically generated.



> Support bootstrap replication of ACID tables
>

[jira] [Commented] (HIVE-19089) Create/Replicate Allocate write-id event

2018-04-08 Thread Hive QA (JIRA)


[ 
https://issues.apache.org/jira/browse/HIVE-19089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16429779#comment-16429779
 ] 

Hive QA commented on HIVE-19089:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12917986/HIVE-19089.04.patch

{color:green}SUCCESS:{color} +1 due to 5 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 101 failed/errored test(s), 13186 tests 
executed
*Failed tests:*
{noformat}
TestBeeLineDriver - did not produce a TEST-*.xml file (likely timed out) 
(batchId=252)
TestCopyUtils - did not produce a TEST-*.xml file (likely timed out) 
(batchId=230)
TestDbNotificationListener - did not produce a TEST-*.xml file (likely timed 
out) (batchId=246)
TestDummy - did not produce a TEST-*.xml file (likely timed out) (batchId=252)
TestExportImport - did not produce a TEST-*.xml file (likely timed out) 
(batchId=230)
TestHCatHiveCompatibility - did not produce a TEST-*.xml file (likely timed 
out) (batchId=246)
TestMiniDruidCliDriver - did not produce a TEST-*.xml file (likely timed out) 
(batchId=252)
TestMiniDruidKafkaCliDriver - did not produce a TEST-*.xml file (likely timed 
out) (batchId=252)
TestNegativeCliDriver - did not produce a TEST-*.xml file (likely timed out) 
(batchId=95)

[jira] [Commented] (HIVE-18988) Support bootstrap replication of ACID tables

2018-04-08 Thread Sankar Hariappan (JIRA)


[ 
https://issues.apache.org/jira/browse/HIVE-18988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16429769#comment-16429769
 ] 

Sankar Hariappan commented on HIVE-18988:
-

Attached 01.patch with
 * Bootstrap dump/load for ACID/MM tables.
 * ReplCopyTask to copy sub-directories too.
 * MoveTask to move the files with sub-directories to avoid missing any data.

> Support bootstrap replication of ACID tables
> 
>
> Key: HIVE-18988
> URL: https://issues.apache.org/jira/browse/HIVE-18988
> Project: Hive
>  Issue Type: Sub-task
>  Components: HiveServer2, repl
>Affects Versions: 3.0.0
>Reporter: Sankar Hariappan
>Assignee: Sankar Hariappan
>Priority: Major
>  Labels: ACID, DR, pull-request-available, replication
> Fix For: 3.0.0
>
> Attachments: HIVE-18988.01.patch
>
>
> Bootstrapping of ACID tables, need special handling to replicate a stable 
> state of data.
>  - If ACID feature enables, then perform bootstrap dump for ACID tables with 
> in read txn.
>  -> Dump table/partition metadata.
>  -> Get the list of valid data files for a table using same logic as read txn 
> do.
>  -> Dump latest ValidWriteIdList as per current read txn.
>  - Find the valid last replication state such that it points to event ID of 
> open_txn event of oldest on-going txn.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-18988) Support bootstrap replication of ACID tables

2018-04-08 Thread Sankar Hariappan (JIRA)


 [ 
https://issues.apache.org/jira/browse/HIVE-18988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sankar Hariappan updated HIVE-18988:

Status: Patch Available  (was: In Progress)

> Support bootstrap replication of ACID tables
> 
>
> Key: HIVE-18988
> URL: https://issues.apache.org/jira/browse/HIVE-18988
> Project: Hive
>  Issue Type: Sub-task
>  Components: HiveServer2, repl
>Affects Versions: 3.0.0
>Reporter: Sankar Hariappan
>Assignee: Sankar Hariappan
>Priority: Major
>  Labels: ACID, DR, pull-request-available, replication
> Fix For: 3.0.0
>
> Attachments: HIVE-18988.01.patch
>
>
> Bootstrapping of ACID tables, need special handling to replicate a stable 
> state of data.
>  - If ACID feature enables, then perform bootstrap dump for ACID tables with 
> in read txn.
>  -> Dump table/partition metadata.
>  -> Get the list of valid data files for a table using same logic as read txn 
> do.
>  -> Dump latest ValidWriteIdList as per current read txn.
>  - Find the valid last replication state such that it points to event ID of 
> open_txn event of oldest on-going txn.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-18988) Support bootstrap replication of ACID tables

2018-04-08 Thread Sankar Hariappan (JIRA)


 [ 
https://issues.apache.org/jira/browse/HIVE-18988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sankar Hariappan updated HIVE-18988:

Attachment: HIVE-18988.01.patch

> Support bootstrap replication of ACID tables
> 
>
> Key: HIVE-18988
> URL: https://issues.apache.org/jira/browse/HIVE-18988
> Project: Hive
>  Issue Type: Sub-task
>  Components: HiveServer2, repl
>Affects Versions: 3.0.0
>Reporter: Sankar Hariappan
>Assignee: Sankar Hariappan
>Priority: Major
>  Labels: ACID, DR, pull-request-available, replication
> Fix For: 3.0.0
>
> Attachments: HIVE-18988.01.patch
>
>
> Bootstrapping of ACID tables, need special handling to replicate a stable 
> state of data.
>  - If ACID feature enables, then perform bootstrap dump for ACID tables with 
> in read txn.
>  -> Dump table/partition metadata.
>  -> Get the list of valid data files for a table using same logic as read txn 
> do.
>  -> Dump latest ValidWriteIdList as per current read txn.
>  - Find the valid last replication state such that it points to event ID of 
> open_txn event of oldest on-going txn.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-18988) Support bootstrap replication of ACID tables

2018-04-08 Thread ASF GitHub Bot (JIRA)


 [ 
https://issues.apache.org/jira/browse/HIVE-18988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ASF GitHub Bot updated HIVE-18988:
--
Labels: ACID DR pull-request-available replication  (was: ACID DR 
replication)

> Support bootstrap replication of ACID tables
> 
>
> Key: HIVE-18988
> URL: https://issues.apache.org/jira/browse/HIVE-18988
> Project: Hive
>  Issue Type: Sub-task
>  Components: HiveServer2, repl
>Affects Versions: 3.0.0
>Reporter: Sankar Hariappan
>Assignee: Sankar Hariappan
>Priority: Major
>  Labels: ACID, DR, pull-request-available, replication
> Fix For: 3.0.0
>
>
> Bootstrapping of ACID tables, need special handling to replicate a stable 
> state of data.
>  - If ACID feature enables, then perform bootstrap dump for ACID tables with 
> in read txn.
>  -> Dump table/partition metadata.
>  -> Get the list of valid data files for a table using same logic as read txn 
> do.
>  -> Dump latest ValidWriteIdList as per current read txn.
>  - Find the valid last replication state such that it points to event ID of 
> open_txn event of oldest on-going txn.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-18988) Support bootstrap replication of ACID tables

2018-04-08 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/HIVE-18988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16429767#comment-16429767
 ] 

ASF GitHub Bot commented on HIVE-18988:
---

GitHub user sankarh opened a pull request:

https://github.com/apache/hive/pull/331

HIVE-18988: Support bootstrap replication of ACID tables.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/sankarh/hive HIVE-18988

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/hive/pull/331.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #331


commit a503722c4fb61a024702977a0c81ef948b98d483
Author: Sankar Hariappan 
Date:   2018-04-05T16:57:43Z

HIVE-18988: Support bootstrap replication of ACID tables.




> Support bootstrap replication of ACID tables
> 
>
> Key: HIVE-18988
> URL: https://issues.apache.org/jira/browse/HIVE-18988
> Project: Hive
>  Issue Type: Sub-task
>  Components: HiveServer2, repl
>Affects Versions: 3.0.0
>Reporter: Sankar Hariappan
>Assignee: Sankar Hariappan
>Priority: Major
>  Labels: ACID, DR, pull-request-available, replication
> Fix For: 3.0.0
>
>
> Bootstrapping of ACID tables, need special handling to replicate a stable 
> state of data.
>  - If ACID feature enables, then perform bootstrap dump for ACID tables with 
> in read txn.
>  -> Dump table/partition metadata.
>  -> Get the list of valid data files for a table using same logic as read txn 
> do.
>  -> Dump latest ValidWriteIdList as per current read txn.
>  - Find the valid last replication state such that it points to event ID of 
> open_txn event of oldest on-going txn.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-19089) Create/Replicate Allocate write-id event

2018-04-08 Thread Hive QA (JIRA)


[ 
https://issues.apache.org/jira/browse/HIVE-19089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16429764#comment-16429764
 ] 

Hive QA commented on HIVE-19089:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m  
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
33s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  8m 
37s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  2m 
58s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
51s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  3m  
8s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
10s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red}  0m 
22s{color} | {color:red} server-extensions in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red}  0m 
21s{color} | {color:red} hcatalog-unit in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red}  0m 
40s{color} | {color:red} ql in the patch failed. {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  2m 
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  2m 
56s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 
12s{color} | {color:red} hcatalog/server-extensions: The patch generated 4 new 
+ 8 unchanged - 0 fixed = 12 total (was 8) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
13s{color} | {color:green} The patch hcatalog-unit passed checkstyle {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
53s{color} | {color:green} ql: The patch generated 0 new + 738 unchanged - 6 
fixed = 738 total (was 744) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 
36s{color} | {color:red} standalone-metastore: The patch generated 8 new + 1560 
unchanged - 12 fixed = 1568 total (was 1572) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red}  0m  
0s{color} | {color:red} The patch has 6 line(s) that end in whitespace. Use git 
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply 
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  2m 
56s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red}  0m 
17s{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 28m 57s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-10079/dev-support/hive-personality.sh
 |
| git revision | master / 6ba33b8 |
| Default Java | 1.8.0_111 |
| mvninstall | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-10079/yetus/patch-mvninstall-hcatalog_server-extensions.txt
 |
| mvninstall | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-10079/yetus/patch-mvninstall-itests_hcatalog-unit.txt
 |
| mvninstall | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-10079/yetus/patch-mvninstall-ql.txt
 |
| checkstyle | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-10079/yetus/diff-checkstyle-hcatalog_server-extensions.txt
 |
| checkstyle | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-10079/yetus/diff-checkstyle-standalone-metastore.txt
 |
| whitespace | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-10079/yetus/whitespace-eol.txt
 |
| asflicense | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-10079/yetus/patch-asflicense-problems.txt
 |

[jira] [Commented] (HIVE-18831) Differentiate errors that are thrown by Spark tasks

2018-04-08 Thread Rui Li (JIRA)


[ 
https://issues.apache.org/jira/browse/HIVE-18831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16429750#comment-16429750
 ] 

Rui Li commented on HIVE-18831:
---

Thanks [~stakiar] for the update. The patch looks good to me overall, although 
it needs rebasing. Left only a minor comments on RB.

> Differentiate errors that are thrown by Spark tasks
> ---
>
> Key: HIVE-18831
> URL: https://issues.apache.org/jira/browse/HIVE-18831
> Project: Hive
>  Issue Type: Sub-task
>  Components: Spark
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>Priority: Major
> Attachments: HIVE-18831.1.patch, HIVE-18831.2.patch, 
> HIVE-18831.3.patch, HIVE-18831.4.patch, HIVE-18831.6.patch, 
> HIVE-18831.7.patch, HIVE-18831.8.WIP.patch, HIVE-18831.9.patch, 
> HIVE-18831.90.patch
>
>
> We propagate exceptions from Spark task failures to the client well, but we 
> don't differentiate between errors from HS2 / RSC vs. errors thrown by 
> individual tasks.
> Main motivation is that when the client sees a propagated Spark exception its 
> difficult to know what part of the excution threw the exception.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-19089) Create/Replicate Allocate write-id event

2018-04-08 Thread mahesh kumar behera (JIRA)


 [ 
https://issues.apache.org/jira/browse/HIVE-19089?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

mahesh kumar behera updated HIVE-19089:
---
Attachment: HIVE-19089.04.patch

> Create/Replicate Allocate write-id event
> 
>
> Key: HIVE-19089
> URL: https://issues.apache.org/jira/browse/HIVE-19089
> Project: Hive
>  Issue Type: Sub-task
>  Components: repl, Transactions
>Affects Versions: 3.0.0
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
>  Labels: ACID, DR, replication
> Fix For: 3.0.0
>
> Attachments: HIVE-19089.01.patch, HIVE-19089.02.patch, 
> HIVE-19089.03.patch, HIVE-19089.04.patch
>
>
> *EVENT_ALLOCATE_WRITE_ID*
> *Source Warehouse:*
>  * Create new event type EVENT_ALLOCATE_WRITE_ID with related message format 
> etc.
>  * Capture this event when allocate a table write ID from the sequence table 
> by ACID operation.
>  * Repl dump should read this event from EventNotificationTable and dump the 
> message.
> *Target Warehouse:*
>  * Repl load should read the event from the dump and get the message.
>  * Validate if source txn ID from the event is there in the source-target txn 
> ID map. If not there, just noop the event.
>  * If valid, then Allocate table write ID from sequence table
> *Extend listener notify event API to add two new parameter , dbconn and 
> sqlgenerator to add the events to notification_log table within the same 
> transaction* 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-17498) Does hive have mr-nativetask support refer to MAPREDUCE-2841

2018-04-08 Thread Igloo (JIRA)


[ 
https://issues.apache.org/jira/browse/HIVE-17498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16429741#comment-16429741
 ] 

Igloo commented on HIVE-17498:
--

!image-2018-04-08-20-00-58-680.png!

> Does hive have mr-nativetask support refer to MAPREDUCE-2841
> 
>
> Key: HIVE-17498
> URL: https://issues.apache.org/jira/browse/HIVE-17498
> Project: Hive
>  Issue Type: Bug
>  Components: Hive
>Reporter: Feng Yuan
>Priority: Major
>
> I try to implement a HivePlatform extends 
> org.apache.hadoop.mapred.nativetask.Platform.
> {code}
> /**
>  * Licensed to the Apache Software Foundation (ASF) under one
>  * or more contributor license agreements.  See the NOTICE file
>  * distributed with this work for additional information
>  * regarding copyright ownership.  The ASF licenses this file
>  * to you under the Apache License, Version 2.0 (the
>  * "License"); you may not use this file except in compliance
>  * with the License.  You may obtain a copy of the License at
>  *
>  * http://www.apache.org/licenses/LICENSE-2.0
>  *
>  * Unless required by applicable law or agreed to in writing, software
>  * distributed under the License is distributed on an "AS IS" BASIS,
>  * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
>  * See the License for the specific language governing permissions and
>  * limitations under the License.
>  */
> package org.apache.hadoop.mapred.nativetask;
> import org.apache.hadoop.hive.ql.io.HiveKey;
> import org.apache.hadoop.mapred.JobConf;
> import org.apache.hadoop.mapred.nativetask.serde.INativeSerializer;
> import org.apache.log4j.Logger;
> import java.io.DataInput;
> import java.io.DataOutput;
> import java.io.IOException;
> public class HivePlatform extends Platform {
>   private static final Logger LOG = Logger.getLogger(HivePlatform.class);
>   public HivePlatform() {
>   }
>   @Override
>   public void init() throws IOException {
> registerKey("org.apache.hadoop.hive.ql.io.HiveKey", 
> HiveKeySerializer.class);
> LOG.info("Hive platform inited");
>   }
>   @Override
>   public String name() {
> return "Hive";
>   }
>   @Override
>   public boolean support(String keyClassName, INativeSerializer serializer, 
> JobConf job) {
> if (keyClassNames.contains(keyClassName) && serializer instanceof 
> INativeComparable) {
>   String nativeComparator = Constants.NATIVE_MAPOUT_KEY_COMPARATOR + "." 
> + keyClassName;
>   job.set(nativeComparator, 
> "HivePlatform.HivePlatform::HiveKeyComparator");
>   if (job.get(Constants.NATIVE_CLASS_LIBRARY_BUILDIN) == null) {
> job.set(Constants.NATIVE_CLASS_LIBRARY_BUILDIN, 
> "HivePlatform=libnativetask.so");
>   }
>   return true;
> } else {
>   return false;
> }
>   }
>   @Override
>   public boolean define(Class comparatorClass) {
> return false;
>   }
>   public static class HiveKeySerializer implements INativeComparable, 
> INativeSerializer {
> public HiveKeySerializer() throws ClassNotFoundException, 
> SecurityException, NoSuchMethodException {
> }
> @Override
> public int getLength(HiveKey w) throws IOException {
>   return 4 + w.getLength();
> }
> @Override
> public void serialize(HiveKey w, DataOutput out) throws IOException {
>   w.write(out);
> }
> @Override
> public void deserialize(DataInput in, int length, HiveKey w ) throws 
> IOException {
>   w.readFields(in);
> }
>   }
> }
> {code}
> and throws exceptions:
> {code}
> Error: java.io.IOException: Initialization of all the collectors failed. 
> Error in last collector was :Native output collector cannot be loaded; at 
> org.apache.hadoop.mapred.MapTask.createSortingCollector(MapTask.java:415) at 
> org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:442) at 
> org.apache.hadoop.mapred.MapTask.run(MapTask.java:343) at 
> org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164) at 
> java.security.AccessController.doPrivileged(Native Method) at 
> javax.security.auth.Subject.doAs(Subject.java:422) at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1700)
>  at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158) Caused by: 
> java.io.IOException: Native output collector cannot be loaded; at 
> org.apache.hadoop.mapred.nativetask.NativeMapOutputCollectorDelegator.init(NativeMapOutputCollectorDelegator.java:165)
>  at org.apache.hadoop.mapred.MapTask.createSortingCollector(MapTask.java:402) 
> ... 7 more Caused by: java.io.IOException: /PartitionBucket.h:56:pool is 
> NULL, or comparator is not set 
> /usr/local/hadoop-2.7.3-yarn/lib/native/libnativetask.so.1.0.0(_ZN10NativeTask15HadoopExceptionC2ERKSs+0x76)
>  [0x7ffdcbba6436] 
>

[jira] [Commented] (HIVE-17498) Does hive have mr-nativetask support refer to MAPREDUCE-2841

2018-04-08 Thread Igloo (JIRA)


[ 
https://issues.apache.org/jira/browse/HIVE-17498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16429740#comment-16429740
 ] 

Igloo commented on HIVE-17498:
--

{quote}diff --git 
a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/util/WritableUtils.cc
 
b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/util/WritableUtils.cc
index 243668117b..0b120e1ac9 100644
--- 
a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/util/WritableUtils.cc
+++ 
b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/util/WritableUtils.cc
@@ -29,6 +29,9 @@ KeyValueType JavaClassToKeyValueType(const std::string & 
clazz) {
 if (clazz == "org.apache.hadoop.io.BytesWritable") {
 return BytesType;
 }
+ if (clazz == "org.apache.hadoop.hive.ql.io.HiveKey") {
+ return BytesType;
+ }
 if (clazz == "org.apache.hadoop.io.ByteWritable") {
 return ByteType;
 }
{quote}
 

 

the patch works for me~ 

> Does hive have mr-nativetask support refer to MAPREDUCE-2841
> 
>
> Key: HIVE-17498
> URL: https://issues.apache.org/jira/browse/HIVE-17498
> Project: Hive
>  Issue Type: Bug
>  Components: Hive
>Reporter: Feng Yuan
>Priority: Major
>
> I try to implement a HivePlatform extends 
> org.apache.hadoop.mapred.nativetask.Platform.
> {code}
> /**
>  * Licensed to the Apache Software Foundation (ASF) under one
>  * or more contributor license agreements.  See the NOTICE file
>  * distributed with this work for additional information
>  * regarding copyright ownership.  The ASF licenses this file
>  * to you under the Apache License, Version 2.0 (the
>  * "License"); you may not use this file except in compliance
>  * with the License.  You may obtain a copy of the License at
>  *
>  * http://www.apache.org/licenses/LICENSE-2.0
>  *
>  * Unless required by applicable law or agreed to in writing, software
>  * distributed under the License is distributed on an "AS IS" BASIS,
>  * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
>  * See the License for the specific language governing permissions and
>  * limitations under the License.
>  */
> package org.apache.hadoop.mapred.nativetask;
> import org.apache.hadoop.hive.ql.io.HiveKey;
> import org.apache.hadoop.mapred.JobConf;
> import org.apache.hadoop.mapred.nativetask.serde.INativeSerializer;
> import org.apache.log4j.Logger;
> import java.io.DataInput;
> import java.io.DataOutput;
> import java.io.IOException;
> public class HivePlatform extends Platform {
>   private static final Logger LOG = Logger.getLogger(HivePlatform.class);
>   public HivePlatform() {
>   }
>   @Override
>   public void init() throws IOException {
> registerKey("org.apache.hadoop.hive.ql.io.HiveKey", 
> HiveKeySerializer.class);
> LOG.info("Hive platform inited");
>   }
>   @Override
>   public String name() {
> return "Hive";
>   }
>   @Override
>   public boolean support(String keyClassName, INativeSerializer serializer, 
> JobConf job) {
> if (keyClassNames.contains(keyClassName) && serializer instanceof 
> INativeComparable) {
>   String nativeComparator = Constants.NATIVE_MAPOUT_KEY_COMPARATOR + "." 
> + keyClassName;
>   job.set(nativeComparator, 
> "HivePlatform.HivePlatform::HiveKeyComparator");
>   if (job.get(Constants.NATIVE_CLASS_LIBRARY_BUILDIN) == null) {
> job.set(Constants.NATIVE_CLASS_LIBRARY_BUILDIN, 
> "HivePlatform=libnativetask.so");
>   }
>   return true;
> } else {
>   return false;
> }
>   }
>   @Override
>   public boolean define(Class comparatorClass) {
> return false;
>   }
>   public static class HiveKeySerializer implements INativeComparable, 
> INativeSerializer {
> public HiveKeySerializer() throws ClassNotFoundException, 
> SecurityException, NoSuchMethodException {
> }
> @Override
> public int getLength(HiveKey w) throws IOException {
>   return 4 + w.getLength();
> }
> @Override
> public void serialize(HiveKey w, DataOutput out) throws IOException {
>   w.write(out);
> }
> @Override
> public void deserialize(DataInput in, int length, HiveKey w ) throws 
> IOException {
>   w.readFields(in);
> }
>   }
> }
> {code}
> and throws exceptions:
> {code}
> Error: java.io.IOException: Initialization of all the collectors failed. 
> Error in last collector was :Native output collector cannot be loaded; at 
> org.apache.hadoop.mapred.MapTask.createSortingCollector(MapTask.java:415) at 
> org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:442) at 
> org.apache.hadoop.mapred.MapTask.run(MapTask.java:343) at 
>

[jira] [Commented] (HIVE-18910) Migrate to Murmur hash for shuffle and bucketing

2018-04-08 Thread Hive QA (JIRA)


[ 
https://issues.apache.org/jira/browse/HIVE-18910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16429717#comment-16429717
 ] 

Hive QA commented on HIVE-18910:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12917969/HIVE-18910.25.patch

{color:green}SUCCESS:{color} +1 due to 14 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 284 failed/errored test(s), 13580 tests 
executed
*Failed tests:*
{noformat}
TestBeeLineDriver - did not produce a TEST-*.xml file (likely timed out) 
(batchId=251)
TestCopyUtils - did not produce a TEST-*.xml file (likely timed out) 
(batchId=229)
TestDbNotificationListener - did not produce a TEST-*.xml file (likely timed 
out) (batchId=245)
TestDummy - did not produce a TEST-*.xml file (likely timed out) (batchId=251)
TestExportImport - did not produce a TEST-*.xml file (likely timed out) 
(batchId=229)
TestHCatHiveCompatibility - did not produce a TEST-*.xml file (likely timed 
out) (batchId=245)
TestMiniDruidCliDriver - did not produce a TEST-*.xml file (likely timed out) 
(batchId=251)
TestMiniDruidKafkaCliDriver - did not produce a TEST-*.xml file (likely timed 
out) (batchId=251)
TestNegativeCliDriver - did not produce a TEST-*.xml file (likely timed out) 
(batchId=94)

[jira] [Commented] (HIVE-18910) Migrate to Murmur hash for shuffle and bucketing

2018-04-08 Thread Hive QA (JIRA)


[ 
https://issues.apache.org/jira/browse/HIVE-18910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16429704#comment-16429704
 ] 

Hive QA commented on HIVE-18910:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m  
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
47s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  7m 
13s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  4m 
37s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  3m 
11s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  4m 
24s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m  
9s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red}  0m 
22s{color} | {color:red} streaming in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red}  1m  
0s{color} | {color:red} ql in the patch failed. {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  4m 
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  4m 
49s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 
15s{color} | {color:red} storage-api: The patch generated 3 new + 97 unchanged 
- 3 fixed = 100 total (was 100) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 
23s{color} | {color:red} serde: The patch generated 150 new + 214 unchanged - 3 
fixed = 364 total (was 217) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 
18s{color} | {color:red} hcatalog/streaming: The patch generated 1 new + 257 
unchanged - 0 fixed = 258 total (was 257) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  1m 
11s{color} | {color:red} ql: The patch generated 26 new + 1586 unchanged - 3 
fixed = 1612 total (was 1589) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red}  0m  
0s{color} | {color:red} The patch 248 line(s) with tabs. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  5m  
5s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red}  0m 
15s{color} | {color:red} The patch generated 2 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 43m 23s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-10078/dev-support/hive-personality.sh
 |
| git revision | master / 6ba33b8 |
| Default Java | 1.8.0_111 |
| mvninstall | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-10078/yetus/patch-mvninstall-hcatalog_streaming.txt
 |
| mvninstall | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-10078/yetus/patch-mvninstall-ql.txt
 |
| checkstyle | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-10078/yetus/diff-checkstyle-storage-api.txt
 |
| checkstyle | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-10078/yetus/diff-checkstyle-serde.txt
 |
| checkstyle | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-10078/yetus/diff-checkstyle-hcatalog_streaming.txt
 |
| checkstyle | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-10078/yetus/diff-checkstyle-ql.txt
 |
| whitespace | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-10078/yetus/whitespace-tabs.txt
 |
| asflicense | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-10078/yetus/patch-asflicense-problems.txt
 |
| modules | C: storage-api serde hbase-handler hcatalog/streaming itests 
itests/hive-blobstore itests/hive-unit ql standalone-metastore U: . |
| Console output |

[jira] [Commented] (HIVE-19045) Vectorization: Disable vectorization in non-vectorized Q files

2018-04-08 Thread Hive QA (JIRA)


[ 
https://issues.apache.org/jira/browse/HIVE-19045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16429692#comment-16429692
 ] 

Hive QA commented on HIVE-19045:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12917968/HIVE-19045.04.patch

{color:red}ERROR:{color} -1 due to no test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 177 failed/errored test(s), 13582 tests 
executed
*Failed tests:*
{noformat}
TestBeeLineDriver - did not produce a TEST-*.xml file (likely timed out) 
(batchId=252)
TestCopyUtils - did not produce a TEST-*.xml file (likely timed out) 
(batchId=230)
TestDbNotificationListener - did not produce a TEST-*.xml file (likely timed 
out) (batchId=246)
TestDummy - did not produce a TEST-*.xml file (likely timed out) (batchId=252)
TestExportImport - did not produce a TEST-*.xml file (likely timed out) 
(batchId=230)
TestHCatHiveCompatibility - did not produce a TEST-*.xml file (likely timed 
out) (batchId=246)
TestMiniDruidCliDriver - did not produce a TEST-*.xml file (likely timed out) 
(batchId=252)
TestMiniDruidKafkaCliDriver - did not produce a TEST-*.xml file (likely timed 
out) (batchId=252)
TestNegativeCliDriver - did not produce a TEST-*.xml file (likely timed out) 
(batchId=96)

[jira] [Commented] (HIVE-17498) Does hive have mr-nativetask support refer to MAPREDUCE-2841

2018-04-08 Thread Igloo (JIRA)


[ 
https://issues.apache.org/jira/browse/HIVE-17498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16429684#comment-16429684
 ] 

Igloo commented on HIVE-17498:
--

[~Feng Yuan] I encountered the exactly same problem, how did you fix it at 
last? 

> Does hive have mr-nativetask support refer to MAPREDUCE-2841
> 
>
> Key: HIVE-17498
> URL: https://issues.apache.org/jira/browse/HIVE-17498
> Project: Hive
>  Issue Type: Bug
>  Components: Hive
>Reporter: Feng Yuan
>Priority: Major
>
> I try to implement a HivePlatform extends 
> org.apache.hadoop.mapred.nativetask.Platform.
> {code}
> /**
>  * Licensed to the Apache Software Foundation (ASF) under one
>  * or more contributor license agreements.  See the NOTICE file
>  * distributed with this work for additional information
>  * regarding copyright ownership.  The ASF licenses this file
>  * to you under the Apache License, Version 2.0 (the
>  * "License"); you may not use this file except in compliance
>  * with the License.  You may obtain a copy of the License at
>  *
>  * http://www.apache.org/licenses/LICENSE-2.0
>  *
>  * Unless required by applicable law or agreed to in writing, software
>  * distributed under the License is distributed on an "AS IS" BASIS,
>  * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
>  * See the License for the specific language governing permissions and
>  * limitations under the License.
>  */
> package org.apache.hadoop.mapred.nativetask;
> import org.apache.hadoop.hive.ql.io.HiveKey;
> import org.apache.hadoop.mapred.JobConf;
> import org.apache.hadoop.mapred.nativetask.serde.INativeSerializer;
> import org.apache.log4j.Logger;
> import java.io.DataInput;
> import java.io.DataOutput;
> import java.io.IOException;
> public class HivePlatform extends Platform {
>   private static final Logger LOG = Logger.getLogger(HivePlatform.class);
>   public HivePlatform() {
>   }
>   @Override
>   public void init() throws IOException {
> registerKey("org.apache.hadoop.hive.ql.io.HiveKey", 
> HiveKeySerializer.class);
> LOG.info("Hive platform inited");
>   }
>   @Override
>   public String name() {
> return "Hive";
>   }
>   @Override
>   public boolean support(String keyClassName, INativeSerializer serializer, 
> JobConf job) {
> if (keyClassNames.contains(keyClassName) && serializer instanceof 
> INativeComparable) {
>   String nativeComparator = Constants.NATIVE_MAPOUT_KEY_COMPARATOR + "." 
> + keyClassName;
>   job.set(nativeComparator, 
> "HivePlatform.HivePlatform::HiveKeyComparator");
>   if (job.get(Constants.NATIVE_CLASS_LIBRARY_BUILDIN) == null) {
> job.set(Constants.NATIVE_CLASS_LIBRARY_BUILDIN, 
> "HivePlatform=libnativetask.so");
>   }
>   return true;
> } else {
>   return false;
> }
>   }
>   @Override
>   public boolean define(Class comparatorClass) {
> return false;
>   }
>   public static class HiveKeySerializer implements INativeComparable, 
> INativeSerializer {
> public HiveKeySerializer() throws ClassNotFoundException, 
> SecurityException, NoSuchMethodException {
> }
> @Override
> public int getLength(HiveKey w) throws IOException {
>   return 4 + w.getLength();
> }
> @Override
> public void serialize(HiveKey w, DataOutput out) throws IOException {
>   w.write(out);
> }
> @Override
> public void deserialize(DataInput in, int length, HiveKey w ) throws 
> IOException {
>   w.readFields(in);
> }
>   }
> }
> {code}
> and throws exceptions:
> {code}
> Error: java.io.IOException: Initialization of all the collectors failed. 
> Error in last collector was :Native output collector cannot be loaded; at 
> org.apache.hadoop.mapred.MapTask.createSortingCollector(MapTask.java:415) at 
> org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:442) at 
> org.apache.hadoop.mapred.MapTask.run(MapTask.java:343) at 
> org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164) at 
> java.security.AccessController.doPrivileged(Native Method) at 
> javax.security.auth.Subject.doAs(Subject.java:422) at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1700)
>  at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158) Caused by: 
> java.io.IOException: Native output collector cannot be loaded; at 
> org.apache.hadoop.mapred.nativetask.NativeMapOutputCollectorDelegator.init(NativeMapOutputCollectorDelegator.java:165)
>  at org.apache.hadoop.mapred.MapTask.createSortingCollector(MapTask.java:402) 
> ... 7 more Caused by: java.io.IOException: /PartitionBucket.h:56:pool is 
> NULL, or comparator is not set 
> /usr/local/hadoop-2.7.3-yarn/lib/native/libnativetask.so.1.0.0(_ZN10NativeTask15HadoopExceptionC2ERKSs+0x76)
>  [0x7ffdcbba6436] 
>

[jira] [Commented] (HIVE-19045) Vectorization: Disable vectorization in non-vectorized Q files

2018-04-08 Thread Hive QA (JIRA)


[ 
https://issues.apache.org/jira/browse/HIVE-19045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16429664#comment-16429664
 ] 

Hive QA commented on HIVE-19045:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m  
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  8m 
41s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
18s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
17s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
16s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
15s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red}  0m 
14s{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 11m 10s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-10077/dev-support/hive-personality.sh
 |
| git revision | master / 6ba33b8 |
| Default Java | 1.8.0_111 |
| asflicense | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-10077/yetus/patch-asflicense-problems.txt
 |
| modules | C: common U: common |
| Console output | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-10077/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |


This message was automatically generated.



> Vectorization: Disable vectorization in non-vectorized Q files
> --
>
> Key: HIVE-19045
> URL: https://issues.apache.org/jira/browse/HIVE-19045
> Project: Hive
>  Issue Type: Bug
>  Components: Hive
>Affects Versions: 3.0.0
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Attachments: HIVE-19045.01.patch, HIVE-19045.02.patch, 
> HIVE-19045.03.patch, HIVE-19045.04.patch
>
>
> In preparation for turning vectorization on by default, explicitly turn off 
> vectorization at the top of the Q files when there exist Vectorization Q 
> files.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-18910) Migrate to Murmur hash for shuffle and bucketing

2018-04-08 Thread Deepak Jaiswal (JIRA)


 [ 
https://issues.apache.org/jira/browse/HIVE-18910?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Deepak Jaiswal updated HIVE-18910:
--
Attachment: HIVE-18910.25.patch

> Migrate to Murmur hash for shuffle and bucketing
> 
>
> Key: HIVE-18910
> URL: https://issues.apache.org/jira/browse/HIVE-18910
> Project: Hive
>  Issue Type: Task
>Reporter: Deepak Jaiswal
>Assignee: Deepak Jaiswal
>Priority: Major
> Attachments: HIVE-18910.1.patch, HIVE-18910.10.patch, 
> HIVE-18910.11.patch, HIVE-18910.12.patch, HIVE-18910.13.patch, 
> HIVE-18910.14.patch, HIVE-18910.15.patch, HIVE-18910.16.patch, 
> HIVE-18910.17.patch, HIVE-18910.18.patch, HIVE-18910.19.patch, 
> HIVE-18910.2.patch, HIVE-18910.20.patch, HIVE-18910.21.patch, 
> HIVE-18910.22.patch, HIVE-18910.23.patch, HIVE-18910.24.patch, 
> HIVE-18910.25.patch, HIVE-18910.3.patch, HIVE-18910.4.patch, 
> HIVE-18910.5.patch, HIVE-18910.6.patch, HIVE-18910.7.patch, 
> HIVE-18910.8.patch, HIVE-18910.9.patch
>
>
> Hive uses JAVA hash which is not as good as murmur for better distribution 
> and efficiency in bucketing a table.
> Migrate to murmur hash but still keep backward compatibility for existing 
> users so that they dont have to reload the existing tables.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-19045) Vectorization: Disable vectorization in non-vectorized Q files

2018-04-08 Thread Matt McCline (JIRA)


 [ 
https://issues.apache.org/jira/browse/HIVE-19045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt McCline updated HIVE-19045:

Attachment: HIVE-19045.04.patch

> Vectorization: Disable vectorization in non-vectorized Q files
> --
>
> Key: HIVE-19045
> URL: https://issues.apache.org/jira/browse/HIVE-19045
> Project: Hive
>  Issue Type: Bug
>  Components: Hive
>Affects Versions: 3.0.0
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Attachments: HIVE-19045.01.patch, HIVE-19045.02.patch, 
> HIVE-19045.03.patch, HIVE-19045.04.patch
>
>
> In preparation for turning vectorization on by default, explicitly turn off 
> vectorization at the top of the Q files when there exist Vectorization Q 
> files.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-19045) Vectorization: Disable vectorization in non-vectorized Q files

2018-04-08 Thread Matt McCline (JIRA)


 [ 
https://issues.apache.org/jira/browse/HIVE-19045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt McCline updated HIVE-19045:

Status: Patch Available  (was: In Progress)

> Vectorization: Disable vectorization in non-vectorized Q files
> --
>
> Key: HIVE-19045
> URL: https://issues.apache.org/jira/browse/HIVE-19045
> Project: Hive
>  Issue Type: Bug
>  Components: Hive
>Affects Versions: 3.0.0
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Attachments: HIVE-19045.01.patch, HIVE-19045.02.patch, 
> HIVE-19045.03.patch, HIVE-19045.04.patch
>
>
> In preparation for turning vectorization on by default, explicitly turn off 
> vectorization at the top of the Q files when there exist Vectorization Q 
> files.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-19045) Vectorization: Disable vectorization in non-vectorized Q files

2018-04-08 Thread Matt McCline (JIRA)


 [ 
https://issues.apache.org/jira/browse/HIVE-19045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt McCline updated HIVE-19045:

Status: In Progress  (was: Patch Available)

> Vectorization: Disable vectorization in non-vectorized Q files
> --
>
> Key: HIVE-19045
> URL: https://issues.apache.org/jira/browse/HIVE-19045
> Project: Hive
>  Issue Type: Bug
>  Components: Hive
>Affects Versions: 3.0.0
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Attachments: HIVE-19045.01.patch, HIVE-19045.02.patch, 
> HIVE-19045.03.patch
>
>
> In preparation for turning vectorization on by default, explicitly turn off 
> vectorization at the top of the Q files when there exist Vectorization Q 
> files.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-19089) Create/Replicate Allocate write-id event

2018-04-08 Thread Hive QA (JIRA)


[ 
https://issues.apache.org/jira/browse/HIVE-19089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16429647#comment-16429647
 ] 

Hive QA commented on HIVE-19089:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12917961/HIVE-19089.03.patch

{color:green}SUCCESS:{color} +1 due to 5 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 145 failed/errored test(s), 13583 tests 
executed
*Failed tests:*
{noformat}
TestBeeLineDriver - did not produce a TEST-*.xml file (likely timed out) 
(batchId=252)
TestCopyUtils - did not produce a TEST-*.xml file (likely timed out) 
(batchId=230)
TestDbNotificationListener - did not produce a TEST-*.xml file (likely timed 
out) (batchId=246)
TestDummy - did not produce a TEST-*.xml file (likely timed out) (batchId=252)
TestExportImport - did not produce a TEST-*.xml file (likely timed out) 
(batchId=230)
TestHCatHiveCompatibility - did not produce a TEST-*.xml file (likely timed 
out) (batchId=246)
TestMiniDruidCliDriver - did not produce a TEST-*.xml file (likely timed out) 
(batchId=252)
TestMiniDruidKafkaCliDriver - did not produce a TEST-*.xml file (likely timed 
out) (batchId=252)
TestNegativeCliDriver - did not produce a TEST-*.xml file (likely timed out) 
(batchId=95)

62 matches

Mail list logo