[jira] [Commented] (HIVE-19417) Modify metastore to have/access persistent tables for stats
[
https://issues.apache.org/jira/browse/HIVE-19417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16505265#comment-16505265
]
Hive QA commented on HIVE-19417:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12926624/HIVE-19417.05.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/11598/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/11598/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-11598/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2018-06-07 20:59:13.414
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-11598/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2018-06-07 20:59:13.417
+ cd apache-github-source-source
+ git fetch origin
>From https://github.com/apache/hive
f04eba3..73aab1a master -> origin/master
9ac67e3..02f4931 branch-3 -> origin/branch-3
+ git reset --hard HEAD
HEAD is now at f04eba3 HIVE-18690: Integrate with Spark OutputMetrics (Sahil
Takiar, reviewed by Aihua Xu)
+ git clean -f -d
Removing upgrade-acid/post-upgrade/
Removing upgrade-acid/pre-upgrade/
+ git checkout master
Already on 'master'
Your branch is behind 'origin/master' by 1 commit, and can be fast-forwarded.
(use "git pull" to update your local branch)
+ git reset --hard origin/master
HEAD is now at 73aab1a HIVE-19799: remove jasper dependency (Prasanth
Jayachandran reviewed by Thejas M Nair)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2018-06-07 20:59:14.429
+ rm -rf ../yetus_PreCommit-HIVE-Build-11598
+ mkdir ../yetus_PreCommit-HIVE-Build-11598
+ git gc
+ cp -R . ../yetus_PreCommit-HIVE-Build-11598
+ mkdir /data/hiveptest/logs/PreCommit-HIVE-Build-11598/yetus
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
error: patch failed:
standalone-metastore/src/main/sql/derby/upgrade-2.3.0-to-3.0.0.derby.sql:254
Falling back to three-way merge...
Applied patch to
'standalone-metastore/src/main/sql/derby/upgrade-2.3.0-to-3.0.0.derby.sql' with
conflicts.
error: patch failed:
standalone-metastore/src/main/sql/derby/upgrade.order.derby:14
Falling back to three-way merge...
Applied patch to 'standalone-metastore/src/main/sql/derby/upgrade.order.derby'
with conflicts.
error: patch failed:
standalone-metastore/src/main/sql/mssql/upgrade-2.3.0-to-3.0.0.mssql.sql:321
Falling back to three-way merge...
Applied patch to
'standalone-metastore/src/main/sql/mssql/upgrade-2.3.0-to-3.0.0.mssql.sql' with
conflicts.
error: patch failed:
standalone-metastore/src/main/sql/mysql/upgrade-2.3.0-to-3.0.0.mysql.sql:292
Falling back to three-way merge...
Applied patch to
'standalone-metastore/src/main/sql/mysql/upgrade-2.3.0-to-3.0.0.mysql.sql' with
conflicts.
error: patch failed:
standalone-metastore/src/main/sql/oracle/upgrade-2.3.0-to-3.0.0.oracle.sql:310
Falling back to three-way merge...
Applied patch to
'standalone-metastore/src/main/sql/oracle/upgrade-2.3.0-to-3.0.0.oracle.sql'
with conflicts.
error: patch failed:
standalone-metastore/src/main/sql/postgres/upgrade-2.3.0-to-3.0.0.postgres.sql:329
Falling back to three-way merge...
Applied patch to
'standalone-metastore/src/main/sql/postgres/upgrade-2.3.0-to-3.0.0.postgres.sql'
with conflicts.
error: patch failed:
standalone-metastore/src/main/thrift/hive_metastore.thrift:1867
Falling back to three-way merge...
Applied patch to 'standalone-metastore/src/main/thrift/hive_metastore.thrift'
with conflicts.
Going to apply patch with: git apply -p0
[jira] [Commented] (HIVE-19417) Modify metastore to have/access persistent tables for stats
[ https://issues.apache.org/jira/browse/HIVE-19417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16502380#comment-16502380 ] Steve Yeom commented on HIVE-19417: --- Create a patch 05 for this jira reflecting the talk of last Thursday. See HIVE-19416 for the talk and the change items. > Modify metastore to have/access persistent tables for stats > --- > > Key: HIVE-19417 > URL: https://issues.apache.org/jira/browse/HIVE-19417 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Affects Versions: 3.0.0 >Reporter: Steve Yeom >Assignee: Steve Yeom >Priority: Major > Attachments: HIVE-19417.01.patch, HIVE-19417.02.patch, > HIVE-19417.03.patch, HIVE-19417.04.patch, HIVE-19417.05.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19417) Modify metastore to have/access persistent tables for stats
[ https://issues.apache.org/jira/browse/HIVE-19417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16495560#comment-16495560 ] Steve Yeom commented on HIVE-19417: --- 1. MetaStore Schema Changes for the patch 04 1.1 New table UPD_TXNS A record is created per transaction per table. - UPDATE_ID: primary key column, generated by Datanucleus when specified at "package.jdo" as datastore-identity - TBL_ID: TBLS.TBL_ID referencing column. A foreign key is created for this column referencing TBLS.TBL_ID. - STATE: this is deleted for the next patch version. - TXN_ID: Transaction id of the transaction to insert the row. - WRITEID_LIST: writeIdList for the table of the transaction 1.2 Modification for TBLS and PARTITIONS tables - A new column TXN_ID: transaction id of the UPD_TXNS. > Modify metastore to have/access persistent tables for stats > --- > > Key: HIVE-19417 > URL: https://issues.apache.org/jira/browse/HIVE-19417 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Affects Versions: 3.0.0 >Reporter: Steve Yeom >Assignee: Steve Yeom >Priority: Major > Attachments: HIVE-19417.01.patch, HIVE-19417.02.patch, > HIVE-19417.03.patch, HIVE-19417.04.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19417) Modify metastore to have/access persistent tables for stats
[
https://issues.apache.org/jira/browse/HIVE-19417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16491837#comment-16491837
]
Hive QA commented on HIVE-19417:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12924969/HIVE-19417.04.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/11237/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/11237/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-11237/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2018-05-26 21:59:34.950
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-11237/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2018-05-26 21:59:34.953
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at f9f4340 HIVE-19684 : Hive stats optimizer wrongly uses stats
against non native tables (Slim Bouguerra via Ashutosh Chauhan)
+ git clean -f -d
+ git checkout master
Already on 'master'
Your branch is up-to-date with 'origin/master'.
+ git reset --hard origin/master
HEAD is now at f9f4340 HIVE-19684 : Hive stats optimizer wrongly uses stats
against non native tables (Slim Bouguerra via Ashutosh Chauhan)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2018-05-26 21:59:36.191
+ rm -rf ../yetus_PreCommit-HIVE-Build-11237
+ mkdir ../yetus_PreCommit-HIVE-Build-11237
+ git gc
+ cp -R . ../yetus_PreCommit-HIVE-Build-11237
+ mkdir /data/hiveptest/logs/PreCommit-HIVE-Build-11237/yetus
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
fatal: corrupt patch at line 21
fatal: corrupt patch at line 21
fatal: corrupt patch at line 21
The patch does not appear to apply with p0, p1, or p2
+ exit 1
'
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12924969 - PreCommit-HIVE-Build
> Modify metastore to have/access persistent tables for stats
> ---
>
> Key: HIVE-19417
> URL: https://issues.apache.org/jira/browse/HIVE-19417
> Project: Hive
> Issue Type: Sub-task
> Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Steve Yeom
>Assignee: Steve Yeom
>Priority: Major
> Attachments: HIVE-19417.01.patch, HIVE-19417.02.patch,
> HIVE-19417.03.patch, HIVE-19417.04.patch
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19417) Modify metastore to have/access persistent tables for stats
[ https://issues.apache.org/jira/browse/HIVE-19417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16490947#comment-16490947 ] Eugene Koifman commented on HIVE-19417: --- [~steveyeom2017] could you provide some explanation about what each filed means/is used for? > Modify metastore to have/access persistent tables for stats > --- > > Key: HIVE-19417 > URL: https://issues.apache.org/jira/browse/HIVE-19417 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Affects Versions: 3.0.0 >Reporter: Steve Yeom >Assignee: Steve Yeom >Priority: Major > Attachments: HIVE-19417.01.patch, HIVE-19417.02.patch, > HIVE-19417.03.patch, HIVE-19417.04.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19417) Modify metastore to have/access persistent tables for stats
[ https://issues.apache.org/jira/browse/HIVE-19417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16490220#comment-16490220 ] Vineet Garg commented on HIVE-19417: FYI HIVE-19323 is adding metastore upgrade and schema scripts. So this JIRA will require HIVE-19323 > Modify metastore to have/access persistent tables for stats > --- > > Key: HIVE-19417 > URL: https://issues.apache.org/jira/browse/HIVE-19417 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Affects Versions: 3.0.0 >Reporter: Steve Yeom >Assignee: Steve Yeom >Priority: Major > Attachments: HIVE-19417.01.patch, HIVE-19417.02.patch, > HIVE-19417.03.patch, HIVE-19417.04.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19417) Modify metastore to have/access persistent tables for stats
[ https://issues.apache.org/jira/browse/HIVE-19417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16483070#comment-16483070 ] Steve Yeom commented on HIVE-19417: --- Hey [~sershe] Thanks for the review. I am following up comments of Eugene and Vineet. > Modify metastore to have/access persistent tables for stats > --- > > Key: HIVE-19417 > URL: https://issues.apache.org/jira/browse/HIVE-19417 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Affects Versions: 3.0.0 >Reporter: Steve Yeom >Assignee: Steve Yeom >Priority: Major > Attachments: HIVE-19417.01.patch, HIVE-19417.02.patch, > HIVE-19417.03.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19417) Modify metastore to have/access persistent tables for stats
[ https://issues.apache.org/jira/browse/HIVE-19417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16483037#comment-16483037 ] Sergey Shelukhin commented on HIVE-19417: - [~steveyeom2017] can you address above comments from [~ekoifman] and [~vgarg]? Other than that it makes sense to me > Modify metastore to have/access persistent tables for stats > --- > > Key: HIVE-19417 > URL: https://issues.apache.org/jira/browse/HIVE-19417 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Affects Versions: 3.0.0 >Reporter: Steve Yeom >Assignee: Steve Yeom >Priority: Major > Attachments: HIVE-19417.01.patch, HIVE-19417.02.patch, > HIVE-19417.03.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19417) Modify metastore to have/access persistent tables for stats
[
https://issues.apache.org/jira/browse/HIVE-19417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16480241#comment-16480241
]
Hive QA commented on HIVE-19417:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12923961/HIVE-19417.03.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/11044/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/11044/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-11044/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2018-05-18 06:48:39.048
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-11044/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2018-05-18 06:48:39.051
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at 0b6bea8 HIVE-19572: Add option to mask stats and data size in q
files (Jesus Camacho Rodriguez, reviewed by Prasanth Jayachandran) (addendum)
+ git clean -f -d
+ git checkout master
Already on 'master'
Your branch is up-to-date with 'origin/master'.
+ git reset --hard origin/master
HEAD is now at 0b6bea8 HIVE-19572: Add option to mask stats and data size in q
files (Jesus Camacho Rodriguez, reviewed by Prasanth Jayachandran) (addendum)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2018-05-18 06:48:40.285
+ rm -rf ../yetus_PreCommit-HIVE-Build-11044
+ mkdir ../yetus_PreCommit-HIVE-Build-11044
+ git gc
+ cp -R . ../yetus_PreCommit-HIVE-Build-11044
+ mkdir /data/hiveptest/logs/PreCommit-HIVE-Build-11044/yetus
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
Going to apply patch with: git apply -p0
/data/hiveptest/working/scratch/build.patch:10128: trailing whitespace.
tmpMap.put(_Fields.WRITE_ID, new
org.apache.thrift.meta_data.FieldMetaData("writeId",
org.apache.thrift.TFieldRequirementType.OPTIONAL,
/data/hiveptest/working/scratch/build.patch:10130: trailing whitespace.
tmpMap.put(_Fields.VALID_WRITE_ID_LIST, new
org.apache.thrift.meta_data.FieldMetaData("validWriteIdList",
org.apache.thrift.TFieldRequirementType.OPTIONAL,
/data/hiveptest/working/scratch/build.patch:10379: trailing whitespace.
} else {
/data/hiveptest/working/scratch/build.patch:10387: trailing whitespace.
} else {
/data/hiveptest/working/scratch/build.patch:10497: trailing whitespace.
*
warning: squelched 34 whitespace errors
warning: 39 lines add whitespace errors.
+ [[ maven == \m\a\v\e\n ]]
+ rm -rf /data/hiveptest/working/maven/org/apache/hive
+ mvn -B clean install -DskipTests -T 4 -q
-Dmaven.repo.local=/data/hiveptest/working/maven
protoc-jar: executing: [/tmp/protoc6821677649077860847.exe, --version]
libprotoc 2.5.0
protoc-jar: executing: [/tmp/protoc6821677649077860847.exe,
-I/data/hiveptest/working/apache-github-source-source/standalone-metastore/src/main/protobuf/org/apache/hadoop/hive/metastore,
--java_out=/data/hiveptest/working/apache-github-source-source/standalone-metastore/target/generated-sources,
/data/hiveptest/working/apache-github-source-source/standalone-metastore/src/main/protobuf/org/apache/hadoop/hive/metastore/metastore.proto]
ANTLR Parser Generator Version 3.5.2
Output file
/data/hiveptest/working/apache-github-source-source/standalone-metastore/target/generated-sources/org/apache/hadoop/hive/metastore/parser/FilterParser.java
does not exist: must build
/data/hiveptest/working/apache-github-source-source/standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/parser/Filter.g
[jira] [Commented] (HIVE-19417) Modify metastore to have/access persistent tables for stats
[ https://issues.apache.org/jira/browse/HIVE-19417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16479600#comment-16479600 ] Vineet Garg commented on HIVE-19417: The patch is touching wrong metastore upgrade scripts and schema scripts e.g. upgrade-2.3.0-to-3.0.0.derby because we have already cut off 3.0 this is wrong. Depending upon what version this patch is targeted for appropriate upgrade/schema scripts should be added and edited. If this is targeted for 3.1 please add upgrade-3.0.0-to-3.1.0..xx scripts for all databases plus hive-schema-3.1.0 for all databases. > Modify metastore to have/access persistent tables for stats > --- > > Key: HIVE-19417 > URL: https://issues.apache.org/jira/browse/HIVE-19417 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Affects Versions: 3.0.0 >Reporter: Steve Yeom >Assignee: Steve Yeom >Priority: Major > Attachments: HIVE-19417.01.patch, HIVE-19417.02.patch, > HIVE-19417.03.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19417) Modify metastore to have/access persistent tables for stats
[ https://issues.apache.org/jira/browse/HIVE-19417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16479350#comment-16479350 ] Eugene Koifman commented on HIVE-19417: --- [~steveyeom2017], could you explain the meaning/intent of the fields in UPD_TXNS. How will COMMIT_SEQID be used? What are the possible values STATE? If you have TBL_ID, why store db_name, tbl_name? > Modify metastore to have/access persistent tables for stats > --- > > Key: HIVE-19417 > URL: https://issues.apache.org/jira/browse/HIVE-19417 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Affects Versions: 3.0.0 >Reporter: Steve Yeom >Assignee: Steve Yeom >Priority: Major > Attachments: HIVE-19417.01.patch, HIVE-19417.02.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19417) Modify metastore to have/access persistent tables for stats
[ https://issues.apache.org/jira/browse/HIVE-19417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16477994#comment-16477994 ] Steve Yeom commented on HIVE-19417: --- While I was talking with Eugene, I realized UPD_PARTTXNS table is not needed. I already made relevant changes for the other patch but will generate a new one for this jira. > Modify metastore to have/access persistent tables for stats > --- > > Key: HIVE-19417 > URL: https://issues.apache.org/jira/browse/HIVE-19417 > Project: Hive > Issue Type: Sub-task > Components: Hive >Affects Versions: 3.0.0 >Reporter: Steve Yeom >Assignee: Steve Yeom >Priority: Major > Attachments: HIVE-19417.01.patch, HIVE-19417.02.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19417) Modify metastore to have/access persistent tables for stats
[ https://issues.apache.org/jira/browse/HIVE-19417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16476532#comment-16476532 ] Sergey Shelukhin commented on HIVE-19417: - Minor comments; overall looks good pending tests. I think [~ekoifman] should review table structures, there seems to be some duplication with ACID tables, it seems to be ok to me. The inserts for the new stuff (Update, and setting the table update_id) will probably need to be in the same DB transaction as the commit of Hive ACID transaction. > Modify metastore to have/access persistent tables for stats > --- > > Key: HIVE-19417 > URL: https://issues.apache.org/jira/browse/HIVE-19417 > Project: Hive > Issue Type: Sub-task > Components: Hive >Affects Versions: 3.0.0 >Reporter: Steve Yeom >Assignee: Steve Yeom >Priority: Major > Attachments: HIVE-19417.01.patch, HIVE-19417.02.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19417) Modify metastore to have/access persistent tables for stats
[
https://issues.apache.org/jira/browse/HIVE-19417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16473485#comment-16473485
]
Hive QA commented on HIVE-19417:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12923106/HIVE-19417.02.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 88 failed/errored test(s), 13580 tests
executed
*Failed tests:*
{noformat}
TestDbNotificationListener - did not produce a TEST-*.xml file (likely timed
out) (batchId=247)
TestHCatHiveCompatibility - did not produce a TEST-*.xml file (likely timed
out) (batchId=247)
TestNegativeCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=95)
[jira] [Commented] (HIVE-19417) Modify metastore to have/access persistent tables for stats
[
https://issues.apache.org/jira/browse/HIVE-19417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16473469#comment-16473469
]
Hive QA commented on HIVE-19417:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
35s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
40s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
51s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
26s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
5s{color} | {color:blue} standalone-metastore in master has 215 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
14s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
9s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
17s{color} | {color:red} standalone-metastore: The patch generated 9 new + 3
unchanged - 0 fixed = 12 total (was 3) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 37 line(s) that end in whitespace. Use
git apply --whitespace=fix <>. Refer
https://git-scm.com/docs/git-apply {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 3m
6s{color} | {color:red} standalone-metastore generated 1 new + 215 unchanged -
0 fixed = 216 total (was 215) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 21m 25s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:standalone-metastore |
| | new
org.apache.hadoop.hive.metastore.model.MPartitionUpdaterTransaction(String,
MTable, MPartition, Timestamp, long, char, long, String) may expose internal
representation by storing an externally mutable object into
MPartitionUpdaterTransaction.creationTime At
MPartitionUpdaterTransaction.java:long, char, long, String) may expose internal
representation by storing an externally mutable object into
MPartitionUpdaterTransaction.creationTime At
MPartitionUpdaterTransaction.java:[line 51] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-10895/dev-support/hive-personality.sh
|
| git revision | master / 6f9368b |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-10895/yetus/diff-checkstyle-standalone-metastore.txt
|
| whitespace |
http://104.198.109.242/logs//PreCommit-HIVE-Build-10895/yetus/whitespace-eol.txt
|
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-10895/yetus/new-findbugs-standalone-metastore.html
|
| modules | C: metastore standalone-metastore U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-10895/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Modify metastore to have/access persistent tables for stats
> ---
>
> Key: HIVE-19417
> URL: https://issues.apache.org/jira/browse/HIVE-19417
> Project: Hive
>
[jira] [Commented] (HIVE-19417) Modify metastore to have/access persistent tables for stats
[ https://issues.apache.org/jira/browse/HIVE-19417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16472810#comment-16472810 ] Steve Yeom commented on HIVE-19417: --- Hi [~sershe] could you look at the patch 02? Thanks, Steve. > Modify metastore to have/access persistent tables for stats > --- > > Key: HIVE-19417 > URL: https://issues.apache.org/jira/browse/HIVE-19417 > Project: Hive > Issue Type: Sub-task > Components: Hive >Affects Versions: 3.0.0 >Reporter: Steve Yeom >Assignee: Steve Yeom >Priority: Major > Attachments: HIVE-19417.01.patch, HIVE-19417.02.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19417) Modify metastore to have/access persistent tables for stats
[ https://issues.apache.org/jira/browse/HIVE-19417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16472763#comment-16472763 ] Steve Yeom commented on HIVE-19417: --- The above p-tests output indicates patch 01 requires master-rebasing. i have rebased and will add patch 02 which is identical to patch 01 except rebased master. > Modify metastore to have/access persistent tables for stats > --- > > Key: HIVE-19417 > URL: https://issues.apache.org/jira/browse/HIVE-19417 > Project: Hive > Issue Type: Sub-task > Components: Hive >Affects Versions: 3.0.0 >Reporter: Steve Yeom >Assignee: Steve Yeom >Priority: Major > Attachments: HIVE-19417.01.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19417) Modify metastore to have/access persistent tables for stats
[
https://issues.apache.org/jira/browse/HIVE-19417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16471486#comment-16471486
]
Hive QA commented on HIVE-19417:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12922684/HIVE-19417.01.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/10815/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/10815/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-10815/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2018-05-11 04:38:25.196
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-10815/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2018-05-11 04:38:25.199
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at 6c364c6 HIVE-19306: Arrow batch serializer (Teddy Choi, reviewed
by Matt McCline and Eric Wohlstadter (non-binding))
+ git clean -f -d
+ git checkout master
Already on 'master'
Your branch is up-to-date with 'origin/master'.
+ git reset --hard origin/master
HEAD is now at 6c364c6 HIVE-19306: Arrow batch serializer (Teddy Choi, reviewed
by Matt McCline and Eric Wohlstadter (non-binding))
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2018-05-11 04:38:28.550
+ rm -rf ../yetus_PreCommit-HIVE-Build-10815
+ mkdir ../yetus_PreCommit-HIVE-Build-10815
+ git gc
+ cp -R . ../yetus_PreCommit-HIVE-Build-10815
+ mkdir /data/hiveptest/logs/PreCommit-HIVE-Build-10815/yetus
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
error: patch failed:
standalone-metastore/src/main/sql/derby/upgrade-2.3.0-to-3.0.0.derby.sql:254
Falling back to three-way merge...
Applied patch to
'standalone-metastore/src/main/sql/derby/upgrade-2.3.0-to-3.0.0.derby.sql' with
conflicts.
error: patch failed:
standalone-metastore/src/main/sql/mssql/upgrade-2.3.0-to-3.0.0.mssql.sql:321
Falling back to three-way merge...
Applied patch to
'standalone-metastore/src/main/sql/mssql/upgrade-2.3.0-to-3.0.0.mssql.sql' with
conflicts.
error: patch failed:
standalone-metastore/src/main/sql/mysql/upgrade-2.3.0-to-3.0.0.mysql.sql:292
Falling back to three-way merge...
Applied patch to
'standalone-metastore/src/main/sql/mysql/upgrade-2.3.0-to-3.0.0.mysql.sql' with
conflicts.
error: patch failed:
standalone-metastore/src/main/sql/oracle/upgrade-2.3.0-to-3.0.0.oracle.sql:310
Falling back to three-way merge...
Applied patch to
'standalone-metastore/src/main/sql/oracle/upgrade-2.3.0-to-3.0.0.oracle.sql'
with conflicts.
error: patch failed:
standalone-metastore/src/main/sql/postgres/upgrade-2.3.0-to-3.0.0.postgres.sql:329
Falling back to three-way merge...
Applied patch to
'standalone-metastore/src/main/sql/postgres/upgrade-2.3.0-to-3.0.0.postgres.sql'
with conflicts.
Going to apply patch with: git apply -p0
/data/hiveptest/working/scratch/build.patch:1317: trailing whitespace.
tmpMap.put(_Fields.WRITE_ID, new
org.apache.thrift.meta_data.FieldMetaData("writeId",
org.apache.thrift.TFieldRequirementType.OPTIONAL,
/data/hiveptest/working/scratch/build.patch:1319: trailing whitespace.
tmpMap.put(_Fields.VALID_WRITE_ID_LIST, new
org.apache.thrift.meta_data.FieldMetaData("validWriteIdList",
org.apache.thrift.TFieldRequirementType.OPTIONAL,
/data/hiveptest/working/scratch/build.patch:1551: trailing whitespace.
} else {
/data/hiveptest/working/scratch/build.patch:1559: trailing whitespace.
} else {
/data/hiveptest/working/scratch/build.patch:1695: trailing whitespace.
[jira] [Commented] (HIVE-19417) Modify metastore to have/access persistent tables for stats
[ https://issues.apache.org/jira/browse/HIVE-19417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16469213#comment-16469213 ] Steve Yeom commented on HIVE-19417: --- I think it will be possible without the rest. Now I am creating a patch 02 and will post it in RB. > Modify metastore to have/access persistent tables for stats > --- > > Key: HIVE-19417 > URL: https://issues.apache.org/jira/browse/HIVE-19417 > Project: Hive > Issue Type: Sub-task > Components: Hive >Affects Versions: 3.0.0 >Reporter: Steve Yeom >Assignee: Steve Yeom >Priority: Major > Attachments: HIVE-19417.01.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19417) Modify metastore to have/access persistent tables for stats
[ https://issues.apache.org/jira/browse/HIVE-19417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16469119#comment-16469119 ] Sergey Shelukhin commented on HIVE-19417: - Can you post an RB, preferably excluding generated files? Will this be possible to commit without the rest of the patch? > Modify metastore to have/access persistent tables for stats > --- > > Key: HIVE-19417 > URL: https://issues.apache.org/jira/browse/HIVE-19417 > Project: Hive > Issue Type: Sub-task > Components: Hive >Affects Versions: 3.0.0 >Reporter: Steve Yeom >Assignee: Steve Yeom >Priority: Major > Attachments: HIVE-19417.01.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19417) Modify metastore to have/access persistent tables for stats
[ https://issues.apache.org/jira/browse/HIVE-19417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16468971#comment-16468971 ] Steve Yeom commented on HIVE-19417: --- Hi [~sershe] Can you look at this patch? This is a partial one. Please let me know if you need a rb. Thanks, Steve. > Modify metastore to have/access persistent tables for stats > --- > > Key: HIVE-19417 > URL: https://issues.apache.org/jira/browse/HIVE-19417 > Project: Hive > Issue Type: Sub-task > Components: Hive >Affects Versions: 3.0.0 >Reporter: Steve Yeom >Assignee: Steve Yeom >Priority: Major > Attachments: HIVE-19417.01.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19417) Modify metastore to have/access persistent tables for stats
[ https://issues.apache.org/jira/browse/HIVE-19417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16468165#comment-16468165 ] Steve Yeom commented on HIVE-19417: --- The patch 01 is for changes for schema, model classes, and thrift changes.. > Modify metastore to have/access persistent tables for stats > --- > > Key: HIVE-19417 > URL: https://issues.apache.org/jira/browse/HIVE-19417 > Project: Hive > Issue Type: Sub-task > Components: Hive >Affects Versions: 3.0.0 >Reporter: Steve Yeom >Assignee: Steve Yeom >Priority: Major > Attachments: HIVE-19417.01.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19417) Modify metastore to have/access persistent tables for stats
[ https://issues.apache.org/jira/browse/HIVE-19417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16466306#comment-16466306 ] Vineet Garg commented on HIVE-19417: Deferring this to next release. Please mark this as blocker if you would like to get this in 3.0 branch. > Modify metastore to have/access persistent tables for stats > --- > > Key: HIVE-19417 > URL: https://issues.apache.org/jira/browse/HIVE-19417 > Project: Hive > Issue Type: Sub-task > Components: Hive >Affects Versions: 3.0.0 >Reporter: Steve Yeom >Assignee: Steve Yeom >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19417) Modify metastore to have/access persistent tables for stats
[ https://issues.apache.org/jira/browse/HIVE-19417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16463226#comment-16463226 ] Steve Yeom commented on HIVE-19417: --- I will add a snapshot table and the reference to it on the existing stats tables. > Modify metastore to have/access persistent tables for stats > --- > > Key: HIVE-19417 > URL: https://issues.apache.org/jira/browse/HIVE-19417 > Project: Hive > Issue Type: Sub-task > Components: Hive >Affects Versions: 3.0.0 >Reporter: Steve Yeom >Assignee: Steve Yeom >Priority: Major > Fix For: 3.0.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
