[ https://issues.apache.org/jira/browse/HIVE-21052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16743169#comment-16743169 ]
Hive QA commented on HIVE-21052: -------------------------------- Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12954984/HIVE-21052.3.patch {color:red}ERROR:{color} -1 due to build exiting with an error Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/15627/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/15627/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-15627/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Tests exited with: NonZeroExitCodeException Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit status 1 and output '+ date '+%Y-%m-%d %T.%3N' 2019-01-15 16:00:38.574 + [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]] + export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64 + JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64 + export PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games + PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games + export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m ' + ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m ' + export 'MAVEN_OPTS=-Xmx1g ' + MAVEN_OPTS='-Xmx1g ' + cd /data/hiveptest/working/ + tee /data/hiveptest/logs/PreCommit-HIVE-Build-15627/source-prep.txt + [[ false == \t\r\u\e ]] + mkdir -p maven ivy + [[ git = \s\v\n ]] + [[ git = \g\i\t ]] + [[ -z master ]] + [[ -d apache-github-source-source ]] + [[ ! -d apache-github-source-source/.git ]] + [[ ! -d apache-github-source-source ]] + date '+%Y-%m-%d %T.%3N' 2019-01-15 16:00:38.577 + cd apache-github-source-source + git fetch origin + git reset --hard HEAD HEAD is now at a3aa074 HIVE-21113: For HPL/SQL that contains boolean expression with NOT, incorrect SQL may be generated (Baoning He, reviewed by Daniel Dai) + git clean -f -d Removing standalone-metastore/metastore-server/src/gen/ + git checkout master Already on 'master' Your branch is up-to-date with 'origin/master'. + git reset --hard origin/master HEAD is now at a3aa074 HIVE-21113: For HPL/SQL that contains boolean expression with NOT, incorrect SQL may be generated (Baoning He, reviewed by Daniel Dai) + git merge --ff-only origin/master Already up-to-date. + date '+%Y-%m-%d %T.%3N' 2019-01-15 16:00:39.260 + rm -rf ../yetus_PreCommit-HIVE-Build-15627 + mkdir ../yetus_PreCommit-HIVE-Build-15627 + git gc + cp -R . ../yetus_PreCommit-HIVE-Build-15627 + mkdir /data/hiveptest/logs/PreCommit-HIVE-Build-15627/yetus + patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh + patchFilePath=/data/hiveptest/working/scratch/build.patch + [[ -f /data/hiveptest/working/scratch/build.patch ]] + chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh + /data/hiveptest/working/scratch/smart-apply-patch.sh /data/hiveptest/working/scratch/build.patch error: patch failed: itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/TestCompactor.java:999 error: repository lacks the necessary blob to fall back on 3-way merge. error: itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/TestCompactor.java: patch does not apply error: patch failed: ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java:2541 error: repository lacks the necessary blob to fall back on 3-way merge. error: ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java: patch does not apply error: patch failed: ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Cleaner.java:395 error: repository lacks the necessary blob to fall back on 3-way merge. error: ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Cleaner.java: patch does not apply error: patch failed: standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/AllocateTableWriteIdsRequest.java:43 Falling back to three-way merge... Applied patch to 'standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/AllocateTableWriteIdsRequest.java' cleanly. error: patch failed: standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/CompactionInfoStruct.java:50 Falling back to three-way merge... Applied patch to 'standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/CompactionInfoStruct.java' with conflicts. error: patch failed: standalone-metastore/metastore-common/src/gen/thrift/gen-php/metastore/ThriftHiveMetastore.php:16520 Falling back to three-way merge... Applied patch to 'standalone-metastore/metastore-common/src/gen/thrift/gen-php/metastore/ThriftHiveMetastore.php' with conflicts. error: patch failed: standalone-metastore/metastore-common/src/gen/thrift/gen-php/metastore/Types.php:18706 Falling back to three-way merge... Applied patch to 'standalone-metastore/metastore-common/src/gen/thrift/gen-php/metastore/Types.php' with conflicts. error: patch failed: standalone-metastore/metastore-common/src/gen/thrift/gen-py/hive_metastore/ttypes.py:13026 Falling back to three-way merge... Applied patch to 'standalone-metastore/metastore-common/src/gen/thrift/gen-py/hive_metastore/ttypes.py' with conflicts. error: patch failed: standalone-metastore/metastore-common/src/gen/thrift/gen-rb/hive_metastore_types.rb:2891 Falling back to three-way merge... Applied patch to 'standalone-metastore/metastore-common/src/gen/thrift/gen-rb/hive_metastore_types.rb' with conflicts. error: patch failed: standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java:2921 Falling back to three-way merge... Applied patch to 'standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java' cleanly. error: patch failed: standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/IMetaStoreClient.java:3005 Falling back to three-way merge... Applied patch to 'standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/IMetaStoreClient.java' cleanly. error: patch failed: standalone-metastore/metastore-common/src/main/thrift/hive_metastore.thrift:977 Falling back to three-way merge... Applied patch to 'standalone-metastore/metastore-common/src/main/thrift/hive_metastore.thrift' with conflicts. error: patch failed: standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/CompactionInfo.java:28 error: repository lacks the necessary blob to fall back on 3-way merge. error: standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/CompactionInfo.java: patch does not apply error: patch failed: standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/CompactionTxnHandler.java:281 error: repository lacks the necessary blob to fall back on 3-way merge. error: standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/CompactionTxnHandler.java: patch does not apply error: patch failed: standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java:1756 error: repository lacks the necessary blob to fall back on 3-way merge. error: standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java: patch does not apply error: patch failed: standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClientPreCatalog.java:2356 Falling back to three-way merge... Applied patch to 'standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClientPreCatalog.java' with conflicts. error: patch failed: streaming/src/java/org/apache/hive/streaming/TransactionBatch.java:207 Falling back to three-way merge... Applied patch to 'streaming/src/java/org/apache/hive/streaming/TransactionBatch.java' cleanly. error: hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/TestCompactor.java: does not exist in index error: src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java: does not exist in index error: src/java/org/apache/hadoop/hive/ql/txn/compactor/Cleaner.java: does not exist in index error: metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/AllocateTableWriteIdsRequest.java: does not exist in index error: metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/CompactionInfoStruct.java: does not exist in index error: metastore-common/src/gen/thrift/gen-php/metastore/ThriftHiveMetastore.php: does not exist in index error: metastore-common/src/gen/thrift/gen-php/metastore/Types.php: does not exist in index error: metastore-common/src/gen/thrift/gen-py/hive_metastore/ttypes.py: does not exist in index error: metastore-common/src/gen/thrift/gen-rb/hive_metastore_types.rb: does not exist in index error: metastore-common/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java: does not exist in index error: metastore-common/src/main/java/org/apache/hadoop/hive/metastore/IMetaStoreClient.java: does not exist in index error: metastore-common/src/main/thrift/hive_metastore.thrift: does not exist in index error: metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/CompactionInfo.java: does not exist in index error: metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/CompactionTxnHandler.java: does not exist in index error: metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java: does not exist in index error: metastore-server/src/test/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClientPreCatalog.java: does not exist in index error: src/java/org/apache/hive/streaming/TransactionBatch.java: does not exist in index error: src/test/java/org/apache/hadoop/hive/ql/txn/compactor/TestCompactor.java: does not exist in index error: java/org/apache/hadoop/hive/ql/io/AcidUtils.java: does not exist in index error: java/org/apache/hadoop/hive/ql/txn/compactor/Cleaner.java: does not exist in index error: src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/AllocateTableWriteIdsRequest.java: does not exist in index error: src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/CompactionInfoStruct.java: does not exist in index error: src/gen/thrift/gen-php/metastore/ThriftHiveMetastore.php: does not exist in index error: src/gen/thrift/gen-php/metastore/Types.php: does not exist in index error: src/gen/thrift/gen-py/hive_metastore/ttypes.py: does not exist in index error: src/gen/thrift/gen-rb/hive_metastore_types.rb: does not exist in index error: src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java: does not exist in index error: src/main/java/org/apache/hadoop/hive/metastore/IMetaStoreClient.java: does not exist in index error: src/main/thrift/hive_metastore.thrift: does not exist in index error: src/main/java/org/apache/hadoop/hive/metastore/txn/CompactionInfo.java: does not exist in index error: src/main/java/org/apache/hadoop/hive/metastore/txn/CompactionTxnHandler.java: does not exist in index error: src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java: does not exist in index error: src/test/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClientPreCatalog.java: does not exist in index error: java/org/apache/hive/streaming/TransactionBatch.java: does not exist in index The patch does not appear to apply with p0, p1, or p2 + result=1 + '[' 1 -ne 0 ']' + rm -rf yetus_PreCommit-HIVE-Build-15627 + exit 1 ' {noformat} This message is automatically generated. ATTACHMENT ID: 12954984 - PreCommit-HIVE-Build > Make sure transactions get cleaned if they are aborted before addPartitions > is called > ------------------------------------------------------------------------------------- > > Key: HIVE-21052 > URL: https://issues.apache.org/jira/browse/HIVE-21052 > Project: Hive > Issue Type: Bug > Components: Transactions > Affects Versions: 3.0.0 > Reporter: Jaume M > Assignee: Jaume M > Priority: Critical > Attachments: Aborted Txn w_Direct Write.pdf, HIVE-21052.1.patch, > HIVE-21052.2.patch, HIVE-21052.3.patch > > > If the transaction is aborted between openTxn and addPartitions and data has > been written on the table the transaction manager will think it's an empty > transaction and no cleaning will be done. > This is currently an issue in the streaming API and in micromanaged tables. > As proposed by [~ekoifman] this can be solved by: > * Writing an entry with a special marker to TXN_COMPONENTS at openTxn and > when addPartitions is called remove this entry from TXN_COMPONENTS and add > the corresponding partition entry to TXN_COMPONENTS. > * If the cleaner finds and entry with a special marker in TXN_COMPONENTS that > specifies that a transaction was opened and it was aborted it must generate > jobs for the worker for every possible partition available. > cc [~ewohlstadter] -- This message was sent by Atlassian JIRA (v7.6.3#76005)