[
https://issues.apache.org/jira/browse/HIVE-11325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14960498#comment-14960498
]
Hive QA commented on HIVE-11325:
--------------------------------
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12746327/HIVE-11325.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/5674/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/5674/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-5674/
Messages:
{noformat}
**** This message was trimmed, see log for full details ****
[WARNING]
/data/hive-ptest/working/apache-github-source-source/cli/src/java/org/apache/hadoop/hive/cli/CliDriver.java:
Recompile with -Xlint:unchecked for details.
[INFO]
[INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) @
hive-cli ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory
/data/hive-ptest/working/apache-github-source-source/cli/src/test/resources
[INFO] Copying 3 resources
[INFO]
[INFO] --- maven-antrun-plugin:1.7:run (setup-test-dirs) @ hive-cli ---
[INFO] Executing tasks
main:
[mkdir] Created dir:
/data/hive-ptest/working/apache-github-source-source/cli/target/tmp
[mkdir] Created dir:
/data/hive-ptest/working/apache-github-source-source/cli/target/warehouse
[mkdir] Created dir:
/data/hive-ptest/working/apache-github-source-source/cli/target/tmp/conf
[copy] Copying 14 files to
/data/hive-ptest/working/apache-github-source-source/cli/target/tmp/conf
[INFO] Executed tasks
[INFO]
[INFO] --- maven-compiler-plugin:3.1:testCompile (default-testCompile) @
hive-cli ---
[INFO] Compiling 4 source files to
/data/hive-ptest/working/apache-github-source-source/cli/target/test-classes
[WARNING]
/data/hive-ptest/working/apache-github-source-source/cli/src/test/org/apache/hadoop/hive/cli/TestCliDriverMethods.java:
/data/hive-ptest/working/apache-github-source-source/cli/src/test/org/apache/hadoop/hive/cli/TestCliDriverMethods.java
uses unchecked or unsafe operations.
[WARNING]
/data/hive-ptest/working/apache-github-source-source/cli/src/test/org/apache/hadoop/hive/cli/TestCliDriverMethods.java:
Recompile with -Xlint:unchecked for details.
[INFO]
[INFO] --- maven-surefire-plugin:2.16:test (default-test) @ hive-cli ---
[INFO] Tests are skipped.
[INFO]
[INFO] --- maven-jar-plugin:2.2:jar (default-jar) @ hive-cli ---
[INFO] Building jar:
/data/hive-ptest/working/apache-github-source-source/cli/target/hive-cli-2.0.0-SNAPSHOT.jar
[INFO]
[INFO] --- maven-site-plugin:3.3:attach-descriptor (attach-descriptor) @
hive-cli ---
[INFO]
[INFO] --- maven-install-plugin:2.4:install (default-install) @ hive-cli ---
[INFO] Installing
/data/hive-ptest/working/apache-github-source-source/cli/target/hive-cli-2.0.0-SNAPSHOT.jar
to
/home/hiveptest/.m2/repository/org/apache/hive/hive-cli/2.0.0-SNAPSHOT/hive-cli-2.0.0-SNAPSHOT.jar
[INFO] Installing
/data/hive-ptest/working/apache-github-source-source/cli/pom.xml to
/home/hiveptest/.m2/repository/org/apache/hive/hive-cli/2.0.0-SNAPSHOT/hive-cli-2.0.0-SNAPSHOT.pom
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] Building Hive Contrib 2.0.0-SNAPSHOT
[INFO] ------------------------------------------------------------------------
[INFO]
[INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ hive-contrib ---
[INFO] Deleting
/data/hive-ptest/working/apache-github-source-source/contrib/target
[INFO] Deleting /data/hive-ptest/working/apache-github-source-source/contrib
(includes = [datanucleus.log, derby.log], excludes = [])
[INFO]
[INFO] --- maven-enforcer-plugin:1.3.1:enforce (enforce-no-snapshots) @
hive-contrib ---
[INFO]
[INFO] --- maven-remote-resources-plugin:1.5:process (default) @ hive-contrib
---
[WARNING] Invalid project model for artifact
[pentaho-aggdesigner-algorithm:org.pentaho:5.1.5-jhyde]. It will be ignored by
the remote resources Mojo.
[INFO]
[INFO] --- maven-resources-plugin:2.6:resources (default-resources) @
hive-contrib ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory
/data/hive-ptest/working/apache-github-source-source/contrib/src/main/resources
[INFO] Copying 3 resources
[INFO]
[INFO] --- maven-antrun-plugin:1.7:run (define-classpath) @ hive-contrib ---
[INFO] Executing tasks
main:
[INFO] Executed tasks
[INFO]
[INFO] --- maven-compiler-plugin:3.1:compile (default-compile) @ hive-contrib
---
[INFO] Compiling 40 source files to
/data/hive-ptest/working/apache-github-source-source/contrib/target/classes
[WARNING]
/data/hive-ptest/working/apache-github-source-source/contrib/src/java/org/apache/hadoop/hive/contrib/udaf/example/UDAFExampleMax.java:
Some input files use or override a deprecated API.
[WARNING]
/data/hive-ptest/working/apache-github-source-source/contrib/src/java/org/apache/hadoop/hive/contrib/udaf/example/UDAFExampleMax.java:
Recompile with -Xlint:deprecation for details.
[WARNING]
/data/hive-ptest/working/apache-github-source-source/contrib/src/java/org/apache/hadoop/hive/contrib/udf/example/UDFExampleStructPrint.java:
/data/hive-ptest/working/apache-github-source-source/contrib/src/java/org/apache/hadoop/hive/contrib/udf/example/UDFExampleStructPrint.java
uses unchecked or unsafe operations.
[WARNING]
/data/hive-ptest/working/apache-github-source-source/contrib/src/java/org/apache/hadoop/hive/contrib/udf/example/UDFExampleStructPrint.java:
Recompile with -Xlint:unchecked for details.
[INFO]
[INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) @
hive-contrib ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory
/data/hive-ptest/working/apache-github-source-source/contrib/src/test/resources
[INFO] Copying 3 resources
[INFO]
[INFO] --- maven-antrun-plugin:1.7:run (setup-test-dirs) @ hive-contrib ---
[INFO] Executing tasks
main:
[mkdir] Created dir:
/data/hive-ptest/working/apache-github-source-source/contrib/target/tmp
[mkdir] Created dir:
/data/hive-ptest/working/apache-github-source-source/contrib/target/warehouse
[mkdir] Created dir:
/data/hive-ptest/working/apache-github-source-source/contrib/target/tmp/conf
[copy] Copying 14 files to
/data/hive-ptest/working/apache-github-source-source/contrib/target/tmp/conf
[INFO] Executed tasks
[INFO]
[INFO] --- maven-compiler-plugin:3.1:testCompile (default-testCompile) @
hive-contrib ---
[INFO] Compiling 2 source files to
/data/hive-ptest/working/apache-github-source-source/contrib/target/test-classes
[WARNING]
/data/hive-ptest/working/apache-github-source-source/contrib/src/test/org/apache/hadoop/hive/contrib/serde2/TestRegexSerDe.java:
/data/hive-ptest/working/apache-github-source-source/contrib/src/test/org/apache/hadoop/hive/contrib/serde2/TestRegexSerDe.java
uses or overrides a deprecated API.
[WARNING]
/data/hive-ptest/working/apache-github-source-source/contrib/src/test/org/apache/hadoop/hive/contrib/serde2/TestRegexSerDe.java:
Recompile with -Xlint:deprecation for details.
[INFO]
[INFO] --- maven-surefire-plugin:2.16:test (default-test) @ hive-contrib ---
[INFO] Tests are skipped.
[INFO]
[INFO] --- maven-jar-plugin:2.2:jar (default-jar) @ hive-contrib ---
[INFO] Building jar:
/data/hive-ptest/working/apache-github-source-source/contrib/target/hive-contrib-2.0.0-SNAPSHOT.jar
[INFO]
[INFO] --- maven-site-plugin:3.3:attach-descriptor (attach-descriptor) @
hive-contrib ---
[INFO]
[INFO] --- maven-install-plugin:2.4:install (default-install) @ hive-contrib ---
[INFO] Installing
/data/hive-ptest/working/apache-github-source-source/contrib/target/hive-contrib-2.0.0-SNAPSHOT.jar
to
/home/hiveptest/.m2/repository/org/apache/hive/hive-contrib/2.0.0-SNAPSHOT/hive-contrib-2.0.0-SNAPSHOT.jar
[INFO] Installing
/data/hive-ptest/working/apache-github-source-source/contrib/pom.xml to
/home/hiveptest/.m2/repository/org/apache/hive/hive-contrib/2.0.0-SNAPSHOT/hive-contrib-2.0.0-SNAPSHOT.pom
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] Building Hive HBase Handler 2.0.0-SNAPSHOT
[INFO] ------------------------------------------------------------------------
[INFO]
[INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ hive-hbase-handler ---
[INFO] Deleting
/data/hive-ptest/working/apache-github-source-source/hbase-handler/target
[INFO] Deleting
/data/hive-ptest/working/apache-github-source-source/hbase-handler (includes =
[datanucleus.log, derby.log], excludes = [])
[INFO]
[INFO] --- maven-enforcer-plugin:1.3.1:enforce (enforce-no-snapshots) @
hive-hbase-handler ---
[INFO]
[INFO] --- maven-remote-resources-plugin:1.5:process (default) @
hive-hbase-handler ---
[WARNING] Invalid project model for artifact
[pentaho-aggdesigner-algorithm:org.pentaho:5.1.5-jhyde]. It will be ignored by
the remote resources Mojo.
[INFO]
[INFO] --- maven-resources-plugin:2.6:resources (default-resources) @
hive-hbase-handler ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory
/data/hive-ptest/working/apache-github-source-source/hbase-handler/src/main/resources
[INFO] Copying 3 resources
[INFO]
[INFO] --- maven-antrun-plugin:1.7:run (define-classpath) @ hive-hbase-handler
---
[INFO] Executing tasks
main:
[INFO] Executed tasks
[INFO]
[INFO] --- maven-compiler-plugin:3.1:compile (default-compile) @
hive-hbase-handler ---
[INFO] Compiling 33 source files to
/data/hive-ptest/working/apache-github-source-source/hbase-handler/target/classes
[INFO] -------------------------------------------------------------
[WARNING] COMPILATION WARNING :
[INFO] -------------------------------------------------------------
[WARNING]
/data/hive-ptest/working/apache-github-source-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/AbstractHBaseKeyFactory.java:
Some input files use or override a deprecated API.
[WARNING]
/data/hive-ptest/working/apache-github-source-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/AbstractHBaseKeyFactory.java:
Recompile with -Xlint:deprecation for details.
[WARNING]
/data/hive-ptest/working/apache-github-source-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSerDeParameters.java:
Some input files use unchecked or unsafe operations.
[WARNING]
/data/hive-ptest/working/apache-github-source-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSerDeParameters.java:
Recompile with -Xlint:unchecked for details.
[INFO] 4 warnings
[INFO] -------------------------------------------------------------
[INFO] -------------------------------------------------------------
[ERROR] COMPILATION ERROR :
[INFO] -------------------------------------------------------------
[ERROR]
/data/hive-ptest/working/apache-github-source-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHFileOutputFormat.java:[164,24]
cannot find symbol
symbol: method isDir()
location: variable srcDir of type org.apache.hadoop.fs.Path
[INFO] 1 error
[INFO] -------------------------------------------------------------
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] Hive .............................................. SUCCESS [13.189s]
[INFO] Hive Shims Common ................................. SUCCESS [15.029s]
[INFO] Hive Shims 0.20S .................................. SUCCESS [3.959s]
[INFO] Hive Shims 0.23 ................................... SUCCESS [11.769s]
[INFO] Hive Shims Scheduler .............................. SUCCESS [2.347s]
[INFO] Hive Shims ........................................ SUCCESS [3.572s]
[INFO] Hive Storage API .................................. SUCCESS [3.830s]
[INFO] Hive Common ....................................... SUCCESS [22.237s]
[INFO] Hive Serde ........................................ SUCCESS [20.788s]
[INFO] Hive Metastore .................................... SUCCESS [1:20.849s]
[INFO] Hive Ant Utilities ................................ SUCCESS [2.109s]
[INFO] Hive Llap Client .................................. SUCCESS [8.562s]
[INFO] Spark Remote Client ............................... SUCCESS [15.956s]
[INFO] Hive Query Language ............................... SUCCESS [3:23.412s]
[INFO] Hive Service ...................................... SUCCESS [15.057s]
[INFO] Hive Accumulo Handler ............................. SUCCESS [7.041s]
[INFO] Hive JDBC ......................................... SUCCESS [23.719s]
[INFO] Hive Beeline ...................................... SUCCESS [4.057s]
[INFO] Hive CLI .......................................... SUCCESS [5.716s]
[INFO] Hive Contrib ...................................... SUCCESS [2.653s]
[INFO] Hive HBase Handler ................................ FAILURE [3.810s]
[INFO] Hive HCatalog ..................................... SKIPPED
[INFO] Hive HCatalog Core ................................ SKIPPED
[INFO] Hive HCatalog Pig Adapter ......................... SKIPPED
[INFO] Hive HCatalog Server Extensions ................... SKIPPED
[INFO] Hive HCatalog Webhcat Java Client ................. SKIPPED
[INFO] Hive HCatalog Webhcat ............................. SKIPPED
[INFO] Hive HCatalog Streaming ........................... SKIPPED
[INFO] Hive HPL/SQL ...................................... SKIPPED
[INFO] Hive HWI .......................................... SKIPPED
[INFO] Hive ODBC ......................................... SKIPPED
[INFO] Hive Shims Aggregator ............................. SKIPPED
[INFO] Hive TestUtils .................................... SKIPPED
[INFO] Hive Packaging .................................... SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 7:52.943s
[INFO] Finished at: Fri Oct 16 06:40:38 EDT 2015
[INFO] Final Memory: 215M/1065M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-compiler-plugin:3.1:compile (default-compile) on
project hive-hbase-handler: Compilation failure
[ERROR]
/data/hive-ptest/working/apache-github-source-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHFileOutputFormat.java:[164,24]
cannot find symbol
[ERROR] symbol: method isDir()
[ERROR] location: variable srcDir of type org.apache.hadoop.fs.Path
[ERROR] -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e
switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please
read the following articles:
[ERROR] [Help 1]
http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR] mvn <goals> -rf :hive-hbase-handler
+ exit 1
'
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12746327 - PreCommit-HIVE-TRUNK-Build
> Infinite loop in HiveHFileOutputFormat
> --------------------------------------
>
> Key: HIVE-11325
> URL: https://issues.apache.org/jira/browse/HIVE-11325
> Project: Hive
> Issue Type: Bug
> Components: HBase Handler
> Affects Versions: 1.0.0
> Reporter: Harsh J
> Assignee: Harsh J
> Attachments: HIVE-11325.patch
>
>
> No idea why {{hbase_handler_bulk.q}} does not catch this if its being run
> regularly in Hive builds, but here's the gist of the issue:
> The condition at
> https://github.com/apache/hive/blob/master/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHFileOutputFormat.java#L152-L164
> indicates that we will infinitely loop until we find a file whose last path
> component (the name) is equal to the column family name.
> In execution, however, the iteration enters an actual infinite loop cause the
> file we end up considering as the srcDir name, is actually the region file,
> whose name will never match the family name.
> This is an example of the IPC the listing loop of a 100% progress task gets
> stuck in:
> {code}
> 2015-07-21 10:32:20,662 TRACE [main] org.apache.hadoop.ipc.ProtobufRpcEngine:
> 1: Call -> cdh54.vm/172.16.29.132:8020: getListing {src:
> "/user/hive/warehouse/hbase_test/_temporary/1/_temporary/attempt_1436935612068_0011_m_000000_0/family/97112ac1c09548ae87bd85af072d2e8c"
> startAfter: "" needLocation: false}
> 2015-07-21 10:32:20,662 DEBUG [IPC Parameter Sending Thread #1]
> org.apache.hadoop.ipc.Client: IPC Client (1551465414) connection to
> cdh54.vm/172.16.29.132:8020 from hive sending #510346
> 2015-07-21 10:32:20,662 DEBUG [IPC Client (1551465414) connection to
> cdh54.vm/172.16.29.132:8020 from hive] org.apache.hadoop.ipc.Client: IPC
> Client (1551465414) connection to cdh54.vm/172.16.29.132:8020 from hive got
> value #510346
> 2015-07-21 10:32:20,662 DEBUG [main] org.apache.hadoop.ipc.ProtobufRpcEngine:
> Call: getListing took 0ms
> 2015-07-21 10:32:20,662 TRACE [main] org.apache.hadoop.ipc.ProtobufRpcEngine:
> 1: Response <- cdh54.vm/172.16.29.132:8020: getListing {dirList {
> partialListing { fileType: IS_FILE path: "" length: 863 permission { perm:
> 4600 } owner: "hive" group: "hive" modification_time: 1437454718130
> access_time: 1437454717973 block_replication: 1 blocksize: 134217728 fileId:
> 33960 childrenNum: 0 storagePolicy: 0 } remainingEntries: 0 }}
> {code}
> The path we are getting out of the listing results is
> {{/user/hive/warehouse/hbase_test/_temporary/1/_temporary/attempt_1436935612068_0011_m_000000_0/family/97112ac1c09548ae87bd85af072d2e8c}},
> but instead of checking the path's parent {{family}} we're instead looping
> infinitely over its hashed filename {{97112ac1c09548ae87bd85af072d2e8c}}
> cause it does not match {{family}}.
> It stays in the infinite loop therefore, until the MR framework kills it away
> due to an idle task timeout (and then since the subsequent task attempts fail
> outright, the job fails).
> While doing a {{getPath().getParent()}} will resolve that, is that infinite
> loop even necessary? Especially given the fact that we throw exceptions if
> there are no entries or there is more than one entry.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
