[jira] [Commented] (HIVE-11887) spark tests break the build on a shared machine, can break HiveQA
[ https://issues.apache.org/jira/browse/HIVE-11887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15266036#comment-15266036 ] Ashutosh Chauhan commented on HIVE-11887: - +1 > spark tests break the build on a shared machine, can break HiveQA > - > > Key: HIVE-11887 > URL: https://issues.apache.org/jira/browse/HIVE-11887 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-11887.01.patch, HIVE-11887.02.patch, > HIVE-11887.patch > > > Spark download creates UDFExampleAdd jar in /tmp; when building on a shared > machine, someone else's jar from a build prevents this jar from being created > (I have no permissions to this file because it was created by a different > user) and the build fails. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11887) spark tests break the build on a shared machine, can break HiveQA
[
https://issues.apache.org/jira/browse/HIVE-11887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15255731#comment-15255731
]
Hive QA commented on HIVE-11887:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12800281/HIVE-11887.02.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 38 failed/errored test(s), 9952 tests
executed
*Failed tests:*
{noformat}
TestHWISessionManager - did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3
org.apache.hadoop.hive.llap.daemon.impl.comparator.TestShortestJobFirstComparator.testWaitQueueComparatorWithinDagPriority
org.apache.hadoop.hive.metastore.TestAuthzApiEmbedAuthorizerInRemote.org.apache.hadoop.hive.metastore.TestAuthzApiEmbedAuthorizerInRemote
org.apache.hadoop.hive.metastore.TestFilterHooks.org.apache.hadoop.hive.metastore.TestFilterHooks
org.apache.hadoop.hive.metastore.TestHiveMetaStorePartitionSpecs.testAddPartitions
org.apache.hadoop.hive.metastore.TestHiveMetaStorePartitionSpecs.testFetchingPartitionsWithDifferentSchemas
org.apache.hadoop.hive.metastore.TestHiveMetaStorePartitionSpecs.testGetPartitionSpecs_WithAndWithoutPartitionGrouping
org.apache.hadoop.hive.metastore.TestMetaStoreEndFunctionListener.testEndFunctionListener
org.apache.hadoop.hive.metastore.TestMetaStoreEventListenerOnlyOnCommit.testEventStatus
org.apache.hadoop.hive.metastore.TestMetaStoreInitListener.testMetaStoreInitListener
org.apache.hadoop.hive.metastore.TestMetaStoreMetrics.org.apache.hadoop.hive.metastore.TestMetaStoreMetrics
org.apache.hadoop.hive.metastore.TestPartitionNameWhitelistValidation.testAppendPartitionWithCommas
org.apache.hadoop.hive.metastore.TestPartitionNameWhitelistValidation.testAppendPartitionWithValidCharacters
org.apache.hadoop.hive.metastore.TestRetryingHMSHandler.testRetryingHMSHandler
org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager.testLockTimeout
org.apache.hadoop.hive.ql.security.TestClientSideAuthorizationProvider.testSimplePrivileges
org.apache.hadoop.hive.ql.security.TestExtendedAcls.org.apache.hadoop.hive.ql.security.TestExtendedAcls
org.apache.hadoop.hive.ql.security.TestFolderPermissions.org.apache.hadoop.hive.ql.security.TestFolderPermissions
org.apache.hadoop.hive.ql.security.TestMetastoreAuthorizationProvider.testSimplePrivileges
org.apache.hadoop.hive.ql.security.TestMultiAuthorizationPreEventListener.org.apache.hadoop.hive.ql.security.TestMultiAuthorizationPreEventListener
org.apache.hadoop.hive.ql.security.TestStorageBasedClientSideAuthorizationProvider.testSimplePrivileges
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationDrops.testDropPartition
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationProvider.testSimplePrivileges
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationProviderWithACL.testSimplePrivileges
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationReads.testReadDbFailure
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationReads.testReadDbSuccess
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationReads.testReadTableFailure
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationReads.testReadTableSuccess
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationReads.testReadTableSuccessWithReadOnly
org.apache.hadoop.hive.thrift.TestHadoopAuthBridge23.testDelegationTokenSharedStore
org.apache.hadoop.hive.thrift.TestHadoopAuthBridge23.testMetastoreProxyUser
org.apache.hadoop.hive.thrift.TestHadoopAuthBridge23.testSaslWithHiveMetaStore
org.apache.hive.beeline.TestSchemaTool.testSchemaInit
org.apache.hive.hcatalog.listener.TestDbNotificationListener.cleanupNotifs
org.apache.hive.hcatalog.listener.TestDbNotificationListener.dropDatabase
org.apache.hive.minikdc.TestJdbcWithDBTokenStore.org.apache.hive.minikdc.TestJdbcWithDBTokenStore
org.apache.hive.service.TestHS2ImpersonationWithRemoteMS.org.apache.hive.service.TestHS2ImpersonationWithRemoteMS
{noformat}
Test results:
http://ec2-54-177-240-2.us-west-1.compute.amazonaws.com/job/PreCommit-HIVE-MASTER-Build/65/testReport
Console output:
http://ec2-54-177-240-2.us-west-1.compute.amazonaws.com/job/PreCommit-HIVE-MASTER-Build/65/console
Test logs:
http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-65/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 38 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12800281 - PreCommit-HIVE-MASTER-Build
> spark tests break the build on a
[jira] [Commented] (HIVE-11887) spark tests break the build on a shared machine, can break HiveQA
[
https://issues.apache.org/jira/browse/HIVE-11887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15208460#comment-15208460
]
Hive QA commented on HIVE-11887:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12794573/HIVE-11887.01.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 5 failed/errored test(s), 9851 tests executed
*Failed tests:*
{noformat}
TestSparkCliDriver-groupby3_map.q-sample2.q-auto_join14.q-and-12-more - did not
produce a TEST-*.xml file
TestSparkCliDriver-groupby_map_ppr_multi_distinct.q-table_access_keys_stats.q-groupby4_noskew.q-and-12-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-join_rc.q-insert1.q-vectorized_rcfile_columnar.q-and-12-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-ppd_join4.q-join9.q-ppd_join3.q-and-12-more - did not
produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_llap_udf
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/7344/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/7344/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-7344/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 5 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12794573 - PreCommit-HIVE-TRUNK-Build
> spark tests break the build on a shared machine, can break HiveQA
> -
>
> Key: HIVE-11887
> URL: https://issues.apache.org/jira/browse/HIVE-11887
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-11887.01.patch, HIVE-11887.patch
>
>
> Spark download creates UDFExampleAdd jar in /tmp; when building on a shared
> machine, someone else's jar from a build prevents this jar from being created
> (I have no permissions to this file because it was created by a different
> user) and the build fails.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-11887) spark tests break the build on a shared machine
[
https://issues.apache.org/jira/browse/HIVE-11887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15204901#comment-15204901
]
Sergey Shelukhin commented on HIVE-11887:
-
This also breaks HiveQA occasionally:
{noformat}
[exec] + sed '/package /d'
/home/hiveptest/54.166.14.147-hiveptest-0/apache-github-source-source/itests/../contrib/src/java/org/apache/hadoop/hive/contrib/udf/example/UDFExampleAdd.java
[exec] + javac -cp
/home/hiveptest/54.166.14.147-hiveptest-0/maven/org/apache/hive/hive-exec/2.1.0-SNAPSHOT/hive-exec-2.1.0-SNAPSHOT.jar
/tmp/UDFExampleAdd.java -d /tmp
[exec] error: error reading /tmp/UDFExampleAdd.java;
/tmp/UDFExampleAdd.java (No such file or directory)
[exec] 1 error
[exec] + jar -cf /tmp/udfexampleadd-1.0.jar -C /tmp UDFExampleAdd.class
[exec] /tmp/UDFExampleAdd.class : no such file or directory
{noformat}
In some JIRA.
I am going to submit a patch to remove this jar for the time being, and disable
tests that depend on it if it's not too many. It can be restored in some
acceptable manner.
> spark tests break the build on a shared machine
> ---
>
> Key: HIVE-11887
> URL: https://issues.apache.org/jira/browse/HIVE-11887
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-11887.patch
>
>
> Spark download creates UDFExampleAdd jar in /tmp; when building on a shared
> machine, someone else's jar from a build prevents this jar from being created
> (I have no permissions to this file because it was created by a different
> user) and the build fails.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-11887) spark tests break the build on a shared machine
[
https://issues.apache.org/jira/browse/HIVE-11887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15134115#comment-15134115
]
Hive QA commented on HIVE-11887:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12785832/HIVE-11887.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 6 failed/errored test(s), 10021 tests
executed
*Failed tests:*
{noformat}
TestMiniTezCliDriver-unionDistinct_1.q-insert_values_non_partitioned.q-selectDistinctStar.q-and-12-more
- did not produce a TEST-*.xml file
TestMiniTezCliDriver-vector_interval_2.q-vectorization_7.q-vectorization_14.q-and-12-more
- did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_char_simple
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_uri_import
org.apache.hive.jdbc.TestMultiSessionsHS2WithLocalClusterSpark.testSparkQuery
org.apache.hive.jdbc.TestSSL.testSSLVersion
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6873/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6873/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-6873/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 6 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12785832 - PreCommit-HIVE-TRUNK-Build
> spark tests break the build on a shared machine
> ---
>
> Key: HIVE-11887
> URL: https://issues.apache.org/jira/browse/HIVE-11887
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-11887.patch
>
>
> Spark download creates UDFExampleAdd jar in /tmp; when building on a shared
> machine, someone else's jar from a build prevents this jar from being created
> (I have no permissions to this file because it was created by a different
> user) and the build fails.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-11887) spark tests break the build on a shared machine
[ https://issues.apache.org/jira/browse/HIVE-11887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14947965#comment-14947965 ] Rui Li commented on HIVE-11887: --- Sorry for the late response. Seems the change was introduced in HIVE-9664. [~nntnag17] do you have any idea for this issue? Thanks. > spark tests break the build on a shared machine > --- > > Key: HIVE-11887 > URL: https://issues.apache.org/jira/browse/HIVE-11887 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin > > Spark download creates UDFExampleAdd jar in /tmp; when building on a shared > machine, someone else's jar from a build prevents this jar from being created > (I have no permissions to this file because it was created by a different > user) and the build fails. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11887) spark tests break the build on a shared machine
[
https://issues.apache.org/jira/browse/HIVE-11887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14908414#comment-14908414
]
Sergey Shelukhin commented on HIVE-11887:
-
It's part of itests build:
{noformat}
$ grep -B 30 -A 2 UDFExa itests/pom.xml
download-spark
generate-sources
run
set -x
/bin/pwd
BASE_DIR=./target
HIVE_ROOT=$BASE_DIR/../../../
DOWNLOAD_DIR=./../thirdparty
download() {
url=$1;
finalName=$2
tarName=$(basename $url)
rm -rf $BASE_DIR/$finalName
if [[ ! -f $DOWNLOAD_DIR/$tarName ]]
then
curl -Sso $DOWNLOAD_DIR/$tarName $url
fi
tar -zxf $DOWNLOAD_DIR/$tarName -C $BASE_DIR
mv
$BASE_DIR/spark-${spark.version}-bin-hadoop2-without-hive $BASE_DIR/$finalName
}
mkdir -p $DOWNLOAD_DIR
download
"http://d3jw87u4immizc.cloudfront.net/spark-tarball/spark-${spark.version}-bin-hadoop2-without-hive.tgz;
"spark"
cp -f $HIVE_ROOT/data/conf/spark/log4j2.xml
$BASE_DIR/spark/conf/
sed '/package /d'
${basedir}/${hive.path.to.root}/contrib/src/java/org/apache/hadoop/hive/contrib/udf/example/UDFExampleAdd.java
> /tmp/UDFExampleAdd.java
javac -cp
${settings.localRepository}/org/apache/hive/hive-exec/${project.version}/hive-exec-${project.version}.jar
/tmp/UDFExampleAdd.java -d /tmp
jar -cf /tmp/udfexampleadd-1.0.jar -C /tmp
UDFExampleAdd.class
{noformat}
See the last two lines
> spark tests break the build on a shared machine
> ---
>
> Key: HIVE-11887
> URL: https://issues.apache.org/jira/browse/HIVE-11887
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>
> Spark download creates UDFExampleAdd jar in /tmp; when building on a shared
> machine, someone else's jar from a build prevents this jar from being created
> (I have no permissions to this file because it was created by a different
> user) and the build fails.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-11887) spark tests break the build on a shared machine
[ https://issues.apache.org/jira/browse/HIVE-11887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14907436#comment-14907436 ] Rui Li commented on HIVE-11887: --- Hi [~sershe], do you mean UDFExampleAdd.jar needs to be created under /tmp when building hive on spark? I just tried building on my side but there isn't such a jar under /tmp. > spark tests break the build on a shared machine > --- > > Key: HIVE-11887 > URL: https://issues.apache.org/jira/browse/HIVE-11887 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin > > Spark download creates UDFExampleAdd jar in /tmp; when building on a shared > machine, someone else's jar from a build prevents this jar from being created > (I have no permissions to this file because it was created by a different > user) and the build fails. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11887) spark tests break the build on a shared machine
[ https://issues.apache.org/jira/browse/HIVE-11887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14904967#comment-14904967 ] Xuefu Zhang commented on HIVE-11887: CC: [~lirui] > spark tests break the build on a shared machine > --- > > Key: HIVE-11887 > URL: https://issues.apache.org/jira/browse/HIVE-11887 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin > > Spark download creates UDFExampleAdd jar in /tmp; when building on a shared > machine, someone else's jar from a build prevents this jar from being created > (I have no permissions to this file because it was created by a different > user) and the build fails. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11887) spark tests break the build on a shared machine
[ https://issues.apache.org/jira/browse/HIVE-11887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14876397#comment-14876397 ] Sergey Shelukhin commented on HIVE-11887: - [~xuefuz] can you guys take a look > spark tests break the build on a shared machine > --- > > Key: HIVE-11887 > URL: https://issues.apache.org/jira/browse/HIVE-11887 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin > > Spark download creates UDFExampleAdd jar in /tmp; when building on a shared > machine, someone else's jar from a build prevents this jar from being created > (I have no permissions to this file because it was created by a different > user) and the build fails. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
