Hm, so they look pretty similar except for minor differences in the actual script run. Is there any reason this should be different? Would it be reasonable to try making the 'new' one work like the 'old' one if the former isn't working?
But I still can't figure out why it causes the same odd error every time on this one PR, which is a minor change to tooltips in the UI. I haven't seen other manually-triggered PR builds fail this way. Really mysterious so far! https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/4964/testReport/ Old: #!/bin/bash set -e # fail on any non-zero exit code set -x export AMPLAB_JENKINS=1 export PATH="$PATH:/home/anaconda/envs/py3k/bin" # Prepend JAVA_HOME/bin to fix issue where Zinc's embedded SBT incremental compiler seems to # ignore our JAVA_HOME and use the system javac instead. export PATH="$JAVA_HOME/bin:$PATH" # Add a pre-downloaded version of Maven to the path so that we avoid the flaky download step. export PATH="/home/jenkins/tools/hudson.tasks.Maven_MavenInstallation/Maven_3.3.9/bin/:$PATH" echo "fixing target dir permissions" chmod -R +w target/* || true # stupid hack by sknapp to ensure that the chmod always exits w/0 and doesn't bork the script echo "running git clean -fdx" git clean -fdx # Configure per-build-executor Ivy caches to avoid SBT Ivy lock contention export HOME="/home/sparkivy/per-executor-caches/$EXECUTOR_NUMBER" mkdir -p "$HOME" export SBT_OPTS="-Duser.home=$HOME -Dsbt.ivy.home=$HOME/.ivy2" export SPARK_VERSIONS_SUITE_IVY_PATH="$HOME/.ivy2" ./dev/run-tests-jenkins # Hack to ensure that at least one JVM suite always runs in order to prevent spurious errors from the # Jenkins JUnit test reporter plugin ./build/sbt unsafe/test > /dev/null 2>&1 New: #!/bin/bash set -e export AMPLAB_JENKINS=1 export PATH="$PATH:/home/anaconda/envs/py3k/bin" git clean -fdx # Prepend JAVA_HOME/bin to fix issue where Zinc's embedded SBT incremental compiler seems to # ignore our JAVA_HOME and use the system javac instead. export PATH="$JAVA_HOME/bin:$PATH" # Add a pre-downloaded version of Maven to the path so that we avoid the flaky download step. export PATH="/home/jenkins/tools/hudson.tasks.Maven_MavenInstallation/Maven_3.3.9/bin/:$PATH" # Configure per-build-executor Ivy caches to avoid SBT Ivy lock contention export HOME="/home/sparkivy/per-executor-caches/$EXECUTOR_NUMBER" mkdir -p "$HOME" export SBT_OPTS="-Duser.home=$HOME -Dsbt.ivy.home=$HOME/.ivy2" export SPARK_VERSIONS_SUITE_IVY_PATH="$HOME/.ivy2" # This is required for tests of backport patches. # We need to download the run-tests-codes.sh file because it's imported by run-tests-jenkins. # When running tests on branch-1.0 (and earlier), the older version of run-tests won't set CURRENT_BLOCK, so # the Jenkins scripts will report all failures as "some tests failed" rather than a more specific # error message. if [ ! -f "dev/run-tests-jenkins" ]; then wget https://raw.githubusercontent.com/apache/spark/master/dev/run-tests-jenkins wget https://raw.githubusercontent.com/apache/spark/master/dev/run-tests-codes.sh mv run-tests-jenkins dev/ mv run-tests-codes.sh dev/ chmod 755 dev/run-tests-jenkins chmod 755 dev/run-tests-codes.sh fi ./dev/run-tests-jenkins On Wed, Dec 4, 2019 at 5:53 PM Shane Knapp <skn...@berkeley.edu> wrote: > > ++yin huai for more insight in to the NewSparkPullRequestBuilder job... > > tbh, i never (or still) really understand the exact use for that job, > except that it's triggered by https://spark-prs.appspot.com/ > > shane > > > On Wed, Dec 4, 2019 at 3:34 PM Sean Owen <sro...@gmail.com> wrote: > > > > BTW does anyone know why there are two PR builder jobs? I'm confused > > about why different ones would execute. > > > > Yes I see NewSparkPullRequestBuilder failing on a variety of PRs. > > I don't think it has anything to do with Hive; these PRs touch > > different parts of code but all not related to this failure. > > > > On Wed, Dec 4, 2019 at 12:40 PM Dongjoon Hyun <dongjoon.h...@gmail.com> > > wrote: > > > > > > Hi, Sean. > > > > > > It seems that there is no failure on your other SQL PR. > > > > > > https://github.com/apache/spark/pull/26748 > > > > > > Does the sequential failure happen only at `NewSparkPullRequestBuilder`? > > > Since `NewSparkPullRequestBuilder` is not the same with > > > `SparkPullRequestBuilder`, > > > there might be a root cause inside it if it happens only at > > > `NewSparkPullRequestBuilder`. > > > > > > For `org.apache.hive.service.ServiceException: Failed to Start > > > HiveServer2`, > > > I've observed them before, but the root cause might be different from > > > this one. > > > > > > BTW, to reduce the scope of investigation, could you try with > > > `[hive-1.2]` tag in your PR? > > > > > > Bests, > > > Dongjoon. > > > > > > > > > On Wed, Dec 4, 2019 at 6:29 AM Sean Owen <sro...@gmail.com> wrote: > > >> > > >> I'm seeing consistent failures in the PR builder when touching SQL code: > > >> > > >> https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/4960/testReport/ > > >> > > >> > > >> org.apache.spark.sql.hive.thriftserver.SparkMetadataOperationSuite.Spark's > > >> own GetSchemasOperation(SparkGetSchemasOperation)14 ms2 > > >> > > >> org.apache.spark.sql.hive.thriftserver.ThriftServerWithSparkContextSuite.(It > > >> is not a test it is a sbt.testing.SuiteSelector) > > >> > > >> Looks like this has failed about 6 builds in the past few days. Has > > >> anyone seen this / has a clue what's causing it? errors are like ... > > >> > > >> java.sql.SQLException: No suitable driver found for > > >> jdbc:hive2://localhost:13694/?a=avalue;b=bvalue#c=cvalue;d=dvalue > > >> > > >> > > >> Caused by: sbt.ForkMain$ForkError: java.lang.RuntimeException: class > > >> org.apache.hadoop.hive.metastore.DefaultMetaStoreFilterHookImpl not > > >> org.apache.hadoop.hive.metastore.MetaStoreFilterHook > > > > -- > Shane Knapp > UC Berkeley EECS Research / RISELab Staff Technical Lead > https://rise.cs.berkeley.edu --------------------------------------------------------------------- To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
