flow0ftime opened a new issue, #24373:
URL: https://github.com/apache/beam/issues/24373
### What happened?
When we ran a CI/CD/UT in GitLab runner environment after upgrading to
Apache Beam Java SDK v2.41.0, we got a 404 not found error when it tried to
delete some nonexistent table IDs, but no error occurred if we ran in the local
environment.
We could see that during this building process. BigQuery created some
temporary tables and successfully deleted them after that. So we do not
understand why it tried to delete some nonexistent table IDs.
We would very much appreciate your help to troubleshoot this issue.
```
1084623 [ERROR] test007 Time elapsed: 143.987 s <<< ERROR!
org.apache.beam.sdk.Pipeline$PipelineExecutionException:
java.io.IOException: Unable to delete table:
temp_table_beam_bq_job_QUERY_issfileoutputpipelinetestroot1021013832c093529f_8d97b954d1ac4cb2938cac9753367693,
aborting after 9 retries.
at
jp.jcblab.jdep.bcbk.output.iss.IssFileOutputPipelineTest.test007(IssFileOutputPipelineTest.java:649)
Caused by: java.io.IOException: Unable to delete table:
temp_table_beam_bq_job_QUERY_issfileoutputpipelinetestroot1021013832c093529f_8d97b954d1ac4cb2938cac9753367693,
aborting after 9 retries.
Caused by:
com.google.api.client.googleapis.json.GoogleJsonResponseException:
404 Not Found
DELETE
https://bigquery.googleapis.com/bigquery/v2/projects/jdep-dev-bcbk/datasets/temp_dataset_beam_bq_job_QUERY_issfileoutputpipelinetestroot1021013832c093529f_8d97b954d1ac4cb2938cac9753367693/tables/temp_table_beam_bq_job_QUERY_issfileoutputpipelinetestroot1021013832c093529f_8d97b954d1ac4cb2938cac9753367693
{
"code" : 404,
"errors" : [ {
"domain" : "global",
"message" : "Not found: Dataset
jdep-dev-bcbk:temp_dataset_beam_bq_job_QUERY_issfileoutputpipelinetestroot1021013832c093529f_8d97b954d1ac4cb2938cac9753367693",
"reason" : "notFound"
} ],
"message" : "Not found: Dataset
jdep-dev-bcbk:temp_dataset_beam_bq_job_QUERY_issfileoutputpipelinetestroot1021013832c093529f_8d97b954d1ac4cb2938cac9753367693",
"status" : "NOT_FOUND"
}
1084626 [INFO] Running
jp.jcblab.jdep.bcbk.output.iss.function.CreateEndRecordFnTest
1085154 [INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time
elapsed: 0.521 s - in
jp.jcblab.jdep.bcbk.output.iss.function.CreateEndRecordFnTest
1085160 [INFO] Running
jp.jcblab.jdep.bcbk.output.iss.function.CreateDataRecordFnTest
1111088 [INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time
elapsed: 25.925 s - in
jp.jcblab.jdep.bcbk.output.iss.function.CreateDataRecordFnTest
1111089 [INFO] Running
jp.jcblab.jdep.bcbk.output.iss.function.CreateTrailerRecordFnTest
1111258 [INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time
elapsed: 0.117 s - in
jp.jcblab.jdep.bcbk.output.iss.function.CreateTrailerRecordFnTest
1111258 [INFO] Running
jp.jcblab.jdep.bcbk.output.iss.function.CreateHeaderRecordFnTest
1111400 [INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time
elapsed: 0.138 s - in
jp.jcblab.jdep.bcbk.output.iss.function.CreateHeaderRecordFnTest
1111980 [INFO]
1111980 [INFO] Results:
1111980 [INFO]
1111980 [ERROR] Errors:
1111980 [ERROR] IssFileOutputPipelineTest.test006:484 » PipelineExecution
java.io.IOException:...
1111980 [ERROR] IssFileOutputPipelineTest.test007:649 » PipelineExecution
java.io.IOException:...
1111980 [INFO]
1111980 [ERROR] Tests run: 14, Failures: 0, Errors: 2, Skipped: 0
1111980 [INFO]
1111986 [INFO]
1111986 [INFO]
------------------------------------------------------------------------
1111986 [INFO] Skipping iss-file-output-pipeline
1111986 [INFO] This project has been banned from the build due to previous
failures.
1111986 [INFO]
------------------------------------------------------------------------
1111986 [INFO]
------------------------------------------------------------------------
1111986 [INFO] BUILD FAILURE
1111986 [INFO]
------------------------------------------------------------------------
1111987 [INFO] Total time: 18:30 min
1111987 [INFO] Finished at: 2022-10-21T01:45:03Z
1111987 [INFO]
------------------------------------------------------------------------
1111988 [ERROR] Failed to execute goal
org.apache.maven.plugins:maven-surefire-plugin:2.22.2:test (default-test) on
project iss-file-output-pipeline: There are test failures.
1111988 [ERROR]
1111988 [ERROR] Please refer to
/builds/bcbk/bcbk-system/file-output/iss-file-output-pipeline/target/surefire-reports
for the individual test results.
1111988 [ERROR] Please refer to dump files (if any exist) [date].dump,
[date]-jvmRun[N].dump and [date].dumpstream.
1111989 [ERROR] -> [Help 1]
org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute
goal org.apache.maven.plugins:maven-surefire-plugin:2.22.2:test (default-test)
on project iss-file-output-pipeline: There are test failures.
Please refer to
/builds/bcbk/bcbk-system/file-output/iss-file-output-pipeline/target/surefire-reports
for the individual test results.
Please refer to dump files (if any exist) [date].dump, [date]-jvmRun[N].dump
and [date].dumpstream.
at org.apache.maven.lifecycle.internal.MojoExecutor.execute
(MojoExecutor.java:215)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute
(MojoExecutor.java:156)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute
(MojoExecutor.java:148)
at
org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject
(LifecycleModuleBuilder.java:117)
at
org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject
(LifecycleModuleBuilder.java:81)
at
org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build
(SingleThreadedBuilder.java:56)
at org.apache.maven.lifecycle.internal.LifecycleStarter.execute
(LifecycleStarter.java:128)
at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:305)
at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:192)
at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:105)
at org.apache.maven.cli.MavenCli.execute (MavenCli.java:957)
at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:289)
at org.apache.maven.cli.MavenCli.main (MavenCli.java:193)
at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
at jdk.internal.reflect.NativeMethodAccessorImpl.invoke
(NativeMethodAccessorImpl.java:62)
at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke
(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke (Method.java:566)
at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced
(Launcher.java:282)
at org.codehaus.plexus.classworlds.launcher.Launcher.launch
(Launcher.java:225)
at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode
(Launcher.java:406)
at org.codehaus.plexus.classworlds.launcher.Launcher.main
(Launcher.java:347)
Caused by: org.apache.maven.plugin.MojoFailureException: There are test
failures.
Please refer to
/builds/bcbk/bcbk-system/file-output/iss-file-output-pipeline/target/surefire-reports
for the individual test results.
Please refer to dump files (if any exist) [date].dump, [date]-jvmRun[N].dump
and [date].dumpstream.
at org.apache.maven.plugin.surefire.SurefireHelper.throwException
(SurefireHelper.java:289)
at org.apache.maven.plugin.surefire.SurefireHelper.reportExecution
(SurefireHelper.java:161)
at org.apache.maven.plugin.surefire.SurefirePlugin.handleSummary
(SurefirePlugin.java:364)
at
org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeAfterPreconditionsChecked
(AbstractSurefireMojo.java:1041)
at org.apache.maven.plugin.surefire.AbstractSurefireMojo.execute
(AbstractSurefireMojo.java:857)
at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo
(DefaultBuildPluginManager.java:137)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute
(MojoExecutor.java:210)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute
(MojoExecutor.java:156)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute
(MojoExecutor.java:148)
at
org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject
(LifecycleModuleBuilder.java:117)
at
org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject
(LifecycleModuleBuilder.java:81)
at
org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build
(SingleThreadedBuilder.java:56)
at org.apache.maven.lifecycle.internal.LifecycleStarter.execute
(LifecycleStarter.java:128)
at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:305)
at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:192)
at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:105)
at org.apache.maven.cli.MavenCli.execute (MavenCli.java:957)
at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:289)
at org.apache.maven.cli.MavenCli.main (MavenCli.java:193)
at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
at jdk.internal.reflect.NativeMethodAccessorImpl.invoke
(NativeMethodAccessorImpl.java:62)
at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke
(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke (Method.java:566)
at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced
(Launcher.java:282)
at org.codehaus.plexus.classworlds.launcher.Launcher.launch
(Launcher.java:225)
at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode
(Launcher.java:406)
at org.codehaus.plexus.classworlds.launcher.Launcher.main
(Launcher.java:347)
1111995 [ERROR]
1111995 [ERROR] Re-run Maven using the -X switch to enable full debug
logging.
1111995 [ERROR]
1111995 [ERROR] For more information about the errors and possible
solutions, please read the following articles:
1111995 [ERROR] [Help 1]
http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
Cleaning up project directory and file based variables
00:00
ERROR: Job failed: command terminated with exit code 1
```
[dataflow_log.txt](https://github.com/apache/beam/files/10102283/dataflow_log.txt)
### Issue Priority
Priority: 2
### Issue Component
Component: beam-community
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
