Hi Israel,
That was it, many thanks! I had it set to "${java.io.tmpdir}". Is the
requirement to use a GCS location documented somewhere?
cheers
Fabian
> Am 31.08.2022 um 11:25 schrieb Israel Herraiz via users
> <[email protected]>:
>
> What are the command line arguments that you are using for those direct
> runner pipelines? For instance, for BigQuery you will need to set
> --tempLocation to a GCS location for the BQ jobs to work.
>
>
> On Wed, 31 Aug 2022 at 09:50, Fabian Peters <[email protected]
> <mailto:[email protected]>> wrote:
> Good morning!
>
> I'm putting together my Dataflow deployment and am running into another
> problem I don't know how to deal with: I'm running a pipeline via Dataflow,
> which contains a "Workflow executor" transform. The workflow contains a
> number of pipelines that have their run configuration set to Beam-Direct. In
> principle, this works fine. (Yeah!)
>
> However, in this setup a BigQuery Output fails with a
> "java.lang.RuntimeException: Failed to create job with prefix
> beam_bq_job_LOAD_sites_FOO_ID, reached max retries: 3, last failed job:
> null." I see the the same when running just the pipeline (or any other with
> BigQuery Output) via Beam-Direct locally, which makes me think that the GCP
> credentials are not being picked up? Is there something I need to configure?
>
> cheers
>
> Fabian
>
> P.S.: Logs from running locally with Beam-Direct:
>
> 2022/08/31 09:30:07 - sites - ERROR: Error starting the Beam pipeline
> 2022/08/31 09:30:07 - sites - ERROR:
> org.apache.hop.core.exception.HopException:
> 2022/08/31 09:30:07 - sites - Error executing pipeline with runner Direct
> 2022/08/31 09:30:07 - sites - java.lang.RuntimeException: Failed to create
> job with prefix
> beam_bq_job_LOAD_sites_65dba39290c04240933e3a982c0c5699_b77cb1586fc969929097729a4a6cdf2a_00001_00000,
> reached max retries: 3, last failed job: null.
> 2022/08/31 09:30:07 - sites -
> 2022/08/31 09:30:07 - sites - at
> org.apache.hop.beam.engines.BeamPipelineEngine.executePipeline(BeamPipelineEngine.java:258)
> 2022/08/31 09:30:07 - sites - at
> org.apache.hop.beam.engines.BeamPipelineEngine.lambda$startThreads$0(BeamPipelineEngine.java:305)
> 2022/08/31 09:30:07 - sites - at
> java.base/java.lang.Thread.run(Thread.java:829)
> 2022/08/31 09:30:07 - sites - Caused by:
> org.apache.beam.sdk.Pipeline$PipelineExecutionException:
> java.lang.RuntimeException: Failed to create job with prefix
> beam_bq_job_LOAD_sites_65dba39290c04240933e3a982c0c5699_b77cb1586fc969929097729a4a6cdf2a_00001_00000,
> reached max retries: 3, last failed job: null.
> 2022/08/31 09:30:07 - sites - at
> org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:373)
> 2022/08/31 09:30:07 - sites - at
> org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:341)
> 2022/08/31 09:30:07 - sites - at
> org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:218)
> 2022/08/31 09:30:07 - sites - at
> org.apache.hop.beam.engines.BeamPipelineEngine.executePipeline(BeamPipelineEngine.java:246)
> 2022/08/31 09:30:07 - sites - ... 2 more
> 2022/08/31 09:30:07 - sites - Caused by: java.lang.RuntimeException: Failed
> to create job with prefix
> beam_bq_job_LOAD_sites_65dba39290c04240933e3a982c0c5699_b77cb1586fc969929097729a4a6cdf2a_00001_00000,
> reached max retries: 3, last failed job: null.
> 2022/08/31 09:30:07 - sites - at
> org.apache.beam.sdk.io.gcp.bigquery.BigQueryHelpers$PendingJob.runJob(BigQueryHelpers.java:199)
> 2022/08/31 09:30:07 - sites - at
> org.apache.beam.sdk.io.gcp.bigquery.BigQueryHelpers$PendingJobManager.waitForDone(BigQueryHelpers.java:152)
> 2022/08/31 09:30:07 - sites - at
> org.apache.beam.sdk.io.gcp.bigquery.WriteTables$WriteTablesDoFn.finishBundle(WriteTables.java:380)
>
