Hello, I am running a pipeline that extracts columns from a bigquery table and writes the extracted data to a tsv file.
The pipeline is stack on this message: oauth2client.client:Refreshing due to a 401 (attempt 1/2) Could you please let me know what might be the reason for that. is it another issue or authentication issue that generating this error. I have copied the logs below Thanks, -- Eila www.orielresearch.org https://www.meetu <https://www.meetup.com/Deep-Learning-In-Production/>p.co <https://www.meetup.com/Deep-Learning-In-Production/> m/Deep-Learning-In-Production/ <https://www.meetup.com/Deep-Learning-In-Production/> Logs: INFO:root:2018-07-09T23:48:42.199Z: JOB_MESSAGE_DETAILED: Fusing consumer writing to TSV files/Write/WriteImpl/WriteBundles/Do into writing to TSV files/Write/WriteImpl/GroupByKey/GroupByWindow INFO:root:2018-07-09T23:48:42.213Z: JOB_MESSAGE_DETAILED: Fusing consumer writing to TSV files/Write/WriteImpl/Map(<lambda at iobase.py:895>) into create more columns INFO:root:2018-07-09T23:48:42.223Z: JOB_MESSAGE_DETAILED: Fusing consumer writing to TSV files/Write/WriteImpl/WindowInto(WindowIntoFn) into writing to TSV files/Write/WriteImpl/Map(<lambda at iobase.py:895>) INFO:root:2018-07-09T23:48:42.245Z: JOB_MESSAGE_DETAILED: Fusing consumer writing to TSV files/Write/WriteImpl/GroupByKey/GroupByWindow into writing to TSV files/Write/WriteImpl/GroupByKey/Read INFO:root:2018-07-09T23:48:42.274Z: JOB_MESSAGE_DETAILED: Fusing consumer writing to TSV files/Write/WriteImpl/InitializeWrite into writing to TSV files/Write/WriteImpl/DoOnce/Read INFO:root:2018-07-09T23:48:42.304Z: JOB_MESSAGE_DEBUG: Workflow config is missing a default resource spec. INFO:root:2018-07-09T23:48:42.332Z: JOB_MESSAGE_DEBUG: Adding StepResource setup and teardown to workflow graph. INFO:root:2018-07-09T23:48:42.365Z: JOB_MESSAGE_DEBUG: Adding workflow start and stop steps. INFO:root:2018-07-09T23:48:42.394Z: JOB_MESSAGE_DEBUG: Assigning stage ids. INFO:root:2018-07-09T23:48:42.532Z: JOB_MESSAGE_DEBUG: Executing wait step start15 INFO:root:Job 2018-07-09_16_48_38-1246984940035065784 is in state JOB_STATE_RUNNING INFO:root:2018-07-09T23:48:42.580Z: JOB_MESSAGE_BASIC: Executing operation writing to TSV files/Write/WriteImpl/DoOnce/Read+writing to TSV files/Write/WriteImpl/InitializeWrite INFO:root:2018-07-09T23:48:42.605Z: JOB_MESSAGE_BASIC: Executing operation writing to TSV files/Write/WriteImpl/GroupByKey/Create INFO:root:2018-07-09T23:48:42.616Z: JOB_MESSAGE_DEBUG: Starting worker pool setup. INFO:root:2018-07-09T23:48:42.650Z: JOB_MESSAGE_BASIC: Starting 1 workers in us-central1-f... INFO:root:2018-07-09T23:48:42.732Z: JOB_MESSAGE_DEBUG: Value "writing to TSV files/Write/WriteImpl/GroupByKey/Session" materialized. INFO:root:2018-07-09T23:48:42.787Z: JOB_MESSAGE_BASIC: Executing operation Extract the rows from dataframe+create more columns+writing to TSV files/Write/WriteImpl/Map(<lambda at iobase.py:895>)+writing to TSV files/Write/WriteImpl/WindowInto(WindowIntoFn)+writing to TSV files/Write/WriteImpl/GroupByKey/Reify+writing to TSV files/Write/WriteImpl/GroupByKey/Write INFO:root:2018-07-09T23:48:43.155Z: JOB_MESSAGE_BASIC: BigQuery export job "dataflow_job_11989624639985116201" started. You can check its status with the bq tool: "bq show -j --project_id=orielresearch-188115 dataflow_job_11989624639985116201". INFO:root:2018-07-09T23:48:53.348Z: JOB_MESSAGE_DETAILED: Autoscaling: Raised the number of workers to 0 based on the rate of progress in the currently running step(s). INFO:root:2018-07-09T23:49:03.919Z: JOB_MESSAGE_DETAILED: Autoscaling: Raised the number of workers to 1 based on the rate of progress in the currently running step(s). INFO:root:2018-07-09T23:49:13.724Z: JOB_MESSAGE_DETAILED: BigQuery export job progress: "dataflow_job_11989624639985116201" observed total of 1 exported files thus far. INFO:root:2018-07-09T23:49:13.760Z: JOB_MESSAGE_BASIC: BigQuery export job finished: "dataflow_job_11989624639985116201" INFO:root:2018-07-09T23:49:21.761Z: JOB_MESSAGE_DETAILED: Workers have started successfully. INFO:root:2018-07-09T23:53:46.857Z: JOB_MESSAGE_BASIC: Autoscaling: Resizing worker pool from 1 to 2. INFO:root:2018-07-09T23:53:52.384Z: JOB_MESSAGE_DETAILED: Autoscaling: Raised the number of workers to 2 based on the rate of progress in the currently running step(s). INFO:root:2018-07-09T23:55:51.970Z: JOB_MESSAGE_DEBUG: Value "writing to TSV files/Write/WriteImpl/DoOnce/Read.out" materialized. INFO:root:2018-07-09T23:55:51.992Z: JOB_MESSAGE_DEBUG: Value "writing to TSV files/Write/WriteImpl/InitializeWrite.out" materialized. INFO:root:2018-07-09T23:55:52.050Z: JOB_MESSAGE_BASIC: Executing operation writing to TSV files/Write/WriteImpl/WriteBundles/SideInput-s8 INFO:root:2018-07-09T23:55:52.081Z: JOB_MESSAGE_BASIC: Executing operation writing to TSV files/Write/WriteImpl/FinalizeWrite/SideInput-s10 INFO:root:2018-07-09T23:55:52.155Z: JOB_MESSAGE_DEBUG: Value "writing to TSV files/Write/WriteImpl/WriteBundles/SideInput-s8.output" materialized. INFO:root:2018-07-09T23:55:52.201Z: JOB_MESSAGE_DEBUG: Value "writing to TSV files/Write/WriteImpl/FinalizeWrite/SideInput-s10.output" materialized. INFO:root:2018-07-09T23:56:16.991Z: JOB_MESSAGE_BASIC: Autoscaling: Resizing worker pool from 2 to 3. INFO:root:2018-07-09T23:56:38.376Z: JOB_MESSAGE_DETAILED: Autoscaling: Raised the number of workers to 3 based on the rate of progress in the currently running step(s). INFO:root:2018-07-09T23:58:47.077Z: JOB_MESSAGE_BASIC: Autoscaling: Resizing worker pool from 3 to 4. INFO:root:2018-07-09T23:59:13.747Z: JOB_MESSAGE_DETAILED: Autoscaling: Raised the number of workers to 4 based on the rate of progress in the currently running step(s). INFO:root:2018-07-10T00:02:17.076Z: JOB_MESSAGE_DETAILED: Autoscaling: Reduced the number of workers to 1 based on the rate of progress in the currently running step(s). INFO:root:2018-07-10T00:02:22.113Z: JOB_MESSAGE_BASIC: Autoscaling: Resizing worker pool from 4 to 1. INFO:oauth2client.client:Refreshing due to a 401 (attempt 1/2)
