See
<https://builds.apache.org/job/beam_PostCommit_Python3_Verify/915/display/redirect?page=changes>
Changes:
[iemejia] [BEAM-7230] Make PoolableDataSourceProvider a static singleton And
------------------------------------------
[...truncated 555.00 KB...]
}
],
"is_wrapper": true
},
"output_name": "out",
"user_name": "WriteToPubSub/ToProtobuf.out"
}
],
"parallel_input": {
"@type": "OutputReference",
"output_name": "out",
"step_name": "s3"
},
"serialized_fn": "ref_AppliedPTransform_WriteToPubSub/ToProtobuf_7",
"user_name": "WriteToPubSub/ToProtobuf"
}
},
{
"kind": "ParallelWrite",
"name": "s5",
"properties": {
"display_data": [],
"encoding": {
"@type": "kind:windowed_value",
"component_encodings": [
{
"@type": "kind:bytes"
},
{
"@type": "kind:global_window"
}
],
"is_wrapper": true
},
"format": "pubsub",
"parallel_input": {
"@type": "OutputReference",
"output_name": "out",
"step_name": "s4"
},
"pubsub_id_label": "id",
"pubsub_serialized_attributes_fn": "",
"pubsub_timestamp_label": "timestamp",
"pubsub_topic":
"projects/apache-beam-testing/topics/psit_topic_output34357799-61b8-432f-9cd0-33808dc56458",
"user_name": "WriteToPubSub/Write/NativeWrite"
}
}
],
"type": "JOB_TYPE_STREAMING"
}
root: INFO: Create job: <Job
createTime: '2019-05-21T16:52:44.770821Z'
currentStateTime: '1970-01-01T00:00:00Z'
id: '2019-05-21_09_52_43-8769910267930579214'
location: 'us-central1'
name: 'beamapp-jenkins-0521165234-551297'
projectId: 'apache-beam-testing'
stageStates: []
startTime: '2019-05-21T16:52:44.770821Z'
steps: []
tempFiles: []
type: TypeValueValuesEnum(JOB_TYPE_STREAMING, 2)>
root: INFO: Created job with id: [2019-05-21_09_52_43-8769910267930579214]
root: INFO: To access the Dataflow monitoring console, please navigate to
https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2019-05-21_09_52_43-8769910267930579214?project=apache-beam-testing
root: INFO: Job 2019-05-21_09_52_43-8769910267930579214 is in state
JOB_STATE_RUNNING
root: INFO: 2019-05-21T16:52:47.348Z: JOB_MESSAGE_DETAILED: Checking
permissions granted to controller Service Account.
root: INFO: 2019-05-21T16:52:48.096Z: JOB_MESSAGE_BASIC: Worker configuration:
n1-standard-4 in us-central1-a.
root: INFO: 2019-05-21T16:52:48.691Z: JOB_MESSAGE_DETAILED: Expanding
SplittableParDo operations into optimizable parts.
root: INFO: 2019-05-21T16:52:48.694Z: JOB_MESSAGE_DETAILED: Expanding
CollectionToSingleton operations into optimizable parts.
root: INFO: 2019-05-21T16:52:48.702Z: JOB_MESSAGE_DETAILED: Expanding
CoGroupByKey operations into optimizable parts.
root: INFO: 2019-05-21T16:52:48.708Z: JOB_MESSAGE_DETAILED: Expanding
SplittableProcessKeyed operations into optimizable parts.
root: INFO: 2019-05-21T16:52:48.710Z: JOB_MESSAGE_DETAILED: Expanding
GroupByKey operations into streaming Read/Write steps
root: INFO: 2019-05-21T16:52:48.715Z: JOB_MESSAGE_DEBUG: Annotating graph with
Autotuner information.
root: INFO: 2019-05-21T16:52:48.726Z: JOB_MESSAGE_DETAILED: Fusing adjacent
ParDo, Read, Write, and Flatten operations
root: INFO: 2019-05-21T16:52:48.729Z: JOB_MESSAGE_DETAILED: Fusing consumer
ReadFromPubSub/Map(_from_proto_str) into ReadFromPubSub/Read
root: INFO: 2019-05-21T16:52:48.732Z: JOB_MESSAGE_DETAILED: Fusing consumer
WriteToPubSub/ToProtobuf into add_attribute
root: INFO: 2019-05-21T16:52:48.735Z: JOB_MESSAGE_DETAILED: Fusing consumer
add_attribute into ReadFromPubSub/Map(_from_proto_str)
root: INFO: 2019-05-21T16:52:48.738Z: JOB_MESSAGE_DETAILED: Fusing consumer
WriteToPubSub/Write/NativeWrite into WriteToPubSub/ToProtobuf
root: INFO: 2019-05-21T16:52:48.746Z: JOB_MESSAGE_DEBUG: Adding StepResource
setup and teardown to workflow graph.
root: INFO: 2019-05-21T16:52:48.762Z: JOB_MESSAGE_DEBUG: Adding workflow start
and stop steps.
root: INFO: 2019-05-21T16:52:48.796Z: JOB_MESSAGE_DEBUG: Assigning stage ids.
root: INFO: 2019-05-21T16:52:48.958Z: JOB_MESSAGE_DEBUG: Executing wait step
start2
root: INFO: 2019-05-21T16:52:48.974Z: JOB_MESSAGE_DEBUG: Starting worker pool
setup.
root: INFO: 2019-05-21T16:52:48.981Z: JOB_MESSAGE_BASIC: Starting 1 workers...
root: INFO: 2019-05-21T16:52:49.811Z: JOB_MESSAGE_DETAILED: Pub/Sub resources
set up for topic
'projects/apache-beam-testing/topics/psit_topic_input34357799-61b8-432f-9cd0-33808dc56458'.
root: INFO: 2019-05-21T16:52:51.147Z: JOB_MESSAGE_BASIC: Executing operation
ReadFromPubSub/Read+ReadFromPubSub/Map(_from_proto_str)+add_attribute+WriteToPubSub/ToProtobuf+WriteToPubSub/Write/NativeWrite
root: WARNING: Timing out on waiting for job
2019-05-21_09_52_43-8769910267930579214 after 182 seconds
google.auth.transport._http_client: DEBUG: Making request: GET
http://169.254.169.254
google.auth.transport._http_client: DEBUG: Making request: GET
http://metadata.google.internal/computeMetadata/v1/project/project-id
google.auth.transport.requests: DEBUG: Making request: GET
http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/?recursive=true
urllib3.connectionpool: DEBUG: Starting new HTTP connection (1):
metadata.google.internal:80
urllib3.connectionpool: DEBUG: http://metadata.google.internal:80 "GET
/computeMetadata/v1/instance/service-accounts/default/?recursive=true HTTP/1.1"
200 144
google.auth.transport.requests: DEBUG: Making request: GET
http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/[email protected]/token
urllib3.connectionpool: DEBUG: http://metadata.google.internal:80 "GET
/computeMetadata/v1/instance/service-accounts/[email protected]/token
HTTP/1.1" 200 176
root: ERROR: Timeout after 300 sec. Received 0 messages from
projects/apache-beam-testing/subscriptions/psit_subscription_output34357799-61b8-432f-9cd0-33808dc56458.
--------------------- >> end captured logging << ---------------------
<https://builds.apache.org/job/beam_PostCommit_Python3_Verify/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:1137:
BeamDeprecationWarning: options is deprecated since First stable release.
References to <pipeline>.options will not be supported
Found:
https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2019-05-21_09_28_09-8908939664341354694?project=apache-beam-testing.
method_to_use = self._compute_method(p, p.options)
Found:
https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2019-05-21_09_43_47-12232068525434500760?project=apache-beam-testing.
Found:
https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2019-05-21_09_52_47-6852120521125735508?project=apache-beam-testing.
<https://builds.apache.org/job/beam_PostCommit_Python3_Verify/ws/src/sdks/python/apache_beam/io/gcp/bigquery_file_loads.py>:545:
BeamDeprecationWarning: options is deprecated since First stable release.
References to <pipeline>.options will not be supported
or p.options.view_as(GoogleCloudOptions).temp_location)
Found:
https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2019-05-21_09_28_05-13782780906343236656?project=apache-beam-testing.
Found:
https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2019-05-21_09_48_29-2309411101392668647?project=apache-beam-testing.
<https://builds.apache.org/job/beam_PostCommit_Python3_Verify/ws/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py>:665:
BeamDeprecationWarning: BigQuerySink is deprecated since 2.11.0. Use
WriteToBigQuery instead.
Found:
https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2019-05-21_09_58_25-8896435460796722674?project=apache-beam-testing.
kms_key=transform.kms_key))
Found:
https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2019-05-21_09_28_18-12830454536881403867?project=apache-beam-testing.
<https://builds.apache.org/job/beam_PostCommit_Python3_Verify/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:1137:
BeamDeprecationWarning: options is deprecated since First stable release.
References to <pipeline>.options will not be supported
Found:
https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2019-05-21_09_44_12-5026498687438197694?project=apache-beam-testing.
Found:
https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2019-05-21_09_52_43-8769910267930579214?project=apache-beam-testing.
method_to_use = self._compute_method(p, p.options)
Found:
https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2019-05-21_09_28_02-9285601410041274558?project=apache-beam-testing.
Found:
https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2019-05-21_09_47_15-5981314052814946132?project=apache-beam-testing.
Found:
https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2019-05-21_09_28_02-18256538628877411467?project=apache-beam-testing.
Found:
https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2019-05-21_09_36_34-15956275196414932718?project=apache-beam-testing.
Found:
https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2019-05-21_09_45_40-18113774607142033342?project=apache-beam-testing.
<https://builds.apache.org/job/beam_PostCommit_Python3_Verify/ws/src/sdks/python/apache_beam/io/fileio_test.py>:218:
FutureWarning: MatchAll is experimental.
| 'GetPath' >> beam.Map(lambda metadata: metadata.path))
<https://builds.apache.org/job/beam_PostCommit_Python3_Verify/ws/src/sdks/python/apache_beam/io/fileio_test.py>:229:
FutureWarning: MatchAll is experimental.
| 'Checksums' >> beam.Map(compute_hash))
<https://builds.apache.org/job/beam_PostCommit_Python3_Verify/ws/src/sdks/python/apache_beam/io/fileio_test.py>:229:
FutureWarning: ReadMatches is experimental.
| 'Checksums' >> beam.Map(compute_hash))
<https://builds.apache.org/job/beam_PostCommit_Python3_Verify/ws/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py>:665:
BeamDeprecationWarning: BigQuerySink is deprecated since 2.11.0. Use
WriteToBigQuery instead.
kms_key=transform.kms_key))
Found:
https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2019-05-21_09_28_02-9264163574105883231?project=apache-beam-testing.
Found:
https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2019-05-21_09_35_55-8774985307648354770?project=apache-beam-testing.
Found:
https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2019-05-21_09_43_47-18009225918290840317?project=apache-beam-testing.
Found:
https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2019-05-21_09_53_20-18332526382415323724?project=apache-beam-testing.
Found:
https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2019-05-21_10_04_14-14452407206926945190?project=apache-beam-testing.
Found:
https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2019-05-21_10_13_10-16420327388046189298?project=apache-beam-testing.
Found:
https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2019-05-21_09_28_03-4804287767337818822?project=apache-beam-testing.
Found:
https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2019-05-21_09_36_15-17559432233157559754?project=apache-beam-testing.
<https://builds.apache.org/job/beam_PostCommit_Python3_Verify/ws/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py>:665:
BeamDeprecationWarning: BigQuerySink is deprecated since 2.11.0. Use
WriteToBigQuery instead.
kms_key=transform.kms_key))
Found:
https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2019-05-21_09_43_53-9649117503357886317?project=apache-beam-testing.
<https://builds.apache.org/job/beam_PostCommit_Python3_Verify/ws/src/sdks/python/apache_beam/io/gcp/big_query_query_to_table_pipeline.py>:73:
BeamDeprecationWarning: BigQuerySink is deprecated since 2.11.0. Use
WriteToBigQuery instead.
kms_key=kms_key))
Found:
https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2019-05-21_09_53_09-6350179846057929639?project=apache-beam-testing.
Found:
https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2019-05-21_09_28_05-17350918912757288652?project=apache-beam-testing.
Found:
https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2019-05-21_09_36_29-16999090721206993168?project=apache-beam-testing.
<https://builds.apache.org/job/beam_PostCommit_Python3_Verify/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:1137:
BeamDeprecationWarning: options is deprecated since First stable release.
References to <pipeline>.options will not be supported
method_to_use = self._compute_method(p, p.options)
<https://builds.apache.org/job/beam_PostCommit_Python3_Verify/ws/src/sdks/python/apache_beam/io/gcp/bigquery_file_loads.py>:545:
BeamDeprecationWarning: options is deprecated since First stable release.
References to <pipeline>.options will not be supported
or p.options.view_as(GoogleCloudOptions).temp_location)
Found:
https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2019-05-21_09_46_02-2117776692623309708?project=apache-beam-testing.
Found:
https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2019-05-21_09_53_37-13751820907756385510?project=apache-beam-testing.
----------------------------------------------------------------------
XML:
<https://builds.apache.org/job/beam_PostCommit_Python3_Verify/ws/src/sdks/python/nosetests.xml>
----------------------------------------------------------------------
Ran 35 tests in 3242.062s
FAILED (SKIP=5, failures=1)
> Task :sdks:python:test-suites:dataflow:py36:postCommitIT FAILED
> Task :sdks:python:test-suites:dataflow:py35:validatesRunnerBatchTests
<https://builds.apache.org/job/beam_PostCommit_Python3_Verify/ws/src/sdks/python/apache_beam/__init__.py>:84:
UserWarning: Running the Apache Beam SDK on Python 3 is not yet fully
supported. You may encounter buggy behavior or missing features.
'Running the Apache Beam SDK on Python 3 is not yet fully supported. '
<https://builds.apache.org/job/beam_PostCommit_Python3_Verify/ws/src/sdks/python/apache_beam/io/gcp/datastore/v1/datastoreio.py>:59:
UserWarning: Datastore IO will support Python 3 after replacing
googledatastore by google-cloud-datastore, see: BEAM-4543.
warnings.warn('Datastore IO will support Python 3 after replacing '
<https://builds.apache.org/job/beam_PostCommit_Python3_Verify/ws/src/sdks/python/apache_beam/io/vcfio.py>:47:
UserWarning: VCF IO will support Python 3 after migration to Nucleus, see:
BEAM-5628.
warnings.warn("VCF IO will support Python 3 after migration to Nucleus, "
Found:
https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2019-05-21_10_22_17-2498160044040734507?project=apache-beam-testing.
Found:
https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2019-05-21_10_30_56-2144902971512643443?project=apache-beam-testing.
Found:
https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2019-05-21_10_22_17-2913717983049823535?project=apache-beam-testing.
Found:
https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2019-05-21_10_30_56-3506472343553493731?project=apache-beam-testing.
Found:
https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2019-05-21_10_22_17-4382972460906972229?project=apache-beam-testing.
Found:
https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2019-05-21_10_30_51-12272582705028594544?project=apache-beam-testing.
Found:
https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2019-05-21_10_22_18-1172755567471521187?project=apache-beam-testing.
Found:
https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2019-05-21_10_30_57-1731251167386942663?project=apache-beam-testing.
Found:
https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2019-05-21_10_22_18-5153873813153813191?project=apache-beam-testing.
Found:
https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2019-05-21_10_32_06-9884526516963364362?project=apache-beam-testing.
Found:
https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2019-05-21_10_22_16-4064757579619856075?project=apache-beam-testing.
Found:
https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2019-05-21_10_31_25-284295912988831607?project=apache-beam-testing.
Found:
https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2019-05-21_10_22_18-17254419135471020122?project=apache-beam-testing.
Found:
https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2019-05-21_10_32_01-1570800744398542288?project=apache-beam-testing.
Found:
https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2019-05-21_10_22_17-3527645016756811297?project=apache-beam-testing.
Found:
https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2019-05-21_10_30_45-3095146113758301403?project=apache-beam-testing.
Found:
https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2019-05-21_10_39_29-10868141696232549324?project=apache-beam-testing.
test_as_dict_twice (apache_beam.transforms.sideinputs_test.SideInputsTest) ...
ok
test_multiple_empty_outputs
(apache_beam.transforms.ptransform_test.PTransformTest) ... ok
test_dofn_lifecycle
(apache_beam.transforms.dofn_lifecycle_test.DoFnLifecycleTest) ... ok
test_flatten_multiple_pcollections_having_multiple_consumers
(apache_beam.transforms.ptransform_test.PTransformTest) ... ok
test_par_do_with_multiple_outputs_and_using_return
(apache_beam.transforms.ptransform_test.PTransformTest) ... ok
test_read_metrics (apache_beam.transforms.ptransform_test.PTransformTest) ... ok
test_undeclared_outputs (apache_beam.transforms.ptransform_test.PTransformTest)
... ok
test_par_do_with_multiple_outputs_and_using_yield
(apache_beam.transforms.ptransform_test.PTransformTest) ... ok
test_as_list_and_as_dict_side_inputs
(apache_beam.transforms.sideinputs_test.SideInputsTest) ... ok
test_as_singleton_with_different_defaults
(apache_beam.transforms.sideinputs_test.SideInputsTest) ... ok
test_as_singleton_without_unique_labels
(apache_beam.transforms.sideinputs_test.SideInputsTest) ... ok
test_empty_singleton_side_input
(apache_beam.transforms.sideinputs_test.SideInputsTest) ... ok
test_flattened_side_input
(apache_beam.transforms.sideinputs_test.SideInputsTest) ... ok
test_as_list_twice (apache_beam.transforms.sideinputs_test.SideInputsTest) ...
ok
test_iterable_side_input
(apache_beam.transforms.sideinputs_test.SideInputsTest) ... ok
test_default_value_singleton_side_input
(apache_beam.transforms.sideinputs_test.SideInputsTest) ... ok
test_multi_valued_singleton_side_input
(apache_beam.transforms.sideinputs_test.SideInputsTest) ... ok
----------------------------------------------------------------------
XML:
<https://builds.apache.org/job/beam_PostCommit_Python3_Verify/ws/src/sdks/python/nosetests.xml>
----------------------------------------------------------------------
Ran 17 tests in 1541.360s
OK
FAILURE: Build completed with 2 failures.
1: Task failed with an exception.
-----------
* Where:
Build file
'<https://builds.apache.org/job/beam_PostCommit_Python3_Verify/ws/src/sdks/python/test-suites/direct/py37/build.gradle'>
line: 46
* What went wrong:
Execution failed for task ':sdks:python:test-suites:direct:py37:postCommitIT'.
> Process 'command 'sh'' finished with non-zero exit value 1
* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug
option to get more log output. Run with --scan to get full insights.
==============================================================================
2: Task failed with an exception.
-----------
* Where:
Build file
'<https://builds.apache.org/job/beam_PostCommit_Python3_Verify/ws/src/sdks/python/test-suites/dataflow/py36/build.gradle'>
line: 47
* What went wrong:
Execution failed for task ':sdks:python:test-suites:dataflow:py36:postCommitIT'.
> Process 'command 'sh'' finished with non-zero exit value 1
* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug
option to get more log output. Run with --scan to get full insights.
==============================================================================
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with
Gradle 6.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See
https://docs.gradle.org/5.2.1/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 1h 21m 20s
78 actionable tasks: 61 executed, 17 from cache
Publishing build scan...
https://gradle.com/s/hvtuwtcyorpfm
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
