See
<https://builds.apache.org/job/beam_PostCommit_Python_Verify/3793/display/redirect>
------------------------------------------
[...truncated 612.35 KB...]
Running setup.py bdist_wheel for pyvcf: finished with status 'done'
Stored in directory:
/root/.cache/pip/wheels/4c/cb/3a/76a743f5fb96613d53bb60ce20e67546af54610a85e1a0fa17
Running setup.py bdist_wheel for proto-google-cloud-datastore-v1: started
Running setup.py bdist_wheel for proto-google-cloud-datastore-v1: finished
with status 'done'
Stored in directory:
/root/.cache/pip/wheels/fe/e5/02/3f0fb27e5e2d7057f9ce3cd88031a0b8a5244a1ad70d46c9c7
Running setup.py bdist_wheel for googledatastore: started
Running setup.py bdist_wheel for googledatastore: finished with status 'done'
Stored in directory:
/root/.cache/pip/wheels/8a/31/1c/7c8967a47bfdf73c6882b4f20758f1ef2f6355c58c741bc285
Running setup.py bdist_wheel for googleapis-common-protos: started
Running setup.py bdist_wheel for googleapis-common-protos: finished with
status 'done'
Stored in directory:
/root/.cache/pip/wheels/93/e0/cb/b06866f012310b96fba79c37f199aaf973a2e97a42ca7ef696
Running setup.py bdist_wheel for gapic-google-cloud-pubsub-v1: started
Running setup.py bdist_wheel for gapic-google-cloud-pubsub-v1: finished with
status 'done'
Stored in directory:
/root/.cache/pip/wheels/0d/f9/0d/f4244cbb1f36aaad82c415b261852f4903dc909f449816e867
Running setup.py bdist_wheel for proto-google-cloud-pubsub-v1: started
Running setup.py bdist_wheel for proto-google-cloud-pubsub-v1: finished with
status 'done'
Stored in directory:
/root/.cache/pip/wheels/bd/01/b1/6ea2adb4534abda38b1c658939beab468ad40d77eb1f85cbd4
Running setup.py bdist_wheel for grpc-google-iam-v1: started
Running setup.py bdist_wheel for grpc-google-iam-v1: finished with status
'done'
Stored in directory:
/root/.cache/pip/wheels/5b/fb/70/a586dc38157ecf8f242cc2207d163545ac0784c05728df7492
Running setup.py bdist_wheel for future: started
Running setup.py bdist_wheel for future: finished with status 'done'
Stored in directory:
/root/.cache/pip/wheels/c2/50/7c/0d83b4baac4f63ff7a765bd16390d2ab43c93587fac9d6017a
Running setup.py bdist_wheel for ply: started
Running setup.py bdist_wheel for ply: finished with status 'done'
Stored in directory:
/root/.cache/pip/wheels/6f/23/3c/7055bf8004cee3c7d06c46318014e0374539094c4e36246e9a
Successfully built apache-beam avro crcmod dill httplib2 oauth2client pyyaml
pyvcf proto-google-cloud-datastore-v1 googledatastore googleapis-common-protos
gapic-google-cloud-pubsub-v1 proto-google-cloud-pubsub-v1 grpc-google-iam-v1
future ply
Installing collected packages: avro, crcmod, dill, six, protobuf, futures,
enum34, grpcio, httplib2, funcsigs, pbr, mock, pyasn1, pyasn1-modules, rsa,
oauth2client, pyyaml, pyvcf, typing, google-apitools, googleapis-common-protos,
proto-google-cloud-datastore-v1, googledatastore, cachetools, google-auth,
google-auth-httplib2, google-cloud-core, future, urllib3, idna, chardet,
requests, ply, google-gax, proto-google-cloud-pubsub-v1, grpc-google-iam-v1,
gapic-google-cloud-pubsub-v1, google-cloud-pubsub, google-cloud-bigquery,
apache-beam
Found existing installation: six 1.11.0
Uninstalling six-1.11.0:
Successfully uninstalled six-1.11.0
Found existing installation: protobuf 3.3.2
Uninstalling protobuf-3.3.2:
Successfully uninstalled protobuf-3.3.2
Successfully installed apache-beam-2.3.0.dev0 avro-1.8.2 cachetools-2.0.1
chardet-3.0.4 crcmod-1.7 dill-0.2.6 enum34-1.1.6 funcsigs-1.0.2 future-0.16.0
futures-3.2.0 gapic-google-cloud-pubsub-v1-0.15.4 google-apitools-0.5.11
google-auth-1.2.1 google-auth-httplib2-0.0.3 google-cloud-bigquery-0.25.0
google-cloud-core-0.25.0 google-cloud-pubsub-0.26.0 google-gax-0.15.16
googleapis-common-protos-1.5.3 googledatastore-7.0.1 grpc-google-iam-v1-0.11.4
grpcio-1.8.2 httplib2-0.9.2 idna-2.6 mock-2.0.0 oauth2client-3.0.0 pbr-3.1.1
ply-3.8 proto-google-cloud-datastore-v1-0.90.4
proto-google-cloud-pubsub-v1-0.15.4 protobuf-3.5.0.post1 pyasn1-0.4.2
pyasn1-modules-0.2.1 pyvcf-0.6.8 pyyaml-3.12 requests-2.18.4 rsa-3.4.2
six-1.10.0 typing-3.6.2 urllib3-1.22
---> ac5277464079
Removing intermediate container b65694bfa6fd
Step 21/21 : CMD bash -cl 'service dropbear start &&
$HVERSION/sbin/start-dfs.sh && python -m apache_beam.examples.wordcount
--output hdfs://tmp/py-wordcount-direct'
---> Running in 077e29af2767
---> eadd99840cd1
Removing intermediate container 077e29af2767
Successfully built eadd99840cd1
Successfully tagged hdfs:jenkins-beam_PostCommit_Python_Verify-3793
real 5m57.629s
user 0m11.408s
sys 0m1.880s
+ case ${1:-""} in
+ docker run -t hdfs:jenkins-beam_PostCommit_Python_Verify-3793
Starting Dropbear SSH server: dropbear.
Starting namenodes on [localhost]
localhost: Warning: Permanently added 'localhost' (ECDSA) to the list of known
hosts.
localhost: starting namenode, logging to
/app/hadoop-2.9.0/logs/hadoop-root-namenode-d8ddd70721e0.out
localhost: starting datanode, logging to
/app/hadoop-2.9.0/logs/hadoop-root-datanode-d8ddd70721e0.out
Starting secondary namenodes [0.0.0.0]
0.0.0.0: Warning: Permanently added '0.0.0.0' (ECDSA) to the list of known
hosts.
0.0.0.0: starting secondarynamenode, logging to
/app/hadoop-2.9.0/logs/hadoop-root-secondarynamenode-d8ddd70721e0.out
No handlers could be found for logger "oauth2client.contrib.multistore_file"
INFO:root:Missing pipeline option (runner). Executing pipeline using the
default runner: DirectRunner.
/app/miniconda/envs/beamenv/lib/python2.7/site-packages/apache_beam/io/gcp/gcsio.py:123:
DeprecationWarning: object() takes no parameters
super(GcsIO, cls).__new__(cls, storage_client))
INFO:root:Starting the size estimation of the input
INFO:oauth2client.transport:Attempting refresh to obtain initial access_token
WARNING:root:Retry with exponential backoff: waiting for 2.90416802545 seconds
before retrying size_of_files_in_glob because we caught exception:
HttpAccessTokenRefreshError: Unable to find the server at
metadata.google.internal
Traceback for above exception (most recent call last):
File
"/app/miniconda/envs/beamenv/lib/python2.7/site-packages/apache_beam/utils/retry.py",
line 175, in wrapper
return fun(*args, **kwargs)
File
"/app/miniconda/envs/beamenv/lib/python2.7/site-packages/apache_beam/io/gcp/gcsio.py",
line 400, in size_of_files_in_glob
response = self.client.objects.List(request)
File
"/app/miniconda/envs/beamenv/lib/python2.7/site-packages/apache_beam/io/gcp/internal/clients/storage/storage_v1_client.py",
line 984, in List
config, request, global_params=global_params)
File
"/app/miniconda/envs/beamenv/lib/python2.7/site-packages/apitools/base/py/base_api.py",
line 721, in _RunMethod
http, http_request, **opts)
File
"/app/miniconda/envs/beamenv/lib/python2.7/site-packages/apitools/base/py/http_wrapper.py",
line 351, in MakeRequest
max_retry_wait, total_wait_sec))
File
"/app/miniconda/envs/beamenv/lib/python2.7/site-packages/apitools/base/py/http_wrapper.py",
line 341, in MakeRequest
check_response_func=check_response_func)
File
"/app/miniconda/envs/beamenv/lib/python2.7/site-packages/apitools/base/py/http_wrapper.py",
line 391, in _MakeRequestNoRetry
redirections=redirections, connection_type=connection_type)
File
"/app/miniconda/envs/beamenv/lib/python2.7/site-packages/oauth2client/transport.py",
line 153, in new_request
credentials._refresh(orig_request_method)
File
"/app/miniconda/envs/beamenv/lib/python2.7/site-packages/oauth2client/contrib/gce.py",
line 138, in _refresh
raise client.HttpAccessTokenRefreshError(str(e))
INFO:root:Starting the size estimation of the input
INFO:oauth2client.transport:Attempting refresh to obtain initial access_token
WARNING:root:Retry with exponential backoff: waiting for 5.71104641037 seconds
before retrying size_of_files_in_glob because we caught exception:
HttpAccessTokenRefreshError: Unable to find the server at
metadata.google.internal
Traceback for above exception (most recent call last):
File
"/app/miniconda/envs/beamenv/lib/python2.7/site-packages/apache_beam/utils/retry.py",
line 175, in wrapper
return fun(*args, **kwargs)
File
"/app/miniconda/envs/beamenv/lib/python2.7/site-packages/apache_beam/io/gcp/gcsio.py",
line 400, in size_of_files_in_glob
response = self.client.objects.List(request)
File
"/app/miniconda/envs/beamenv/lib/python2.7/site-packages/apache_beam/io/gcp/internal/clients/storage/storage_v1_client.py",
line 984, in List
config, request, global_params=global_params)
File
"/app/miniconda/envs/beamenv/lib/python2.7/site-packages/apitools/base/py/base_api.py",
line 721, in _RunMethod
http, http_request, **opts)
File
"/app/miniconda/envs/beamenv/lib/python2.7/site-packages/apitools/base/py/http_wrapper.py",
line 351, in MakeRequest
max_retry_wait, total_wait_sec))
File
"/app/miniconda/envs/beamenv/lib/python2.7/site-packages/apitools/base/py/http_wrapper.py",
line 341, in MakeRequest
check_response_func=check_response_func)
File
"/app/miniconda/envs/beamenv/lib/python2.7/site-packages/apitools/base/py/http_wrapper.py",
line 391, in _MakeRequestNoRetry
redirections=redirections, connection_type=connection_type)
File
"/app/miniconda/envs/beamenv/lib/python2.7/site-packages/oauth2client/transport.py",
line 153, in new_request
credentials._refresh(orig_request_method)
File
"/app/miniconda/envs/beamenv/lib/python2.7/site-packages/oauth2client/contrib/gce.py",
line 138, in _refresh
raise client.HttpAccessTokenRefreshError(str(e))
INFO:root:Starting the size estimation of the input
INFO:oauth2client.transport:Attempting refresh to obtain initial access_token
WARNING:root:Retry with exponential backoff: waiting for 15.0658980488 seconds
before retrying size_of_files_in_glob because we caught exception:
HttpAccessTokenRefreshError: Unable to find the server at
metadata.google.internal
Traceback for above exception (most recent call last):
File
"/app/miniconda/envs/beamenv/lib/python2.7/site-packages/apache_beam/utils/retry.py",
line 175, in wrapper
return fun(*args, **kwargs)
File
"/app/miniconda/envs/beamenv/lib/python2.7/site-packages/apache_beam/io/gcp/gcsio.py",
line 400, in size_of_files_in_glob
response = self.client.objects.List(request)
File
"/app/miniconda/envs/beamenv/lib/python2.7/site-packages/apache_beam/io/gcp/internal/clients/storage/storage_v1_client.py",
line 984, in List
config, request, global_params=global_params)
File
"/app/miniconda/envs/beamenv/lib/python2.7/site-packages/apitools/base/py/base_api.py",
line 721, in _RunMethod
http, http_request, **opts)
File
"/app/miniconda/envs/beamenv/lib/python2.7/site-packages/apitools/base/py/http_wrapper.py",
line 351, in MakeRequest
max_retry_wait, total_wait_sec))
File
"/app/miniconda/envs/beamenv/lib/python2.7/site-packages/apitools/base/py/http_wrapper.py",
line 341, in MakeRequest
check_response_func=check_response_func)
File
"/app/miniconda/envs/beamenv/lib/python2.7/site-packages/apitools/base/py/http_wrapper.py",
line 391, in _MakeRequestNoRetry
redirections=redirections, connection_type=connection_type)
File
"/app/miniconda/envs/beamenv/lib/python2.7/site-packages/oauth2client/transport.py",
line 153, in new_request
credentials._refresh(orig_request_method)
File
"/app/miniconda/envs/beamenv/lib/python2.7/site-packages/oauth2client/contrib/gce.py",
line 138, in _refresh
raise client.HttpAccessTokenRefreshError(str(e))
INFO:root:Starting the size estimation of the input
INFO:oauth2client.transport:Attempting refresh to obtain initial access_token
WARNING:root:Retry with exponential backoff: waiting for 32.6740196595 seconds
before retrying size_of_files_in_glob because we caught exception:
HttpAccessTokenRefreshError: Unable to find the server at
metadata.google.internal
Traceback for above exception (most recent call last):
File
"/app/miniconda/envs/beamenv/lib/python2.7/site-packages/apache_beam/utils/retry.py",
line 175, in wrapper
return fun(*args, **kwargs)
File
"/app/miniconda/envs/beamenv/lib/python2.7/site-packages/apache_beam/io/gcp/gcsio.py",
line 400, in size_of_files_in_glob
response = self.client.objects.List(request)
File
"/app/miniconda/envs/beamenv/lib/python2.7/site-packages/apache_beam/io/gcp/internal/clients/storage/storage_v1_client.py",
line 984, in List
config, request, global_params=global_params)
File
"/app/miniconda/envs/beamenv/lib/python2.7/site-packages/apitools/base/py/base_api.py",
line 721, in _RunMethod
http, http_request, **opts)
File
"/app/miniconda/envs/beamenv/lib/python2.7/site-packages/apitools/base/py/http_wrapper.py",
line 351, in MakeRequest
max_retry_wait, total_wait_sec))
File
"/app/miniconda/envs/beamenv/lib/python2.7/site-packages/apitools/base/py/http_wrapper.py",
line 341, in MakeRequest
check_response_func=check_response_func)
File
"/app/miniconda/envs/beamenv/lib/python2.7/site-packages/apitools/base/py/http_wrapper.py",
line 391, in _MakeRequestNoRetry
redirections=redirections, connection_type=connection_type)
File
"/app/miniconda/envs/beamenv/lib/python2.7/site-packages/oauth2client/transport.py",
line 153, in new_request
credentials._refresh(orig_request_method)
File
"/app/miniconda/envs/beamenv/lib/python2.7/site-packages/oauth2client/contrib/gce.py",
line 138, in _refresh
raise client.HttpAccessTokenRefreshError(str(e))
INFO:root:Starting the size estimation of the input
INFO:oauth2client.transport:Attempting refresh to obtain initial access_token
WARNING:root:Retry with exponential backoff: waiting for 46.121141371 seconds
before retrying size_of_files_in_glob because we caught exception:
HttpAccessTokenRefreshError: Unable to find the server at
metadata.google.internal
Traceback for above exception (most recent call last):
File
"/app/miniconda/envs/beamenv/lib/python2.7/site-packages/apache_beam/utils/retry.py",
line 175, in wrapper
return fun(*args, **kwargs)
File
"/app/miniconda/envs/beamenv/lib/python2.7/site-packages/apache_beam/io/gcp/gcsio.py",
line 400, in size_of_files_in_glob
response = self.client.objects.List(request)
File
"/app/miniconda/envs/beamenv/lib/python2.7/site-packages/apache_beam/io/gcp/internal/clients/storage/storage_v1_client.py",
line 984, in List
config, request, global_params=global_params)
File
"/app/miniconda/envs/beamenv/lib/python2.7/site-packages/apitools/base/py/base_api.py",
line 721, in _RunMethod
http, http_request, **opts)
File
"/app/miniconda/envs/beamenv/lib/python2.7/site-packages/apitools/base/py/http_wrapper.py",
line 351, in MakeRequest
max_retry_wait, total_wait_sec))
File
"/app/miniconda/envs/beamenv/lib/python2.7/site-packages/apitools/base/py/http_wrapper.py",
line 341, in MakeRequest
check_response_func=check_response_func)
File
"/app/miniconda/envs/beamenv/lib/python2.7/site-packages/apitools/base/py/http_wrapper.py",
line 391, in _MakeRequestNoRetry
redirections=redirections, connection_type=connection_type)
File
"/app/miniconda/envs/beamenv/lib/python2.7/site-packages/oauth2client/transport.py",
line 153, in new_request
credentials._refresh(orig_request_method)
File
"/app/miniconda/envs/beamenv/lib/python2.7/site-packages/oauth2client/contrib/gce.py",
line 138, in _refresh
raise client.HttpAccessTokenRefreshError(str(e))
INFO:root:Starting the size estimation of the input
INFO:oauth2client.transport:Attempting refresh to obtain initial access_token
WARNING:root:Retry with exponential backoff: waiting for 136.93061493 seconds
before retrying size_of_files_in_glob because we caught exception:
HttpAccessTokenRefreshError: Unable to find the server at
metadata.google.internal
Traceback for above exception (most recent call last):
File
"/app/miniconda/envs/beamenv/lib/python2.7/site-packages/apache_beam/utils/retry.py",
line 175, in wrapper
return fun(*args, **kwargs)
File
"/app/miniconda/envs/beamenv/lib/python2.7/site-packages/apache_beam/io/gcp/gcsio.py",
line 400, in size_of_files_in_glob
response = self.client.objects.List(request)
File
"/app/miniconda/envs/beamenv/lib/python2.7/site-packages/apache_beam/io/gcp/internal/clients/storage/storage_v1_client.py",
line 984, in List
config, request, global_params=global_params)
File
"/app/miniconda/envs/beamenv/lib/python2.7/site-packages/apitools/base/py/base_api.py",
line 721, in _RunMethod
http, http_request, **opts)
File
"/app/miniconda/envs/beamenv/lib/python2.7/site-packages/apitools/base/py/http_wrapper.py",
line 351, in MakeRequest
max_retry_wait, total_wait_sec))
File
"/app/miniconda/envs/beamenv/lib/python2.7/site-packages/apitools/base/py/http_wrapper.py",
line 341, in MakeRequest
check_response_func=check_response_func)
File
"/app/miniconda/envs/beamenv/lib/python2.7/site-packages/apitools/base/py/http_wrapper.py",
line 391, in _MakeRequestNoRetry
redirections=redirections, connection_type=connection_type)
File
"/app/miniconda/envs/beamenv/lib/python2.7/site-packages/oauth2client/transport.py",
line 153, in new_request
credentials._refresh(orig_request_method)
File
"/app/miniconda/envs/beamenv/lib/python2.7/site-packages/oauth2client/contrib/gce.py",
line 138, in _refresh
raise client.HttpAccessTokenRefreshError(str(e))
INFO:root:Starting the size estimation of the input
INFO:oauth2client.transport:Attempting refresh to obtain initial access_token
WARNING:root:Retry with exponential backoff: waiting for 302.968003931 seconds
before retrying size_of_files_in_glob because we caught exception:
HttpAccessTokenRefreshError: Unable to find the server at
metadata.google.internal
Traceback for above exception (most recent call last):
File
"/app/miniconda/envs/beamenv/lib/python2.7/site-packages/apache_beam/utils/retry.py",
line 175, in wrapper
return fun(*args, **kwargs)
File
"/app/miniconda/envs/beamenv/lib/python2.7/site-packages/apache_beam/io/gcp/gcsio.py",
line 400, in size_of_files_in_glob
response = self.client.objects.List(request)
File
"/app/miniconda/envs/beamenv/lib/python2.7/site-packages/apache_beam/io/gcp/internal/clients/storage/storage_v1_client.py",
line 984, in List
config, request, global_params=global_params)
File
"/app/miniconda/envs/beamenv/lib/python2.7/site-packages/apitools/base/py/base_api.py",
line 721, in _RunMethod
http, http_request, **opts)
File
"/app/miniconda/envs/beamenv/lib/python2.7/site-packages/apitools/base/py/http_wrapper.py",
line 351, in MakeRequest
max_retry_wait, total_wait_sec))
File
"/app/miniconda/envs/beamenv/lib/python2.7/site-packages/apitools/base/py/http_wrapper.py",
line 341, in MakeRequest
check_response_func=check_response_func)
File
"/app/miniconda/envs/beamenv/lib/python2.7/site-packages/apitools/base/py/http_wrapper.py",
line 391, in _MakeRequestNoRetry
redirections=redirections, connection_type=connection_type)
File
"/app/miniconda/envs/beamenv/lib/python2.7/site-packages/oauth2client/transport.py",
line 153, in new_request
credentials._refresh(orig_request_method)
File
"/app/miniconda/envs/beamenv/lib/python2.7/site-packages/oauth2client/contrib/gce.py",
line 138, in _refresh
raise client.HttpAccessTokenRefreshError(str(e))
INFO:root:Starting the size estimation of the input
INFO:oauth2client.transport:Attempting refresh to obtain initial access_token
Traceback (most recent call last):
File "/app/miniconda/envs/beamenv/lib/python2.7/runpy.py", line 174, in
_run_module_as_main
"__main__", fname, loader, pkg_name)
File "/app/miniconda/envs/beamenv/lib/python2.7/runpy.py", line 72, in
_run_code
exec code in run_globals
File
"/app/miniconda/envs/beamenv/lib/python2.7/site-packages/apache_beam/examples/wordcount.py",
line 134, in <module>
run()
File
"/app/miniconda/envs/beamenv/lib/python2.7/site-packages/apache_beam/examples/wordcount.py",
line 88, in run
lines = p | 'read' >> ReadFromText(known_args.input)
File
"/app/miniconda/envs/beamenv/lib/python2.7/site-packages/apache_beam/io/textio.py",
line 520, in __init__
skip_header_lines=skip_header_lines)
File
"/app/miniconda/envs/beamenv/lib/python2.7/site-packages/apache_beam/io/textio.py",
line 115, in __init__
validate=validate)
File
"/app/miniconda/envs/beamenv/lib/python2.7/site-packages/apache_beam/io/filebasedsource.py",
line 120, in __init__
self._validate()
File
"/app/miniconda/envs/beamenv/lib/python2.7/site-packages/apache_beam/options/value_provider.py",
line 107, in _f
return fnc(self, *args, **kwargs)
File
"/app/miniconda/envs/beamenv/lib/python2.7/site-packages/apache_beam/io/filebasedsource.py",
line 177, in _validate
match_result = FileSystems.match([pattern], limits=[1])[0]
File
"/app/miniconda/envs/beamenv/lib/python2.7/site-packages/apache_beam/io/filesystems.py",
line 131, in match
return filesystem.match(patterns, limits)
File
"/app/miniconda/envs/beamenv/lib/python2.7/site-packages/apache_beam/io/gcp/gcsfilesystem.py",
line 138, in match
raise BeamIOError("Match operation failed", exceptions)
apache_beam.io.filesystem.BeamIOError: Match operation failed with exceptions
{'gs://dataflow-samples/shakespeare/kinglear.txt':
HttpAccessTokenRefreshError('Unable to find the server at
metadata.google.internal',)}
Build step 'Execute shell' marked build as failure
