See
<https://builds.apache.org/job/beam_PostCommit_Python37/1608/display/redirect?page=changes>
Changes:
[github] [BEAM-9269] Add commit deadline for Spanner writes. (#10752)
------------------------------------------
[...truncated 136.86 KB...]
datanode_1 | 20/02/12 17:54:56 INFO impl.FsDatasetImpl: Volume reference is
released.
datanode_1 | 20/02/12 17:54:56 INFO impl.FsDatasetImpl: Adding block pool
BP-2096143976-172.25.0.2-1581530092904
datanode_1 | 20/02/12 17:54:56 INFO impl.FsDatasetImpl: Scanning block pool
BP-2096143976-172.25.0.2-1581530092904 on volume /hadoop/dfs/data/current...
datanode_1 | 20/02/12 17:54:56 INFO impl.FsDatasetImpl: Time taken to scan
block pool BP-2096143976-172.25.0.2-1581530092904 on /hadoop/dfs/data/current:
13ms
datanode_1 | 20/02/12 17:54:56 INFO impl.FsDatasetImpl: Total time to scan all
replicas for block pool BP-2096143976-172.25.0.2-1581530092904: 14ms
datanode_1 | 20/02/12 17:54:56 INFO impl.FsDatasetImpl: Adding replicas to map
for block pool BP-2096143976-172.25.0.2-1581530092904 on volume
/hadoop/dfs/data/current...
datanode_1 | 20/02/12 17:54:56 INFO impl.BlockPoolSlice: Replica Cache file:
/hadoop/dfs/data/current/BP-2096143976-172.25.0.2-1581530092904/current/replicas
doesn't exist
datanode_1 | 20/02/12 17:54:56 INFO impl.FsDatasetImpl: Time to add replicas
to map for block pool BP-2096143976-172.25.0.2-1581530092904 on volume
/hadoop/dfs/data/current: 1ms
datanode_1 | 20/02/12 17:54:56 INFO impl.FsDatasetImpl: Total time to add all
replicas to map: 1ms
datanode_1 | 20/02/12 17:54:56 INFO datanode.VolumeScanner: Now scanning bpid
BP-2096143976-172.25.0.2-1581530092904 on volume /hadoop/dfs/data
datanode_1 | 20/02/12 17:54:56 INFO datanode.VolumeScanner:
VolumeScanner(/hadoop/dfs/data, DS-a7714fb9-2fb0-42e3-bbe9-a33f791a268a):
finished scanning block pool BP-2096143976-172.25.0.2-1581530092904
datanode_1 | 20/02/12 17:54:56 INFO datanode.DirectoryScanner: Periodic
Directory Tree Verification scan starting at 2/12/20 9:27 PM with interval of
21600000ms
datanode_1 | 20/02/12 17:54:56 INFO datanode.DataNode: Block pool
BP-2096143976-172.25.0.2-1581530092904 (Datanode Uuid
5a3711a3-81ff-44bd-bfb8-ebb1a7dcefbf) service to namenode/172.25.0.2:8020
beginning handshake with NN
datanode_1 | 20/02/12 17:54:56 INFO datanode.VolumeScanner:
VolumeScanner(/hadoop/dfs/data, DS-a7714fb9-2fb0-42e3-bbe9-a33f791a268a): no
suitable block pools found to scan. Waiting 1814399969 ms.
namenode_1 | 20/02/12 17:54:56 INFO hdfs.StateChange: BLOCK* registerDatanode:
from DatanodeRegistration(172.25.0.3:50010,
datanodeUuid=5a3711a3-81ff-44bd-bfb8-ebb1a7dcefbf, infoPort=50075,
infoSecurePort=0, ipcPort=50020,
storageInfo=lv=-57;cid=CID-67ee2219-5c3c-4ca3-b4bc-3a3b97d7b0a3;nsid=1556663080;c=1581530092904)
storage 5a3711a3-81ff-44bd-bfb8-ebb1a7dcefbf
namenode_1 | 20/02/12 17:54:56 INFO net.NetworkTopology: Adding a new node:
/default-rack/172.25.0.3:50010
namenode_1 | 20/02/12 17:54:56 INFO blockmanagement.BlockReportLeaseManager:
Registered DN 5a3711a3-81ff-44bd-bfb8-ebb1a7dcefbf (172.25.0.3:50010).
datanode_1 | 20/02/12 17:54:56 INFO datanode.DataNode: Block pool Block pool
BP-2096143976-172.25.0.2-1581530092904 (Datanode Uuid
5a3711a3-81ff-44bd-bfb8-ebb1a7dcefbf) service to namenode/172.25.0.2:8020
successfully registered with NN
datanode_1 | 20/02/12 17:54:56 INFO datanode.DataNode: For namenode
namenode/172.25.0.2:8020 using BLOCKREPORT_INTERVAL of 21600000msec
CACHEREPORT_INTERVAL of 10000msec Initial delay: 0msec; heartBeatInterval=3000
> Task :runners:google-cloud-dataflow-java:worker:shadowJar
> Task :sdks:python:test-suites:direct:py37:hdfsIntegrationTest
namenode_1 | 20/02/12 17:54:56 INFO blockmanagement.DatanodeDescriptor: Adding
new storage ID DS-a7714fb9-2fb0-42e3-bbe9-a33f791a268a for DN 172.25.0.3:50010
namenode_1 | 20/02/12 17:54:57 INFO BlockStateChange: BLOCK* processReport
0xb47aa00e1d382dcc: Processing first storage report for
DS-a7714fb9-2fb0-42e3-bbe9-a33f791a268a from datanode
5a3711a3-81ff-44bd-bfb8-ebb1a7dcefbf
namenode_1 | 20/02/12 17:54:57 INFO BlockStateChange: BLOCK* processReport
0xb47aa00e1d382dcc: from storage DS-a7714fb9-2fb0-42e3-bbe9-a33f791a268a node
DatanodeRegistration(172.25.0.3:50010,
datanodeUuid=5a3711a3-81ff-44bd-bfb8-ebb1a7dcefbf, infoPort=50075,
infoSecurePort=0, ipcPort=50020,
storageInfo=lv=-57;cid=CID-67ee2219-5c3c-4ca3-b4bc-3a3b97d7b0a3;nsid=1556663080;c=1581530092904),
blocks: 0, hasStaleStorage: false, processing time: 1 msecs,
invalidatedBlocks: 0
datanode_1 | 20/02/12 17:54:57 INFO datanode.DataNode: Successfully sent block
report 0xb47aa00e1d382dcc, containing 1 storage report(s), of which we sent 1.
The reports had 0 total blocks and used 1 RPC(s). This took 4 msec to generate
and 48 msecs for RPC and NN processing. Got back one command: FinalizeCommand/5.
datanode_1 | 20/02/12 17:54:57 INFO datanode.DataNode: Got finalize command
for block pool BP-2096143976-172.25.0.2-1581530092904
test_1 | hdfs_integration_test create:
/app/sdks/python/target/.tox/hdfs_integration_test
test_1 | hdfs_integration_test installdeps: -rbuild-requirements.txt,
gsutil==4.47, holdup==1.8.0
> Task :runners:flink:1.9:job-server:shadowJar
> Task :runners:spark:job-server:shadowJar
> Task :sdks:python:test-suites:direct:py37:hdfsIntegrationTest
test_1 | hdfs_integration_test inst:
/app/sdks/python/target/.tox/.tmp/package/1/apache-beam-2.20.0.dev0.zip
test_1 | ERROR: invocation failed (exit code 1), logfile:
/app/sdks/python/target/.tox/hdfs_integration_test/log/hdfs_integration_test-2.log
test_1 | ================================== log start
===================================
test_1 | Processing
./target/.tox/.tmp/package/1/apache-beam-2.20.0.dev0.zip
test_1 | Requirement already satisfied: crcmod<2.0,>=1.7 in
./target/.tox/hdfs_integration_test/lib/python3.7/site-packages (from
apache-beam==2.20.0.dev0) (1.7)
test_1 | Collecting dill<0.3.2,>=0.3.1.1
test_1 | Downloading dill-0.3.1.1.tar.gz (151 kB)
test_1 | Collecting fastavro<0.22,>=0.21.4
test_1 | Downloading fastavro-0.21.24-cp37-cp37m-manylinux1_x86_64.whl
(1.2 MB)
test_1 | Requirement already satisfied: future<1.0.0,>=0.16.0 in
./target/.tox/hdfs_integration_test/lib/python3.7/site-packages (from
apache-beam==2.20.0.dev0) (0.16.0)
test_1 | Requirement already satisfied: grpcio<2,>=1.12.1 in
./target/.tox/hdfs_integration_test/lib/python3.7/site-packages (from
apache-beam==2.20.0.dev0) (1.27.1)
test_1 | Collecting hdfs<3.0.0,>=2.1.0
test_1 | Downloading hdfs-2.5.8.tar.gz (41 kB)
test_1 | Collecting httplib2<=0.12.0,>=0.8
test_1 | Downloading httplib2-0.12.0.tar.gz (218 kB)
test_1 | Requirement already satisfied: mock<3.0.0,>=1.0.1 in
./target/.tox/hdfs_integration_test/lib/python3.7/site-packages (from
apache-beam==2.20.0.dev0) (2.0.0)
test_1 | Collecting numpy<2,>=1.14.3
test_1 | Downloading numpy-1.18.1-cp37-cp37m-manylinux1_x86_64.whl (20.1
MB)
test_1 | Collecting pymongo<4.0.0,>=3.8.0
test_1 | Downloading pymongo-3.10.1-cp37-cp37m-manylinux2014_x86_64.whl
(462 kB)
test_1 | Collecting oauth2client<4,>=2.0.1
test_1 | Downloading oauth2client-3.0.0.tar.gz (77 kB)
test_1 | Requirement already satisfied: protobuf<4,>=3.5.0.post1 in
./target/.tox/hdfs_integration_test/lib/python3.7/site-packages (from
apache-beam==2.20.0.dev0) (3.11.3)
test_1 | Collecting pydot<2,>=1.2.0
test_1 | Downloading pydot-1.4.1-py2.py3-none-any.whl (19 kB)
test_1 | Collecting python-dateutil<3,>=2.8.0
test_1 | Downloading python_dateutil-2.8.1-py2.py3-none-any.whl (227 kB)
test_1 | Collecting pytz>=2018.3
test_1 | Downloading pytz-2019.3-py2.py3-none-any.whl (509 kB)
test_1 | Collecting typing<3.8.0,>=3.7.0
test_1 | Downloading typing-3.7.4.1-py3-none-any.whl (25 kB)
test_1 | Collecting typing-extensions<3.8.0,>=3.7.0
test_1 | Downloading typing_extensions-3.7.4.1-py3-none-any.whl (20 kB)
test_1 | Collecting avro-python3<2.0.0,>=1.8.1
test_1 | Downloading avro-python3-1.9.2.tar.gz (37 kB)
test_1 | ERROR: Command errored out with exit status 1:
test_1 | command:
/app/sdks/python/target/.tox/hdfs_integration_test/bin/python -c 'import sys,
setuptools, tokenize; sys.argv[0] =
'"'"'/tmp/pip-install-xi0n50zd/avro-python3/setup.py'"'"';
__file__='"'"'/tmp/pip-install-xi0n50zd/avro-python3/setup.py'"'"';f=getattr(tokenize,
'"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"',
'"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info
--egg-base /tmp/pip-install-xi0n50zd/avro-python3/pip-egg-info
test_1 | cwd: /tmp/pip-install-xi0n50zd/avro-python3/
test_1 | Complete output (5 lines):
test_1 | Traceback (most recent call last):
test_1 | File "<string>", line 1, in <module>
test_1 | File "/tmp/pip-install-xi0n50zd/avro-python3/setup.py",
line 41, in <module>
test_1 | import pycodestyle
test_1 | ModuleNotFoundError: No module named 'pycodestyle'
test_1 | ----------------------------------------
test_1 | ERROR: Command errored out with exit status 1: python setup.py
egg_info Check the logs for full command output.
test_1 |
test_1 | =================================== log end
====================================
test_1 | ___________________________________ summary
____________________________________
test_1 | ERROR: hdfs_integration_test: InvocationError for command
/app/sdks/python/target/.tox/hdfs_integration_test/bin/python
target/.tox/hdfs_integration_test/bin/pip install --retries 10 --exists-action
w
'/app/sdks/python/target/.tox/.tmp/package/1/apache-beam-2.20.0.dev0.zip[gcp]'
(exited with code 1)
hdfs_it-jenkins-beam_postcommit_python37-1608_test_1 exited with code 1
Stopping hdfs_it-jenkins-beam_postcommit_python37-1608_datanode_1 ...
Stopping hdfs_it-jenkins-beam_postcommit_python37-1608_namenode_1 ...
Stopping hdfs_it-jenkins-beam_postcommit_python37-1608_datanode_1 ... done
Stopping hdfs_it-jenkins-beam_postcommit_python37-1608_namenode_1 ... done
Aborting on container exit...
+ finally
+ docker-compose -p hdfs_IT-jenkins-beam_PostCommit_Python37-1608 --no-ansi down
Removing hdfs_it-jenkins-beam_postcommit_python37-1608_test_1 ...
Removing hdfs_it-jenkins-beam_postcommit_python37-1608_datanode_1 ...
Removing hdfs_it-jenkins-beam_postcommit_python37-1608_namenode_1 ...
Removing hdfs_it-jenkins-beam_postcommit_python37-1608_datanode_1 ... done
Removing hdfs_it-jenkins-beam_postcommit_python37-1608_test_1 ... done
Removing hdfs_it-jenkins-beam_postcommit_python37-1608_namenode_1 ... done
Removing network hdfs_it-jenkins-beam_postcommit_python37-1608_test_net
real 0m0.991s
user 0m0.603s
sys 0m0.100s
> Task :sdks:python:test-suites:direct:py37:hdfsIntegrationTest FAILED
> Task :sdks:python:test-suites:direct:py37:setupVirtualenv
Using base prefix '/usr'
New python executable in
<https://builds.apache.org/job/beam_PostCommit_Python37/ws/src/build/gradleenv/1398941891/bin/python3.7>
Also creating executable in
<https://builds.apache.org/job/beam_PostCommit_Python37/ws/src/build/gradleenv/1398941891/bin/python>
Installing setuptools, pip, wheel...
done.
Running virtualenv with interpreter /usr/bin/python3.7
Collecting tox==3.11.1
Using cached tox-3.11.1-py2.py3-none-any.whl (76 kB)
Collecting grpcio-tools<=1.14.2,>=1.3.5
Using cached grpcio_tools-1.14.2-cp37-cp37m-manylinux1_x86_64.whl (22.2 MB)
Processing
/home/jenkins/.cache/pip/wheels/bf/c9/a3/c538d90ef17cf7823fa51fc701a7a7a910a80f6a405bf15b1a/future-0.16.0-cp37-none-any.whl
Collecting filelock<4,>=3.0.0
Using cached filelock-3.0.12-py3-none-any.whl (7.6 kB)
Collecting virtualenv>=14.0.0
Using cached virtualenv-20.0.3-py2.py3-none-any.whl (4.6 MB)
Requirement already satisfied, skipping upgrade: setuptools>=30.0.0 in
<https://builds.apache.org/job/beam_PostCommit_Python37/ws/src/build/gradleenv/1398941891/lib/python3.7/site-packages>
(from tox==3.11.1) (45.2.0)
Collecting pluggy<1,>=0.3.0
Using cached pluggy-0.13.1-py2.py3-none-any.whl (18 kB)
Collecting py<2,>=1.4.17
Using cached py-1.8.1-py2.py3-none-any.whl (83 kB)
Collecting toml>=0.9.4
Using cached toml-0.10.0-py2.py3-none-any.whl (25 kB)
Collecting six<2,>=1.0.0
Using cached six-1.14.0-py2.py3-none-any.whl (10 kB)
Collecting grpcio>=1.14.2
Using cached grpcio-1.27.1-cp37-cp37m-manylinux2010_x86_64.whl (2.7 MB)
Collecting protobuf>=3.5.0.post1
Using cached protobuf-3.11.3-cp37-cp37m-manylinux1_x86_64.whl (1.3 MB)
Collecting importlib-metadata<2,>=0.12; python_version < "3.8"
Using cached importlib_metadata-1.5.0-py2.py3-none-any.whl (30 kB)
Collecting appdirs<2,>=1.4.3
Using cached appdirs-1.4.3-py2.py3-none-any.whl (12 kB)
Processing
/home/jenkins/.cache/pip/wheels/a2/19/da/a15d4e2bedf3062c739b190d5cb5b7b2ecfbccb6b0d93c861b/distlib-0.3.0-py3-none-any.whl
Collecting zipp>=0.5
Using cached zipp-2.2.0-py36-none-any.whl (4.6 kB)
Installing collected packages: filelock, zipp, importlib-metadata, appdirs,
distlib, six, virtualenv, pluggy, py, toml, tox, grpcio, protobuf,
grpcio-tools, future
Successfully installed appdirs-1.4.3 distlib-0.3.0 filelock-3.0.12
future-0.16.0 grpcio-1.27.1 grpcio-tools-1.14.2 importlib-metadata-1.5.0
pluggy-0.13.1 protobuf-3.11.3 py-1.8.1 six-1.14.0 toml-0.10.0 tox-3.11.1
virtualenv-20.0.3 zipp-2.2.0
> Task :sdks:python:test-suites:direct:py37:installGcpTest
Processing
<https://builds.apache.org/job/beam_PostCommit_Python37/ws/src/sdks/python/build/apache-beam.tar.gz>
Processing
/home/jenkins/.cache/pip/wheels/50/24/4d/4580ca4a299f1ad6fd63443e6e584cb21e9a07988e4aa8daac/crcmod-1.7-cp37-cp37m-linux_x86_64.whl
Processing
/home/jenkins/.cache/pip/wheels/59/b1/91/f02e76c732915c4015ab4010f3015469866c1eb9b14058d8e7/dill-0.3.1.1-cp37-none-any.whl
Collecting fastavro<0.22,>=0.21.4
Using cached fastavro-0.21.24-cp37-cp37m-manylinux1_x86_64.whl (1.2 MB)
Requirement already satisfied: future<1.0.0,>=0.16.0 in
<https://builds.apache.org/job/beam_PostCommit_Python37/ws/src/build/gradleenv/1398941891/lib/python3.7/site-packages>
(from apache-beam==2.20.0.dev0) (0.16.0)
Requirement already satisfied: grpcio<2,>=1.12.1 in
<https://builds.apache.org/job/beam_PostCommit_Python37/ws/src/build/gradleenv/1398941891/lib/python3.7/site-packages>
(from apache-beam==2.20.0.dev0) (1.27.1)
Processing
/home/jenkins/.cache/pip/wheels/fe/a7/05/23e3699975fc20f8a30e00ac1e515ab8c61168e982abe4ce70/hdfs-2.5.8-cp37-none-any.whl
Processing
/home/jenkins/.cache/pip/wheels/6d/41/4b/2b369d6e2b7eaebcdd423516d3fb659c7658c16a2be8fd04ec/httplib2-0.12.0-cp37-none-any.whl
Collecting mock<3.0.0,>=1.0.1
Using cached mock-2.0.0-py2.py3-none-any.whl (56 kB)
Collecting numpy<2,>=1.14.3
Using cached numpy-1.18.1-cp37-cp37m-manylinux1_x86_64.whl (20.1 MB)
Collecting pymongo<4.0.0,>=3.8.0
Using cached pymongo-3.10.1-cp37-cp37m-manylinux2014_x86_64.whl (462 kB)
Processing
/home/jenkins/.cache/pip/wheels/48/f7/87/b932f09c6335dbcf45d916937105a372ab14f353a9ca431d7d/oauth2client-3.0.0-cp37-none-any.whl
Requirement already satisfied: protobuf<4,>=3.5.0.post1 in
<https://builds.apache.org/job/beam_PostCommit_Python37/ws/src/build/gradleenv/1398941891/lib/python3.7/site-packages>
(from apache-beam==2.20.0.dev0) (3.11.3)
Collecting pydot<2,>=1.2.0
Using cached pydot-1.4.1-py2.py3-none-any.whl (19 kB)
Collecting python-dateutil<3,>=2.8.0
Using cached python_dateutil-2.8.1-py2.py3-none-any.whl (227 kB)
Collecting pytz>=2018.3
Using cached pytz-2019.3-py2.py3-none-any.whl (509 kB)
Collecting typing<3.8.0,>=3.7.0
Using cached typing-3.7.4.1-py3-none-any.whl (25 kB)
Collecting typing-extensions<3.8.0,>=3.7.0
Using cached typing_extensions-3.7.4.1-py3-none-any.whl (20 kB)
Collecting avro-python3<2.0.0,>=1.8.1
Using cached avro-python3-1.9.2.tar.gz (37 kB)
ERROR: Command errored out with exit status 1:
command:
<https://builds.apache.org/job/beam_PostCommit_Python37/ws/src/build/gradleenv/1398941891/bin/python3.7>
-c 'import sys, setuptools, tokenize; sys.argv[0] =
'"'"'/tmp/pip-install-bcm19np9/avro-python3/setup.py'"'"';
__file__='"'"'/tmp/pip-install-bcm19np9/avro-python3/setup.py'"'"';f=getattr(tokenize,
'"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"',
'"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info
--egg-base /tmp/pip-install-bcm19np9/avro-python3/pip-egg-info
cwd: /tmp/pip-install-bcm19np9/avro-python3/
Complete output (5 lines):
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/tmp/pip-install-bcm19np9/avro-python3/setup.py", line 41, in
<module>
import pycodestyle
ModuleNotFoundError: No module named 'pycodestyle'
----------------------------------------
ERROR: Command errored out with exit status 1: python setup.py egg_info Check
the logs for full command output.
> Task :sdks:python:test-suites:direct:py37:installGcpTest FAILED
FAILURE: Build completed with 4 failures.
1: Task failed with an exception.
-----------
* What went wrong:
Execution failed for task
':sdks:python:test-suites:dataflow:py37:installGcpTest'.
> Process 'command 'sh'' finished with non-zero exit value 1
* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug
option to get more log output. Run with --scan to get full insights.
==============================================================================
2: Task failed with an exception.
-----------
* What went wrong:
Execution failed for task
':sdks:python:test-suites:portable:py37:installGcpTest'.
> Process 'command 'sh'' finished with non-zero exit value 1
* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug
option to get more log output. Run with --scan to get full insights.
==============================================================================
3: Task failed with an exception.
-----------
* Where:
Build file
'<https://builds.apache.org/job/beam_PostCommit_Python37/ws/src/sdks/python/test-suites/direct/py37/build.gradle'>
line: 60
* What went wrong:
Execution failed for task
':sdks:python:test-suites:direct:py37:hdfsIntegrationTest'.
> Process 'command 'sh'' finished with non-zero exit value 1
* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug
option to get more log output. Run with --scan to get full insights.
==============================================================================
4: Task failed with an exception.
-----------
* What went wrong:
Execution failed for task ':sdks:python:test-suites:direct:py37:installGcpTest'.
> Process 'command 'sh'' finished with non-zero exit value 1
* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug
option to get more log output. Run with --scan to get full insights.
==============================================================================
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with
Gradle 6.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See
https://docs.gradle.org/5.2.1/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 2m 24s
79 actionable tasks: 59 executed, 20 from cache
Publishing build scan...
https://gradle.com/s/oi22fmbdztwco
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
