Rein Houthooft created BEAM-7054:
------------------------------------
Summary: PortableRunner on Flink cluster crashes
Key: BEAM-7054
URL: https://issues.apache.org/jira/browse/BEAM-7054
Project: Beam
Issue Type: Bug
Components: examples-python, runner-flink
Affects Versions: 2.11.0
Environment: Ubuntu 18.04, Beam 2.11.0, Flink 1.7.2, Docker 18.09.4,
on a 2-node cluster with identical VM images
Reporter: Rein Houthooft
h2. Setup
2-node cluster: `master` and `worker`. `master` runs Flink JobManager, `worker`
runs Flink TaskManager.
h2. Problem description
When running Flink TaskManager on `master` and running
{code}
python sdks/python/apache_beam/examples/wordcount.py --input=gs://<input file
location>-output=gs://<output file location>--runner=PortableRunner
--job_endpoint=localhost:8099 --environment_config='<docker image location>'
{code}
on `master`, the job executes correctly.
However, when running the Flink TaskManger on `worker` (the second node), the
`worker` outputs:
{code:java}
java.util.concurrent.ExecutionException: java.io.FileNotFoundException:
/tmp/beam-artifact-staging/job_8ff51268-48fb-42e3-b14c-f2e387e26e5a/MANIFEST
(No such file or directory)
{code}
and crashes (full log in [1]).
The docker image has been created via:
{code}
./gradlew :beam-sdks-python-container:docker
{code}
The job server has been set up on `master` via:
{code}
./gradlew :beam-runners-flink-1.7-job-server:runShadow
-PflinkMasterUrl=localhost:8081
{code}
which logs
{code:java}
> Task :beam-runners-flink-1.7-job-server:runShadow
Listening for transport dt_socket at address: 5005
[main] INFO org.apache.beam.runners.flink.FlinkJobServerDriver -
ArtifactStagingService started on localhost:8098
[main] INFO org.apache.beam.runners.flink.FlinkJobServerDriver - JobService
started on localhost:8099
<============-> 98% EXECUTING [3m 0s]
> :beam-runners-flink-1.7-job-server:runShadow
> IDLE
> IDLE
> IDLE
{code}
The default Flink example
{code:java}
./bin/flink run ./examples/batch/WordCount.jarworks correctly on the 2-node
cluster.
{code}
works correctly on the 2-node cluster.
h2. Expected behavior
Either the job executes correctly and returns as in the single-node setup or
additional configuration is needed, which is not reflected in the docs. Please
let me know if I missed something;.
----
[1]
{code:java}
tail -111111f
/root/flink-1.7.2/log/flink-root-taskexecutor-0-VM-0-16-ubuntu.log
2019-04-11 13:11:59,497 INFO
org.apache.flink.runtime.taskexecutor.TaskManagerRunner -
--------------------------------------------------------------------------------
2019-04-11 13:11:59,498 INFO
org.apache.flink.runtime.taskexecutor.TaskManagerRunner - Starting
TaskManager (Version: 1.7.2, Rev:ceba8af, Date:11.02.2019 @ 14:17:09 UTC)
2019-04-11 13:11:59,498 INFO
org.apache.flink.runtime.taskexecutor.TaskManagerRunner - OS current
user: root
2019-04-11 13:11:59,499 INFO
org.apache.flink.runtime.taskexecutor.TaskManagerRunner - Current
Hadoop/Kerberos user: <no hadoop dependency found>
2019-04-11 13:11:59,499 INFO
org.apache.flink.runtime.taskexecutor.TaskManagerRunner - JVM: OpenJDK
64-Bit Server VM - Oracle Corporation - 1.8/25.191-b12
2019-04-11 13:11:59,499 INFO
org.apache.flink.runtime.taskexecutor.TaskManagerRunner - Maximum heap
size: 922 MiBytes
2019-04-11 13:11:59,499 INFO
org.apache.flink.runtime.taskexecutor.TaskManagerRunner - JAVA_HOME:
(not set)
2019-04-11 13:11:59,499 INFO
org.apache.flink.runtime.taskexecutor.TaskManagerRunner - No Hadoop
Dependency available
2019-04-11 13:11:59,499 INFO
org.apache.flink.runtime.taskexecutor.TaskManagerRunner - JVM Options:
2019-04-11 13:11:59,499 INFO
org.apache.flink.runtime.taskexecutor.TaskManagerRunner - -XX:+UseG1GC
2019-04-11 13:11:59,499 INFO
org.apache.flink.runtime.taskexecutor.TaskManagerRunner - -Xms922M
2019-04-11 13:11:59,499 INFO
org.apache.flink.runtime.taskexecutor.TaskManagerRunner - -Xmx922M
2019-04-11 13:11:59,500 INFO
org.apache.flink.runtime.taskexecutor.TaskManagerRunner -
-XX:MaxDirectMemorySize=8388607T
2019-04-11 13:11:59,500 INFO
org.apache.flink.runtime.taskexecutor.TaskManagerRunner -
-Dlog.file=/root/flink-1.7.2/log/flink-root-taskexecutor-0-VM-0-16-ubuntu.log
2019-04-11 13:11:59,500 INFO
org.apache.flink.runtime.taskexecutor.TaskManagerRunner -
-Dlog4j.configuration=file:/root/flink-1.7.2/conf/log4j.properties
2019-04-11 13:11:59,500 INFO
org.apache.flink.runtime.taskexecutor.TaskManagerRunner -
-Dlogback.configurationFile=file:/root/flink-1.7.2/conf/logback.xml
2019-04-11 13:11:59,500 INFO
org.apache.flink.runtime.taskexecutor.TaskManagerRunner - Program
Arguments:
2019-04-11 13:11:59,500 INFO
org.apache.flink.runtime.taskexecutor.TaskManagerRunner - --configDir
2019-04-11 13:11:59,500 INFO
org.apache.flink.runtime.taskexecutor.TaskManagerRunner -
/root/flink-1.7.2/conf
2019-04-11 13:11:59,500 INFO
org.apache.flink.runtime.taskexecutor.TaskManagerRunner - Classpath:
/root/flink-1.7.2/lib/flink-python_2.11-1.7.2.jar:/root/flink-1.7.2/lib/log4j-1.2.17.jar:/root/flink-1.7.2/lib/slf4j-log4j12-1.7.15.jar:/root/flink-1.7.2/lib/flink-dist_2.11-1.7.2.jar:::
2019-04-11 13:11:59,500 INFO
org.apache.flink.runtime.taskexecutor.TaskManagerRunner -
--------------------------------------------------------------------------------
2019-04-11 13:11:59,502 INFO
org.apache.flink.runtime.taskexecutor.TaskManagerRunner - Registered UNIX
signal handlers for [TERM, HUP, INT]
2019-04-11 13:11:59,505 INFO
org.apache.flink.runtime.taskexecutor.TaskManagerRunner - Maximum number
of open file descriptors is 1048576.
2019-04-11 13:11:59,517 INFO
org.apache.flink.configuration.GlobalConfiguration - Loading
configuration property: jobmanager.rpc.address, 172.26.0.4
2019-04-11 13:11:59,517 INFO
org.apache.flink.configuration.GlobalConfiguration - Loading
configuration property: jobmanager.rpc.port, 6123
2019-04-11 13:11:59,517 INFO
org.apache.flink.configuration.GlobalConfiguration - Loading
configuration property: jobmanager.heap.size, 1024m
2019-04-11 13:11:59,517 INFO
org.apache.flink.configuration.GlobalConfiguration - Loading
configuration property: taskmanager.heap.size, 1024m
2019-04-11 13:11:59,517 INFO
org.apache.flink.configuration.GlobalConfiguration - Loading
configuration property: taskmanager.numberOfTaskSlots, 4
2019-04-11 13:11:59,517 INFO
org.apache.flink.configuration.GlobalConfiguration - Loading
configuration property: parallelism.default, 2
2019-04-11 13:11:59,518 INFO
org.apache.flink.configuration.GlobalConfiguration - Loading
configuration property: rest.port, 8081
2019-04-11 13:11:59,523 INFO org.apache.flink.core.fs.FileSystem
- Hadoop is not in the classpath/dependencies. The extended set of
supported File Systems via Hadoop is not available.
2019-04-11 13:11:59,547 INFO
org.apache.flink.runtime.security.modules.HadoopModuleFactory - Cannot create
Hadoop Security Module because Hadoop cannot be found in the Classpath.
2019-04-11 13:11:59,568 INFO org.apache.flink.runtime.security.SecurityUtils
- Cannot install HadoopSecurityContext because Hadoop cannot be
found in the Classpath.
2019-04-11 13:11:59,754 WARN org.apache.flink.configuration.Configuration
- Config uses deprecated configuration key 'jobmanager.rpc.address'
instead of proper key 'rest.address'
2019-04-11 13:11:59,758 INFO
org.apache.flink.runtime.util.LeaderRetrievalUtils - Trying to
select the network interface and address to use by connecting to the leading
JobManager.
2019-04-11 13:11:59,759 INFO
org.apache.flink.runtime.util.LeaderRetrievalUtils - TaskManager
will try to connect for 10000 milliseconds before falling back to heuristics
2019-04-11 13:11:59,762 INFO org.apache.flink.runtime.net.ConnectionUtils
- Retrieved new target address /172.26.0.4:6123.
2019-04-11 13:11:59,769 INFO
org.apache.flink.runtime.taskexecutor.TaskManagerRunner - TaskManager
will use hostname/address '172.26.0.16' (172.26.0.16) for communication.
2019-04-11 13:11:59,772 INFO
org.apache.flink.runtime.rpc.akka.AkkaRpcServiceUtils - Trying to start
actor system at 172.26.0.16:0
2019-04-11 13:12:00,135 INFO akka.event.slf4j.Slf4jLogger
- Slf4jLogger started
2019-04-11 13:12:00,201 INFO akka.remote.Remoting
- Starting remoting
2019-04-11 13:12:00,303 INFO akka.remote.Remoting
- Remoting started; listening on addresses
:[akka.tcp://[email protected]:39493]
2019-04-11 13:12:00,310 INFO
org.apache.flink.runtime.rpc.akka.AkkaRpcServiceUtils - Actor system
started at akka.tcp://[email protected]:39493
2019-04-11 13:12:00,316 INFO
org.apache.flink.runtime.taskexecutor.TaskManagerRunner - Trying to start
actor system at 172.26.0.16:0
2019-04-11 13:12:00,332 INFO akka.event.slf4j.Slf4jLogger
- Slf4jLogger started
2019-04-11 13:12:00,338 INFO akka.remote.Remoting
- Starting remoting
2019-04-11 13:12:00,354 INFO
org.apache.flink.runtime.taskexecutor.TaskManagerRunner - Actor system
started at akka.tcp://[email protected]:41573
2019-04-11 13:12:00,356 INFO akka.remote.Remoting
- Remoting started; listening on addresses
:[akka.tcp://[email protected]:41573]
2019-04-11 13:12:00,367 INFO
org.apache.flink.runtime.metrics.MetricRegistryImpl - No metrics
reporter configured, no metrics will be exposed/reported.
2019-04-11 13:12:00,373 INFO org.apache.flink.runtime.blob.PermanentBlobCache
- Created BLOB cache storage directory
/tmp/blobStore-b1adf8f9-213f-4fcf-86ee-be140bc76181
2019-04-11 13:12:00,376 INFO org.apache.flink.runtime.blob.TransientBlobCache
- Created BLOB cache storage directory
/tmp/blobStore-3bdadd52-faad-4807-b06a-6e6924f1b1d0
2019-04-11 13:12:00,380 INFO
org.apache.flink.runtime.taskexecutor.TaskManagerRunner - Starting
TaskManager with ResourceID: 76eff45c35decbc12d43e53d88883a7f
2019-04-11 13:12:00,384 INFO
org.apache.flink.runtime.io.network.netty.NettyConfig - NettyConfig
[server address: /172.26.0.16, server port: 0, ssl enabled: false, memory
segment size (bytes): 32768, transport type: NIO, number of server threads: 4
(manual), number of client threads: 4 (manual), server connect backlog: 0 (use
Netty's default), client connect timeout (sec): 120, send/receive buffer size
(bytes): 0 (use Netty's default)]
2019-04-11 13:12:00,423 INFO
org.apache.flink.runtime.taskexecutor.TaskManagerServices - Temporary file
directory '/tmp': total 49 GB, usable 38 GB (77.55% usable)
2019-04-11 13:12:00,492 INFO
org.apache.flink.runtime.io.network.buffer.NetworkBufferPool - Allocated 102
MB for network buffer pool (number of memory segments: 3278, bytes per segment:
32768).
2019-04-11 13:12:00,542 INFO
org.apache.flink.runtime.query.QueryableStateUtils - Could not load
Queryable State Client Proxy. Probable reason: flink-queryable-state-runtime is
not in the classpath. To enable Queryable State, please move the
flink-queryable-state-runtime jar from the opt to the lib folder.
2019-04-11 13:12:00,542 INFO
org.apache.flink.runtime.query.QueryableStateUtils - Could not load
Queryable State Server. Probable reason: flink-queryable-state-runtime is not
in the classpath. To enable Queryable State, please move the
flink-queryable-state-runtime jar from the opt to the lib folder.
2019-04-11 13:12:00,543 INFO
org.apache.flink.runtime.io.network.NetworkEnvironment - Starting the
network environment and its components.
2019-04-11 13:12:00,578 INFO
org.apache.flink.runtime.io.network.netty.NettyClient - Successful
initialization (took 33 ms).
2019-04-11 13:12:00,624 INFO
org.apache.flink.runtime.io.network.netty.NettyServer - Successful
initialization (took 45 ms). Listening on SocketAddress /172.26.0.16:33565.
2019-04-11 13:12:00,625 WARN
org.apache.flink.runtime.taskmanager.TaskManagerLocation - No hostname
could be resolved for the IP address 172.26.0.16, using IP address as host
name. Local input split assignment (such as for HDFS files) may be impacted.
2019-04-11 13:12:00,625 INFO
org.apache.flink.runtime.taskexecutor.TaskManagerServices - Limiting
managed memory to 0.7 of the currently free heap space (640 MB), memory will be
allocated lazily.
2019-04-11 13:12:00,629 INFO
org.apache.flink.runtime.io.disk.iomanager.IOManager - I/O manager
uses directory /tmp/flink-io-eb728574-61d9-4170-a020-5eeed20b8551 for spill
files.
2019-04-11 13:12:00,693 INFO
org.apache.flink.runtime.taskexecutor.TaskManagerConfiguration - Messages have
a max timeout of 10000 ms
2019-04-11 13:12:00,701 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcService
- Starting RPC endpoint for
org.apache.flink.runtime.taskexecutor.TaskExecutor at
akka://flink/user/taskmanager_0 .
2019-04-11 13:12:00,717 INFO
org.apache.flink.runtime.taskexecutor.JobLeaderService - Start job
leader service.
2019-04-11 13:12:00,718 INFO
org.apache.flink.runtime.taskexecutor.TaskExecutor - Connecting to
ResourceManager
akka.tcp://[email protected]:6123/user/resourcemanager(00000000000000000000000000000000).
2019-04-11 13:12:00,718 INFO org.apache.flink.runtime.filecache.FileCache
- User file cache uses directory
/tmp/flink-dist-cache-d80719fe-a81a-47b6-9560-59766fb86afe
2019-04-11 13:12:00,907 INFO
org.apache.flink.runtime.taskexecutor.TaskExecutor - Resolved
ResourceManager address, beginning registration
2019-04-11 13:12:00,907 INFO
org.apache.flink.runtime.taskexecutor.TaskExecutor - Registration at
ResourceManager attempt 1 (timeout=100ms)
2019-04-11 13:12:00,965 INFO
org.apache.flink.runtime.taskexecutor.TaskExecutor - Successful
registration at resource manager
akka.tcp://[email protected]:6123/user/resourcemanager under registration id
28630de4ae4181854d3b7143d9a37af1.
2019-04-11 13:22:13,394 INFO
org.apache.flink.runtime.taskexecutor.TaskExecutor - Receive slot
request AllocationID\{8810509925a2184bdabc6d58bd808b17} for job
a0f3ed507058c39f6261f1e8953d9363 from resource manager with leader id
00000000000000000000000000000000.
2019-04-11 13:22:13,395 INFO
org.apache.flink.runtime.taskexecutor.TaskExecutor - Allocated slot
for AllocationID\{8810509925a2184bdabc6d58bd808b17}.
2019-04-11 13:22:13,396 INFO
org.apache.flink.runtime.taskexecutor.JobLeaderService - Add job
a0f3ed507058c39f6261f1e8953d9363 for job leader monitoring.
2019-04-11 13:22:13,397 INFO
org.apache.flink.runtime.taskexecutor.JobLeaderService - Try to register
at job manager akka.tcp://[email protected]:6123/user/jobmanager_0 with leader
id 00000000-0000-0000-0000-000000000000.
2019-04-11 13:22:13,414 INFO
org.apache.flink.runtime.taskexecutor.JobLeaderService - Resolved
JobManager address, beginning registration
2019-04-11 13:22:13,414 INFO
org.apache.flink.runtime.taskexecutor.JobLeaderService - Registration at
JobManager attempt 1 (timeout=100ms)
2019-04-11 13:22:13,433 INFO
org.apache.flink.runtime.taskexecutor.JobLeaderService - Successful
registration at job manager akka.tcp://[email protected]:6123/user/jobmanager_0
for job a0f3ed507058c39f6261f1e8953d9363.
2019-04-11 13:22:13,434 INFO
org.apache.flink.runtime.taskexecutor.TaskExecutor - Establish
JobManager connection for job a0f3ed507058c39f6261f1e8953d9363.
2019-04-11 13:22:13,437 INFO
org.apache.flink.runtime.taskexecutor.TaskExecutor - Offer reserved
slots to the leader of job a0f3ed507058c39f6261f1e8953d9363.
2019-04-11 13:22:13,454 INFO
org.apache.flink.runtime.taskexecutor.slot.TaskSlotTable - Activate slot
AllocationID\{8810509925a2184bdabc6d58bd808b17}.
2019-04-11 13:22:13,481 INFO
org.apache.flink.runtime.taskexecutor.TaskExecutor - Received task
DataSource (Impulse) (1/1).
2019-04-11 13:22:13,482 INFO org.apache.flink.runtime.taskmanager.Task
- DataSource (Impulse) (1/1) (ad6774374f4abba501b8f319af3d9ac1)
switched from CREATED to DEPLOYING.
2019-04-11 13:22:13,482 INFO
org.apache.flink.runtime.taskexecutor.TaskExecutor - Received task
DataSource (Impulse) (1/1).
2019-04-11 13:22:13,483 INFO org.apache.flink.runtime.taskmanager.Task
- Creating FileSystem stream leak safety net for task DataSource
(Impulse) (1/1) (ad6774374f4abba501b8f319af3d9ac1) [DEPLOYING]
2019-04-11 13:22:13,483 INFO org.apache.flink.runtime.taskmanager.Task
- DataSource (Impulse) (1/1) (94014197941e34830c0d79aff847f000)
switched from CREATED to DEPLOYING.
2019-04-11 13:22:13,484 INFO org.apache.flink.runtime.taskmanager.Task
- Creating FileSystem stream leak safety net for task DataSource
(Impulse) (1/1) (94014197941e34830c0d79aff847f000) [DEPLOYING]
2019-04-11 13:22:13,486 INFO org.apache.flink.runtime.taskmanager.Task
- Loading JAR files for task DataSource (Impulse) (1/1)
(94014197941e34830c0d79aff847f000) [DEPLOYING].
2019-04-11 13:22:13,487 INFO org.apache.flink.runtime.taskmanager.Task
- Loading JAR files for task DataSource (Impulse) (1/1)
(ad6774374f4abba501b8f319af3d9ac1) [DEPLOYING].
2019-04-11 13:22:13,489 INFO org.apache.flink.runtime.blob.BlobClient
- Downloading
a0f3ed507058c39f6261f1e8953d9363/p-f784164b3f975e48e5bdfa7f321e70fbe4b0e186-ca919340fcac9cb4b2e93b7b6225711b
from /172.26.0.4:45541
2019-04-11 13:22:14,085 INFO org.apache.flink.runtime.taskmanager.Task
- Registering task at network: DataSource (Impulse) (1/1)
(94014197941e34830c0d79aff847f000) [DEPLOYING].
2019-04-11 13:22:14,085 INFO org.apache.flink.runtime.taskmanager.Task
- Registering task at network: DataSource (Impulse) (1/1)
(ad6774374f4abba501b8f319af3d9ac1) [DEPLOYING].
2019-04-11 13:22:14,103 INFO org.apache.flink.runtime.taskmanager.Task
- DataSource (Impulse) (1/1) (94014197941e34830c0d79aff847f000)
switched from DEPLOYING to RUNNING.
2019-04-11 13:22:14,103 INFO org.apache.flink.runtime.taskmanager.Task
- DataSource (Impulse) (1/1) (ad6774374f4abba501b8f319af3d9ac1)
switched from DEPLOYING to RUNNING.
2019-04-11 13:22:14,325 INFO org.apache.flink.runtime.taskmanager.Task
- DataSource (Impulse) (1/1) (94014197941e34830c0d79aff847f000)
switched from RUNNING to FINISHED.
2019-04-11 13:22:14,325 INFO org.apache.flink.runtime.taskmanager.Task
- Freeing task resources for DataSource (Impulse) (1/1)
(94014197941e34830c0d79aff847f000).
2019-04-11 13:22:14,326 INFO org.apache.flink.runtime.taskmanager.Task
- Ensuring all FileSystem streams are closed for task DataSource
(Impulse) (1/1) (94014197941e34830c0d79aff847f000) [FINISHED]
2019-04-11 13:22:14,326 INFO org.apache.flink.runtime.taskmanager.Task
- DataSource (Impulse) (1/1) (ad6774374f4abba501b8f319af3d9ac1)
switched from RUNNING to FINISHED.
2019-04-11 13:22:14,326 INFO org.apache.flink.runtime.taskmanager.Task
- Freeing task resources for DataSource (Impulse) (1/1)
(ad6774374f4abba501b8f319af3d9ac1).
2019-04-11 13:22:14,327 INFO org.apache.flink.runtime.taskmanager.Task
- Ensuring all FileSystem streams are closed for task DataSource
(Impulse) (1/1) (ad6774374f4abba501b8f319af3d9ac1) [FINISHED]
2019-04-11 13:22:14,327 INFO
org.apache.flink.runtime.taskexecutor.TaskExecutor - Un-registering
task and sending final execution state FINISHED to JobManager for task
DataSource (Impulse) 94014197941e34830c0d79aff847f000.
2019-04-11 13:22:14,333 INFO
org.apache.flink.runtime.taskexecutor.TaskExecutor - Un-registering
task and sending final execution state FINISHED to JobManager for task
DataSource (Impulse) ad6774374f4abba501b8f319af3d9ac1.
2019-04-11 13:22:14,341 INFO
org.apache.flink.runtime.taskexecutor.TaskExecutor - Received task
CHAIN MapPartition (MapPartition at
[2]write/Write/WriteImpl/DoOnce/\{FlatMap(<lambda at core.py:2123>),
Map(decode)}) -> FlatMap (FlatMap at ExtractOutput[0]) (1/1).
2019-04-11 13:22:14,343 INFO
org.apache.flink.runtime.taskexecutor.TaskExecutor - Received task
CHAIN MapPartition (MapPartition at [1]read/Read/Split) -> FlatMap (FlatMap at
ExtractOutput[0]) (1/1).
2019-04-11 13:22:14,343 INFO org.apache.flink.runtime.taskmanager.Task
- CHAIN MapPartition (MapPartition at
[2]write/Write/WriteImpl/DoOnce/\{FlatMap(<lambda at core.py:2123>),
Map(decode)}) -> FlatMap (FlatMap at ExtractOutput[0]) (1/1)
(c3350586427cb04999d1c23038e2b1fc) switched from CREATED to DEPLOYING.
2019-04-11 13:22:14,344 INFO org.apache.flink.runtime.taskmanager.Task
- Creating FileSystem stream leak safety net for task CHAIN
MapPartition (MapPartition at [2]write/Write/WriteImpl/DoOnce/\{FlatMap(<lambda
at core.py:2123>), Map(decode)}) -> FlatMap (FlatMap at ExtractOutput[0]) (1/1)
(c3350586427cb04999d1c23038e2b1fc) [DEPLOYING]
2019-04-11 13:22:14,348 INFO org.apache.flink.runtime.taskmanager.Task
- Loading JAR files for task CHAIN MapPartition (MapPartition at
[2]write/Write/WriteImpl/DoOnce/\{FlatMap(<lambda at core.py:2123>),
Map(decode)}) -> FlatMap (FlatMap at ExtractOutput[0]) (1/1)
(c3350586427cb04999d1c23038e2b1fc) [DEPLOYING].
2019-04-11 13:22:14,350 INFO org.apache.flink.runtime.taskmanager.Task
- Registering task at network: CHAIN MapPartition (MapPartition at
[2]write/Write/WriteImpl/DoOnce/\{FlatMap(<lambda at core.py:2123>),
Map(decode)}) -> FlatMap (FlatMap at ExtractOutput[0]) (1/1)
(c3350586427cb04999d1c23038e2b1fc) [DEPLOYING].
2019-04-11 13:22:14,350 INFO org.apache.flink.runtime.taskmanager.Task
- CHAIN MapPartition (MapPartition at
[2]write/Write/WriteImpl/DoOnce/\{FlatMap(<lambda at core.py:2123>),
Map(decode)}) -> FlatMap (FlatMap at ExtractOutput[0]) (1/1)
(c3350586427cb04999d1c23038e2b1fc) switched from DEPLOYING to RUNNING.
2019-04-11 13:22:14,352 WARN org.apache.flink.metrics.MetricGroup
- The operator name MapPartition (MapPartition at
[2]write/Write/WriteImpl/DoOnce/\{FlatMap(<lambda at core.py:2123>),
Map(decode)}) exceeded the 80 characters length limit and was truncated.
2019-04-11 13:22:14,352 INFO org.apache.flink.runtime.taskmanager.Task
- CHAIN MapPartition (MapPartition at [1]read/Read/Split) ->
FlatMap (FlatMap at ExtractOutput[0]) (1/1) (c9bb2f2368faf10656e3ef9ce40e578f)
switched from CREATED to DEPLOYING.
2019-04-11 13:22:14,353 INFO org.apache.flink.runtime.taskmanager.Task
- Creating FileSystem stream leak safety net for task CHAIN
MapPartition (MapPartition at [1]read/Read/Split) -> FlatMap (FlatMap at
ExtractOutput[0]) (1/1) (c9bb2f2368faf10656e3ef9ce40e578f) [DEPLOYING]
2019-04-11 13:22:14,356 INFO org.apache.flink.runtime.taskmanager.Task
- Loading JAR files for task CHAIN MapPartition (MapPartition at
[1]read/Read/Split) -> FlatMap (FlatMap at ExtractOutput[0]) (1/1)
(c9bb2f2368faf10656e3ef9ce40e578f) [DEPLOYING].
2019-04-11 13:22:14,356 INFO org.apache.flink.runtime.taskmanager.Task
- Registering task at network: CHAIN MapPartition (MapPartition at
[1]read/Read/Split) -> FlatMap (FlatMap at ExtractOutput[0]) (1/1)
(c9bb2f2368faf10656e3ef9ce40e578f) [DEPLOYING].
2019-04-11 13:22:14,365 INFO org.apache.flink.runtime.taskmanager.Task
- CHAIN MapPartition (MapPartition at [1]read/Read/Split) ->
FlatMap (FlatMap at ExtractOutput[0]) (1/1) (c9bb2f2368faf10656e3ef9ce40e578f)
switched from DEPLOYING to RUNNING.
2019-04-11 13:22:15,156 INFO
org.apache.flink.configuration.GlobalConfiguration - Loading
configuration property: jobmanager.rpc.address, 172.26.0.4
2019-04-11 13:22:15,156 INFO
org.apache.flink.configuration.GlobalConfiguration - Loading
configuration property: jobmanager.rpc.port, 6123
2019-04-11 13:22:15,156 INFO
org.apache.flink.configuration.GlobalConfiguration - Loading
configuration property: jobmanager.heap.size, 1024m
2019-04-11 13:22:15,156 INFO
org.apache.flink.configuration.GlobalConfiguration - Loading
configuration property: taskmanager.heap.size, 1024m
2019-04-11 13:22:15,156 INFO
org.apache.flink.configuration.GlobalConfiguration - Loading
configuration property: taskmanager.numberOfTaskSlots, 4
2019-04-11 13:22:15,156 INFO
org.apache.flink.configuration.GlobalConfiguration - Loading
configuration property: parallelism.default, 2
2019-04-11 13:22:15,157 INFO
org.apache.flink.configuration.GlobalConfiguration - Loading
configuration property: rest.port, 8081
2019-04-11 13:22:16,544 INFO
org.apache.beam.runners.fnexecution.artifact.BeamFileSystemArtifactRetrievalService
- GetManifest for
/tmp/beam-artifact-staging/job_8ff51268-48fb-42e3-b14c-f2e387e26e5a/MANIFEST
2019-04-11 13:22:16,545 INFO
org.apache.beam.runners.fnexecution.artifact.BeamFileSystemArtifactRetrievalService
- Loading manifest for retrieval token
/tmp/beam-artifact-staging/job_8ff51268-48fb-42e3-b14c-f2e387e26e5a/MANIFEST
2019-04-11 13:22:16,552 INFO
org.apache.beam.runners.fnexecution.artifact.BeamFileSystemArtifactRetrievalService
- GetManifest for
/tmp/beam-artifact-staging/job_8ff51268-48fb-42e3-b14c-f2e387e26e5a/MANIFEST
failed
java.util.concurrent.ExecutionException: java.io.FileNotFoundException:
/tmp/beam-artifact-staging/job_8ff51268-48fb-42e3-b14c-f2e387e26e5a/MANIFEST
(No such file or directory)
at
org.apache.beam.vendor.guava.v20_0.com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:500)
at
org.apache.beam.vendor.guava.v20_0.com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:459)
at
org.apache.beam.vendor.guava.v20_0.com.google.common.util.concurrent.AbstractFuture$TrustedFuture.get(AbstractFuture.java:76)
at
org.apache.beam.vendor.guava.v20_0.com.google.common.util.concurrent.Uninterruptibles.getUninterruptibly(Uninterruptibles.java:142)
at
org.apache.beam.vendor.guava.v20_0.com.google.common.cache.LocalCache$Segment.getAndRecordStats(LocalCache.java:2373)
at
org.apache.beam.vendor.guava.v20_0.com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2337)
at
org.apache.beam.vendor.guava.v20_0.com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2295)
at
org.apache.beam.vendor.guava.v20_0.com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2208)
at
org.apache.beam.vendor.guava.v20_0.com.google.common.cache.LocalCache.get(LocalCache.java:4053)
at
org.apache.beam.vendor.guava.v20_0.com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:4057)
at
org.apache.beam.vendor.guava.v20_0.com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4986)
at
org.apache.beam.runners.fnexecution.artifact.BeamFileSystemArtifactRetrievalService.getManifest(BeamFileSystemArtifactRetrievalService.java:80)
at
org.apache.beam.model.jobmanagement.v1.ArtifactRetrievalServiceGrpc$MethodHandlers.invoke(ArtifactRetrievalServiceGrpc.java:298)
at
org.apache.beam.vendor.grpc.v1p13p1.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171)
at
org.apache.beam.vendor.grpc.v1p13p1.io.grpc.PartialForwardingServerCallListener.onHalfClose(PartialForwardingServerCallListener.java:35)
at
org.apache.beam.vendor.grpc.v1p13p1.io.grpc.ForwardingServerCallListener.onHalfClose(ForwardingServerCallListener.java:23)
at
org.apache.beam.vendor.grpc.v1p13p1.io.grpc.ForwardingServerCallListener$SimpleForwardingServerCallListener.onHalfClose(ForwardingServerCallListener.java:40)
at
org.apache.beam.vendor.grpc.v1p13p1.io.grpc.Contexts$ContextualizedServerCallListener.onHalfClose(Contexts.java:86)
at
org.apache.beam.vendor.grpc.v1p13p1.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283)
at
org.apache.beam.vendor.grpc.v1p13p1.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:707)
at
org.apache.beam.vendor.grpc.v1p13p1.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
at
org.apache.beam.vendor.grpc.v1p13p1.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.FileNotFoundException:
/tmp/beam-artifact-staging/job_8ff51268-48fb-42e3-b14c-f2e387e26e5a/MANIFEST
(No such file or directory)
at java.io.FileInputStream.open0(Native Method)
at java.io.FileInputStream.open(FileInputStream.java:195)
at java.io.FileInputStream.<init>(FileInputStream.java:138)
at org.apache.beam.sdk.io.LocalFileSystem.open(LocalFileSystem.java:114)
at org.apache.beam.sdk.io.LocalFileSystem.open(LocalFileSystem.java:81)
at org.apache.beam.sdk.io.FileSystems.open(FileSystems.java:251)
at
org.apache.beam.runners.fnexecution.artifact.BeamFileSystemArtifactRetrievalService.loadManifest(BeamFileSystemArtifactRetrievalService.java:185)
at
org.apache.beam.runners.fnexecution.artifact.BeamFileSystemArtifactRetrievalService.loadManifest(BeamFileSystemArtifactRetrievalService.java:180)
at
org.apache.beam.runners.fnexecution.artifact.BeamFileSystemArtifactRetrievalService$1.load(BeamFileSystemArtifactRetrievalService.java:171)
at
org.apache.beam.runners.fnexecution.artifact.BeamFileSystemArtifactRetrievalService$1.load(BeamFileSystemArtifactRetrievalService.java:168)
at
org.apache.beam.vendor.guava.v20_0.com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3628)
at
org.apache.beam.vendor.guava.v20_0.com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2336)
... 19 more
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
