[ 
https://issues.apache.org/jira/browse/FLINK-7523?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Till Rohrmann closed FLINK-7523.
--------------------------------
       Resolution: Fixed
    Fix Version/s: 1.4.0

Fixed via ff1660629fea73886bf9c9f802c80dd9bf84c83d

> test instability in LocalStreamEnvironmentITCase#testRunIsolatedJob
> -------------------------------------------------------------------
>
>                 Key: FLINK-7523
>                 URL: https://issues.apache.org/jira/browse/FLINK-7523
>             Project: Flink
>          Issue Type: Bug
>          Components: Distributed Coordination
>    Affects Versions: 1.4.0
>            Reporter: Nico Kruber
>            Assignee: Till Rohrmann
>            Priority: Critical
>              Labels: flip-6, test-stability
>             Fix For: 1.4.0
>
>
> In a branch with unrelated changes that was created based on the current 
> master, {{LocalStreamEnvironmentITCase#testRunIsolatedJob}} seems to fail 
> sporadically with
> {code}
> Running 
> org.apache.flink.streaming.api.environment.LocalStreamEnvironmentITCase
> Tests run: 2, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 2.662 sec <<< 
> FAILURE! - in 
> org.apache.flink.streaming.api.environment.LocalStreamEnvironmentITCase
> testRunIsolatedJob(org.apache.flink.streaming.api.environment.LocalStreamEnvironmentITCase)
>   Time elapsed: 1.797 sec  <<< ERROR!
> java.util.ConcurrentModificationException: null
>       at java.util.HashMap$HashIterator.nextNode(HashMap.java:1437)
>       at java.util.HashMap$ValueIterator.next(HashMap.java:1466)
>       at 
> org.apache.flink.runtime.resourcemanager.JobLeaderIdService.clear(JobLeaderIdService.java:120)
>       at 
> org.apache.flink.runtime.resourcemanager.JobLeaderIdService.stop(JobLeaderIdService.java:98)
>       at 
> org.apache.flink.runtime.resourcemanager.ResourceManagerRuntimeServices.shutDown(ResourceManagerRuntimeServices.java:51)
>       at 
> org.apache.flink.runtime.resourcemanager.ResourceManagerRunner.shutDownInternally(ResourceManagerRunner.java:107)
>       at 
> org.apache.flink.runtime.resourcemanager.ResourceManagerRunner.shutDown(ResourceManagerRunner.java:94)
>       at 
> org.apache.flink.runtime.minicluster.MiniCluster.shutdownInternally(MiniCluster.java:341)
>       at 
> org.apache.flink.runtime.minicluster.MiniCluster.shutdown(MiniCluster.java:309)
>       at 
> org.apache.flink.streaming.api.environment.Flip6LocalStreamEnvironment.execute(Flip6LocalStreamEnvironment.java:124)
>       at 
> org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.execute(StreamExecutionEnvironment.java:1501)
>       at 
> org.apache.flink.streaming.api.environment.LocalStreamEnvironmentITCase.testRunIsolatedJob(LocalStreamEnvironmentITCase.java:45)
> {code}
> https://travis-ci.org/NicoK/flink/jobs/268340083
> https://transfer.sh/10CvKr/449.1.tar.gz
> relevant parts of the log:
> {code}
> ================================================================================
> Test 
> testRunIsolatedJob(org.apache.flink.streaming.api.environment.LocalStreamEnvironmentITCase)
>  is running.
> --------------------------------------------------------------------------------
> 12:38:58,611 INFO  
> org.apache.flink.streaming.api.environment.Flip6LocalStreamEnvironment  - 
> Running job on local embedded Flink mini cluster
> 12:38:58,614 INFO  org.apache.flink.runtime.minicluster.MiniCluster           
>    - Starting Flink Mini Cluster
> 12:38:58,814 INFO  org.apache.flink.runtime.minicluster.MiniCluster           
>    - Starting Metrics Registry
> 12:38:58,822 INFO  org.apache.flink.runtime.metrics.MetricRegistry            
>    - No metrics reporter configured, no metrics will be exposed/reported.
> 12:38:58,823 INFO  org.apache.flink.runtime.minicluster.MiniCluster           
>    - Starting RPC Service(s)
> 12:38:59,191 INFO  akka.event.slf4j.Slf4jLogger                               
>    - Slf4jLogger started
> 12:38:59,211 INFO  org.apache.flink.runtime.minicluster.MiniCluster           
>    - Starting high-availability services
> 12:38:59,227 INFO  org.apache.flink.runtime.blob.BlobServer                   
>    - Created BLOB server storage directory 
> /tmp/blobStore-afe2e280-9b06-4f47-ad6f-a8aa60054470
> 12:38:59,235 INFO  org.apache.flink.runtime.blob.BlobServer                   
>    - Started BLOB server at 0.0.0.0:40558 - max concurrent requests: 50 - max 
> backlog: 1000
> 12:38:59,236 INFO  org.apache.flink.runtime.minicluster.MiniCluster           
>    - Starting 1 ResourceManger(s)
> 12:38:59,251 INFO  org.apache.flink.runtime.rpc.akka.AkkaRpcService           
>    - Starting RPC endpoint for 
> org.apache.flink.runtime.resourcemanager.StandaloneResourceManager at 
> akka://flink/user/resourcemanager_0 .
> 12:38:59,266 INFO  
> org.apache.flink.runtime.highavailability.nonha.embedded.EmbeddedLeaderService
>   - Proposing leadership to contender 
> org.apache.flink.runtime.resourcemanager.StandaloneResourceManager@32c4e8b2 @ 
> akka://flink/user/resourcemanager_0
> 12:38:59,268 INFO  
> org.apache.flink.runtime.resourcemanager.StandaloneResourceManager  - 
> ResourceManager akka://flink/user/resourcemanager_0 was granted leadership 
> with leader session ID c79740f8-fa00-4eb8-84d2-10138390ada8
> 12:38:59,268 INFO  
> org.apache.flink.runtime.resourcemanager.slotmanager.SlotManager  - Starting 
> the SlotManager.
> 12:38:59,269 INFO  
> org.apache.flink.runtime.highavailability.nonha.embedded.EmbeddedLeaderService
>   - Received confirmation of leadership for leader 
> akka://flink/user/resourcemanager_0 , 
> session=c79740f8-fa00-4eb8-84d2-10138390ada8
> 12:38:59,270 INFO  org.apache.flink.runtime.minicluster.MiniCluster           
>    - Starting 1 TaskManger(s)
> 12:38:59,278 INFO  org.apache.flink.runtime.taskexecutor.TaskManagerServices  
>    - Temporary file directory '/tmp': total 29 GB, usable 18 GB (62.07% 
> usable)
> 12:38:59,407 INFO  
> org.apache.flink.runtime.io.network.buffer.NetworkBufferPool  - Allocated 196 
> MB for network buffer pool (number of memory segments: 6295, bytes per 
> segment: 32768).
> 12:38:59,417 INFO  org.apache.flink.runtime.io.network.NetworkEnvironment     
>    - Starting the network environment and its components.
> 12:38:59,419 INFO  org.apache.flink.runtime.taskexecutor.TaskManagerServices  
>    - Limiting managed memory to 414 MB, memory will be allocated lazily.
> 12:38:59,422 INFO  org.apache.flink.runtime.io.disk.iomanager.IOManager       
>    - I/O manager uses directory 
> /tmp/flink-io-b4293dd8-c902-46f3-b6f8-0f9730cf84f5 for spill files.
> 12:38:59,480 INFO  org.apache.flink.runtime.metrics.MetricRegistry            
>    - No metrics reporter configured, no metrics will be exposed/reported.
> 12:38:59,531 INFO  org.apache.flink.runtime.filecache.FileCache               
>    - User file cache uses directory 
> /tmp/flink-dist-cache-541aefeb-81f2-4c89-9530-1d83f18f67ba
> 12:38:59,539 INFO  
> org.apache.flink.runtime.taskexecutor.TaskManagerConfiguration  - Messages 
> have a max timeout of 10000 ms
> 12:38:59,545 INFO  org.apache.flink.runtime.rpc.akka.AkkaRpcService           
>    - Starting RPC endpoint for 
> org.apache.flink.runtime.taskexecutor.TaskExecutor at 
> akka://flink/user/taskmanager_0 .
> 12:38:59,551 INFO  org.apache.flink.runtime.taskexecutor.TaskExecutor         
>    - Attempting to register at ResourceManager 
> akka://flink/user/resourcemanager_0
> 12:38:59,551 INFO  org.apache.flink.runtime.taskexecutor.JobLeaderService     
>    - Start job leader service.
> 12:38:59,551 INFO  org.apache.flink.runtime.minicluster.MiniCluster           
>    - Starting job dispatcher(s) for 1 JobManger(s)
> 12:38:59,552 INFO  
> org.apache.flink.runtime.minicluster.MiniClusterJobDispatcher  - Creating 
> JobMaster services
> 12:38:59,556 INFO  org.apache.flink.runtime.minicluster.MiniCluster           
>    - Flink Mini Cluster started successfully
> 12:38:59,574 INFO  org.apache.flink.runtime.taskexecutor.TaskExecutor         
>    - Resolved ResourceManager address, beginning registration
> 12:38:59,575 INFO  org.apache.flink.runtime.taskexecutor.TaskExecutor         
>    - Registration at ResourceManager attempt 1 (timeout=100ms)
> 12:38:59,591 INFO  
> org.apache.flink.runtime.resourcemanager.slotmanager.SlotManager  - Register 
> TaskManager 839f9983d3f82bee54580acea5a5d9dc at the SlotManager.
> 12:38:59,593 INFO  org.apache.flink.runtime.taskexecutor.TaskExecutor         
>    - Successful registration at resource manager 
> akka://flink/user/resourcemanager_0 under registration id 
> 839f9983d3f82bee54580acea5a5d9dc.
> 12:38:59,594 INFO  
> org.apache.flink.runtime.minicluster.MiniClusterJobDispatcher  - Received job 
> for blocking execution: Flink Streaming Job (4edcbd9b42d9cca157794e9e2cdc29d3)
> 12:38:59,594 INFO  
> org.apache.flink.runtime.minicluster.MiniClusterJobDispatcher  - Starting 1 
> JobMaster(s) for job Flink Streaming Job (4edcbd9b42d9cca157794e9e2cdc29d3)
> 12:38:59,603 INFO  org.apache.flink.runtime.rpc.akka.AkkaRpcService           
>    - Starting RPC endpoint for org.apache.flink.runtime.jobmaster.JobMaster 
> at akka://flink/user/jobmanager_1 .
> 12:38:59,616 INFO  org.apache.flink.runtime.jobmaster.JobMaster               
>    - Initializing job Flink Streaming Job (4edcbd9b42d9cca157794e9e2cdc29d3).
> 12:38:59,625 INFO  org.apache.flink.runtime.jobmaster.JobMaster               
>    - Using restart strategy 
> FixedDelayRestartStrategy(maxNumberRestartAttempts=2147483647, 
> delayBetweenRestartAttempts=5) for Flink Streaming Job 
> (4edcbd9b42d9cca157794e9e2cdc29d3).
> 12:38:59,629 INFO  org.apache.flink.runtime.rpc.akka.AkkaRpcService           
>    - Starting RPC endpoint for org.apache.flink.runtime.instance.SlotPool at 
> akka://flink/user/11533ce9-593d-4fd9-ac88-6999a1490795 .
> 12:38:59,659 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph     
>    - Job recovers via failover strategy: full graph restart
> 12:38:59,704 INFO  org.apache.flink.runtime.jobmaster.JobMaster               
>    - Running initialization on master for job Flink Streaming Job 
> (4edcbd9b42d9cca157794e9e2cdc29d3).
> 12:38:59,704 INFO  org.apache.flink.runtime.jobmaster.JobMaster               
>    - Successfully ran initialization on master in 0 ms.
> 12:38:59,722 INFO  org.apache.flink.runtime.jobmaster.JobMaster               
>    - No state backend has been configured, using default state backend 
> (Memory / JobManager)
> 12:38:59,730 INFO  
> org.apache.flink.runtime.highavailability.nonha.embedded.EmbeddedLeaderService
>   - Proposing leadership to contender 
> org.apache.flink.runtime.jobmaster.JobManagerRunner@4bc28c33 @ 
> akka://flink/user/jobmanager_1
> 12:38:59,730 INFO  org.apache.flink.runtime.jobmaster.JobManagerRunner        
>    - JobManager runner for job Flink Streaming Job 
> (4edcbd9b42d9cca157794e9e2cdc29d3) was granted leadership with session id 
> 1c41035e-ef27-436e-bd0a-434042290346 at akka://flink/user/jobmanager_1.
> 12:38:59,730 INFO  
> org.apache.flink.runtime.highavailability.nonha.embedded.EmbeddedLeaderService
>   - Received confirmation of leadership for leader 
> akka://flink/user/jobmanager_1 , session=1c41035e-ef27-436e-bd0a-434042290346
> 12:38:59,731 INFO  org.apache.flink.runtime.jobmaster.JobMaster               
>    - JobManager started as leader 1c41035e-ef27-436e-bd0a-434042290346 for 
> job 4edcbd9b42d9cca157794e9e2cdc29d3
> 12:38:59,731 INFO  org.apache.flink.runtime.jobmaster.JobMaster               
>    - Starting execution of job Flink Streaming Job 
> (4edcbd9b42d9cca157794e9e2cdc29d3)
> 12:38:59,733 INFO  org.apache.flink.runtime.jobmaster.JobMaster               
>    - Attempting to register at ResourceManager 
> akka://flink/user/resourcemanager_0
> 12:38:59,734 INFO  org.apache.flink.runtime.jobmaster.JobMaster               
>    - Resolved ResourceManager address, beginning registration
> 12:38:59,734 INFO  org.apache.flink.runtime.jobmaster.JobMaster               
>    - Registration at ResourceManager attempt 1 (timeout=100ms)
> 12:38:59,736 INFO  
> org.apache.flink.runtime.resourcemanager.StandaloneResourceManager  - 
> Registering job manager 
> 1c41035e-ef27-436e-bd0a-434042290346@akka://flink/user/jobmanager_1 for job 
> 4edcbd9b42d9cca157794e9e2cdc29d3.
> 12:38:59,739 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph     
>    - Job Flink Streaming Job (4edcbd9b42d9cca157794e9e2cdc29d3) switched from 
> state CREATED to RUNNING.
> 12:38:59,740 INFO  org.apache.flink.runtime.jobmaster.JobMaster               
>    - Status of job 4edcbd9b42d9cca157794e9e2cdc29d3 (Flink Streaming Job) 
> changed to RUNNING.
> 12:38:59,741 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph     
>    - Source: Sequence Source (1/3) (fb89023001ac7ae37953fbac1a00652c) 
> switched from CREATED to SCHEDULED.
> 12:38:59,741 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph     
>    - Source: Sequence Source (2/3) (987b738a072a184f2075b127f21df794) 
> switched from CREATED to SCHEDULED.
> 12:38:59,741 INFO  org.apache.flink.runtime.instance.SlotPool                 
>    - Cannot serve slot request, no ResourceManager connected. Adding as 
> pending request 36ff442eabcce42e2da5f1425560fb76
> 12:38:59,742 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph     
>    - Source: Sequence Source (3/3) (e36e2811c6f205facefd1195a2a3d183) 
> switched from CREATED to SCHEDULED.
> 12:38:59,742 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph     
>    - Filter -> Sink: Unnamed (1/3) (6ac1d973b37ddf0809005158a83892a4) 
> switched from CREATED to SCHEDULED.
> 12:38:59,742 INFO  org.apache.flink.runtime.instance.SlotPool                 
>    - Cannot serve slot request, no ResourceManager connected. Adding as 
> pending request 4077791c30fcc3ec7783f40d00613714
> 12:38:59,742 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph     
>    - Filter -> Sink: Unnamed (2/3) (f1a2286709bf333f4dd8f24b4813a964) 
> switched from CREATED to SCHEDULED.
> 12:38:59,742 INFO  org.apache.flink.runtime.instance.SlotPool                 
>    - Cannot serve slot request, no ResourceManager connected. Adding as 
> pending request 301c6e92cb81140887b965994fac704e
> 12:38:59,742 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph     
>    - Filter -> Sink: Unnamed (3/3) (6735fc301957d96c817e09b14c35ac27) 
> switched from CREATED to SCHEDULED.
> 12:38:59,743 INFO  org.apache.flink.runtime.instance.SlotPool                 
>    - Cannot serve slot request, no ResourceManager connected. Adding as 
> pending request 89ba38d0df4387549f1e874d3334ad97
> 12:38:59,743 INFO  org.apache.flink.runtime.instance.SlotPool                 
>    - Cannot serve slot request, no ResourceManager connected. Adding as 
> pending request 5d472abf5d10a7c4ee69e011b6959353
> 12:38:59,743 INFO  org.apache.flink.runtime.instance.SlotPool                 
>    - Cannot serve slot request, no ResourceManager connected. Adding as 
> pending request 7173302fdb6a5d654e02ea35943345f3
> 12:38:59,744 INFO  
> org.apache.flink.runtime.resourcemanager.StandaloneResourceManager  - 
> Registered job manager 
> 1c41035e-ef27-436e-bd0a-434042290346@akka://flink/user/jobmanager_1 for job 
> 4edcbd9b42d9cca157794e9e2cdc29d3.
> 12:38:59,746 INFO  org.apache.flink.runtime.jobmaster.JobMaster               
>    - JobManager successfully registered at ResourceManager, leader id: 
> c79740f8-fa00-4eb8-84d2-10138390ada8.
> 12:38:59,746 INFO  org.apache.flink.runtime.instance.SlotPool                 
>    - Requesting slot with profile ResourceProfile{cpuCores=-1.0, 
> heapMemoryInMB=-1, directMemoryInMB=0, nativeMemoryInMB=0} from resource 
> manager (request = 4077791c30fcc3ec7783f40d00613714).
> 12:38:59,746 INFO  
> org.apache.flink.runtime.resourcemanager.StandaloneResourceManager  - Request 
> slot with profile ResourceProfile{cpuCores=-1.0, heapMemoryInMB=-1, 
> directMemoryInMB=0, nativeMemoryInMB=0} for job 
> 4edcbd9b42d9cca157794e9e2cdc29d3 with allocation id 
> 4077791c30fcc3ec7783f40d00613714.
> 12:38:59,747 INFO  org.apache.flink.runtime.taskexecutor.TaskExecutor         
>    - Receive slot request 4077791c30fcc3ec7783f40d00613714 for job 
> 4edcbd9b42d9cca157794e9e2cdc29d3 from resource manager with leader id 
> c79740f8-fa00-4eb8-84d2-10138390ada8.
> 12:38:59,748 INFO  org.apache.flink.runtime.instance.SlotPool                 
>    - Requesting slot with profile ResourceProfile{cpuCores=-1.0, 
> heapMemoryInMB=-1, directMemoryInMB=0, nativeMemoryInMB=0} from resource 
> manager (request = 36ff442eabcce42e2da5f1425560fb76).
> 12:38:59,748 INFO  org.apache.flink.runtime.instance.SlotPool                 
>    - Requesting slot with profile ResourceProfile{cpuCores=-1.0, 
> heapMemoryInMB=-1, directMemoryInMB=0, nativeMemoryInMB=0} from resource 
> manager (request = 5d472abf5d10a7c4ee69e011b6959353).
> 12:38:59,749 INFO  org.apache.flink.runtime.instance.SlotPool                 
>    - Requesting slot with profile ResourceProfile{cpuCores=-1.0, 
> heapMemoryInMB=-1, directMemoryInMB=0, nativeMemoryInMB=0} from resource 
> manager (request = 7173302fdb6a5d654e02ea35943345f3).
> 12:38:59,749 INFO  org.apache.flink.runtime.instance.SlotPool                 
>    - Requesting slot with profile ResourceProfile{cpuCores=-1.0, 
> heapMemoryInMB=-1, directMemoryInMB=0, nativeMemoryInMB=0} from resource 
> manager (request = 301c6e92cb81140887b965994fac704e).
> 12:38:59,749 INFO  org.apache.flink.runtime.instance.SlotPool                 
>    - Requesting slot with profile ResourceProfile{cpuCores=-1.0, 
> heapMemoryInMB=-1, directMemoryInMB=0, nativeMemoryInMB=0} from resource 
> manager (request = 89ba38d0df4387549f1e874d3334ad97).
> 12:38:59,750 INFO  
> org.apache.flink.runtime.resourcemanager.StandaloneResourceManager  - Request 
> slot with profile ResourceProfile{cpuCores=-1.0, heapMemoryInMB=-1, 
> directMemoryInMB=0, nativeMemoryInMB=0} for job 
> 4edcbd9b42d9cca157794e9e2cdc29d3 with allocation id 
> 36ff442eabcce42e2da5f1425560fb76.
> 12:38:59,751 INFO  
> org.apache.flink.runtime.resourcemanager.StandaloneResourceManager  - Request 
> slot with profile ResourceProfile{cpuCores=-1.0, heapMemoryInMB=-1, 
> directMemoryInMB=0, nativeMemoryInMB=0} for job 
> 4edcbd9b42d9cca157794e9e2cdc29d3 with allocation id 
> 5d472abf5d10a7c4ee69e011b6959353.
> 12:38:59,752 INFO  org.apache.flink.runtime.taskexecutor.TaskExecutor         
>    - Allocated slot for 4077791c30fcc3ec7783f40d00613714.
> 12:38:59,752 INFO  org.apache.flink.runtime.taskexecutor.JobLeaderService     
>    - Add job 4edcbd9b42d9cca157794e9e2cdc29d3 for job leader monitoring.
> 12:38:59,753 INFO  org.apache.flink.runtime.taskexecutor.TaskExecutor         
>    - Receive slot request 36ff442eabcce42e2da5f1425560fb76 for job 
> 4edcbd9b42d9cca157794e9e2cdc29d3 from resource manager with leader id 
> c79740f8-fa00-4eb8-84d2-10138390ada8.
> 12:38:59,753 INFO  org.apache.flink.runtime.taskexecutor.TaskExecutor         
>    - Allocated slot for 36ff442eabcce42e2da5f1425560fb76.
> 12:38:59,753 INFO  org.apache.flink.runtime.taskexecutor.JobLeaderService     
>    - Add job 4edcbd9b42d9cca157794e9e2cdc29d3 for job leader monitoring.
> 12:38:59,754 INFO  org.apache.flink.runtime.taskexecutor.TaskExecutor         
>    - Receive slot request 5d472abf5d10a7c4ee69e011b6959353 for job 
> 4edcbd9b42d9cca157794e9e2cdc29d3 from resource manager with leader id 
> c79740f8-fa00-4eb8-84d2-10138390ada8.
> 12:38:59,754 INFO  org.apache.flink.runtime.taskexecutor.JobLeaderService     
>    - Try to register at job manager akka://flink/user/jobmanager_1 with 
> leader id 1c41035e-ef27-436e-bd0a-434042290346.
> 12:38:59,754 INFO  org.apache.flink.runtime.taskexecutor.TaskExecutor         
>    - Allocated slot for 5d472abf5d10a7c4ee69e011b6959353.
> 12:38:59,754 INFO  org.apache.flink.runtime.taskexecutor.JobLeaderService     
>    - Add job 4edcbd9b42d9cca157794e9e2cdc29d3 for job leader monitoring.
> 12:38:59,754 INFO  org.apache.flink.runtime.taskexecutor.JobLeaderService     
>    - Try to register at job manager akka://flink/user/jobmanager_1 with 
> leader id 1c41035e-ef27-436e-bd0a-434042290346.
> 12:38:59,754 INFO  org.apache.flink.runtime.taskexecutor.JobLeaderService     
>    - Try to register at job manager akka://flink/user/jobmanager_1 with 
> leader id 1c41035e-ef27-436e-bd0a-434042290346.
> 12:38:59,755 INFO  org.apache.flink.runtime.taskexecutor.JobLeaderService     
>    - Resolved JobManager address, beginning registration
> 12:38:59,755 INFO  org.apache.flink.runtime.taskexecutor.JobLeaderService     
>    - Resolved JobManager address, beginning registration
> 12:38:59,755 INFO  org.apache.flink.runtime.taskexecutor.JobLeaderService     
>    - Registration at JobManager attempt 1 (timeout=100ms)
> 12:38:59,755 INFO  
> org.apache.flink.runtime.resourcemanager.StandaloneResourceManager  - Request 
> slot with profile ResourceProfile{cpuCores=-1.0, heapMemoryInMB=-1, 
> directMemoryInMB=0, nativeMemoryInMB=0} for job 
> 4edcbd9b42d9cca157794e9e2cdc29d3 with allocation id 
> 7173302fdb6a5d654e02ea35943345f3.
> 12:38:59,755 INFO  
> org.apache.flink.runtime.resourcemanager.StandaloneResourceManager  - Request 
> slot with profile ResourceProfile{cpuCores=-1.0, heapMemoryInMB=-1, 
> directMemoryInMB=0, nativeMemoryInMB=0} for job 
> 4edcbd9b42d9cca157794e9e2cdc29d3 with allocation id 
> 301c6e92cb81140887b965994fac704e.
> 12:38:59,755 INFO  
> org.apache.flink.runtime.resourcemanager.StandaloneResourceManager  - Request 
> slot with profile ResourceProfile{cpuCores=-1.0, heapMemoryInMB=-1, 
> directMemoryInMB=0, nativeMemoryInMB=0} for job 
> 4edcbd9b42d9cca157794e9e2cdc29d3 with allocation id 
> 89ba38d0df4387549f1e874d3334ad97.
> 12:38:59,756 INFO  org.apache.flink.runtime.taskexecutor.TaskExecutor         
>    - Receive slot request 7173302fdb6a5d654e02ea35943345f3 for job 
> 4edcbd9b42d9cca157794e9e2cdc29d3 from resource manager with leader id 
> c79740f8-fa00-4eb8-84d2-10138390ada8.
> 12:38:59,756 INFO  org.apache.flink.runtime.taskexecutor.TaskExecutor         
>    - Allocated slot for 7173302fdb6a5d654e02ea35943345f3.
> 12:38:59,756 INFO  org.apache.flink.runtime.taskexecutor.JobLeaderService     
>    - Add job 4edcbd9b42d9cca157794e9e2cdc29d3 for job leader monitoring.
> 12:38:59,756 INFO  org.apache.flink.runtime.taskexecutor.TaskExecutor         
>    - Receive slot request 301c6e92cb81140887b965994fac704e for job 
> 4edcbd9b42d9cca157794e9e2cdc29d3 from resource manager with leader id 
> c79740f8-fa00-4eb8-84d2-10138390ada8.
> 12:38:59,756 INFO  org.apache.flink.runtime.taskexecutor.TaskExecutor         
>    - Allocated slot for 301c6e92cb81140887b965994fac704e.
> 12:38:59,756 INFO  org.apache.flink.runtime.taskexecutor.JobLeaderService     
>    - Add job 4edcbd9b42d9cca157794e9e2cdc29d3 for job leader monitoring.
> 12:38:59,757 INFO  org.apache.flink.runtime.taskexecutor.TaskExecutor         
>    - Receive slot request 89ba38d0df4387549f1e874d3334ad97 for job 
> 4edcbd9b42d9cca157794e9e2cdc29d3 from resource manager with leader id 
> c79740f8-fa00-4eb8-84d2-10138390ada8.
> 12:38:59,757 INFO  org.apache.flink.runtime.taskexecutor.TaskExecutor         
>    - Allocated slot for 89ba38d0df4387549f1e874d3334ad97.
> 12:38:59,757 INFO  org.apache.flink.runtime.taskexecutor.JobLeaderService     
>    - Try to register at job manager akka://flink/user/jobmanager_1 with 
> leader id 1c41035e-ef27-436e-bd0a-434042290346.
> 12:38:59,757 INFO  org.apache.flink.runtime.taskexecutor.JobLeaderService     
>    - Try to register at job manager akka://flink/user/jobmanager_1 with 
> leader id 1c41035e-ef27-436e-bd0a-434042290346.
> 12:38:59,758 INFO  org.apache.flink.runtime.taskexecutor.JobLeaderService     
>    - Resolved JobManager address, beginning registration
> 12:38:59,758 INFO  org.apache.flink.runtime.taskexecutor.JobLeaderService     
>    - Registration at JobManager attempt 1 (timeout=100ms)
> 12:38:59,758 INFO  org.apache.flink.runtime.taskexecutor.JobLeaderService     
>    - Resolved JobManager address, beginning registration
> 12:38:59,758 INFO  org.apache.flink.runtime.taskexecutor.JobLeaderService     
>    - Registration at JobManager attempt 1 (timeout=100ms)
> 12:38:59,758 INFO  org.apache.flink.runtime.taskexecutor.JobLeaderService     
>    - Successful registration at job manager akka://flink/user/jobmanager_1 
> for job 4edcbd9b42d9cca157794e9e2cdc29d3.
> 12:38:59,758 INFO  org.apache.flink.runtime.taskexecutor.JobLeaderService     
>    - Successful registration at job manager akka://flink/user/jobmanager_1 
> for job 4edcbd9b42d9cca157794e9e2cdc29d3.
> 12:38:59,759 INFO  org.apache.flink.runtime.taskexecutor.JobLeaderService     
>    - Successful registration at job manager akka://flink/user/jobmanager_1 
> for job 4edcbd9b42d9cca157794e9e2cdc29d3.
> 12:38:59,755 INFO  org.apache.flink.runtime.taskexecutor.JobLeaderService     
>    - Resolved JobManager address, beginning registration
> 12:38:59,759 INFO  org.apache.flink.runtime.taskexecutor.JobLeaderService     
>    - Registration at JobManager attempt 1 (timeout=100ms)
> 12:38:59,759 INFO  org.apache.flink.runtime.taskexecutor.JobLeaderService     
>    - Successful registration at job manager akka://flink/user/jobmanager_1 
> for job 4edcbd9b42d9cca157794e9e2cdc29d3.
> 12:38:59,757 INFO  org.apache.flink.runtime.taskexecutor.JobLeaderService     
>    - Add job 4edcbd9b42d9cca157794e9e2cdc29d3 for job leader monitoring.
> 12:38:59,755 INFO  org.apache.flink.runtime.taskexecutor.JobLeaderService     
>    - Registration at JobManager attempt 1 (timeout=100ms)
> 12:38:59,762 INFO  org.apache.flink.runtime.taskexecutor.JobLeaderService     
>    - Try to register at job manager akka://flink/user/jobmanager_1 with 
> leader id 1c41035e-ef27-436e-bd0a-434042290346.
> 12:38:59,763 INFO  org.apache.flink.runtime.taskexecutor.JobLeaderService     
>    - Resolved JobManager address, beginning registration
> 12:38:59,763 INFO  org.apache.flink.runtime.taskexecutor.JobLeaderService     
>    - Registration at JobManager attempt 1 (timeout=100ms)
> 12:38:59,763 INFO  org.apache.flink.runtime.taskexecutor.JobLeaderService     
>    - Successful registration at job manager akka://flink/user/jobmanager_1 
> for job 4edcbd9b42d9cca157794e9e2cdc29d3.
> 12:38:59,764 INFO  org.apache.flink.runtime.taskexecutor.JobLeaderService     
>    - Successful registration at job manager akka://flink/user/jobmanager_1 
> for job 4edcbd9b42d9cca157794e9e2cdc29d3.
> 12:38:59,762 INFO  org.apache.flink.runtime.taskexecutor.TaskExecutor         
>    - Establish JobManager connection for job 4edcbd9b42d9cca157794e9e2cdc29d3.
> 12:38:59,766 INFO  org.apache.flink.runtime.blob.BlobCache                    
>    - Created BLOB cache storage directory 
> /tmp/blobStore-e1aca126-a878-48d8-928f-24a8cdec3821
> 12:38:59,770 INFO  org.apache.flink.runtime.taskexecutor.TaskExecutor         
>    - Offer reserved slots to the leader of job 
> 4edcbd9b42d9cca157794e9e2cdc29d3.
> 12:38:59,771 INFO  org.apache.flink.runtime.taskexecutor.slot.TaskSlotTable   
>    - Activate slot 4077791c30fcc3ec7783f40d00613714.
> 12:38:59,771 INFO  org.apache.flink.runtime.taskexecutor.slot.TaskSlotTable   
>    - Activate slot 36ff442eabcce42e2da5f1425560fb76.
> 12:38:59,771 INFO  org.apache.flink.runtime.taskexecutor.slot.TaskSlotTable   
>    - Activate slot 5d472abf5d10a7c4ee69e011b6959353.
> 12:38:59,771 INFO  org.apache.flink.runtime.taskexecutor.slot.TaskSlotTable   
>    - Activate slot 7173302fdb6a5d654e02ea35943345f3.
> 12:38:59,771 INFO  org.apache.flink.runtime.taskexecutor.slot.TaskSlotTable   
>    - Activate slot 301c6e92cb81140887b965994fac704e.
> 12:38:59,771 INFO  org.apache.flink.runtime.taskexecutor.slot.TaskSlotTable   
>    - Activate slot 89ba38d0df4387549f1e874d3334ad97.
> 12:38:59,784 INFO  org.apache.flink.runtime.taskexecutor.TaskExecutor         
>    - Establish JobManager connection for job 4edcbd9b42d9cca157794e9e2cdc29d3.
> 12:38:59,784 INFO  org.apache.flink.runtime.blob.BlobCache                    
>    - Created BLOB cache storage directory 
> /tmp/blobStore-d89c78c4-1314-49e6-bb8c-f3687e219c3d
> 12:38:59,786 INFO  org.apache.flink.runtime.taskexecutor.TaskExecutor         
>    - The target with resource ID 226145e3fb640091eca783ef73dc1333 is already 
> been monitored.
> 12:38:59,786 INFO  org.apache.flink.runtime.taskexecutor.TaskExecutor         
>    - Establish JobManager connection for job 4edcbd9b42d9cca157794e9e2cdc29d3.
> 12:38:59,786 INFO  org.apache.flink.runtime.blob.BlobCache                    
>    - Created BLOB cache storage directory 
> /tmp/blobStore-468c0558-8977-4261-82b8-f07292f8415a
> 12:38:59,787 INFO  org.apache.flink.runtime.taskexecutor.TaskExecutor         
>    - The target with resource ID 226145e3fb640091eca783ef73dc1333 is already 
> been monitored.
> 12:38:59,787 INFO  org.apache.flink.runtime.taskexecutor.TaskExecutor         
>    - Establish JobManager connection for job 4edcbd9b42d9cca157794e9e2cdc29d3.
> 12:38:59,787 INFO  org.apache.flink.runtime.blob.BlobCache                    
>    - Created BLOB cache storage directory 
> /tmp/blobStore-d7b3c569-0b58-4601-af32-bbb889d863b8
> 12:38:59,788 INFO  org.apache.flink.runtime.taskexecutor.TaskExecutor         
>    - The target with resource ID 226145e3fb640091eca783ef73dc1333 is already 
> been monitored.
> 12:38:59,788 INFO  org.apache.flink.runtime.taskexecutor.TaskExecutor         
>    - Establish JobManager connection for job 4edcbd9b42d9cca157794e9e2cdc29d3.
> 12:38:59,789 INFO  org.apache.flink.runtime.blob.BlobCache                    
>    - Created BLOB cache storage directory 
> /tmp/blobStore-d3f356e2-c965-4400-9677-37dbb2f23971
> 12:38:59,789 INFO  org.apache.flink.runtime.taskexecutor.TaskExecutor         
>    - The target with resource ID 226145e3fb640091eca783ef73dc1333 is already 
> been monitored.
> 12:38:59,789 INFO  org.apache.flink.runtime.taskexecutor.TaskExecutor         
>    - Establish JobManager connection for job 4edcbd9b42d9cca157794e9e2cdc29d3.
> 12:38:59,789 INFO  org.apache.flink.runtime.blob.BlobCache                    
>    - Created BLOB cache storage directory 
> /tmp/blobStore-d000c3bb-0423-46b7-94ff-4533fb496aaf
> 12:38:59,790 INFO  org.apache.flink.runtime.taskexecutor.TaskExecutor         
>    - The target with resource ID 226145e3fb640091eca783ef73dc1333 is already 
> been monitored.
> 12:38:59,798 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph     
>    - Source: Sequence Source (1/3) (fb89023001ac7ae37953fbac1a00652c) 
> switched from SCHEDULED to DEPLOYING.
> 12:38:59,798 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph     
>    - Deploying Source: Sequence Source (1/3) (attempt #0) to localhost
> 12:38:59,807 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph     
>    - Source: Sequence Source (2/3) (987b738a072a184f2075b127f21df794) 
> switched from SCHEDULED to DEPLOYING.
> 12:38:59,807 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph     
>    - Deploying Source: Sequence Source (2/3) (attempt #0) to localhost
> 12:38:59,807 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph     
>    - Source: Sequence Source (3/3) (e36e2811c6f205facefd1195a2a3d183) 
> switched from SCHEDULED to DEPLOYING.
> 12:38:59,808 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph     
>    - Deploying Source: Sequence Source (3/3) (attempt #0) to localhost
> 12:38:59,808 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph     
>    - Filter -> Sink: Unnamed (1/3) (6ac1d973b37ddf0809005158a83892a4) 
> switched from SCHEDULED to DEPLOYING.
> 12:38:59,808 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph     
>    - Deploying Filter -> Sink: Unnamed (1/3) (attempt #0) to localhost
> 12:38:59,810 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph     
>    - Filter -> Sink: Unnamed (2/3) (f1a2286709bf333f4dd8f24b4813a964) 
> switched from SCHEDULED to DEPLOYING.
> 12:38:59,810 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph     
>    - Deploying Filter -> Sink: Unnamed (2/3) (attempt #0) to localhost
> 12:38:59,811 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph     
>    - Filter -> Sink: Unnamed (3/3) (6735fc301957d96c817e09b14c35ac27) 
> switched from SCHEDULED to DEPLOYING.
> 12:38:59,811 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph     
>    - Deploying Filter -> Sink: Unnamed (3/3) (attempt #0) to localhost
> 12:38:59,835 INFO  org.apache.flink.runtime.taskexecutor.TaskExecutor         
>    - Received task Source: Sequence Source (1/3).
> 12:38:59,838 INFO  org.apache.flink.runtime.taskexecutor.TaskExecutor         
>    - Received task Source: Sequence Source (2/3).
> 12:38:59,841 INFO  org.apache.flink.runtime.taskmanager.Task                  
>    - Source: Sequence Source (1/3) (fb89023001ac7ae37953fbac1a00652c) 
> switched from CREATED to DEPLOYING.
> 12:38:59,842 INFO  org.apache.flink.runtime.taskmanager.Task                  
>    - Creating FileSystem stream leak safety net for task Source: Sequence 
> Source (1/3) (fb89023001ac7ae37953fbac1a00652c) [DEPLOYING]
> 12:38:59,847 INFO  org.apache.flink.runtime.taskexecutor.TaskExecutor         
>    - Received task Source: Sequence Source (3/3).
> 12:38:59,847 INFO  org.apache.flink.runtime.taskmanager.Task                  
>    - Source: Sequence Source (2/3) (987b738a072a184f2075b127f21df794) 
> switched from CREATED to DEPLOYING.
> 12:38:59,848 INFO  org.apache.flink.runtime.taskmanager.Task                  
>    - Creating FileSystem stream leak safety net for task Source: Sequence 
> Source (2/3) (987b738a072a184f2075b127f21df794) [DEPLOYING]
> 12:38:59,862 INFO  org.apache.flink.runtime.taskmanager.Task                  
>    - Loading JAR files for task Source: Sequence Source (1/3) 
> (fb89023001ac7ae37953fbac1a00652c) [DEPLOYING].
> 12:38:59,866 INFO  org.apache.flink.runtime.taskmanager.Task                  
>    - Registering task at network: Source: Sequence Source (1/3) 
> (fb89023001ac7ae37953fbac1a00652c) [DEPLOYING].
> 12:38:59,867 INFO  org.apache.flink.runtime.taskmanager.Task                  
>    - Loading JAR files for task Source: Sequence Source (2/3) 
> (987b738a072a184f2075b127f21df794) [DEPLOYING].
> 12:38:59,870 INFO  org.apache.flink.runtime.taskmanager.Task                  
>    - Registering task at network: Source: Sequence Source (2/3) 
> (987b738a072a184f2075b127f21df794) [DEPLOYING].
> 12:38:59,875 INFO  org.apache.flink.runtime.taskmanager.Task                  
>    - Source: Sequence Source (3/3) (e36e2811c6f205facefd1195a2a3d183) 
> switched from CREATED to DEPLOYING.
> 12:38:59,875 INFO  org.apache.flink.runtime.taskmanager.Task                  
>    - Creating FileSystem stream leak safety net for task Source: Sequence 
> Source (3/3) (e36e2811c6f205facefd1195a2a3d183) [DEPLOYING]
> 12:38:59,875 INFO  org.apache.flink.runtime.taskmanager.Task                  
>    - Loading JAR files for task Source: Sequence Source (3/3) 
> (e36e2811c6f205facefd1195a2a3d183) [DEPLOYING].
> 12:38:59,876 INFO  org.apache.flink.runtime.taskmanager.Task                  
>    - Registering task at network: Source: Sequence Source (3/3) 
> (e36e2811c6f205facefd1195a2a3d183) [DEPLOYING].
> 12:38:59,876 INFO  org.apache.flink.runtime.taskexecutor.TaskExecutor         
>    - Received task Filter -> Sink: Unnamed (1/3).
> 12:38:59,879 INFO  org.apache.flink.runtime.taskmanager.Task                  
>    - Filter -> Sink: Unnamed (1/3) (6ac1d973b37ddf0809005158a83892a4) 
> switched from CREATED to DEPLOYING.
> 12:38:59,879 INFO  org.apache.flink.runtime.taskmanager.Task                  
>    - Creating FileSystem stream leak safety net for task Filter -> Sink: 
> Unnamed (1/3) (6ac1d973b37ddf0809005158a83892a4) [DEPLOYING]
> 12:38:59,880 INFO  org.apache.flink.runtime.taskmanager.Task                  
>    - Loading JAR files for task Filter -> Sink: Unnamed (1/3) 
> (6ac1d973b37ddf0809005158a83892a4) [DEPLOYING].
> 12:38:59,881 INFO  org.apache.flink.runtime.taskexecutor.TaskExecutor         
>    - Received task Filter -> Sink: Unnamed (2/3).
> 12:38:59,881 INFO  org.apache.flink.runtime.taskmanager.Task                  
>    - Registering task at network: Filter -> Sink: Unnamed (1/3) 
> (6ac1d973b37ddf0809005158a83892a4) [DEPLOYING].
> 12:38:59,886 INFO  org.apache.flink.runtime.taskmanager.Task                  
>    - Source: Sequence Source (3/3) (e36e2811c6f205facefd1195a2a3d183) 
> switched from DEPLOYING to RUNNING.
> 12:38:59,887 INFO  org.apache.flink.runtime.taskmanager.Task                  
>    - Source: Sequence Source (1/3) (fb89023001ac7ae37953fbac1a00652c) 
> switched from DEPLOYING to RUNNING.
> 12:38:59,887 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph     
>    - Source: Sequence Source (1/3) (fb89023001ac7ae37953fbac1a00652c) 
> switched from DEPLOYING to RUNNING.
> 12:38:59,890 INFO  org.apache.flink.runtime.taskexecutor.TaskExecutor         
>    - Received task Filter -> Sink: Unnamed (3/3).
> 12:38:59,891 INFO  org.apache.flink.runtime.taskmanager.Task                  
>    - Source: Sequence Source (2/3) (987b738a072a184f2075b127f21df794) 
> switched from DEPLOYING to RUNNING.
> 12:38:59,891 INFO  org.apache.flink.runtime.taskmanager.Task                  
>    - Filter -> Sink: Unnamed (2/3) (f1a2286709bf333f4dd8f24b4813a964) 
> switched from CREATED to DEPLOYING.
> 12:38:59,891 INFO  org.apache.flink.runtime.taskmanager.Task                  
>    - Filter -> Sink: Unnamed (1/3) (6ac1d973b37ddf0809005158a83892a4) 
> switched from DEPLOYING to RUNNING.
> 12:38:59,891 INFO  org.apache.flink.runtime.taskmanager.Task                  
>    - Creating FileSystem stream leak safety net for task Filter -> Sink: 
> Unnamed (2/3) (f1a2286709bf333f4dd8f24b4813a964) [DEPLOYING]
> 12:38:59,891 INFO  org.apache.flink.runtime.taskmanager.Task                  
>    - Loading JAR files for task Filter -> Sink: Unnamed (2/3) 
> (f1a2286709bf333f4dd8f24b4813a964) [DEPLOYING].
> 12:38:59,892 INFO  org.apache.flink.runtime.taskmanager.Task                  
>    - Registering task at network: Filter -> Sink: Unnamed (2/3) 
> (f1a2286709bf333f4dd8f24b4813a964) [DEPLOYING].
> 12:38:59,892 INFO  org.apache.flink.runtime.taskmanager.Task                  
>    - Filter -> Sink: Unnamed (2/3) (f1a2286709bf333f4dd8f24b4813a964) 
> switched from DEPLOYING to RUNNING.
> 12:38:59,892 INFO  org.apache.flink.streaming.runtime.tasks.StreamTask        
>    - No state backend has been configured, using default state backend 
> (Memory / JobManager)
> 12:38:59,892 INFO  org.apache.flink.streaming.runtime.tasks.StreamTask        
>    - No state backend has been configured, using default state backend 
> (Memory / JobManager)
> 12:38:59,893 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph     
>    - Source: Sequence Source (2/3) (987b738a072a184f2075b127f21df794) 
> switched from DEPLOYING to RUNNING.
> 12:38:59,893 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph     
>    - Filter -> Sink: Unnamed (1/3) (6ac1d973b37ddf0809005158a83892a4) 
> switched from DEPLOYING to RUNNING.
> 12:38:59,893 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph     
>    - Filter -> Sink: Unnamed (2/3) (f1a2286709bf333f4dd8f24b4813a964) 
> switched from DEPLOYING to RUNNING.
> 12:38:59,894 INFO  org.apache.flink.runtime.taskmanager.Task                  
>    - Filter -> Sink: Unnamed (3/3) (6735fc301957d96c817e09b14c35ac27) 
> switched from CREATED to DEPLOYING.
> 12:38:59,894 INFO  org.apache.flink.runtime.taskmanager.Task                  
>    - Creating FileSystem stream leak safety net for task Filter -> Sink: 
> Unnamed (3/3) (6735fc301957d96c817e09b14c35ac27) [DEPLOYING]
> 12:38:59,894 INFO  org.apache.flink.runtime.taskmanager.Task                  
>    - Loading JAR files for task Filter -> Sink: Unnamed (3/3) 
> (6735fc301957d96c817e09b14c35ac27) [DEPLOYING].
> 12:38:59,895 INFO  org.apache.flink.streaming.runtime.tasks.StreamTask        
>    - No state backend has been configured, using default state backend 
> (Memory / JobManager)
> 12:38:59,896 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph     
>    - Source: Sequence Source (3/3) (e36e2811c6f205facefd1195a2a3d183) 
> switched from DEPLOYING to RUNNING.
> 12:38:59,899 INFO  org.apache.flink.streaming.runtime.tasks.StreamTask        
>    - No state backend has been configured, using default state backend 
> (Memory / JobManager)
> 12:38:59,901 INFO  org.apache.flink.streaming.runtime.tasks.StreamTask        
>    - No state backend has been configured, using default state backend 
> (Memory / JobManager)
> 12:38:59,903 INFO  org.apache.flink.runtime.taskmanager.Task                  
>    - Registering task at network: Filter -> Sink: Unnamed (3/3) 
> (6735fc301957d96c817e09b14c35ac27) [DEPLOYING].
> 12:38:59,903 INFO  org.apache.flink.runtime.taskmanager.Task                  
>    - Filter -> Sink: Unnamed (3/3) (6735fc301957d96c817e09b14c35ac27) 
> switched from DEPLOYING to RUNNING.
> 12:38:59,904 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph     
>    - Filter -> Sink: Unnamed (3/3) (6735fc301957d96c817e09b14c35ac27) 
> switched from DEPLOYING to RUNNING.
> 12:38:59,904 INFO  org.apache.flink.streaming.runtime.tasks.StreamTask        
>    - No state backend has been configured, using default state backend 
> (Memory / JobManager)
> 12:38:59,964 INFO  org.apache.flink.runtime.taskmanager.Task                  
>    - Source: Sequence Source (1/3) (fb89023001ac7ae37953fbac1a00652c) 
> switched from RUNNING to FINISHED.
> 12:38:59,964 INFO  org.apache.flink.runtime.taskmanager.Task                  
>    - Freeing task resources for Source: Sequence Source (1/3) 
> (fb89023001ac7ae37953fbac1a00652c).
> 12:38:59,964 INFO  org.apache.flink.runtime.taskmanager.Task                  
>    - Ensuring all FileSystem streams are closed for task Source: Sequence 
> Source (1/3) (fb89023001ac7ae37953fbac1a00652c) [FINISHED]
> 12:38:59,965 INFO  org.apache.flink.runtime.taskmanager.Task                  
>    - Filter -> Sink: Unnamed (1/3) (6ac1d973b37ddf0809005158a83892a4) 
> switched from RUNNING to FINISHED.
> 12:38:59,965 INFO  org.apache.flink.runtime.taskmanager.Task                  
>    - Freeing task resources for Filter -> Sink: Unnamed (1/3) 
> (6ac1d973b37ddf0809005158a83892a4).
> 12:38:59,965 INFO  org.apache.flink.runtime.taskmanager.Task                  
>    - Ensuring all FileSystem streams are closed for task Filter -> Sink: 
> Unnamed (1/3) (6ac1d973b37ddf0809005158a83892a4) [FINISHED]
> 12:38:59,966 INFO  org.apache.flink.runtime.taskmanager.Task                  
>    - Source: Sequence Source (2/3) (987b738a072a184f2075b127f21df794) 
> switched from RUNNING to FINISHED.
> 12:38:59,966 INFO  org.apache.flink.runtime.taskmanager.Task                  
>    - Freeing task resources for Source: Sequence Source (2/3) 
> (987b738a072a184f2075b127f21df794).
> 12:38:59,966 INFO  org.apache.flink.runtime.taskmanager.Task                  
>    - Ensuring all FileSystem streams are closed for task Source: Sequence 
> Source (2/3) (987b738a072a184f2075b127f21df794) [FINISHED]
> 12:38:59,966 INFO  org.apache.flink.runtime.taskexecutor.TaskExecutor         
>    - Un-registering task and sending final execution state FINISHED to 
> JobManager for task Source: Sequence Source fb89023001ac7ae37953fbac1a00652c.
> 12:38:59,967 INFO  org.apache.flink.runtime.taskmanager.Task                  
>    - Filter -> Sink: Unnamed (2/3) (f1a2286709bf333f4dd8f24b4813a964) 
> switched from RUNNING to FINISHED.
> 12:38:59,967 INFO  org.apache.flink.runtime.taskmanager.Task                  
>    - Freeing task resources for Filter -> Sink: Unnamed (2/3) 
> (f1a2286709bf333f4dd8f24b4813a964).
> 12:38:59,967 INFO  org.apache.flink.runtime.taskmanager.Task                  
>    - Ensuring all FileSystem streams are closed for task Filter -> Sink: 
> Unnamed (2/3) (f1a2286709bf333f4dd8f24b4813a964) [FINISHED]
> 12:38:59,973 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph     
>    - Source: Sequence Source (1/3) (fb89023001ac7ae37953fbac1a00652c) 
> switched from RUNNING to FINISHED.
> 12:38:59,974 INFO  org.apache.flink.runtime.taskexecutor.TaskExecutor         
>    - Un-registering task and sending final execution state FINISHED to 
> JobManager for task Filter -> Sink: Unnamed 6ac1d973b37ddf0809005158a83892a4.
> 12:38:59,975 INFO  org.apache.flink.runtime.taskmanager.Task                  
>    - Source: Sequence Source (3/3) (e36e2811c6f205facefd1195a2a3d183) 
> switched from RUNNING to FINISHED.
> 12:38:59,975 INFO  org.apache.flink.runtime.taskmanager.Task                  
>    - Freeing task resources for Source: Sequence Source (3/3) 
> (e36e2811c6f205facefd1195a2a3d183).
> 12:38:59,975 INFO  org.apache.flink.runtime.taskmanager.Task                  
>    - Ensuring all FileSystem streams are closed for task Source: Sequence 
> Source (3/3) (e36e2811c6f205facefd1195a2a3d183) [FINISHED]
> 12:38:59,980 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph     
>    - Filter -> Sink: Unnamed (1/3) (6ac1d973b37ddf0809005158a83892a4) 
> switched from RUNNING to FINISHED.
> 12:38:59,980 INFO  org.apache.flink.runtime.taskexecutor.TaskExecutor         
>    - Un-registering task and sending final execution state FINISHED to 
> JobManager for task Source: Sequence Source 987b738a072a184f2075b127f21df794.
> 12:38:59,981 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph     
>    - Source: Sequence Source (2/3) (987b738a072a184f2075b127f21df794) 
> switched from RUNNING to FINISHED.
> 12:38:59,982 INFO  org.apache.flink.runtime.taskexecutor.TaskExecutor         
>    - Un-registering task and sending final execution state FINISHED to 
> JobManager for task Filter -> Sink: Unnamed f1a2286709bf333f4dd8f24b4813a964.
> 12:38:59,983 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph     
>    - Filter -> Sink: Unnamed (2/3) (f1a2286709bf333f4dd8f24b4813a964) 
> switched from RUNNING to FINISHED.
> 12:38:59,984 INFO  org.apache.flink.runtime.taskexecutor.TaskExecutor         
>    - Un-registering task and sending final execution state FINISHED to 
> JobManager for task Source: Sequence Source e36e2811c6f205facefd1195a2a3d183.
> 12:38:59,985 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph     
>    - Source: Sequence Source (3/3) (e36e2811c6f205facefd1195a2a3d183) 
> switched from RUNNING to FINISHED.
> 12:38:59,999 INFO  org.apache.flink.runtime.taskmanager.Task                  
>    - Filter -> Sink: Unnamed (3/3) (6735fc301957d96c817e09b14c35ac27) 
> switched from RUNNING to FINISHED.
> 12:39:00,000 INFO  org.apache.flink.runtime.taskmanager.Task                  
>    - Freeing task resources for Filter -> Sink: Unnamed (3/3) 
> (6735fc301957d96c817e09b14c35ac27).
> 12:39:00,000 INFO  org.apache.flink.runtime.taskmanager.Task                  
>    - Ensuring all FileSystem streams are closed for task Filter -> Sink: 
> Unnamed (3/3) (6735fc301957d96c817e09b14c35ac27) [FINISHED]
> 12:39:00,000 INFO  org.apache.flink.runtime.taskexecutor.TaskExecutor         
>    - Un-registering task and sending final execution state FINISHED to 
> JobManager for task Filter -> Sink: Unnamed 6735fc301957d96c817e09b14c35ac27.
> 12:39:00,002 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph     
>    - Filter -> Sink: Unnamed (3/3) (6735fc301957d96c817e09b14c35ac27) 
> switched from RUNNING to FINISHED.
> 12:39:00,002 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph     
>    - Job Flink Streaming Job (4edcbd9b42d9cca157794e9e2cdc29d3) switched from 
> state RUNNING to FINISHED.
> 12:39:00,002 INFO  org.apache.flink.runtime.checkpoint.CheckpointCoordinator  
>    - Stopping checkpoint coordinator for job 4edcbd9b42d9cca157794e9e2cdc29d3
> 12:39:00,002 INFO  
> org.apache.flink.runtime.checkpoint.StandaloneCompletedCheckpointStore  - 
> Shutting down
> 12:39:00,002 INFO  org.apache.flink.runtime.jobmaster.JobMaster               
>    - Status of job 4edcbd9b42d9cca157794e9e2cdc29d3 (Flink Streaming Job) 
> changed to FINISHED.
> 12:39:00,006 INFO  
> org.apache.flink.runtime.resourcemanager.StandaloneResourceManager  - 
> Disconnect job manager 
> 1c41035e-ef27-436e-bd0a-434042290346@akka://flink/user/jobmanager_1 for job 
> 4edcbd9b42d9cca157794e9e2cdc29d3 from the resource manager.
> 12:39:00,006 INFO  org.apache.flink.runtime.taskexecutor.TaskExecutor         
>    - JobManager for job 4edcbd9b42d9cca157794e9e2cdc29d3 with leader id 
> 1c41035e-ef27-436e-bd0a-434042290346 lost leadership.
> 12:39:00,007 INFO  org.apache.flink.runtime.taskexecutor.TaskExecutor         
>    - Close JobManager connection for job 4edcbd9b42d9cca157794e9e2cdc29d3.
> 12:39:00,007 INFO  org.apache.flink.runtime.taskexecutor.TaskExecutor         
>    - JobManager for job 4edcbd9b42d9cca157794e9e2cdc29d3 with leader id 
> 1c41035e-ef27-436e-bd0a-434042290346 lost leadership.
> 12:39:00,007 INFO  org.apache.flink.runtime.taskexecutor.TaskExecutor         
>    - JobManager for job 4edcbd9b42d9cca157794e9e2cdc29d3 with leader id 
> 1c41035e-ef27-436e-bd0a-434042290346 lost leadership.
> 12:39:00,007 INFO  org.apache.flink.runtime.taskexecutor.TaskExecutor         
>    - JobManager for job 4edcbd9b42d9cca157794e9e2cdc29d3 with leader id 
> 1c41035e-ef27-436e-bd0a-434042290346 lost leadership.
> 12:39:00,007 INFO  org.apache.flink.runtime.taskexecutor.TaskExecutor         
>    - JobManager for job 4edcbd9b42d9cca157794e9e2cdc29d3 with leader id 
> 1c41035e-ef27-436e-bd0a-434042290346 lost leadership.
> 12:39:00,008 INFO  org.apache.flink.runtime.taskexecutor.TaskExecutor         
>    - JobManager for job 4edcbd9b42d9cca157794e9e2cdc29d3 with leader id 
> 1c41035e-ef27-436e-bd0a-434042290346 lost leadership.
> 12:39:00,008 INFO  org.apache.flink.runtime.rpc.akka.AkkaRpcService           
>    - Trigger shut down of RPC endpoint akka://flink/user/jobmanager_1.
> 12:39:00,008 INFO  org.apache.flink.runtime.blob.BlobCache                    
>    - Shutting down BlobCache
> 12:39:00,013 INFO  org.apache.flink.runtime.blob.BlobServer                   
>    - Stopped BLOB server at 0.0.0.0:40558
> 12:39:00,014 INFO  org.apache.flink.runtime.minicluster.MiniCluster           
>    - Shutting down Flink Mini Cluster
> 12:39:00,014 INFO  
> org.apache.flink.runtime.minicluster.MiniClusterJobDispatcher  - Shutting 
> down the job dispatcher
> 12:39:00,014 INFO  org.apache.flink.runtime.rpc.akka.AkkaRpcService           
>    - Trigger shut down of RPC endpoint akka://flink/user/resourcemanager_0.
> 12:39:00,015 INFO  
> org.apache.flink.runtime.resourcemanager.slotmanager.SlotManager  - Closing 
> the SlotManager.
> 12:39:00,015 INFO  
> org.apache.flink.runtime.resourcemanager.slotmanager.SlotManager  - 
> Suspending the SlotManager.
> 12:39:00,014 INFO  org.apache.flink.runtime.jobmaster.JobMaster               
>    - Close ResourceManager connection ee59ee8ea0ec7be58b8874633e57e365.
> java.lang.Exception: Job leader lost leadership.
>       at 
> org.apache.flink.runtime.resourcemanager.ResourceManager.jobLeaderLostLeadership(ResourceManager.java:796)
>       at 
> org.apache.flink.runtime.resourcemanager.ResourceManager$JobLeaderIdActionsImpl$1.run(ResourceManager.java:1013)
>       at 
> org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRunAsync(AkkaRpcActor.java:287)
>       at 
> org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleMessage(AkkaRpcActor.java:140)
>       at 
> org.apache.flink.runtime.rpc.akka.AkkaRpcActor.access$000(AkkaRpcActor.java:71)
>       at 
> org.apache.flink.runtime.rpc.akka.AkkaRpcActor$1.apply(AkkaRpcActor.java:119)
>       at 
> akka.actor.ActorCell$$anonfun$become$1.applyOrElse(ActorCell.scala:534)
>       at akka.actor.Actor$class.aroundReceive(Actor.scala:467)
>       at akka.actor.UntypedActor.aroundReceive(UntypedActor.scala:97)
>       at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516)
>       at akka.actor.ActorCell.invoke(ActorCell.scala:487)
>       at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:238)
>       at akka.dispatch.Mailbox.run(Mailbox.scala:220)
>       at 
> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:397)
>       at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
>       at 
> scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
>       at 
> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>       at 
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
> 12:39:00,015 INFO  org.apache.flink.runtime.taskexecutor.TaskExecutor         
>    - Close JobManager connection for job 4edcbd9b42d9cca157794e9e2cdc29d3.
> 12:39:00,026 INFO  org.apache.flink.runtime.taskexecutor.TaskExecutor         
>    - Close JobManager connection for job 4edcbd9b42d9cca157794e9e2cdc29d3.
> 12:39:00,026 INFO  org.apache.flink.runtime.taskexecutor.TaskExecutor         
>    - Close JobManager connection for job 4edcbd9b42d9cca157794e9e2cdc29d3.
> 12:39:00,026 INFO  org.apache.flink.runtime.taskexecutor.TaskExecutor         
>    - Close JobManager connection for job 4edcbd9b42d9cca157794e9e2cdc29d3.
> 12:39:00,026 INFO  org.apache.flink.runtime.taskexecutor.TaskExecutor         
>    - Close JobManager connection for job 4edcbd9b42d9cca157794e9e2cdc29d3.
> 12:39:00,026 INFO  org.apache.flink.runtime.taskexecutor.TaskExecutor         
>    - Current ResourceManager akka://flink/user/resourcemanager_0 lost leader 
> status. Waiting for new ResourceManager leader.
> 12:39:00,025 INFO  org.apache.flink.runtime.rpc.akka.AkkaRpcService           
>    - Trigger shut down of RPC endpoint akka://flink/user/taskmanager_0.
> 12:39:00,026 INFO  org.apache.flink.runtime.rpc.akka.AkkaRpcService           
>    - Stopping Akka RPC service.
> 12:39:00,027 INFO  org.apache.flink.runtime.taskexecutor.TaskExecutor         
>    - Stopping TaskManager akka://flink/user/taskmanager_0.
> 12:39:00,028 INFO  org.apache.flink.runtime.io.disk.iomanager.IOManager       
>    - I/O manager removed spill file directory 
> /tmp/flink-io-b4293dd8-c902-46f3-b6f8-0f9730cf84f5
> 12:39:00,028 INFO  org.apache.flink.runtime.io.network.NetworkEnvironment     
>    - Shutting down the network environment and its components.
> 12:39:00,046 INFO  org.apache.flink.runtime.taskexecutor.TaskExecutor         
>    - Stopped TaskManager akka://flink/user/taskmanager_0.
> 12:39:00,066 INFO  org.apache.flink.runtime.jobmaster.JobMaster               
>    - Heartbeat of TaskManager with id 3b5cc819-a053-4360-a0d8-db0e29e00fc8 
> timed out.
> 12:39:00,068 ERROR 
> org.apache.flink.streaming.api.environment.LocalStreamEnvironmentITCase  - 
> --------------------------------------------------------------------------------
> Test 
> testRunIsolatedJob(org.apache.flink.streaming.api.environment.LocalStreamEnvironmentITCase)
>  failed with:
> java.util.ConcurrentModificationException
>       at java.util.HashMap$HashIterator.nextNode(HashMap.java:1437)
>       at java.util.HashMap$ValueIterator.next(HashMap.java:1466)
>       at 
> org.apache.flink.runtime.resourcemanager.JobLeaderIdService.clear(JobLeaderIdService.java:120)
>       at 
> org.apache.flink.runtime.resourcemanager.JobLeaderIdService.stop(JobLeaderIdService.java:98)
>       at 
> org.apache.flink.runtime.resourcemanager.ResourceManagerRuntimeServices.shutDown(ResourceManagerRuntimeServices.java:51)
>       at 
> org.apache.flink.runtime.resourcemanager.ResourceManagerRunner.shutDownInternally(ResourceManagerRunner.java:107)
>       at 
> org.apache.flink.runtime.resourcemanager.ResourceManagerRunner.shutDown(ResourceManagerRunner.java:94)
>       at 
> org.apache.flink.runtime.minicluster.MiniCluster.shutdownInternally(MiniCluster.java:341)
>       at 
> org.apache.flink.runtime.minicluster.MiniCluster.shutdown(MiniCluster.java:309)
>       at 
> org.apache.flink.streaming.api.environment.Flip6LocalStreamEnvironment.execute(Flip6LocalStreamEnvironment.java:124)
>       at 
> org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.execute(StreamExecutionEnvironment.java:1501)
>       at 
> org.apache.flink.streaming.api.environment.LocalStreamEnvironmentITCase.testRunIsolatedJob(LocalStreamEnvironmentITCase.java:45)
>       at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>       at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>       at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>       at java.lang.reflect.Method.invoke(Method.java:498)
>       at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
>       at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
>       at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
>       at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
>       at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
>       at org.junit.rules.RunRules.evaluate(RunRules.java:20)
>       at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
>       at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
>       at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
>       at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
>       at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
>       at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
>       at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
>       at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
>       at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
>       at 
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:283)
>       at 
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:173)
>       at 
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:153)
>       at 
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:128)
>       at 
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:203)
>       at 
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:155)
>       at 
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:103)
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Reply via email to