See
<https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark_Gradle/789/display/redirect>
------------------------------------------
[...truncated 26.32 MB...]
[task-result-getter-1] INFO org.apache.spark.scheduler.TaskSetManager -
Finished task 1.0 in stage 546.0 (TID 459) in 15 ms on localhost (executor
driver) (1/5)
[task-result-getter-0] INFO org.apache.spark.scheduler.TaskSetManager -
Finished task 0.0 in stage 546.0 (TID 458) in 16 ms on localhost (executor
driver) (2/5)
[task-result-getter-2] INFO org.apache.spark.scheduler.TaskSetManager -
Finished task 3.0 in stage 546.0 (TID 461) in 15 ms on localhost (executor
driver) (3/5)
[task-result-getter-3] INFO org.apache.spark.scheduler.TaskSetManager -
Finished task 2.0 in stage 546.0 (TID 460) in 15 ms on localhost (executor
driver) (4/5)
[Executor task launch worker for task 462] INFO
org.apache.spark.executor.Executor - Finished task 4.0 in stage 546.0 (TID
462). 59423 bytes result sent to driver
[task-result-getter-1] INFO org.apache.spark.scheduler.TaskSetManager -
Finished task 4.0 in stage 546.0 (TID 462) in 13 ms on localhost (executor
driver) (5/5)
[task-result-getter-1] INFO org.apache.spark.scheduler.TaskSchedulerImpl -
Removed TaskSet 546.0, whose tasks have all completed, from pool
[dag-scheduler-event-loop] INFO org.apache.spark.scheduler.DAGScheduler -
ShuffleMapStage 546 (mapToPair at GroupCombineFunctions.java:59) finished in
0.037 s
[dag-scheduler-event-loop] INFO org.apache.spark.scheduler.DAGScheduler -
looking for newly runnable stages
[dag-scheduler-event-loop] INFO org.apache.spark.scheduler.DAGScheduler -
running: Set()
[dag-scheduler-event-loop] INFO org.apache.spark.scheduler.DAGScheduler -
waiting: Set(ResultStage 547)
[dag-scheduler-event-loop] INFO org.apache.spark.scheduler.DAGScheduler -
failed: Set()
[dag-scheduler-event-loop] INFO org.apache.spark.scheduler.DAGScheduler -
Submitting ResultStage 547 (MapPartitionsRDD[2773] at map at
TranslationUtils.java:129), which has no missing parents
[dag-scheduler-event-loop] INFO org.apache.spark.storage.memory.MemoryStore
- Block broadcast_107 stored as values in memory (estimated size 208.7 KB, free
13.5 GB)
[dag-scheduler-event-loop] INFO org.apache.spark.storage.memory.MemoryStore
- Block broadcast_107_piece0 stored as bytes in memory (estimated size 53.2 KB,
free 13.5 GB)
[dispatcher-event-loop-0] INFO org.apache.spark.storage.BlockManagerInfo -
Added broadcast_107_piece0 in memory on localhost:38605 (size: 53.2 KB, free:
13.5 GB)
[dag-scheduler-event-loop] INFO org.apache.spark.SparkContext - Created
broadcast 107 from broadcast at DAGScheduler.scala:1039
[dag-scheduler-event-loop] INFO org.apache.spark.scheduler.DAGScheduler -
Submitting 4 missing tasks from ResultStage 547 (MapPartitionsRDD[2773] at map
at TranslationUtils.java:129) (first 15 tasks are for partitions Vector(0, 1,
2, 3))
[dag-scheduler-event-loop] INFO
org.apache.spark.scheduler.TaskSchedulerImpl - Adding task set 547.0 with 4
tasks
[dispatcher-event-loop-1] INFO org.apache.spark.scheduler.TaskSetManager -
Starting task 0.0 in stage 547.0 (TID 463, localhost, executor driver,
partition 0, PROCESS_LOCAL, 8308 bytes)
[dispatcher-event-loop-1] INFO org.apache.spark.scheduler.TaskSetManager -
Starting task 1.0 in stage 547.0 (TID 464, localhost, executor driver,
partition 1, PROCESS_LOCAL, 8308 bytes)
[dispatcher-event-loop-1] INFO org.apache.spark.scheduler.TaskSetManager -
Starting task 2.0 in stage 547.0 (TID 465, localhost, executor driver,
partition 2, PROCESS_LOCAL, 8308 bytes)
[dispatcher-event-loop-1] INFO org.apache.spark.scheduler.TaskSetManager -
Starting task 3.0 in stage 547.0 (TID 466, localhost, executor driver,
partition 3, PROCESS_LOCAL, 8308 bytes)
[Executor task launch worker for task 463] INFO
org.apache.spark.executor.Executor - Running task 0.0 in stage 547.0 (TID 463)
[Executor task launch worker for task 465] INFO
org.apache.spark.executor.Executor - Running task 2.0 in stage 547.0 (TID 465)
[Executor task launch worker for task 464] INFO
org.apache.spark.executor.Executor - Running task 1.0 in stage 547.0 (TID 464)
[Executor task launch worker for task 466] INFO
org.apache.spark.executor.Executor - Running task 3.0 in stage 547.0 (TID 466)
[Executor task launch worker for task 464] INFO
org.apache.spark.storage.ShuffleBlockFetcherIterator - Getting 0 non-empty
blocks out of 5 blocks
[Executor task launch worker for task 465] INFO
org.apache.spark.storage.ShuffleBlockFetcherIterator - Getting 0 non-empty
blocks out of 5 blocks
[Executor task launch worker for task 464] INFO
org.apache.spark.storage.ShuffleBlockFetcherIterator - Started 0 remote fetches
in 0 ms
[Executor task launch worker for task 466] INFO
org.apache.spark.storage.ShuffleBlockFetcherIterator - Getting 0 non-empty
blocks out of 5 blocks
[Executor task launch worker for task 465] INFO
org.apache.spark.storage.ShuffleBlockFetcherIterator - Started 0 remote fetches
in 0 ms
[Executor task launch worker for task 466] INFO
org.apache.spark.storage.ShuffleBlockFetcherIterator - Started 0 remote fetches
in 1 ms
[Executor task launch worker for task 464] INFO
org.apache.spark.storage.BlockManager - Found block rdd_2441_1 locally
[Executor task launch worker for task 466] INFO
org.apache.spark.storage.BlockManager - Found block rdd_2441_3 locally
[Executor task launch worker for task 465] INFO
org.apache.spark.storage.BlockManager - Found block rdd_2441_2 locally
[Executor task launch worker for task 463] INFO
org.apache.spark.storage.ShuffleBlockFetcherIterator - Getting 0 non-empty
blocks out of 5 blocks
[Executor task launch worker for task 463] INFO
org.apache.spark.storage.ShuffleBlockFetcherIterator - Started 0 remote fetches
in 0 ms
[Executor task launch worker for task 463] INFO
org.apache.spark.storage.BlockManager - Found block rdd_2441_0 locally
[Executor task launch worker for task 466] INFO
org.apache.spark.storage.memory.MemoryStore - Block rdd_2756_3 stored as bytes
in memory (estimated size 4.0 B, free 13.5 GB)
[Executor task launch worker for task 464] INFO
org.apache.spark.storage.memory.MemoryStore - Block rdd_2756_1 stored as bytes
in memory (estimated size 4.0 B, free 13.5 GB)
[Executor task launch worker for task 465] INFO
org.apache.spark.storage.memory.MemoryStore - Block rdd_2756_2 stored as bytes
in memory (estimated size 4.0 B, free 13.5 GB)
[dispatcher-event-loop-2] INFO org.apache.spark.storage.BlockManagerInfo -
Added rdd_2756_1 in memory on localhost:38605 (size: 4.0 B, free: 13.5 GB)
[dispatcher-event-loop-2] INFO org.apache.spark.storage.BlockManagerInfo -
Added rdd_2756_2 in memory on localhost:38605 (size: 4.0 B, free: 13.5 GB)
[dispatcher-event-loop-2] INFO org.apache.spark.storage.BlockManagerInfo -
Added rdd_2756_3 in memory on localhost:38605 (size: 4.0 B, free: 13.5 GB)
[Executor task launch worker for task 463] INFO
org.apache.spark.storage.memory.MemoryStore - Block rdd_2756_0 stored as bytes
in memory (estimated size 4.0 B, free 13.5 GB)
[dispatcher-event-loop-1] INFO org.apache.spark.storage.BlockManagerInfo -
Added rdd_2756_0 in memory on localhost:38605 (size: 4.0 B, free: 13.5 GB)
[Executor task launch worker for task 464] INFO
org.apache.spark.executor.Executor - Finished task 1.0 in stage 547.0 (TID
464). 59881 bytes result sent to driver
[Executor task launch worker for task 465] INFO
org.apache.spark.executor.Executor - Finished task 2.0 in stage 547.0 (TID
465). 59881 bytes result sent to driver
[Executor task launch worker for task 466] INFO
org.apache.spark.executor.Executor - Finished task 3.0 in stage 547.0 (TID
466). 59881 bytes result sent to driver
[Executor task launch worker for task 463] INFO
org.apache.spark.executor.Executor - Finished task 0.0 in stage 547.0 (TID
463). 59881 bytes result sent to driver
[task-result-getter-0] INFO org.apache.spark.scheduler.TaskSetManager -
Finished task 2.0 in stage 547.0 (TID 465) in 15 ms on localhost (executor
driver) (1/4)
[task-result-getter-2] INFO org.apache.spark.scheduler.TaskSetManager -
Finished task 1.0 in stage 547.0 (TID 464) in 15 ms on localhost (executor
driver) (2/4)
[task-result-getter-1] INFO org.apache.spark.scheduler.TaskSetManager -
Finished task 0.0 in stage 547.0 (TID 463) in 15 ms on localhost (executor
driver) (3/4)
[task-result-getter-3] INFO org.apache.spark.scheduler.TaskSetManager -
Finished task 3.0 in stage 547.0 (TID 466) in 15 ms on localhost (executor
driver) (4/4)
[task-result-getter-3] INFO org.apache.spark.scheduler.TaskSchedulerImpl -
Removed TaskSet 547.0, whose tasks have all completed, from pool
[dag-scheduler-event-loop] INFO org.apache.spark.scheduler.DAGScheduler -
ResultStage 547 (foreach at UnboundedDataset.java:81) finished in 0.028 s
[streaming-job-executor-0] INFO org.apache.spark.scheduler.DAGScheduler -
Job 34 finished: foreach at UnboundedDataset.java:81, took 0.096656 s
[JobScheduler] INFO org.apache.spark.streaming.scheduler.JobScheduler -
Finished job streaming job 1529136921000 ms.2 from job set of time
1529136921000 ms
[JobScheduler] INFO org.apache.spark.streaming.scheduler.JobScheduler -
Starting job streaming job 1529136921000 ms.3 from job set of time
1529136921000 ms
[streaming-job-executor-0] INFO org.apache.spark.SparkContext - Starting
job: foreach at UnboundedDataset.java:81
[dag-scheduler-event-loop] INFO org.apache.spark.scheduler.DAGScheduler -
Registering RDD 2787 (mapToPair at GroupCombineFunctions.java:59)
[dag-scheduler-event-loop] INFO org.apache.spark.scheduler.DAGScheduler -
Registering RDD 2815 (mapToPair at GroupCombineFunctions.java:59)
[dag-scheduler-event-loop] INFO org.apache.spark.scheduler.DAGScheduler -
Got job 35 (foreach at UnboundedDataset.java:81) with 4 output partitions
[dag-scheduler-event-loop] INFO org.apache.spark.scheduler.DAGScheduler -
Final stage: ResultStage 575 (foreach at UnboundedDataset.java:81)
[dag-scheduler-event-loop] INFO org.apache.spark.scheduler.DAGScheduler -
Parents of final stage: List(ShuffleMapStage 574, ShuffleMapStage 571,
ShuffleMapStage 560, ShuffleMapStage 561, ShuffleMapStage 568, ShuffleMapStage
566, ShuffleMapStage 552, ShuffleMapStage 567, ShuffleMapStage 559)
[dag-scheduler-event-loop] INFO org.apache.spark.scheduler.DAGScheduler -
Missing parents: List(ShuffleMapStage 574)
[dag-scheduler-event-loop] INFO org.apache.spark.scheduler.DAGScheduler -
Submitting ShuffleMapStage 573 (MapPartitionsRDD[2787] at mapToPair at
GroupCombineFunctions.java:59), which has no missing parents
[dag-scheduler-event-loop] INFO org.apache.spark.storage.memory.MemoryStore
- Block broadcast_108 stored as values in memory (estimated size 140.5 KB, free
13.5 GB)
[dag-scheduler-event-loop] INFO org.apache.spark.storage.memory.MemoryStore
- Block broadcast_108_piece0 stored as bytes in memory (estimated size 32.2 KB,
free 13.5 GB)
[dispatcher-event-loop-2] INFO org.apache.spark.storage.BlockManagerInfo -
Added broadcast_108_piece0 in memory on localhost:38605 (size: 32.2 KB, free:
13.5 GB)
[dag-scheduler-event-loop] INFO org.apache.spark.SparkContext - Created
broadcast 108 from broadcast at DAGScheduler.scala:1039
[dag-scheduler-event-loop] INFO org.apache.spark.scheduler.DAGScheduler -
Submitting 4 missing tasks from ShuffleMapStage 573 (MapPartitionsRDD[2787] at
mapToPair at GroupCombineFunctions.java:59) (first 15 tasks are for partitions
Vector(0, 1, 2, 3))
[dag-scheduler-event-loop] INFO
org.apache.spark.scheduler.TaskSchedulerImpl - Adding task set 573.0 with 4
tasks
[dispatcher-event-loop-3] INFO org.apache.spark.scheduler.TaskSetManager -
Starting task 0.0 in stage 573.0 (TID 467, localhost, executor driver,
partition 0, PROCESS_LOCAL, 8297 bytes)
[dispatcher-event-loop-3] INFO org.apache.spark.scheduler.TaskSetManager -
Starting task 1.0 in stage 573.0 (TID 468, localhost, executor driver,
partition 1, PROCESS_LOCAL, 8297 bytes)
[dispatcher-event-loop-3] INFO org.apache.spark.scheduler.TaskSetManager -
Starting task 2.0 in stage 573.0 (TID 469, localhost, executor driver,
partition 2, PROCESS_LOCAL, 8297 bytes)
[dispatcher-event-loop-3] INFO org.apache.spark.scheduler.TaskSetManager -
Starting task 3.0 in stage 573.0 (TID 470, localhost, executor driver,
partition 3, PROCESS_LOCAL, 8297 bytes)
[Executor task launch worker for task 467] INFO
org.apache.spark.executor.Executor - Running task 0.0 in stage 573.0 (TID 467)
[Executor task launch worker for task 468] INFO
org.apache.spark.executor.Executor - Running task 1.0 in stage 573.0 (TID 468)
[Executor task launch worker for task 469] INFO
org.apache.spark.executor.Executor - Running task 2.0 in stage 573.0 (TID 469)
[Executor task launch worker for task 470] INFO
org.apache.spark.executor.Executor - Running task 3.0 in stage 573.0 (TID 470)
[Executor task launch worker for task 468] INFO
org.apache.spark.storage.BlockManager - Found block rdd_2559_1 locally
[Executor task launch worker for task 470] INFO
org.apache.spark.storage.BlockManager - Found block rdd_2559_3 locally
[Executor task launch worker for task 469] INFO
org.apache.spark.storage.BlockManager - Found block rdd_2559_2 locally
[Executor task launch worker for task 467] INFO
org.apache.spark.storage.BlockManager - Found block rdd_2559_0 locally
[Executor task launch worker for task 470] INFO
org.apache.spark.executor.Executor - Finished task 3.0 in stage 573.0 (TID
470). 59466 bytes result sent to driver
[Executor task launch worker for task 468] INFO
org.apache.spark.executor.Executor - Finished task 1.0 in stage 573.0 (TID
468). 59466 bytes result sent to driver
[Executor task launch worker for task 469] INFO
org.apache.spark.executor.Executor - Finished task 2.0 in stage 573.0 (TID
469). 59509 bytes result sent to driver
[Executor task launch worker for task 467] INFO
org.apache.spark.executor.Executor - Finished task 0.0 in stage 573.0 (TID
467). 59509 bytes result sent to driver
[task-result-getter-0] INFO org.apache.spark.scheduler.TaskSetManager -
Finished task 3.0 in stage 573.0 (TID 470) in 12 ms on localhost (executor
driver) (1/4)
[task-result-getter-1] INFO org.apache.spark.scheduler.TaskSetManager -
Finished task 2.0 in stage 573.0 (TID 469) in 12 ms on localhost (executor
driver) (2/4)
[task-result-getter-2] INFO org.apache.spark.scheduler.TaskSetManager -
Finished task 1.0 in stage 573.0 (TID 468) in 12 ms on localhost (executor
driver) (3/4)
[task-result-getter-3] INFO org.apache.spark.scheduler.TaskSetManager -
Finished task 0.0 in stage 573.0 (TID 467) in 12 ms on localhost (executor
driver) (4/4)
[task-result-getter-3] INFO org.apache.spark.scheduler.TaskSchedulerImpl -
Removed TaskSet 573.0, whose tasks have all completed, from pool
[dag-scheduler-event-loop] INFO org.apache.spark.scheduler.DAGScheduler -
ShuffleMapStage 573 (mapToPair at GroupCombineFunctions.java:59) finished in
0.022 s
[dag-scheduler-event-loop] INFO org.apache.spark.scheduler.DAGScheduler -
looking for newly runnable stages
[dag-scheduler-event-loop] INFO org.apache.spark.scheduler.DAGScheduler -
running: Set()
[dag-scheduler-event-loop] INFO org.apache.spark.scheduler.DAGScheduler -
waiting: Set(ShuffleMapStage 574, ResultStage 575)
[dag-scheduler-event-loop] INFO org.apache.spark.scheduler.DAGScheduler -
failed: Set()
[dag-scheduler-event-loop] INFO org.apache.spark.scheduler.DAGScheduler -
Submitting ShuffleMapStage 574 (MapPartitionsRDD[2815] at mapToPair at
GroupCombineFunctions.java:59), which has no missing parents
[dag-scheduler-event-loop] INFO org.apache.spark.storage.memory.MemoryStore
- Block broadcast_109 stored as values in memory (estimated size 174.0 KB, free
13.5 GB)
[dag-scheduler-event-loop] INFO org.apache.spark.storage.memory.MemoryStore
- Block broadcast_109_piece0 stored as bytes in memory (estimated size 42.0 KB,
free 13.5 GB)
[dispatcher-event-loop-1] INFO org.apache.spark.storage.BlockManagerInfo -
Added broadcast_109_piece0 in memory on localhost:38605 (size: 42.0 KB, free:
13.5 GB)
[dag-scheduler-event-loop] INFO org.apache.spark.SparkContext - Created
broadcast 109 from broadcast at DAGScheduler.scala:1039
[dag-scheduler-event-loop] INFO org.apache.spark.scheduler.DAGScheduler -
Submitting 5 missing tasks from ShuffleMapStage 574 (MapPartitionsRDD[2815] at
mapToPair at GroupCombineFunctions.java:59) (first 15 tasks are for partitions
Vector(0, 1, 2, 3, 4))
[dag-scheduler-event-loop] INFO
org.apache.spark.scheduler.TaskSchedulerImpl - Adding task set 574.0 with 5
tasks
[dispatcher-event-loop-3] INFO org.apache.spark.scheduler.TaskSetManager -
Starting task 0.0 in stage 574.0 (TID 471, localhost, executor driver,
partition 0, PROCESS_LOCAL, 8436 bytes)
[dispatcher-event-loop-3] INFO org.apache.spark.scheduler.TaskSetManager -
Starting task 1.0 in stage 574.0 (TID 472, localhost, executor driver,
partition 1, PROCESS_LOCAL, 8436 bytes)
[dispatcher-event-loop-3] INFO org.apache.spark.scheduler.TaskSetManager -
Starting task 2.0 in stage 574.0 (TID 473, localhost, executor driver,
partition 2, PROCESS_LOCAL, 8436 bytes)
[dispatcher-event-loop-3] INFO org.apache.spark.scheduler.TaskSetManager -
Starting task 3.0 in stage 574.0 (TID 474, localhost, executor driver,
partition 3, PROCESS_LOCAL, 8436 bytes)
[Executor task launch worker for task 471] INFO
org.apache.spark.executor.Executor - Running task 0.0 in stage 574.0 (TID 471)
[Executor task launch worker for task 473] INFO
org.apache.spark.executor.Executor - Running task 2.0 in stage 574.0 (TID 473)
[Executor task launch worker for task 474] INFO
org.apache.spark.executor.Executor - Running task 3.0 in stage 574.0 (TID 474)
[Executor task launch worker for task 472] INFO
org.apache.spark.executor.Executor - Running task 1.0 in stage 574.0 (TID 472)
[Executor task launch worker for task 472] INFO
org.apache.spark.storage.ShuffleBlockFetcherIterator - Getting 0 non-empty
blocks out of 4 blocks
[Executor task launch worker for task 474] INFO
org.apache.spark.storage.ShuffleBlockFetcherIterator - Getting 0 non-empty
blocks out of 4 blocks
[Executor task launch worker for task 472] INFO
org.apache.spark.storage.ShuffleBlockFetcherIterator - Started 0 remote fetches
in 1 ms
[Executor task launch worker for task 474] INFO
org.apache.spark.storage.ShuffleBlockFetcherIterator - Started 0 remote fetches
in 1 ms
[Executor task launch worker for task 473] INFO
org.apache.spark.storage.ShuffleBlockFetcherIterator - Getting 0 non-empty
blocks out of 4 blocks
[Executor task launch worker for task 473] INFO
org.apache.spark.storage.ShuffleBlockFetcherIterator - Started 0 remote fetches
in 0 ms
[Executor task launch worker for task 472] INFO
org.apache.spark.storage.BlockManager - Found block rdd_2484_1 locally
[Executor task launch worker for task 474] INFO
org.apache.spark.storage.BlockManager - Found block rdd_2484_3 locally
[Executor task launch worker for task 473] INFO
org.apache.spark.storage.BlockManager - Found block rdd_2484_2 locally
[Executor task launch worker for task 471] INFO
org.apache.spark.storage.ShuffleBlockFetcherIterator - Getting 0 non-empty
blocks out of 4 blocks
[Executor task launch worker for task 471] INFO
org.apache.spark.storage.ShuffleBlockFetcherIterator - Started 0 remote fetches
in 0 ms
[Executor task launch worker for task 471] INFO
org.apache.spark.storage.BlockManager - Found block rdd_2484_0 locally
[Executor task launch worker for task 472] INFO
org.apache.spark.storage.memory.MemoryStore - Block rdd_2799_1 stored as bytes
in memory (estimated size 4.0 B, free 13.5 GB)
[Executor task launch worker for task 473] INFO
org.apache.spark.storage.memory.MemoryStore - Block rdd_2799_2 stored as bytes
in memory (estimated size 4.0 B, free 13.5 GB)
[Executor task launch worker for task 474] INFO
org.apache.spark.storage.memory.MemoryStore - Block rdd_2799_3 stored as bytes
in memory (estimated size 4.0 B, free 13.5 GB)
[dispatcher-event-loop-2] INFO org.apache.spark.storage.BlockManagerInfo -
Added rdd_2799_2 in memory on localhost:38605 (size: 4.0 B, free: 13.5 GB)
[Executor task launch worker for task 471] INFO
org.apache.spark.storage.memory.MemoryStore - Block rdd_2799_0 stored as bytes
in memory (estimated size 4.0 B, free 13.5 GB)
[dispatcher-event-loop-2] INFO org.apache.spark.storage.BlockManagerInfo -
Added rdd_2799_1 in memory on localhost:38605 (size: 4.0 B, free: 13.5 GB)
[dispatcher-event-loop-2] INFO org.apache.spark.storage.BlockManagerInfo -
Added rdd_2799_3 in memory on localhost:38605 (size: 4.0 B, free: 13.5 GB)
[dispatcher-event-loop-2] INFO org.apache.spark.storage.BlockManagerInfo -
Added rdd_2799_0 in memory on localhost:38605 (size: 4.0 B, free: 13.5 GB)
[Executor task launch worker for task 474] INFO
org.apache.spark.executor.Executor - Finished task 3.0 in stage 574.0 (TID
474). 59939 bytes result sent to driver
[Executor task launch worker for task 472] INFO
org.apache.spark.executor.Executor - Finished task 1.0 in stage 574.0 (TID
472). 59939 bytes result sent to driver
[Executor task launch worker for task 473] INFO
org.apache.spark.executor.Executor - Finished task 2.0 in stage 574.0 (TID
473). 59939 bytes result sent to driver
[Executor task launch worker for task 471] INFO
org.apache.spark.executor.Executor - Finished task 0.0 in stage 574.0 (TID
471). 59939 bytes result sent to driver
[dispatcher-event-loop-1] INFO org.apache.spark.scheduler.TaskSetManager -
Starting task 4.0 in stage 574.0 (TID 475, localhost, executor driver,
partition 4, PROCESS_LOCAL, 7968 bytes)
[Executor task launch worker for task 475] INFO
org.apache.spark.executor.Executor - Running task 4.0 in stage 574.0 (TID 475)
[task-result-getter-0] INFO org.apache.spark.scheduler.TaskSetManager -
Finished task 1.0 in stage 574.0 (TID 472) in 15 ms on localhost (executor
driver) (1/5)
[task-result-getter-1] INFO org.apache.spark.scheduler.TaskSetManager -
Finished task 3.0 in stage 574.0 (TID 474) in 15 ms on localhost (executor
driver) (2/5)
[task-result-getter-3] INFO org.apache.spark.scheduler.TaskSetManager -
Finished task 0.0 in stage 574.0 (TID 471) in 15 ms on localhost (executor
driver) (3/5)
[task-result-getter-2] INFO org.apache.spark.scheduler.TaskSetManager -
Finished task 2.0 in stage 574.0 (TID 473) in 15 ms on localhost (executor
driver) (4/5)
[Executor task launch worker for task 475] INFO
org.apache.spark.executor.Executor - Finished task 4.0 in stage 574.0 (TID
475). 59423 bytes result sent to driver
[task-result-getter-0] INFO org.apache.spark.scheduler.TaskSetManager -
Finished task 4.0 in stage 574.0 (TID 475) in 13 ms on localhost (executor
driver) (5/5)
[task-result-getter-0] INFO org.apache.spark.scheduler.TaskSchedulerImpl -
Removed TaskSet 574.0, whose tasks have all completed, from pool
[dag-scheduler-event-loop] INFO org.apache.spark.scheduler.DAGScheduler -
ShuffleMapStage 574 (mapToPair at GroupCombineFunctions.java:59) finished in
0.037 s
[dag-scheduler-event-loop] INFO org.apache.spark.scheduler.DAGScheduler -
looking for newly runnable stages
[dag-scheduler-event-loop] INFO org.apache.spark.scheduler.DAGScheduler -
running: Set()
[dag-scheduler-event-loop] INFO org.apache.spark.scheduler.DAGScheduler -
waiting: Set(ResultStage 575)
[dag-scheduler-event-loop] INFO org.apache.spark.scheduler.DAGScheduler -
failed: Set()
[dag-scheduler-event-loop] INFO org.apache.spark.scheduler.DAGScheduler -
Submitting ResultStage 575 (MapPartitionsRDD[2844] at map at
TranslationUtils.java:129), which has no missing parents
[dag-scheduler-event-loop] INFO org.apache.spark.storage.memory.MemoryStore
- Block broadcast_110 stored as values in memory (estimated size 208.7 KB, free
13.5 GB)
[dag-scheduler-event-loop] INFO org.apache.spark.storage.memory.MemoryStore
- Block broadcast_110_piece0 stored as bytes in memory (estimated size 53.4 KB,
free 13.5 GB)
[dispatcher-event-loop-0] INFO org.apache.spark.storage.BlockManagerInfo -
Added broadcast_110_piece0 in memory on localhost:38605 (size: 53.4 KB, free:
13.5 GB)
[dag-scheduler-event-loop] INFO org.apache.spark.SparkContext - Created
broadcast 110 from broadcast at DAGScheduler.scala:1039
[dag-scheduler-event-loop] INFO org.apache.spark.scheduler.DAGScheduler -
Submitting 4 missing tasks from ResultStage 575 (MapPartitionsRDD[2844] at map
at TranslationUtils.java:129) (first 15 tasks are for partitions Vector(0, 1,
2, 3))
[dag-scheduler-event-loop] INFO
org.apache.spark.scheduler.TaskSchedulerImpl - Adding task set 575.0 with 4
tasks
[dispatcher-event-loop-3] INFO org.apache.spark.scheduler.TaskSetManager -
Starting task 0.0 in stage 575.0 (TID 476, localhost, executor driver,
partition 0, PROCESS_LOCAL, 8308 bytes)
[dispatcher-event-loop-3] INFO org.apache.spark.scheduler.TaskSetManager -
Starting task 1.0 in stage 575.0 (TID 477, localhost, executor driver,
partition 1, PROCESS_LOCAL, 8308 bytes)
[dispatcher-event-loop-3] INFO org.apache.spark.scheduler.TaskSetManager -
Starting task 2.0 in stage 575.0 (TID 478, localhost, executor driver,
partition 2, PROCESS_LOCAL, 8308 bytes)
[dispatcher-event-loop-3] INFO org.apache.spark.scheduler.TaskSetManager -
Starting task 3.0 in stage 575.0 (TID 479, localhost, executor driver,
partition 3, PROCESS_LOCAL, 8308 bytes)
[Executor task launch worker for task 476] INFO
org.apache.spark.executor.Executor - Running task 0.0 in stage 575.0 (TID 476)
[Executor task launch worker for task 478] INFO
org.apache.spark.executor.Executor - Running task 2.0 in stage 575.0 (TID 478)
[Executor task launch worker for task 477] INFO
org.apache.spark.executor.Executor - Running task 1.0 in stage 575.0 (TID 477)
[Executor task launch worker for task 479] INFO
org.apache.spark.executor.Executor - Running task 3.0 in stage 575.0 (TID 479)
[Executor task launch worker for task 476] INFO
org.apache.spark.storage.ShuffleBlockFetcherIterator - Getting 0 non-empty
blocks out of 5 blocks
[Executor task launch worker for task 478] INFO
org.apache.spark.storage.ShuffleBlockFetcherIterator - Getting 0 non-empty
blocks out of 5 blocks
[Executor task launch worker for task 478] INFO
org.apache.spark.storage.ShuffleBlockFetcherIterator - Started 0 remote fetches
in 0 ms
[Executor task launch worker for task 476] INFO
org.apache.spark.storage.ShuffleBlockFetcherIterator - Started 0 remote fetches
in 0 ms
[Executor task launch worker for task 478] INFO
org.apache.spark.storage.BlockManager - Found block rdd_2512_2 locally
[Executor task launch worker for task 476] INFO
org.apache.spark.storage.BlockManager - Found block rdd_2512_0 locally
[Executor task launch worker for task 478] INFO
org.apache.spark.storage.memory.MemoryStore - Block rdd_2827_2 stored as bytes
in memory (estimated size 4.0 B, free 13.5 GB)
[Executor task launch worker for task 476] INFO
org.apache.spark.storage.memory.MemoryStore - Block rdd_2827_0 stored as bytes
in memory (estimated size 4.0 B, free 13.5 GB)
[dispatcher-event-loop-0] INFO org.apache.spark.storage.BlockManagerInfo -
Added rdd_2827_2 in memory on localhost:38605 (size: 4.0 B, free: 13.5 GB)
[dispatcher-event-loop-0] INFO org.apache.spark.storage.BlockManagerInfo -
Added rdd_2827_0 in memory on localhost:38605 (size: 4.0 B, free: 13.5 GB)
[Executor task launch worker for task 477] INFO
org.apache.spark.storage.ShuffleBlockFetcherIterator - Getting 0 non-empty
blocks out of 5 blocks
[Executor task launch worker for task 479] INFO
org.apache.spark.storage.ShuffleBlockFetcherIterator - Getting 0 non-empty
blocks out of 5 blocks
[Executor task launch worker for task 477] INFO
org.apache.spark.storage.ShuffleBlockFetcherIterator - Started 0 remote fetches
in 0 ms
[Executor task launch worker for task 479] INFO
org.apache.spark.storage.ShuffleBlockFetcherIterator - Started 0 remote fetches
in 0 ms
[Executor task launch worker for task 477] INFO
org.apache.spark.storage.BlockManager - Found block rdd_2512_1 locally
[Executor task launch worker for task 479] INFO
org.apache.spark.storage.BlockManager - Found block rdd_2512_3 locally
[Executor task launch worker for task 479] INFO
org.apache.spark.storage.memory.MemoryStore - Block rdd_2827_3 stored as bytes
in memory (estimated size 4.0 B, free 13.5 GB)
[Executor task launch worker for task 477] INFO
org.apache.spark.storage.memory.MemoryStore - Block rdd_2827_1 stored as bytes
in memory (estimated size 4.0 B, free 13.5 GB)
[dispatcher-event-loop-3] INFO org.apache.spark.storage.BlockManagerInfo -
Added rdd_2827_3 in memory on localhost:38605 (size: 4.0 B, free: 13.5 GB)
[dispatcher-event-loop-3] INFO org.apache.spark.storage.BlockManagerInfo -
Added rdd_2827_1 in memory on localhost:38605 (size: 4.0 B, free: 13.5 GB)
[Executor task launch worker for task 478] INFO
org.apache.spark.executor.Executor - Finished task 2.0 in stage 575.0 (TID
478). 59881 bytes result sent to driver
[Executor task launch worker for task 476] INFO
org.apache.spark.executor.Executor - Finished task 0.0 in stage 575.0 (TID
476). 59881 bytes result sent to driver
[task-result-getter-1] INFO org.apache.spark.scheduler.TaskSetManager -
Finished task 2.0 in stage 575.0 (TID 478) in 16 ms on localhost (executor
driver) (1/4)
[task-result-getter-3] INFO org.apache.spark.scheduler.TaskSetManager -
Finished task 0.0 in stage 575.0 (TID 476) in 17 ms on localhost (executor
driver) (2/4)
[Executor task launch worker for task 479] INFO
org.apache.spark.executor.Executor - Finished task 3.0 in stage 575.0 (TID
479). 59881 bytes result sent to driver
[Executor task launch worker for task 477] INFO
org.apache.spark.executor.Executor - Finished task 1.0 in stage 575.0 (TID
477). 59881 bytes result sent to driver
[task-result-getter-0] INFO org.apache.spark.scheduler.TaskSetManager -
Finished task 1.0 in stage 575.0 (TID 477) in 20 ms on localhost (executor
driver) (3/4)
[task-result-getter-2] INFO org.apache.spark.scheduler.TaskSetManager -
Finished task 3.0 in stage 575.0 (TID 479) in 20 ms on localhost (executor
driver) (4/4)
[task-result-getter-2] INFO org.apache.spark.scheduler.TaskSchedulerImpl -
Removed TaskSet 575.0, whose tasks have all completed, from pool
[dag-scheduler-event-loop] INFO org.apache.spark.scheduler.DAGScheduler -
ResultStage 575 (foreach at UnboundedDataset.java:81) finished in 0.033 s
[streaming-job-executor-0] INFO org.apache.spark.scheduler.DAGScheduler -
Job 35 finished: foreach at UnboundedDataset.java:81, took 0.100900 s
[JobScheduler] INFO org.apache.spark.streaming.scheduler.JobScheduler -
Finished job streaming job 1529136921000 ms.3 from job set of time
1529136921000 ms
[JobScheduler] INFO org.apache.spark.streaming.scheduler.JobScheduler -
Total delay: 5.341 s for time 1529136921000 ms (execution: 0.397 s)
[Test worker] INFO org.apache.spark.streaming.scheduler.JobScheduler -
Stopped JobScheduler
[Test worker] INFO org.spark_project.jetty.server.handler.ContextHandler -
Stopped
o.s.j.s.ServletContextHandler@482d9b39{/streaming,null,UNAVAILABLE,@Spark}
[Test worker] INFO org.spark_project.jetty.server.handler.ContextHandler -
Stopped
o.s.j.s.ServletContextHandler@3a02d45a{/streaming/batch,null,UNAVAILABLE,@Spark}
[Test worker] INFO org.spark_project.jetty.server.handler.ContextHandler -
Stopped
o.s.j.s.ServletContextHandler@6f4887c3{/static/streaming,null,UNAVAILABLE,@Spark}
[Test worker] INFO org.apache.spark.streaming.StreamingContext -
StreamingContext stopped successfully
[Test worker] INFO org.spark_project.jetty.server.AbstractConnector -
Stopped Spark@72aaaa24{HTTP/1.1,[http/1.1]}{127.0.0.1:4040}
[Test worker] INFO org.apache.spark.ui.SparkUI - Stopped Spark web UI at
http://localhost:4040
[dispatcher-event-loop-3] INFO
org.apache.spark.MapOutputTrackerMasterEndpoint -
MapOutputTrackerMasterEndpoint stopped!
[Test worker] INFO org.apache.spark.storage.memory.MemoryStore -
MemoryStore cleared
[Test worker] INFO org.apache.spark.storage.BlockManager - BlockManager
stopped
[Test worker] INFO org.apache.spark.storage.BlockManagerMaster -
BlockManagerMaster stopped
[dispatcher-event-loop-1] INFO
org.apache.spark.scheduler.OutputCommitCoordinator$OutputCommitCoordinatorEndpoint
- OutputCommitCoordinator stopped!
[Test worker] INFO org.apache.spark.SparkContext - Successfully stopped
SparkContext
Gradle Test Executor 277 finished executing tests.
> Task :beam-runners-spark:validatesRunnerStreaming
[Thread-4] INFO org.apache.spark.util.ShutdownHookManager - Shutdown hook called
[Thread-4] INFO org.apache.spark.util.ShutdownHookManager - Deleting directory
/tmp/spark-5cfffacb-c846-4571-a47e-a459dcc35630
Finished generating test XML results (0.095 secs) into:
<https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark_Gradle/ws/src/runners/spark/build/test-results/validatesRunnerStreaming>
Generating HTML test report...
Finished generating test html results (0.097 secs) into:
<https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark_Gradle/ws/src/runners/spark/build/reports/tests/validatesRunnerStreaming>
Packing task ':beam-runners-spark:validatesRunnerStreaming'
:beam-runners-spark:validatesRunnerStreaming (Thread[Task worker for ':' Thread
14,5,main]) completed. Took 59.532 secs.
FAILURE: Build failed with an exception.
* What went wrong:
Execution failed for task ':beam-runners-spark:validatesRunnerBatch'.
> There were failing tests. See the report at:
> file://<https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark_Gradle/ws/src/runners/spark/build/reports/tests/validatesRunnerBatch/index.html>
* Try:
Run with --stacktrace option to get the stack trace. Run with --debug option to
get more log output. Run with --scan to get full insights.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with
Gradle 5.0.
See
https://docs.gradle.org/4.8/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 5m 24s
39 actionable tasks: 35 executed, 4 from cache
Publishing build scan...
https://gradle.com/s/kdkf6cmtigxco
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
Recording test results
