hansva commented on issue #4752: URL: https://github.com/apache/hop/issues/4752#issuecomment-2569139330
seems to be working here... ``` Downloads/spark-3.5.4-bin-hadoop3/bin on ☁️ (eu-west-1) on ☁️ [email protected] took 16s ❯ ./spark-submit \ --master spark://MacBook-Hans.local:7077 \ --class org.apache.hop.beam.run.MainBeam \ --driver-java-options '-DPROJECT_HOME=/Users/hans/git/hop/assemblies/client/target/hop/config/projects/samples' \ /Users/hans/git/hop/assemblies/client/target/hop/fat-jar.jar \ /Users/hans/git/hop/assemblies/client/target/hop/config/projects/samples/beam/pipelines/input-process-output.hpl \ /Users/hans/git/hop/assemblies/client/target/hop/metadata.json \ Spark 25/01/03 13:16:56 WARN Utils: Your hostname, MacBook-Hans.local resolves to a loopback address: 127.0.0.1; using 192.168.86.198 instead (on interface en0) 25/01/03 13:16:56 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another address >>>>>> Initializing Hop Hop configuration file not found, not serializing: /Users/hans/Downloads/spark-3.5.4-bin-hadoop3/bin/config/hop-config.json Argument 1 : Pipeline filename (.hpl) : /Users/hans/git/hop/assemblies/client/target/hop/config/projects/samples/beam/pipelines/input-process-output.hpl Argument 2 : Environment state filename: (.json) : /Users/hans/git/hop/assemblies/client/target/hop/metadata.json Argument 3 : Pipeline run configuration : Spark >>>>>> Loading pipeline metadata >>>>>> Building Apache Beam Pipeline... >>>>>> Pipeline executing starting... 2025/01/03 13:17:01 - General - Created Apache Beam pipeline with name 'input-process-output' 2025/01/03 13:17:01 - General - Handled transform (INPUT) : Customers 2025/01/03 13:17:01 - General - Handled generic transform (TRANSFORM) : Only CA, gets data from 1 previous transform(s), targets=0, infos=0 2025/01/03 13:17:01 - General - Handled generic transform (TRANSFORM) : Limit fields, re-order, gets data from 1 previous transform(s), targets=0, infos=0 2025/01/03 13:17:01 - General - Handled transform (OUTPUT) : input-process-output, gets data from Limit fields, re-order 2025/01/03 13:17:01 - General - Executing this pipeline using the Beam Pipeline Engine with run configuration 'Spark' 25/01/03 13:17:01 INFO SparkRunner: Executing pipeline using the SparkRunner. 25/01/03 13:17:01 INFO SparkContextFactory: Creating a brand new Spark Context. 25/01/03 13:17:01 INFO SparkContext: Running Spark version 3.5.4 25/01/03 13:17:01 INFO SparkContext: OS info Mac OS X, 15.2, aarch64 25/01/03 13:17:01 INFO SparkContext: Java version 17.0.11 25/01/03 13:17:01 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 25/01/03 13:17:02 INFO ResourceUtils: ============================================================== 25/01/03 13:17:02 INFO ResourceUtils: No custom resources configured for spark.driver. 25/01/03 13:17:02 INFO ResourceUtils: ============================================================== 25/01/03 13:17:02 INFO SparkContext: Submitted application: BeamSparkPipelineRunConfiguration 25/01/03 13:17:02 INFO ResourceProfile: Default ResourceProfile created, executor resources: Map(memory -> name: memory, amount: 1024, script: , vendor: , offHeap -> name: offHeap, amount: 0, script: , vendor: ), task resources: Map(cpus -> name: cpus, amount: 1.0) 25/01/03 13:17:02 INFO ResourceProfile: Limiting resource is cpu 25/01/03 13:17:02 INFO ResourceProfileManager: Added ResourceProfile id: 0 25/01/03 13:17:02 INFO SecurityManager: Changing view acls to: hans 25/01/03 13:17:02 INFO SecurityManager: Changing modify acls to: hans 25/01/03 13:17:02 INFO SecurityManager: Changing view acls groups to: 25/01/03 13:17:02 INFO SecurityManager: Changing modify acls groups to: 25/01/03 13:17:02 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: hans; groups with view permissions: EMPTY; users with modify permissions: hans; groups with modify permissions: EMPTY 25/01/03 13:17:02 INFO Utils: Successfully started service 'sparkDriver' on port 55057. 25/01/03 13:17:02 INFO SparkEnv: Registering MapOutputTracker 25/01/03 13:17:02 INFO SparkEnv: Registering BlockManagerMaster 25/01/03 13:17:02 INFO BlockManagerMasterEndpoint: Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information 25/01/03 13:17:02 INFO BlockManagerMasterEndpoint: BlockManagerMasterEndpoint up 25/01/03 13:17:02 INFO SparkEnv: Registering BlockManagerMasterHeartbeat 25/01/03 13:17:02 INFO DiskBlockManager: Created local directory at /private/var/folders/v0/hjm6kpgd0076ncdkpf8rg3dw0000gn/T/blockmgr-58d26b5c-d34a-4682-8fe7-b5beb2ab0873 25/01/03 13:17:02 INFO MemoryStore: MemoryStore started with capacity 434.4 MiB 25/01/03 13:17:02 INFO SparkEnv: Registering OutputCommitCoordinator 25/01/03 13:17:02 INFO JettyUtils: Start Jetty 0.0.0.0:4040 for SparkUI 25/01/03 13:17:02 INFO Utils: Successfully started service 'SparkUI' on port 4040. 25/01/03 13:17:02 INFO SparkContext: Added JAR /Users/hans/git/hop/assemblies/client/target/hop/fat-jar.jar at spark://192.168.86.198:55057/jars/fat-jar.jar with timestamp 1735906621944 25/01/03 13:17:02 INFO StandaloneAppClient$ClientEndpoint: Connecting to master spark://MacBook-Hans.local:7077... 25/01/03 13:17:02 INFO TransportClientFactory: Successfully created connection to MacBook-Hans.local/127.0.0.1:7077 after 14 ms (0 ms spent in bootstraps) 25/01/03 13:17:02 INFO StandaloneSchedulerBackend: Connected to Spark cluster with app ID app-20250103131702-0002 25/01/03 13:17:02 INFO StandaloneAppClient$ClientEndpoint: Executor added: app-20250103131702-0002/0 on worker-20250103125214-192.168.86.198-54182 (192.168.86.198:54182) with 12 core(s) 25/01/03 13:17:02 INFO StandaloneSchedulerBackend: Granted executor ID app-20250103131702-0002/0 on hostPort 192.168.86.198:54182 with 12 core(s), 1024.0 MiB RAM 25/01/03 13:17:02 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 55059. 25/01/03 13:17:02 INFO NettyBlockTransferService: Server created on 192.168.86.198:55059 25/01/03 13:17:02 INFO BlockManager: Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy 25/01/03 13:17:02 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(driver, 192.168.86.198, 55059, None) 25/01/03 13:17:02 INFO BlockManagerMasterEndpoint: Registering block manager 192.168.86.198:55059 with 434.4 MiB RAM, BlockManagerId(driver, 192.168.86.198, 55059, None) 25/01/03 13:17:02 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(driver, 192.168.86.198, 55059, None) 25/01/03 13:17:02 INFO BlockManager: Initialized BlockManager: BlockManagerId(driver, 192.168.86.198, 55059, None) 25/01/03 13:17:02 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20250103131702-0002/0 is now RUNNING 25/01/03 13:17:02 INFO StandaloneSchedulerBackend: SchedulerBackend is ready for scheduling beginning after reached minRegisteredResourcesRatio: 0.0 25/01/03 13:17:02 INFO SparkRunner$Evaluator: Entering directly-translatable composite transform: 'BeamOutputTransform/TextIO.Write/WriteFiles/GatherTempFileResults/Consolidate/Reshuffle' 25/01/03 13:17:02 INFO SparkRunner$Evaluator: Entering directly-translatable composite transform: 'BeamOutputTransform/TextIO.Write/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle' 25/01/03 13:17:02 INFO MetricsAccumulator: Instantiated metrics accumulator: MetricQueryResults() 25/01/03 13:17:02 INFO SparkRunner$Evaluator: Evaluating Read(CompressedSource) 25/01/03 13:17:02 INFO SparkRunner$Evaluator: Evaluating org.apache.hop.beam.core.fn.StringToHopFn@7aa06fac 25/01/03 13:17:02 INFO SparkRunner$Evaluator: Evaluating org.apache.hop.beam.core.transform.TransformFn@2ea82cef 25/01/03 13:17:02 INFO SparkRunner$Evaluator: Evaluating org.apache.hop.beam.core.transform.TransformFn@5cea34e6 25/01/03 13:17:02 INFO SparkRunner$Evaluator: Evaluating org.apache.hop.beam.core.fn.HopToStringFn@526fbb80 25/01/03 13:17:02 INFO SparkRunner$Evaluator: Evaluating Window.Assign 25/01/03 13:17:02 INFO SparkRunner$Evaluator: Evaluating org.apache.beam.sdk.io.WriteFiles$WriteUnshardedTempFilesFn@2162e4a 25/01/03 13:17:02 INFO SparkRunner$Evaluator: Evaluating GroupByKey 25/01/03 13:17:02 INFO FileBasedSource: Splitting filepattern /Users/hans/git/hop/assemblies/client/target/hop/config/projects/samples/beam/input/customers-noheader-1k.txt into bundles of size 50000 took 3 ms and produced 1 files and 2 bundles 25/01/03 13:17:02 INFO SparkRunner$Evaluator: Evaluating org.apache.beam.sdk.io.WriteFiles$WriteShardsIntoTempFilesFn@3199c2c1 25/01/03 13:17:02 INFO SparkRunner$Evaluator: Evaluating org.apache.beam.sdk.io.WriteFiles$WriteUnshardedBundlesToTempFiles$1@7ae736a8 25/01/03 13:17:02 INFO SparkRunner$Evaluator: Evaluating Flatten.PCollections 25/01/03 13:17:02 INFO SparkRunner$Evaluator: Evaluating org.apache.beam.sdk.transforms.Reshuffle$AssignShardFn@4623c0d3 25/01/03 13:17:02 INFO SparkRunner$Evaluator: Entering directly-translatable composite transform: 'BeamOutputTransform/TextIO.Write/WriteFiles/GatherTempFileResults/Consolidate/Reshuffle' 25/01/03 13:17:02 INFO SparkRunner$Evaluator: Evaluating Reshuffle 25/01/03 13:17:03 INFO SparkRunner$Evaluator: Evaluating org.apache.beam.sdk.transforms.MapElements$2@2182ebc7 25/01/03 13:17:03 INFO SparkRunner$Evaluator: Evaluating org.apache.beam.sdk.transforms.MapElements$2@447cdbaa 25/01/03 13:17:03 INFO SparkRunner$Evaluator: Evaluating View.CreatePCollectionView 25/01/03 13:17:03 INFO SparkContext: Starting job: collect at BoundedDataset.java:96 25/01/03 13:17:03 INFO DAGScheduler: Registering RDD 20 (mapToPair at GroupNonMergingWindowsFunctions.java:273) as input to shuffle 1 25/01/03 13:17:03 INFO DAGScheduler: Registering RDD 37 (repartition at GroupCombineFunctions.java:191) as input to shuffle 0 25/01/03 13:17:03 INFO DAGScheduler: Got job 0 (collect at BoundedDataset.java:96) with 4 output partitions 25/01/03 13:17:03 INFO DAGScheduler: Final stage: ResultStage 2 (collect at BoundedDataset.java:96) 25/01/03 13:17:03 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 1) 25/01/03 13:17:03 INFO DAGScheduler: Missing parents: List(ShuffleMapStage 1) 25/01/03 13:17:03 INFO DAGScheduler: Submitting ShuffleMapStage 0 (MapPartitionsRDD[20] at mapToPair at GroupNonMergingWindowsFunctions.java:273), which has no missing parents 25/01/03 13:17:03 INFO MemoryStore: Block broadcast_0 stored as values in memory (estimated size 215.7 KiB, free 434.2 MiB) 25/01/03 13:17:03 INFO MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 69.0 KiB, free 434.1 MiB) 25/01/03 13:17:03 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on 192.168.86.198:55059 (size: 69.0 KiB, free: 434.3 MiB) 25/01/03 13:17:03 INFO SparkContext: Created broadcast 0 from broadcast at DAGScheduler.scala:1585 25/01/03 13:17:03 INFO DAGScheduler: Submitting 2 missing tasks from ShuffleMapStage 0 (MapPartitionsRDD[20] at mapToPair at GroupNonMergingWindowsFunctions.java:273) (first 15 tasks are for partitions Vector(0, 1)) 25/01/03 13:17:03 INFO TaskSchedulerImpl: Adding task set 0.0 with 2 tasks resource profile 0 25/01/03 13:17:03 INFO StandaloneSchedulerBackend$StandaloneDriverEndpoint: Registered executor NettyRpcEndpointRef(spark-client://Executor) (192.168.86.198:55061) with ID 0, ResourceProfileId 0 25/01/03 13:17:03 INFO BlockManagerMasterEndpoint: Registering block manager 192.168.86.198:55063 with 434.4 MiB RAM, BlockManagerId(0, 192.168.86.198, 55063, None) 25/01/03 13:17:05 INFO TaskSetManager: Starting task 0.0 in stage 0.0 (TID 0) (192.168.86.198, executor 0, partition 0, PROCESS_LOCAL, 10914 bytes) 25/01/03 13:17:05 INFO TaskSetManager: Starting task 1.0 in stage 0.0 (TID 1) (192.168.86.198, executor 0, partition 1, PROCESS_LOCAL, 10914 bytes) 25/01/03 13:17:05 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on 192.168.86.198:55063 (size: 69.0 KiB, free: 434.3 MiB) 25/01/03 13:17:10 INFO BlockManagerInfo: Added rdd_13_1 in memory on 192.168.86.198:55063 (size: 616.0 B, free: 434.3 MiB) 25/01/03 13:17:10 INFO BlockManagerInfo: Added rdd_13_0 in memory on 192.168.86.198:55063 (size: 616.0 B, free: 434.3 MiB) 25/01/03 13:17:10 INFO TaskSetManager: Finished task 1.0 in stage 0.0 (TID 1) in 5259 ms on 192.168.86.198 (executor 0) (1/2) 25/01/03 13:17:10 INFO TaskSetManager: Finished task 0.0 in stage 0.0 (TID 0) in 5267 ms on 192.168.86.198 (executor 0) (2/2) 25/01/03 13:17:10 INFO TaskSchedulerImpl: Removed TaskSet 0.0, whose tasks have all completed, from pool 25/01/03 13:17:10 INFO DAGScheduler: ShuffleMapStage 0 (mapToPair at GroupNonMergingWindowsFunctions.java:273) finished in 7,352 s 25/01/03 13:17:10 INFO DAGScheduler: looking for newly runnable stages 25/01/03 13:17:10 INFO DAGScheduler: running: Set() 25/01/03 13:17:10 INFO DAGScheduler: waiting: Set(ShuffleMapStage 1, ResultStage 2) 25/01/03 13:17:10 INFO DAGScheduler: failed: Set() 25/01/03 13:17:10 INFO DAGScheduler: Submitting ShuffleMapStage 1 (MapPartitionsRDD[37] at repartition at GroupCombineFunctions.java:191), which has no missing parents 25/01/03 13:17:10 INFO MemoryStore: Block broadcast_1 stored as values in memory (estimated size 225.4 KiB, free 433.9 MiB) 25/01/03 13:17:10 INFO MemoryStore: Block broadcast_1_piece0 stored as bytes in memory (estimated size 72.3 KiB, free 433.8 MiB) 25/01/03 13:17:10 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on 192.168.86.198:55059 (size: 72.3 KiB, free: 434.3 MiB) 25/01/03 13:17:10 INFO SparkContext: Created broadcast 1 from broadcast at DAGScheduler.scala:1585 25/01/03 13:17:10 INFO DAGScheduler: Submitting 4 missing tasks from ShuffleMapStage 1 (MapPartitionsRDD[37] at repartition at GroupCombineFunctions.java:191) (first 15 tasks are for partitions Vector(0, 1, 2, 3)) 25/01/03 13:17:10 INFO TaskSchedulerImpl: Adding task set 1.0 with 4 tasks resource profile 0 25/01/03 13:17:10 INFO TaskSetManager: Starting task 0.0 in stage 1.0 (TID 2) (192.168.86.198, executor 0, partition 0, PROCESS_LOCAL, 11023 bytes) 25/01/03 13:17:10 INFO TaskSetManager: Starting task 1.0 in stage 1.0 (TID 3) (192.168.86.198, executor 0, partition 1, PROCESS_LOCAL, 11023 bytes) 25/01/03 13:17:10 INFO TaskSetManager: Starting task 2.0 in stage 1.0 (TID 4) (192.168.86.198, executor 0, partition 2, PROCESS_LOCAL, 9091 bytes) 25/01/03 13:17:10 INFO TaskSetManager: Starting task 3.0 in stage 1.0 (TID 5) (192.168.86.198, executor 0, partition 3, PROCESS_LOCAL, 9091 bytes) 25/01/03 13:17:10 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on 192.168.86.198:55063 (size: 72.3 KiB, free: 434.3 MiB) 25/01/03 13:17:10 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 1 to 192.168.86.198:55061 25/01/03 13:17:10 INFO TaskSetManager: Finished task 0.0 in stage 1.0 (TID 2) in 85 ms on 192.168.86.198 (executor 0) (1/4) 25/01/03 13:17:10 INFO TaskSetManager: Finished task 1.0 in stage 1.0 (TID 3) in 84 ms on 192.168.86.198 (executor 0) (2/4) 25/01/03 13:17:10 INFO BlockManagerInfo: Added rdd_24_1 in memory on 192.168.86.198:55063 (size: 16.0 B, free: 434.3 MiB) 25/01/03 13:17:10 INFO BlockManagerInfo: Added rdd_24_0 in memory on 192.168.86.198:55063 (size: 16.0 B, free: 434.3 MiB) 25/01/03 13:17:10 INFO TaskSetManager: Finished task 3.0 in stage 1.0 (TID 5) in 499 ms on 192.168.86.198 (executor 0) (3/4) 25/01/03 13:17:10 INFO TaskSetManager: Finished task 2.0 in stage 1.0 (TID 4) in 499 ms on 192.168.86.198 (executor 0) (4/4) 25/01/03 13:17:10 INFO TaskSchedulerImpl: Removed TaskSet 1.0, whose tasks have all completed, from pool 25/01/03 13:17:10 INFO DAGScheduler: ShuffleMapStage 1 (repartition at GroupCombineFunctions.java:191) finished in 0,514 s 25/01/03 13:17:10 INFO DAGScheduler: looking for newly runnable stages 25/01/03 13:17:10 INFO DAGScheduler: running: Set() 25/01/03 13:17:10 INFO DAGScheduler: waiting: Set(ResultStage 2) 25/01/03 13:17:10 INFO DAGScheduler: failed: Set() 25/01/03 13:17:10 INFO DAGScheduler: Submitting ResultStage 2 (MapPartitionsRDD[48] at map at BoundedDataset.java:95), which has no missing parents 25/01/03 13:17:10 INFO MemoryStore: Block broadcast_2 stored as values in memory (estimated size 21.4 KiB, free 433.8 MiB) 25/01/03 13:17:10 INFO MemoryStore: Block broadcast_2_piece0 stored as bytes in memory (estimated size 9.0 KiB, free 433.8 MiB) 25/01/03 13:17:10 INFO BlockManagerInfo: Added broadcast_2_piece0 in memory on 192.168.86.198:55059 (size: 9.0 KiB, free: 434.3 MiB) 25/01/03 13:17:10 INFO SparkContext: Created broadcast 2 from broadcast at DAGScheduler.scala:1585 25/01/03 13:17:10 INFO DAGScheduler: Submitting 4 missing tasks from ResultStage 2 (MapPartitionsRDD[48] at map at BoundedDataset.java:95) (first 15 tasks are for partitions Vector(0, 1, 2, 3)) 25/01/03 13:17:10 INFO TaskSchedulerImpl: Adding task set 2.0 with 4 tasks resource profile 0 25/01/03 13:17:10 INFO TaskSetManager: Starting task 1.0 in stage 2.0 (TID 6) (192.168.86.198, executor 0, partition 1, NODE_LOCAL, 9269 bytes) 25/01/03 13:17:10 INFO TaskSetManager: Starting task 0.0 in stage 2.0 (TID 7) (192.168.86.198, executor 0, partition 0, PROCESS_LOCAL, 9269 bytes) 25/01/03 13:17:10 INFO TaskSetManager: Starting task 2.0 in stage 2.0 (TID 8) (192.168.86.198, executor 0, partition 2, PROCESS_LOCAL, 9269 bytes) 25/01/03 13:17:10 INFO TaskSetManager: Starting task 3.0 in stage 2.0 (TID 9) (192.168.86.198, executor 0, partition 3, PROCESS_LOCAL, 9269 bytes) 25/01/03 13:17:10 INFO BlockManagerInfo: Added broadcast_2_piece0 in memory on 192.168.86.198:55063 (size: 9.0 KiB, free: 434.3 MiB) 25/01/03 13:17:11 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 0 to 192.168.86.198:55061 25/01/03 13:17:11 INFO BlockManagerInfo: Added rdd_44_0 in memory on 192.168.86.198:55063 (size: 16.0 B, free: 434.3 MiB) 25/01/03 13:17:11 INFO BlockManagerInfo: Added rdd_44_3 in memory on 192.168.86.198:55063 (size: 16.0 B, free: 434.3 MiB) 25/01/03 13:17:11 INFO BlockManagerInfo: Added rdd_44_2 in memory on 192.168.86.198:55063 (size: 16.0 B, free: 434.3 MiB) 25/01/03 13:17:11 INFO TaskSetManager: Finished task 3.0 in stage 2.0 (TID 9) in 62 ms on 192.168.86.198 (executor 0) (1/4) 25/01/03 13:17:11 INFO TaskSetManager: Finished task 2.0 in stage 2.0 (TID 8) in 62 ms on 192.168.86.198 (executor 0) (2/4) 25/01/03 13:17:11 INFO TaskSetManager: Finished task 0.0 in stage 2.0 (TID 7) in 63 ms on 192.168.86.198 (executor 0) (3/4) 25/01/03 13:17:11 INFO BlockManagerInfo: Added rdd_44_1 in memory on 192.168.86.198:55063 (size: 808.0 B, free: 434.3 MiB) 25/01/03 13:17:11 INFO TaskSetManager: Finished task 1.0 in stage 2.0 (TID 6) in 80 ms on 192.168.86.198 (executor 0) (4/4) 25/01/03 13:17:11 INFO TaskSchedulerImpl: Removed TaskSet 2.0, whose tasks have all completed, from pool 25/01/03 13:17:11 INFO DAGScheduler: ResultStage 2 (collect at BoundedDataset.java:96) finished in 0,087 s 25/01/03 13:17:11 INFO DAGScheduler: Job 0 is finished. Cancelling potential speculative or zombie tasks for this job 25/01/03 13:17:11 INFO TaskSchedulerImpl: Killing all running tasks in stage 2: Stage finished 25/01/03 13:17:11 INFO DAGScheduler: Job 0 finished: collect at BoundedDataset.java:96, took 8,010176 s 25/01/03 13:17:11 INFO SparkRunner$Evaluator: Evaluating Impulse 25/01/03 13:17:11 INFO SparkRunner$Evaluator: Evaluating org.apache.beam.sdk.transforms.MapElements$2@685e8e17 25/01/03 13:17:11 INFO SparkRunner$Evaluator: Evaluating org.apache.beam.sdk.transforms.Reify$ReifyView$1@5d6e77a4 25/01/03 13:17:11 INFO MemoryStore: Block broadcast_3 stored as values in memory (estimated size 448.0 B, free 433.8 MiB) 25/01/03 13:17:11 INFO MemoryStore: Block broadcast_3_piece0 stored as bytes in memory (estimated size 330.0 B, free 433.8 MiB) 25/01/03 13:17:11 INFO BlockManagerInfo: Added broadcast_3_piece0 in memory on 192.168.86.198:55059 (size: 330.0 B, free: 434.3 MiB) 25/01/03 13:17:11 INFO SparkContext: Created broadcast 3 from broadcast at SideInputBroadcast.java:62 25/01/03 13:17:11 INFO SparkRunner$Evaluator: Evaluating org.apache.beam.sdk.transforms.MapElements$2@72139933 25/01/03 13:17:11 INFO SparkRunner$Evaluator: Evaluating org.apache.beam.sdk.transforms.MapElements$2@4f0b02a3 25/01/03 13:17:11 INFO SparkRunner$Evaluator: Evaluating org.apache.beam.sdk.io.WriteFiles$FinalizeTempFileBundles$FinalizeFn@280d1d8d 25/01/03 13:17:11 INFO SparkRunner$Evaluator: Evaluating org.apache.beam.sdk.transforms.Reshuffle$AssignShardFn@5abfb698 25/01/03 13:17:11 INFO SparkRunner$Evaluator: Entering directly-translatable composite transform: 'BeamOutputTransform/TextIO.Write/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle' 25/01/03 13:17:11 INFO SparkRunner$Evaluator: Evaluating Reshuffle 25/01/03 13:17:11 INFO SparkRunner$Evaluator: Evaluating org.apache.beam.sdk.transforms.MapElements$2@64a0a1c6 25/01/03 13:17:11 INFO SparkContext: Starting job: foreach at BoundedDataset.java:127 25/01/03 13:17:11 INFO DAGScheduler: Got job 1 (foreach at BoundedDataset.java:127) with 2 output partitions 25/01/03 13:17:11 INFO DAGScheduler: Final stage: ResultStage 3 (foreach at BoundedDataset.java:127) 25/01/03 13:17:11 INFO DAGScheduler: Parents of final stage: List() 25/01/03 13:17:11 INFO DAGScheduler: Missing parents: List() 25/01/03 13:17:11 INFO DAGScheduler: Submitting ResultStage 3 (BeamOutputTransform/TextIO.Write/WriteFiles/WriteUnshardedBundlesToTempFiles/WriteUnshardedBundles.out2 MapPartitionsRDD[19] at values at TransformTranslator.java:472), which has no missing parents 25/01/03 13:17:11 INFO MemoryStore: Block broadcast_4 stored as values in memory (estimated size 213.7 KiB, free 433.6 MiB) 25/01/03 13:17:11 INFO MemoryStore: Block broadcast_4_piece0 stored as bytes in memory (estimated size 68.1 KiB, free 433.5 MiB) 25/01/03 13:17:11 INFO BlockManagerInfo: Added broadcast_4_piece0 in memory on 192.168.86.198:55059 (size: 68.1 KiB, free: 434.2 MiB) 25/01/03 13:17:11 INFO SparkContext: Created broadcast 4 from broadcast at DAGScheduler.scala:1585 25/01/03 13:17:11 INFO DAGScheduler: Submitting 2 missing tasks from ResultStage 3 (BeamOutputTransform/TextIO.Write/WriteFiles/WriteUnshardedBundlesToTempFiles/WriteUnshardedBundles.out2 MapPartitionsRDD[19] at values at TransformTranslator.java:472) (first 15 tasks are for partitions Vector(0, 1)) 25/01/03 13:17:11 INFO TaskSchedulerImpl: Adding task set 3.0 with 2 tasks resource profile 0 25/01/03 13:17:11 INFO TaskSetManager: Starting task 0.0 in stage 3.0 (TID 10) (192.168.86.198, executor 0, partition 0, PROCESS_LOCAL, 10925 bytes) 25/01/03 13:17:11 INFO TaskSetManager: Starting task 1.0 in stage 3.0 (TID 11) (192.168.86.198, executor 0, partition 1, PROCESS_LOCAL, 10925 bytes) 25/01/03 13:17:11 INFO BlockManagerInfo: Added broadcast_4_piece0 in memory on 192.168.86.198:55063 (size: 68.1 KiB, free: 434.2 MiB) 25/01/03 13:17:11 INFO TaskSetManager: Finished task 1.0 in stage 3.0 (TID 11) in 27 ms on 192.168.86.198 (executor 0) (1/2) 25/01/03 13:17:11 INFO TaskSetManager: Finished task 0.0 in stage 3.0 (TID 10) in 28 ms on 192.168.86.198 (executor 0) (2/2) 25/01/03 13:17:11 INFO TaskSchedulerImpl: Removed TaskSet 3.0, whose tasks have all completed, from pool 25/01/03 13:17:11 INFO DAGScheduler: ResultStage 3 (foreach at BoundedDataset.java:127) finished in 0,033 s 25/01/03 13:17:11 INFO DAGScheduler: Job 1 is finished. Cancelling potential speculative or zombie tasks for this job 25/01/03 13:17:11 INFO TaskSchedulerImpl: Killing all running tasks in stage 3: Stage finished 25/01/03 13:17:11 INFO DAGScheduler: Job 1 finished: foreach at BoundedDataset.java:127, took 0,035680 s 25/01/03 13:17:11 INFO SparkContext: Starting job: foreach at BoundedDataset.java:127 25/01/03 13:17:11 INFO DAGScheduler: Got job 2 (foreach at BoundedDataset.java:127) with 2 output partitions 25/01/03 13:17:11 INFO DAGScheduler: Final stage: ResultStage 5 (foreach at BoundedDataset.java:127) 25/01/03 13:17:11 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 4) 25/01/03 13:17:11 INFO DAGScheduler: Missing parents: List() 25/01/03 13:17:11 INFO DAGScheduler: Submitting ResultStage 5 (BeamOutputTransform/TextIO.Write/WriteFiles/WriteUnshardedBundlesToTempFiles/WriteUnwritten.out1 MapPartitionsRDD[28] at values at TransformTranslator.java:472), which has no missing parents 25/01/03 13:17:11 INFO MemoryStore: Block broadcast_5 stored as values in memory (estimated size 221.9 KiB, free 433.3 MiB) 25/01/03 13:17:11 INFO MemoryStore: Block broadcast_5_piece0 stored as bytes in memory (estimated size 71.3 KiB, free 433.2 MiB) 25/01/03 13:17:11 INFO BlockManagerInfo: Added broadcast_5_piece0 in memory on 192.168.86.198:55059 (size: 71.3 KiB, free: 434.1 MiB) 25/01/03 13:17:11 INFO SparkContext: Created broadcast 5 from broadcast at DAGScheduler.scala:1585 25/01/03 13:17:11 INFO DAGScheduler: Submitting 2 missing tasks from ResultStage 5 (BeamOutputTransform/TextIO.Write/WriteFiles/WriteUnshardedBundlesToTempFiles/WriteUnwritten.out1 MapPartitionsRDD[28] at values at TransformTranslator.java:472) (first 15 tasks are for partitions Vector(0, 1)) 25/01/03 13:17:11 INFO TaskSchedulerImpl: Adding task set 5.0 with 2 tasks resource profile 0 25/01/03 13:17:11 INFO TaskSetManager: Starting task 0.0 in stage 5.0 (TID 12) (192.168.86.198, executor 0, partition 0, PROCESS_LOCAL, 8993 bytes) 25/01/03 13:17:11 INFO TaskSetManager: Starting task 1.0 in stage 5.0 (TID 13) (192.168.86.198, executor 0, partition 1, PROCESS_LOCAL, 8993 bytes) 25/01/03 13:17:11 INFO BlockManagerInfo: Added broadcast_5_piece0 in memory on 192.168.86.198:55063 (size: 71.3 KiB, free: 434.1 MiB) 25/01/03 13:17:11 INFO TaskSetManager: Finished task 1.0 in stage 5.0 (TID 13) in 18 ms on 192.168.86.198 (executor 0) (1/2) 25/01/03 13:17:11 INFO TaskSetManager: Finished task 0.0 in stage 5.0 (TID 12) in 18 ms on 192.168.86.198 (executor 0) (2/2) 25/01/03 13:17:11 INFO TaskSchedulerImpl: Removed TaskSet 5.0, whose tasks have all completed, from pool 25/01/03 13:17:11 INFO DAGScheduler: ResultStage 5 (foreach at BoundedDataset.java:127) finished in 0,024 s 25/01/03 13:17:11 INFO DAGScheduler: Job 2 is finished. Cancelling potential speculative or zombie tasks for this job 25/01/03 13:17:11 INFO TaskSchedulerImpl: Killing all running tasks in stage 5: Stage finished 25/01/03 13:17:11 INFO DAGScheduler: Job 2 finished: foreach at BoundedDataset.java:127, took 0,027143 s 25/01/03 13:17:11 INFO SparkContext: Starting job: foreach at BoundedDataset.java:127 25/01/03 13:17:11 INFO DAGScheduler: Got job 3 (foreach at BoundedDataset.java:127) with 4 output partitions 25/01/03 13:17:11 INFO DAGScheduler: Final stage: ResultStage 8 (foreach at BoundedDataset.java:127) 25/01/03 13:17:11 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 7) 25/01/03 13:17:11 INFO DAGScheduler: Missing parents: List() 25/01/03 13:17:11 INFO DAGScheduler: Submitting ResultStage 8 (BeamOutputTransform/TextIO.Write/WriteFiles/GatherTempFileResults/View.AsIterable/MapElements/Map/ParMultiDo(Anonymous).output MapPartitionsRDD[47] at values at TransformTranslator.java:472), which has no missing parents 25/01/03 13:17:11 INFO MemoryStore: Block broadcast_6 stored as values in memory (estimated size 22.5 KiB, free 433.2 MiB) 25/01/03 13:17:11 INFO MemoryStore: Block broadcast_6_piece0 stored as bytes in memory (estimated size 9.4 KiB, free 433.2 MiB) 25/01/03 13:17:11 INFO BlockManagerInfo: Added broadcast_6_piece0 in memory on 192.168.86.198:55059 (size: 9.4 KiB, free: 434.1 MiB) 25/01/03 13:17:11 INFO SparkContext: Created broadcast 6 from broadcast at DAGScheduler.scala:1585 25/01/03 13:17:11 INFO DAGScheduler: Submitting 4 missing tasks from ResultStage 8 (BeamOutputTransform/TextIO.Write/WriteFiles/GatherTempFileResults/View.AsIterable/MapElements/Map/ParMultiDo(Anonymous).output MapPartitionsRDD[47] at values at TransformTranslator.java:472) (first 15 tasks are for partitions Vector(0, 1, 2, 3)) 25/01/03 13:17:11 INFO TaskSchedulerImpl: Adding task set 8.0 with 4 tasks resource profile 0 25/01/03 13:17:11 INFO TaskSetManager: Starting task 0.0 in stage 8.0 (TID 14) (192.168.86.198, executor 0, partition 0, PROCESS_LOCAL, 9269 bytes) 25/01/03 13:17:11 INFO TaskSetManager: Starting task 1.0 in stage 8.0 (TID 15) (192.168.86.198, executor 0, partition 1, PROCESS_LOCAL, 9269 bytes) 25/01/03 13:17:11 INFO TaskSetManager: Starting task 2.0 in stage 8.0 (TID 16) (192.168.86.198, executor 0, partition 2, PROCESS_LOCAL, 9269 bytes) 25/01/03 13:17:11 INFO TaskSetManager: Starting task 3.0 in stage 8.0 (TID 17) (192.168.86.198, executor 0, partition 3, PROCESS_LOCAL, 9269 bytes) 25/01/03 13:17:11 INFO BlockManagerInfo: Added broadcast_6_piece0 in memory on 192.168.86.198:55063 (size: 9.4 KiB, free: 434.1 MiB) 25/01/03 13:17:11 INFO TaskSetManager: Finished task 2.0 in stage 8.0 (TID 16) in 17 ms on 192.168.86.198 (executor 0) (1/4) 25/01/03 13:17:11 INFO TaskSetManager: Finished task 0.0 in stage 8.0 (TID 14) in 19 ms on 192.168.86.198 (executor 0) (2/4) 25/01/03 13:17:11 INFO TaskSetManager: Finished task 3.0 in stage 8.0 (TID 17) in 18 ms on 192.168.86.198 (executor 0) (3/4) 25/01/03 13:17:11 INFO TaskSetManager: Finished task 1.0 in stage 8.0 (TID 15) in 18 ms on 192.168.86.198 (executor 0) (4/4) 25/01/03 13:17:11 INFO TaskSchedulerImpl: Removed TaskSet 8.0, whose tasks have all completed, from pool 25/01/03 13:17:11 INFO DAGScheduler: ResultStage 8 (foreach at BoundedDataset.java:127) finished in 0,023 s 25/01/03 13:17:11 INFO DAGScheduler: Job 3 is finished. Cancelling potential speculative or zombie tasks for this job 25/01/03 13:17:11 INFO TaskSchedulerImpl: Killing all running tasks in stage 8: Stage finished 25/01/03 13:17:11 INFO DAGScheduler: Job 3 finished: foreach at BoundedDataset.java:127, took 0,027300 s 25/01/03 13:17:11 INFO SparkContext: Starting job: foreach at BoundedDataset.java:127 25/01/03 13:17:11 INFO DAGScheduler: Registering RDD 70 (repartition at GroupCombineFunctions.java:191) as input to shuffle 2 25/01/03 13:17:11 INFO DAGScheduler: Got job 4 (foreach at BoundedDataset.java:127) with 12 output partitions 25/01/03 13:17:11 INFO DAGScheduler: Final stage: ResultStage 10 (foreach at BoundedDataset.java:127) 25/01/03 13:17:11 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 9) 25/01/03 13:17:11 INFO DAGScheduler: Missing parents: List(ShuffleMapStage 9) 25/01/03 13:17:11 INFO DAGScheduler: Submitting ShuffleMapStage 9 (MapPartitionsRDD[70] at repartition at GroupCombineFunctions.java:191), which has no missing parents 25/01/03 13:17:11 INFO MemoryStore: Block broadcast_7 stored as values in memory (estimated size 32.5 KiB, free 433.2 MiB) 25/01/03 13:17:11 INFO MemoryStore: Block broadcast_7_piece0 stored as bytes in memory (estimated size 13.1 KiB, free 433.2 MiB) 25/01/03 13:17:11 INFO BlockManagerInfo: Added broadcast_7_piece0 in memory on 192.168.86.198:55059 (size: 13.1 KiB, free: 434.1 MiB) 25/01/03 13:17:11 INFO SparkContext: Created broadcast 7 from broadcast at DAGScheduler.scala:1585 25/01/03 13:17:11 INFO DAGScheduler: Submitting 12 missing tasks from ShuffleMapStage 9 (MapPartitionsRDD[70] at repartition at GroupCombineFunctions.java:191) (first 15 tasks are for partitions Vector(0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11)) 25/01/03 13:17:11 INFO TaskSchedulerImpl: Adding task set 9.0 with 12 tasks resource profile 0 25/01/03 13:17:11 INFO TaskSetManager: Starting task 0.0 in stage 9.0 (TID 18) (192.168.86.198, executor 0, partition 0, PROCESS_LOCAL, 9160 bytes) 25/01/03 13:17:11 INFO TaskSetManager: Starting task 1.0 in stage 9.0 (TID 19) (192.168.86.198, executor 0, partition 1, PROCESS_LOCAL, 9160 bytes) 25/01/03 13:17:11 INFO TaskSetManager: Starting task 2.0 in stage 9.0 (TID 20) (192.168.86.198, executor 0, partition 2, PROCESS_LOCAL, 9160 bytes) 25/01/03 13:17:11 INFO TaskSetManager: Starting task 3.0 in stage 9.0 (TID 21) (192.168.86.198, executor 0, partition 3, PROCESS_LOCAL, 9160 bytes) 25/01/03 13:17:11 INFO TaskSetManager: Starting task 4.0 in stage 9.0 (TID 22) (192.168.86.198, executor 0, partition 4, PROCESS_LOCAL, 9160 bytes) 25/01/03 13:17:11 INFO TaskSetManager: Starting task 5.0 in stage 9.0 (TID 23) (192.168.86.198, executor 0, partition 5, PROCESS_LOCAL, 9160 bytes) 25/01/03 13:17:11 INFO TaskSetManager: Starting task 6.0 in stage 9.0 (TID 24) (192.168.86.198, executor 0, partition 6, PROCESS_LOCAL, 9160 bytes) 25/01/03 13:17:11 INFO TaskSetManager: Starting task 7.0 in stage 9.0 (TID 25) (192.168.86.198, executor 0, partition 7, PROCESS_LOCAL, 9160 bytes) 25/01/03 13:17:11 INFO TaskSetManager: Starting task 8.0 in stage 9.0 (TID 26) (192.168.86.198, executor 0, partition 8, PROCESS_LOCAL, 9160 bytes) 25/01/03 13:17:11 INFO TaskSetManager: Starting task 9.0 in stage 9.0 (TID 27) (192.168.86.198, executor 0, partition 9, PROCESS_LOCAL, 9160 bytes) 25/01/03 13:17:11 INFO TaskSetManager: Starting task 10.0 in stage 9.0 (TID 28) (192.168.86.198, executor 0, partition 10, PROCESS_LOCAL, 9160 bytes) 25/01/03 13:17:11 INFO TaskSetManager: Starting task 11.0 in stage 9.0 (TID 29) (192.168.86.198, executor 0, partition 11, PROCESS_LOCAL, 9171 bytes) 25/01/03 13:17:11 INFO BlockManagerInfo: Added broadcast_7_piece0 in memory on 192.168.86.198:55063 (size: 13.1 KiB, free: 434.1 MiB) 25/01/03 13:17:11 INFO TaskSetManager: Finished task 3.0 in stage 9.0 (TID 21) in 52 ms on 192.168.86.198 (executor 0) (1/12) 25/01/03 13:17:11 INFO TaskSetManager: Finished task 10.0 in stage 9.0 (TID 28) in 50 ms on 192.168.86.198 (executor 0) (2/12) 25/01/03 13:17:11 INFO TaskSetManager: Finished task 5.0 in stage 9.0 (TID 23) in 53 ms on 192.168.86.198 (executor 0) (3/12) 25/01/03 13:17:11 INFO TaskSetManager: Finished task 9.0 in stage 9.0 (TID 27) in 51 ms on 192.168.86.198 (executor 0) (4/12) 25/01/03 13:17:11 INFO TaskSetManager: Finished task 2.0 in stage 9.0 (TID 20) in 53 ms on 192.168.86.198 (executor 0) (5/12) 25/01/03 13:17:11 INFO TaskSetManager: Finished task 8.0 in stage 9.0 (TID 26) in 51 ms on 192.168.86.198 (executor 0) (6/12) 25/01/03 13:17:11 INFO TaskSetManager: Finished task 1.0 in stage 9.0 (TID 19) in 54 ms on 192.168.86.198 (executor 0) (7/12) 25/01/03 13:17:11 INFO TaskSetManager: Finished task 7.0 in stage 9.0 (TID 25) in 53 ms on 192.168.86.198 (executor 0) (8/12) 25/01/03 13:17:11 INFO TaskSetManager: Finished task 6.0 in stage 9.0 (TID 24) in 55 ms on 192.168.86.198 (executor 0) (9/12) 25/01/03 13:17:11 INFO TaskSetManager: Finished task 4.0 in stage 9.0 (TID 22) in 55 ms on 192.168.86.198 (executor 0) (10/12) 25/01/03 13:17:11 INFO TaskSetManager: Finished task 0.0 in stage 9.0 (TID 18) in 57 ms on 192.168.86.198 (executor 0) (11/12) 25/01/03 13:17:11 INFO BlockManagerInfo: Added broadcast_3_piece0 in memory on 192.168.86.198:55063 (size: 330.0 B, free: 434.1 MiB) 25/01/03 13:17:11 INFO TaskSetManager: Finished task 11.0 in stage 9.0 (TID 29) in 107 ms on 192.168.86.198 (executor 0) (12/12) 25/01/03 13:17:11 INFO TaskSchedulerImpl: Removed TaskSet 9.0, whose tasks have all completed, from pool 25/01/03 13:17:11 INFO DAGScheduler: ShuffleMapStage 9 (repartition at GroupCombineFunctions.java:191) finished in 0,115 s 25/01/03 13:17:11 INFO DAGScheduler: looking for newly runnable stages 25/01/03 13:17:11 INFO DAGScheduler: running: Set() 25/01/03 13:17:11 INFO DAGScheduler: waiting: Set(ResultStage 10) 25/01/03 13:17:11 INFO DAGScheduler: failed: Set() 25/01/03 13:17:11 INFO DAGScheduler: Submitting ResultStage 10 (BeamOutputTransform/TextIO.Write/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Values/Values/Map/ParMultiDo(Anonymous).output MapPartitionsRDD[77] at values at TransformTranslator.java:472), which has no missing parents 25/01/03 13:17:11 INFO MemoryStore: Block broadcast_8 stored as values in memory (estimated size 21.2 KiB, free 433.1 MiB) 25/01/03 13:17:11 INFO MemoryStore: Block broadcast_8_piece0 stored as bytes in memory (estimated size 8.9 KiB, free 433.1 MiB) 25/01/03 13:17:11 INFO BlockManagerInfo: Added broadcast_8_piece0 in memory on 192.168.86.198:55059 (size: 8.9 KiB, free: 434.1 MiB) 25/01/03 13:17:11 INFO SparkContext: Created broadcast 8 from broadcast at DAGScheduler.scala:1585 25/01/03 13:17:11 INFO DAGScheduler: Submitting 12 missing tasks from ResultStage 10 (BeamOutputTransform/TextIO.Write/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Values/Values/Map/ParMultiDo(Anonymous).output MapPartitionsRDD[77] at values at TransformTranslator.java:472) (first 15 tasks are for partitions Vector(0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11)) 25/01/03 13:17:11 INFO TaskSchedulerImpl: Adding task set 10.0 with 12 tasks resource profile 0 25/01/03 13:17:11 INFO TaskSetManager: Starting task 3.0 in stage 10.0 (TID 30) (192.168.86.198, executor 0, partition 3, NODE_LOCAL, 9269 bytes) 25/01/03 13:17:11 INFO TaskSetManager: Starting task 4.0 in stage 10.0 (TID 31) (192.168.86.198, executor 0, partition 4, NODE_LOCAL, 9269 bytes) 25/01/03 13:17:11 INFO TaskSetManager: Starting task 0.0 in stage 10.0 (TID 32) (192.168.86.198, executor 0, partition 0, PROCESS_LOCAL, 9269 bytes) 25/01/03 13:17:11 INFO TaskSetManager: Starting task 1.0 in stage 10.0 (TID 33) (192.168.86.198, executor 0, partition 1, PROCESS_LOCAL, 9269 bytes) 25/01/03 13:17:11 INFO TaskSetManager: Starting task 2.0 in stage 10.0 (TID 34) (192.168.86.198, executor 0, partition 2, PROCESS_LOCAL, 9269 bytes) 25/01/03 13:17:11 INFO TaskSetManager: Starting task 5.0 in stage 10.0 (TID 35) (192.168.86.198, executor 0, partition 5, PROCESS_LOCAL, 9269 bytes) 25/01/03 13:17:11 INFO TaskSetManager: Starting task 6.0 in stage 10.0 (TID 36) (192.168.86.198, executor 0, partition 6, PROCESS_LOCAL, 9269 bytes) 25/01/03 13:17:11 INFO TaskSetManager: Starting task 7.0 in stage 10.0 (TID 37) (192.168.86.198, executor 0, partition 7, PROCESS_LOCAL, 9269 bytes) 25/01/03 13:17:11 INFO TaskSetManager: Starting task 8.0 in stage 10.0 (TID 38) (192.168.86.198, executor 0, partition 8, PROCESS_LOCAL, 9269 bytes) 25/01/03 13:17:11 INFO TaskSetManager: Starting task 9.0 in stage 10.0 (TID 39) (192.168.86.198, executor 0, partition 9, PROCESS_LOCAL, 9269 bytes) 25/01/03 13:17:11 INFO TaskSetManager: Starting task 10.0 in stage 10.0 (TID 40) (192.168.86.198, executor 0, partition 10, PROCESS_LOCAL, 9269 bytes) 25/01/03 13:17:11 INFO TaskSetManager: Starting task 11.0 in stage 10.0 (TID 41) (192.168.86.198, executor 0, partition 11, PROCESS_LOCAL, 9269 bytes) 25/01/03 13:17:11 INFO BlockManagerInfo: Added broadcast_8_piece0 in memory on 192.168.86.198:55063 (size: 8.9 KiB, free: 434.1 MiB) 25/01/03 13:17:11 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 2 to 192.168.86.198:55061 25/01/03 13:17:11 INFO TaskSetManager: Finished task 6.0 in stage 10.0 (TID 36) in 21 ms on 192.168.86.198 (executor 0) (1/12) 25/01/03 13:17:11 INFO TaskSetManager: Finished task 3.0 in stage 10.0 (TID 30) in 23 ms on 192.168.86.198 (executor 0) (2/12) 25/01/03 13:17:11 INFO TaskSetManager: Finished task 5.0 in stage 10.0 (TID 35) in 22 ms on 192.168.86.198 (executor 0) (3/12) 25/01/03 13:17:11 INFO TaskSetManager: Finished task 1.0 in stage 10.0 (TID 33) in 23 ms on 192.168.86.198 (executor 0) (4/12) 25/01/03 13:17:11 INFO TaskSetManager: Finished task 10.0 in stage 10.0 (TID 40) in 21 ms on 192.168.86.198 (executor 0) (5/12) 25/01/03 13:17:11 INFO TaskSetManager: Finished task 0.0 in stage 10.0 (TID 32) in 24 ms on 192.168.86.198 (executor 0) (6/12) 25/01/03 13:17:11 INFO TaskSetManager: Finished task 11.0 in stage 10.0 (TID 41) in 22 ms on 192.168.86.198 (executor 0) (7/12) 25/01/03 13:17:11 INFO TaskSetManager: Finished task 2.0 in stage 10.0 (TID 34) in 24 ms on 192.168.86.198 (executor 0) (8/12) 25/01/03 13:17:11 INFO TaskSetManager: Finished task 9.0 in stage 10.0 (TID 39) in 24 ms on 192.168.86.198 (executor 0) (9/12) 25/01/03 13:17:11 INFO TaskSetManager: Finished task 8.0 in stage 10.0 (TID 38) in 24 ms on 192.168.86.198 (executor 0) (10/12) 25/01/03 13:17:11 INFO TaskSetManager: Finished task 7.0 in stage 10.0 (TID 37) in 24 ms on 192.168.86.198 (executor 0) (11/12) 25/01/03 13:17:11 INFO TaskSetManager: Finished task 4.0 in stage 10.0 (TID 31) in 25 ms on 192.168.86.198 (executor 0) (12/12) 25/01/03 13:17:11 INFO TaskSchedulerImpl: Removed TaskSet 10.0, whose tasks have all completed, from pool 25/01/03 13:17:11 INFO DAGScheduler: ResultStage 10 (foreach at BoundedDataset.java:127) finished in 0,030 s 25/01/03 13:17:11 INFO DAGScheduler: Job 4 is finished. Cancelling potential speculative or zombie tasks for this job 25/01/03 13:17:11 INFO TaskSchedulerImpl: Killing all running tasks in stage 10: Stage finished 25/01/03 13:17:11 INFO DAGScheduler: Job 4 finished: foreach at BoundedDataset.java:127, took 0,152777 s 25/01/03 13:17:11 INFO SparkRunner: Batch pipeline execution complete. 25/01/03 13:17:11 INFO SparkContext: SparkContext is stopping with exitCode 0. 25/01/03 13:17:11 INFO SparkUI: Stopped Spark web UI at http://192.168.86.198:4040 25/01/03 13:17:11 INFO StandaloneSchedulerBackend: Shutting down all executors 25/01/03 13:17:11 INFO StandaloneSchedulerBackend$StandaloneDriverEndpoint: Asking each executor to shut down 25/01/03 13:17:11 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped! 25/01/03 13:17:11 INFO MemoryStore: MemoryStore cleared 25/01/03 13:17:11 INFO BlockManager: BlockManager stopped 25/01/03 13:17:11 INFO BlockManagerMaster: BlockManagerMaster stopped 25/01/03 13:17:11 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped! 25/01/03 13:17:11 INFO SparkContext: Successfully stopped SparkContext 2025/01/03 13:17:11 - General - Beam pipeline execution has finished. >>>>>> Execution finished... 25/01/03 13:17:11 INFO ShutdownHookManager: Shutdown hook called 25/01/03 13:17:11 INFO ShutdownHookManager: Deleting directory /private/var/folders/v0/hjm6kpgd0076ncdkpf8rg3dw0000gn/T/spark-5a592189-8383-44ae-8574-70c2aaabba19 25/01/03 13:17:11 INFO ShutdownHookManager: Deleting directory /private/var/folders/v0/hjm6kpgd0076ncdkpf8rg3dw0000gn/T/spark-611dd39c-8451-4aed-9c91-ea906a4e9e04 ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
