[ https://issues.apache.org/jira/browse/SPARK-1895?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14133964#comment-14133964 ]
Sean Owen commented on SPARK-1895: ---------------------------------- Can anyone still reproduce this? I know test temp file cleanup was improved in 1.0.x, and am not sure I have heard of this since. > Run tests on windows > -------------------- > > Key: SPARK-1895 > URL: https://issues.apache.org/jira/browse/SPARK-1895 > Project: Spark > Issue Type: Bug > Components: PySpark, Windows > Affects Versions: 0.9.1 > Environment: spark-0.9.1-bin-hadoop1 > Reporter: stribog > Priority: Trivial > > bin\pyspark python\pyspark\rdd.py > Sometimes tests complete without error >_<. > Last tests fail log: > .... > 14/05/21 18:31:40 INFO Executor: Running task ID 321 > 14/05/21 18:31:40 INFO Executor: Running task ID 324 > 14/05/21 18:31:40 INFO Executor: Running task ID 322 > 14/05/21 18:31:40 INFO Executor: Running task ID 323 > 14/05/21 18:31:40 INFO PythonRDD: Times: total = 241, boot = 240, init = 1, > finish = 0 > 14/05/21 18:31:40 INFO Executor: Serialized size of result for 324 is 607 > 14/05/21 18:31:40 INFO Executor: Sending result for 324 directly to driver > 14/05/21 18:31:40 INFO Executor: Finished task ID 324 > 14/05/21 18:31:40 INFO TaskSetManager: Finished TID 324 in 248 ms on > localhost (progress: 1/4) > 14/05/21 18:31:40 INFO DAGScheduler: Completed ResultTask(80, 3) > 14/05/21 18:31:40 INFO PythonRDD: Times: total = 518, boot = 516, init = 2, > finish = 0 > 14/05/21 18:31:40 INFO Executor: Serialized size of result for 323 is 607 > 14/05/21 18:31:40 INFO Executor: Sending result for 323 directly to driver > 14/05/21 18:31:40 INFO Executor: Finished task ID 323 > 14/05/21 18:31:40 INFO TaskSetManager: Finished TID 323 in 528 ms on > localhost (progress: 2/4) > 14/05/21 18:31:40 INFO DAGScheduler: Completed ResultTask(80, 2) > 14/05/21 18:31:41 INFO PythonRDD: Times: total = 776, boot = 774, init = 2, > finish = 0 > 14/05/21 18:31:41 INFO Executor: Serialized size of result for 322 is 607 > 14/05/21 18:31:41 INFO Executor: Sending result for 322 directly to driver > 14/05/21 18:31:41 INFO Executor: Finished task ID 322 > 14/05/21 18:31:41 INFO TaskSetManager: Finished TID 322 in 785 ms on > localhost (progress: 3/4) > 14/05/21 18:31:41 INFO DAGScheduler: Completed ResultTask(80, 1) > 14/05/21 18:31:41 INFO PythonRDD: Times: total = 1043, boot = 1042, init = 1, > finish = 0 > 14/05/21 18:31:41 INFO Executor: Serialized size of result for 321 is 607 > 14/05/21 18:31:41 INFO Executor: Sending result for 321 directly to driver > 14/05/21 18:31:41 INFO Executor: Finished task ID 321 > 14/05/21 18:31:41 INFO TaskSetManager: Finished TID 321 in 1049 ms on > localhost (progress: 4/4) > 14/05/21 18:31:41 INFO DAGScheduler: Completed ResultTask(80, 0) > 14/05/21 18:31:41 INFO TaskSchedulerImpl: Removed TaskSet 80.0, whose tasks > have all completed, from pool > 14/05/21 18:31:41 INFO DAGScheduler: Stage 80 (top at <doctest > __main__.RDD.top[0]>:1) finished in 1,051 s > 14/05/21 18:31:41 INFO SparkContext: Job finished: top at <doctest > __main__.RDD.top[0]>:1, took 1.053832912 s > 14/05/21 18:31:41 INFO SparkContext: Starting job: top at <doctest > __main__.RDD.top[1]>:1 > 14/05/21 18:31:41 INFO DAGScheduler: Got job 63 (top at <doctest > __main__.RDD.top[1]>:1) with 4 output partitions (allowLocal=false) > 14/05/21 18:31:41 INFO DAGScheduler: Final stage: Stage 81 (top at <doctest > __main__.RDD.top[1]>:1) > 14/05/21 18:31:41 INFO DAGScheduler: Parents of final stage: List() > 14/05/21 18:31:41 INFO DAGScheduler: Missing parents: List() > 14/05/21 18:31:41 INFO DAGScheduler: Submitting Stage 81 (PythonRDD[213] at > top at <doctest __main__.RDD.top[1]>:1), which has no missing parents > 14/05/21 18:31:41 INFO DAGScheduler: Submitting 4 missing tasks from Stage 81 > (PythonRDD[213] at top at <doctest __main__.RDD.top[1]>:1) > 14/05/21 18:31:41 INFO TaskSchedulerImpl: Adding task set 81.0 with 4 tasks > 14/05/21 18:31:41 INFO TaskSetManager: Starting task 81.0:0 as TID 325 on > executor localhost: localhost (PROCESS_LOCAL) > 14/05/21 18:31:41 INFO TaskSetManager: Serialized task 81.0:0 as 2594 bytes > in 0 ms > 14/05/21 18:31:41 INFO TaskSetManager: Starting task 81.0:1 as TID 326 on > executor localhost: localhost (PROCESS_LOCAL) > 14/05/21 18:31:41 INFO TaskSetManager: Serialized task 81.0:1 as 2594 bytes > in 0 ms > 14/05/21 18:31:41 INFO TaskSetManager: Starting task 81.0:2 as TID 327 on > executor localhost: localhost (PROCESS_LOCAL) > 14/05/21 18:31:41 INFO TaskSetManager: Serialized task 81.0:2 as 2594 bytes > in 0 ms > 14/05/21 18:31:41 INFO TaskSetManager: Starting task 81.0:3 as TID 328 on > executor localhost: localhost (PROCESS_LOCAL) > 14/05/21 18:31:41 INFO TaskSetManager: Serialized task 81.0:3 as 2609 bytes > in 1 ms > 14/05/21 18:31:41 INFO Executor: Running task ID 326 > 14/05/21 18:31:41 INFO Executor: Running task ID 328 > 14/05/21 18:31:41 INFO Executor: Running task ID 327 > 14/05/21 18:31:41 INFO Executor: Running task ID 325 > 14/05/21 18:31:41 INFO CacheManager: Partition rdd_212_3 not found, computing > it > 14/05/21 18:31:41 INFO MemoryStore: ensureFreeSpace(152) called with > curMem=1120, maxMem=311387750 > 14/05/21 18:31:41 INFO MemoryStore: Block rdd_212_3 stored as values to > memory (estimated size 152.0 B, free 297.0 MB) > 14/05/21 18:31:41 INFO BlockManagerMasterActor$BlockManagerInfo: Added > rdd_212_3 in memory on stribog-pc:37187 (size: 152.0 B, free: 297.0 MB) > 14/05/21 18:31:41 INFO BlockManagerMaster: Updated info of block rdd_212_3 > 14/05/21 18:31:41 INFO PythonRDD: Times: total = 245, boot = 241, init = 4, > finish = 0 > 14/05/21 18:31:41 INFO Executor: Serialized size of result for 328 is 610 > 14/05/21 18:31:41 INFO Executor: Sending result for 328 directly to driver > 14/05/21 18:31:41 INFO Executor: Finished task ID 328 > 14/05/21 18:31:41 INFO TaskSetManager: Finished TID 328 in 251 ms on > localhost (progress: 1/4) > 14/05/21 18:31:41 INFO DAGScheduler: Completed ResultTask(81, 3) > 14/05/21 18:31:41 INFO CacheManager: Partition rdd_212_0 not found, computing > it > 14/05/21 18:31:41 INFO MemoryStore: ensureFreeSpace(128) called with > curMem=1272, maxMem=311387750 > 14/05/21 18:31:41 INFO MemoryStore: Block rdd_212_0 stored as values to > memory (estimated size 128.0 B, free 297.0 MB) > 14/05/21 18:31:41 INFO BlockManagerMasterActor$BlockManagerInfo: Added > rdd_212_0 in memory on stribog-pc:37187 (size: 128.0 B, free: 297.0 MB) > 14/05/21 18:31:41 INFO BlockManagerMaster: Updated info of block rdd_212_0 > 14/05/21 18:31:41 INFO PythonRDD: Times: total = 484, boot = 481, init = 3, > finish = 0 > 14/05/21 18:31:41 INFO Executor: Serialized size of result for 325 is 607 > 14/05/21 18:31:41 INFO Executor: Sending result for 325 directly to driver > 14/05/21 18:31:41 INFO Executor: Finished task ID 325 > 14/05/21 18:31:41 INFO DAGScheduler: Completed ResultTask(81, 0) > 14/05/21 18:31:41 INFO TaskSetManager: Finished TID 325 in 493 ms on > localhost (progress: 2/4) > 14/05/21 18:31:42 INFO CacheManager: Partition rdd_212_2 not found, computing > it > 14/05/21 18:31:42 INFO MemoryStore: ensureFreeSpace(128) called with > curMem=1400, maxMem=311387750 > 14/05/21 18:31:42 INFO MemoryStore: Block rdd_212_2 stored as values to > memory (estimated size 128.0 B, free 297.0 MB) > 14/05/21 18:31:42 INFO BlockManagerMasterActor$BlockManagerInfo: Added > rdd_212_2 in memory on stribog-pc:37187 (size: 128.0 B, free: 297.0 MB) > 14/05/21 18:31:42 INFO BlockManagerMaster: Updated info of block rdd_212_2 > 14/05/21 18:31:42 INFO PythonRDD: Times: total = 728, boot = 724, init = 4, > finish = 0 > 14/05/21 18:31:42 INFO Executor: Serialized size of result for 327 is 607 > 14/05/21 18:31:42 INFO Executor: Sending result for 327 directly to driver > 14/05/21 18:31:42 INFO Executor: Finished task ID 327 > 14/05/21 18:31:42 INFO TaskSetManager: Finished TID 327 in 735 ms on > localhost (progress: 3/4) > 14/05/21 18:31:42 INFO DAGScheduler: Completed ResultTask(81, 2) > 14/05/21 18:31:42 INFO CacheManager: Partition rdd_212_1 not found, computing > it > 14/05/21 18:31:42 INFO MemoryStore: ensureFreeSpace(128) called with > curMem=1528, maxMem=311387750 > 14/05/21 18:31:42 INFO MemoryStore: Block rdd_212_1 stored as values to > memory (estimated size 128.0 B, free 297.0 MB) > 14/05/21 18:31:42 INFO BlockManagerMasterActor$BlockManagerInfo: Added > rdd_212_1 in memory on stribog-pc:37187 (size: 128.0 B, free: 297.0 MB) > 14/05/21 18:31:42 INFO BlockManagerMaster: Updated info of block rdd_212_1 > 14/05/21 18:31:42 INFO PythonRDD: Times: total = 1487, boot = 1484, init = 3, > finish = 0 > 14/05/21 18:31:42 INFO Executor: Serialized size of result for 326 is 607 > 14/05/21 18:31:42 INFO Executor: Sending result for 326 directly to driver > 14/05/21 18:31:42 INFO Executor: Finished task ID 326 > 14/05/21 18:31:42 INFO TaskSetManager: Finished TID 326 in 1493 ms on > localhost (progress: 4/4) > 14/05/21 18:31:42 INFO TaskSchedulerImpl: Removed TaskSet 81.0, whose tasks > have all completed, from pool > 14/05/21 18:31:42 INFO DAGScheduler: Completed ResultTask(81, 1) > 14/05/21 18:31:42 INFO DAGScheduler: Stage 81 (top at <doctest > __main__.RDD.top[1]>:1) finished in 1,496 s > 14/05/21 18:31:42 INFO SparkContext: Job finished: top at <doctest > __main__.RDD.top[1]>:1, took 1.498387373 s > 14/05/21 18:31:42 INFO SparkContext: Starting job: collect at <doctest > __main__.RDD.union[1]>:1 > 14/05/21 18:31:42 INFO DAGScheduler: Got job 64 (collect at <doctest > __main__.RDD.union[1]>:1) with 8 output partitions (allowLocal=false) > 14/05/21 18:31:42 INFO DAGScheduler: Final stage: Stage 82 (collect at > <doctest __main__.RDD.union[1]>:1) > 14/05/21 18:31:42 INFO DAGScheduler: Parents of final stage: List() > 14/05/21 18:31:42 INFO DAGScheduler: Missing parents: List() > 14/05/21 18:31:42 INFO DAGScheduler: Submitting Stage 82 (UnionRDD[215] at > union at null:-2), which has no missing parents > 14/05/21 18:31:42 INFO DAGScheduler: Submitting 8 missing tasks from Stage 82 > (UnionRDD[215] at union at null:-2) > 14/05/21 18:31:42 INFO TaskSchedulerImpl: Adding task set 82.0 with 8 tasks > 14/05/21 18:31:42 INFO TaskSetManager: Starting task 82.0:0 as TID 329 on > executor localhost: localhost (PROCESS_LOCAL) > 14/05/21 18:31:42 INFO TaskSetManager: Serialized task 82.0:0 as 2208 bytes > in 0 ms > 14/05/21 18:31:42 INFO TaskSetManager: Starting task 82.0:1 as TID 330 on > executor localhost: localhost (PROCESS_LOCAL) > 14/05/21 18:31:42 INFO TaskSetManager: Serialized task 82.0:1 as 2208 bytes > in 0 ms > 14/05/21 18:31:42 INFO TaskSetManager: Starting task 82.0:2 as TID 331 on > executor localhost: localhost (PROCESS_LOCAL) > 14/05/21 18:31:42 INFO TaskSetManager: Serialized task 82.0:2 as 2208 bytes > in 0 ms > 14/05/21 18:31:42 INFO TaskSetManager: Starting task 82.0:3 as TID 332 on > executor localhost: localhost (PROCESS_LOCAL) > 14/05/21 18:31:42 INFO TaskSetManager: Serialized task 82.0:3 as 2208 bytes > in 0 ms > 14/05/21 18:31:42 INFO Executor: Running task ID 330 > 14/05/21 18:31:42 INFO Executor: Running task ID 331 > 14/05/21 18:31:42 INFO Executor: Running task ID 332 > 14/05/21 18:31:42 INFO Executor: Serialized size of result for 331 is 537 > 14/05/21 18:31:42 INFO Executor: Sending result for 331 directly to driver > 14/05/21 18:31:42 INFO Executor: Finished task ID 331 > 14/05/21 18:31:42 INFO Executor: Serialized size of result for 332 is 537 > 14/05/21 18:31:42 INFO Executor: Running task ID 329 > 14/05/21 18:31:42 INFO Executor: Sending result for 332 directly to driver > 14/05/21 18:31:42 INFO TaskSetManager: Starting task 82.0:4 as TID 333 on > executor localhost: localhost (PROCESS_LOCAL) > 14/05/21 18:31:42 INFO Executor: Serialized size of result for 330 is 537 > 14/05/21 18:31:42 INFO Executor: Sending result for 330 directly to driver > 14/05/21 18:31:42 INFO Executor: Finished task ID 330 > 14/05/21 18:31:42 INFO Executor: Finished task ID 332 > 14/05/21 18:31:42 INFO TaskSetManager: Serialized task 82.0:4 as 2208 bytes > in 0 ms > 14/05/21 18:31:42 INFO Executor: Running task ID 333 > 14/05/21 18:31:42 INFO TaskSetManager: Finished TID 331 in 3 ms on localhost > (progress: 1/8) > 14/05/21 18:31:42 INFO DAGScheduler: Completed ResultTask(82, 2) > 14/05/21 18:31:42 INFO TaskSetManager: Finished TID 332 in 2 ms on localhost > (progress: 2/8) > 14/05/21 18:31:42 INFO Executor: Serialized size of result for 329 is 537 > 14/05/21 18:31:42 INFO Executor: Serialized size of result for 333 is 537 > 14/05/21 18:31:42 INFO DAGScheduler: Completed ResultTask(82, 3) > 14/05/21 18:31:42 INFO Executor: Sending result for 333 directly to driver > 14/05/21 18:31:42 INFO TaskSetManager: Starting task 82.0:5 as TID 334 on > executor localhost: localhost (PROCESS_LOCAL) > 14/05/21 18:31:42 INFO Executor: Sending result for 329 directly to driver > 14/05/21 18:31:42 INFO Executor: Finished task ID 333 > 14/05/21 18:31:42 INFO Executor: Finished task ID 329 > 14/05/21 18:31:42 INFO TaskSetManager: Serialized task 82.0:5 as 2208 bytes > in 0 ms > 14/05/21 18:31:42 INFO Executor: Running task ID 334 > 14/05/21 18:31:42 INFO TaskSetManager: Starting task 82.0:6 as TID 335 on > executor localhost: localhost (PROCESS_LOCAL) > 14/05/21 18:31:42 INFO TaskSetManager: Serialized task 82.0:6 as 2208 bytes > in 0 ms > 14/05/21 18:31:42 INFO TaskSetManager: Starting task 82.0:7 as TID 336 on > executor localhost: localhost (PROCESS_LOCAL) > 14/05/21 18:31:42 INFO Executor: Running task ID 335 > 14/05/21 18:31:42 INFO Executor: Serialized size of result for 334 is 537 > 14/05/21 18:31:42 INFO TaskSetManager: Serialized task 82.0:7 as 2208 bytes > in 0 ms > 14/05/21 18:31:42 INFO Executor: Sending result for 334 directly to driver > 14/05/21 18:31:42 INFO Executor: Finished task ID 334 > 14/05/21 18:31:42 INFO DAGScheduler: Completed ResultTask(82, 1) > 14/05/21 18:31:42 INFO TaskSetManager: Finished TID 330 in 5 ms on localhost > (progress: 3/8) > 14/05/21 18:31:42 INFO Executor: Serialized size of result for 335 is 537 > 14/05/21 18:31:42 INFO Executor: Sending result for 335 directly to driver > 14/05/21 18:31:42 INFO Executor: Running task ID 336 > 14/05/21 18:31:42 INFO Executor: Finished task ID 335 > 14/05/21 18:31:42 INFO DAGScheduler: Completed ResultTask(82, 5) > 14/05/21 18:31:42 INFO TaskSetManager: Finished TID 334 in 2 ms on localhost > (progress: 4/8) > 14/05/21 18:31:42 INFO TaskSetManager: Finished TID 333 in 5 ms on localhost > (progress: 5/8) > 14/05/21 18:31:42 INFO DAGScheduler: Completed ResultTask(82, 4) > 14/05/21 18:31:42 INFO Executor: Serialized size of result for 336 is 537 > 14/05/21 18:31:42 INFO Executor: Sending result for 336 directly to driver > 14/05/21 18:31:42 INFO TaskSetManager: Finished TID 335 in 3 ms on localhost > (progress: 6/8) > 14/05/21 18:31:42 INFO DAGScheduler: Completed ResultTask(82, 6) > 14/05/21 18:31:42 INFO TaskSetManager: Finished TID 329 in 7 ms on localhost > (progress: 7/8) > 14/05/21 18:31:42 INFO Executor: Finished task ID 336 > 14/05/21 18:31:42 INFO TaskSetManager: Finished TID 336 in 3 ms on localhost > (progress: 8/8) > 14/05/21 18:31:42 INFO DAGScheduler: Completed ResultTask(82, 0) > 14/05/21 18:31:42 INFO TaskSchedulerImpl: Removed TaskSet 82.0, whose tasks > have all completed, from pool > 14/05/21 18:31:42 INFO DAGScheduler: Completed ResultTask(82, 7) > 14/05/21 18:31:42 INFO DAGScheduler: Stage 82 (collect at <doctest > __main__.RDD.union[1]>:1) finished in 0,008 s > 14/05/21 18:31:42 INFO SparkContext: Job finished: collect at <doctest > __main__.RDD.union[1]>:1, took 0.010936769 s > 14/05/21 18:31:42 INFO SparkContext: Starting job: variance at <doctest > __main__.RDD.variance[0]>:1 > 14/05/21 18:31:42 INFO DAGScheduler: Got job 65 (variance at <doctest > __main__.RDD.variance[0]>:1) with 4 output partitions (allowLocal=false) > 14/05/21 18:31:42 INFO DAGScheduler: Final stage: Stage 83 (variance at > <doctest __main__.RDD.variance[0]>:1) > 14/05/21 18:31:42 INFO DAGScheduler: Parents of final stage: List() > 14/05/21 18:31:42 INFO DAGScheduler: Missing parents: List() > 14/05/21 18:31:42 INFO DAGScheduler: Submitting Stage 83 (PythonRDD[217] at > variance at <doctest __main__.RDD.variance[0]>:1), which has no missing > parents > 14/05/21 18:31:42 INFO DAGScheduler: Submitting 4 missing tasks from Stage 83 > (PythonRDD[217] at variance at <doctest __main__.RDD.variance[0]>:1) > 14/05/21 18:31:42 INFO TaskSchedulerImpl: Adding task set 83.0 with 4 tasks > 14/05/21 18:31:42 INFO TaskSetManager: Starting task 83.0:0 as TID 337 on > executor localhost: localhost (PROCESS_LOCAL) > 14/05/21 18:31:42 INFO TaskSetManager: Serialized task 83.0:0 as 2486 bytes > in 0 ms > 14/05/21 18:31:42 INFO TaskSetManager: Starting task 83.0:1 as TID 338 on > executor localhost: localhost (PROCESS_LOCAL) > 14/05/21 18:31:42 INFO TaskSetManager: Serialized task 83.0:1 as 2514 bytes > in 0 ms > 14/05/21 18:31:42 INFO TaskSetManager: Starting task 83.0:2 as TID 339 on > executor localhost: localhost (PROCESS_LOCAL) > 14/05/21 18:31:42 INFO TaskSetManager: Serialized task 83.0:2 as 2514 bytes > in 0 ms > 14/05/21 18:31:42 INFO TaskSetManager: Starting task 83.0:3 as TID 340 on > executor localhost: localhost (PROCESS_LOCAL) > 14/05/21 18:31:42 INFO TaskSetManager: Serialized task 83.0:3 as 2514 bytes > in 0 ms > 14/05/21 18:31:42 INFO Executor: Running task ID 338 > 14/05/21 18:31:42 INFO Executor: Running task ID 340 > 14/05/21 18:31:42 INFO Executor: Running task ID 339 > 14/05/21 18:31:42 INFO Executor: Running task ID 337 > 14/05/21 18:31:43 INFO PythonRDD: Times: total = 265, boot = 264, init = 1, > finish = 0 > 14/05/21 18:31:43 INFO Executor: Serialized size of result for 340 is 682 > 14/05/21 18:31:43 INFO Executor: Sending result for 340 directly to driver > 14/05/21 18:31:43 INFO Executor: Finished task ID 340 > 14/05/21 18:31:43 INFO TaskSetManager: Finished TID 340 in 278 ms on > localhost (progress: 1/4) > 14/05/21 18:31:43 INFO DAGScheduler: Completed ResultTask(83, 3) > 14/05/21 18:31:43 INFO PythonRDD: Times: total = 519, boot = 517, init = 2, > finish = 0 > 14/05/21 18:31:43 INFO Executor: Serialized size of result for 337 is 681 > 14/05/21 18:31:43 INFO Executor: Sending result for 337 directly to driver > 14/05/21 18:31:43 INFO Executor: Finished task ID 337 > 14/05/21 18:31:43 INFO TaskSetManager: Finished TID 337 in 527 ms on > localhost (progress: 2/4) > 14/05/21 18:31:43 INFO DAGScheduler: Completed ResultTask(83, 0) > 14/05/21 18:31:43 INFO PythonRDD: Times: total = 779, boot = 778, init = 1, > finish = 0 > 14/05/21 18:31:43 INFO Executor: Serialized size of result for 338 is 682 > 14/05/21 18:31:43 INFO Executor: Sending result for 338 directly to driver > 14/05/21 18:31:43 INFO Executor: Finished task ID 338 > 14/05/21 18:31:43 INFO DAGScheduler: Completed ResultTask(83, 1) > 14/05/21 18:31:43 INFO TaskSetManager: Finished TID 338 in 786 ms on > localhost (progress: 3/4) > 14/05/21 18:31:43 INFO PythonRDD: Times: total = 1025, boot = 1023, init = 1, > finish = 1 > 14/05/21 18:31:43 INFO Executor: Serialized size of result for 339 is 682 > 14/05/21 18:31:43 INFO Executor: Sending result for 339 directly to driver > 14/05/21 18:31:43 INFO Executor: Finished task ID 339 > 14/05/21 18:31:43 INFO DAGScheduler: Completed ResultTask(83, 2) > 14/05/21 18:31:43 INFO TaskSetManager: Finished TID 339 in 1030 ms on > localhost (progress: 4/4) > 14/05/21 18:31:43 INFO TaskSchedulerImpl: Removed TaskSet 83.0, whose tasks > have all completed, from pool > 14/05/21 18:31:43 INFO DAGScheduler: Stage 83 (variance at <doctest > __main__.RDD.variance[0]>:1) finished in 1,032 s > 14/05/21 18:31:43 INFO SparkContext: Job finished: variance at <doctest > __main__.RDD.variance[0]>:1, took 1.034616593 s > 14/05/21 18:31:44 INFO SparkContext: Starting job: collect at <doctest > __main__.RDD.zip[2]>:1 > 14/05/21 18:31:44 INFO DAGScheduler: Got job 66 (collect at <doctest > __main__.RDD.zip[2]>:1) with 4 output partitions (allowLocal=false) > 14/05/21 18:31:44 INFO DAGScheduler: Final stage: Stage 84 (collect at > <doctest __main__.RDD.zip[2]>:1) > 14/05/21 18:31:44 INFO DAGScheduler: Parents of final stage: List() > 14/05/21 18:31:44 INFO DAGScheduler: Missing parents: List() > 14/05/21 18:31:44 INFO DAGScheduler: Submitting Stage 84 (ZippedRDD[220] at > zip at null:-2), which has no missing parents > 14/05/21 18:31:44 INFO DAGScheduler: Submitting 4 missing tasks from Stage 84 > (ZippedRDD[220] at zip at null:-2) > 14/05/21 18:31:44 INFO TaskSchedulerImpl: Adding task set 84.0 with 4 tasks > 14/05/21 18:31:44 INFO TaskSetManager: Starting task 84.0:0 as TID 341 on > executor localhost: localhost (PROCESS_LOCAL) > 14/05/21 18:31:44 INFO TaskSetManager: Serialized task 84.0:0 as 1581 bytes > in 1 ms > 14/05/21 18:31:44 INFO TaskSetManager: Starting task 84.0:1 as TID 342 on > executor localhost: localhost (PROCESS_LOCAL) > 14/05/21 18:31:44 INFO TaskSetManager: Serialized task 84.0:1 as 1581 bytes > in 0 ms > 14/05/21 18:31:44 INFO TaskSetManager: Starting task 84.0:2 as TID 343 on > executor localhost: localhost (PROCESS_LOCAL) > 14/05/21 18:31:44 INFO TaskSetManager: Serialized task 84.0:2 as 1581 bytes > in 0 ms > 14/05/21 18:31:44 INFO TaskSetManager: Starting task 84.0:3 as TID 344 on > executor localhost: localhost (PROCESS_LOCAL) > 14/05/21 18:31:44 INFO TaskSetManager: Serialized task 84.0:3 as 1612 bytes > in 0 ms > 14/05/21 18:31:44 INFO Executor: Running task ID 341 > 14/05/21 18:31:44 INFO Executor: Running task ID 343 > 14/05/21 18:31:44 INFO Executor: Running task ID 344 > 14/05/21 18:31:44 INFO Executor: Running task ID 342 > 14/05/21 18:31:44 INFO Executor: Serialized size of result for 343 is 630 > 14/05/21 18:31:44 INFO Executor: Serialized size of result for 344 is 667 > 14/05/21 18:31:44 INFO Executor: Serialized size of result for 341 is 630 > 14/05/21 18:31:44 INFO Executor: Sending result for 344 directly to driver > 14/05/21 18:31:44 INFO Executor: Sending result for 343 directly to driver > 14/05/21 18:31:44 INFO Executor: Finished task ID 343 > 14/05/21 18:31:44 INFO Executor: Finished task ID 344 > 14/05/21 18:31:44 INFO Executor: Sending result for 341 directly to driver > 14/05/21 18:31:44 INFO Executor: Serialized size of result for 342 is 630 > 14/05/21 18:31:44 INFO TaskSetManager: Finished TID 344 in 3 ms on localhost > (progress: 1/4) > 14/05/21 18:31:44 INFO DAGScheduler: Completed ResultTask(84, 3) > 14/05/21 18:31:44 INFO DAGScheduler: Completed ResultTask(84, 2) > 14/05/21 18:31:44 INFO Executor: Finished task ID 341 > 14/05/21 18:31:44 INFO TaskSetManager: Finished TID 343 in 4 ms on localhost > (progress: 2/4) > 14/05/21 18:31:44 INFO Executor: Sending result for 342 directly to driver > 14/05/21 18:31:44 INFO Executor: Finished task ID 342 > 14/05/21 18:31:44 INFO TaskSetManager: Finished TID 341 in 5 ms on localhost > (progress: 3/4) > 14/05/21 18:31:44 INFO DAGScheduler: Completed ResultTask(84, 0) > 14/05/21 18:31:44 INFO DAGScheduler: Completed ResultTask(84, 1) > 14/05/21 18:31:44 INFO TaskSetManager: Finished TID 342 in 5 ms on localhost > (progress: 4/4) > 14/05/21 18:31:44 INFO DAGScheduler: Stage 84 (collect at <doctest > __main__.RDD.zip[2]>:1) finished in 0,006 s > 14/05/21 18:31:44 INFO TaskSchedulerImpl: Removed TaskSet 84.0, whose tasks > have all completed, from pool > 14/05/21 18:31:44 INFO SparkContext: Job finished: collect at <doctest > __main__.RDD.zip[2]>:1, took 0.010312512 s > 14/05/21 18:31:45 INFO MapOutputTrackerMasterActor: MapOutputTrackerActor > stopped! > 14/05/21 18:31:45 INFO ConnectionManager: Selector thread was interrupted! > 14/05/21 18:31:45 INFO ConnectionManager: ConnectionManager stopped > 14/05/21 18:31:45 INFO MemoryStore: MemoryStore cleared > 14/05/21 18:31:45 INFO BlockManager: BlockManager stopped > 14/05/21 18:31:45 INFO BlockManagerMasterActor: Stopping BlockManagerMaster > 14/05/21 18:31:45 INFO BlockManagerMaster: BlockManagerMaster stopped > 14/05/21 18:31:45 INFO SparkContext: Successfully stopped SparkContext > 14/05/21 18:31:45 INFO RemoteActorRefProvider$RemotingTerminator: Shutting > down remote daemon. > 14/05/21 18:31:45 INFO RemoteActorRefProvider$RemotingTerminator: Remote > daemon shut down; proceeding with flushing remote transports. > 14/05/21 18:31:45 INFO Remoting: Remoting shut down > 14/05/21 18:31:45 INFO RemoteActorRefProvider$RemotingTerminator: Remoting > shut down. > Exception in thread "delete Spark temp dir > C:\Temp\spark-b8e5c57a-dad1-481b-a20e-cd0e6a2ced75" > java.io.IOException: Failed to delete: > C:\Temp\spark-b8e5c57a-dad1-481b-a20e-cd0e6 > a2ced75\tmpbdykfu > at org.apache.spark.util.Utils$.deleteRecursively(Utils.scala:483) > at > org.apache.spark.util.Utils$$anonfun$deleteRecursively$1.apply(Utils.scala:479) > at > org.apache.spark.util.Utils$$anonfun$deleteRecursively$1.apply(Utils.scala:478) > at > scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33) > at > scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:34) > at org.apache.spark.util.Utils$.deleteRecursively(Utils.scala:478) > at org.apache.spark.util.Utils$$anon$4.run(Utils.scala:212) -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org