skonto edited a comment on issue #23514: [SPARK-24902][K8s] Add PV integration tests URL: https://github.com/apache/spark/pull/23514#issuecomment-469497166 @shaneknapp works fine with virtual box: ``` ./minikube mount --9p-version=9p2000.L --uid=185 --gid=0 /tmp/test:/tmp/test ssh -i ~/.minikube/machines/minikube/id_rsa docker@$(minikube ip) $ls /tmp -al ... drwxrwxr-x 1 185 root 4096 Mar 5 01:22 test $ ls /tmp/test -al total 5 drwxr-xr-x 1 185 root 4096 Mar 5 01:23 dfs_read_write_test -rw-rw-r-- 1 185 root 8 Mar 5 01:21 tmp1583665121256859192.txt kubectl get pods -n spark NAME READY STATUS RESTARTS AGE spark-test-app-5ed2764629864476b13d10c76f74cdb7 0/1 Completed 0 2m49s ``` spark log: ``` kubectl logs spark-test-app-5ed2764629864476b13d10c76f74cdb7 -n spark ++ id -u + myuid=185 ++ id -g + mygid=0 + set +e ++ getent passwd 185 + uidentry= + set -e + '[' -z '' ']' + '[' -w /etc/passwd ']' + echo '185:x:185:0:anonymous uid:/opt/spark:/bin/false' + SPARK_CLASSPATH=':/opt/spark/jars/*' + env + grep SPARK_JAVA_OPT_ + sort -t_ -k4 -n + sed 's/[^=]*=\(.*\)/\1/g' + readarray -t SPARK_EXECUTOR_JAVA_OPTS + '[' -n '' ']' + '[' -n '' ']' + '[' '' == 2 ']' + '[' '' == 3 ']' + '[' -z ']' + case "$1" in + shift 1 + CMD=("$SPARK_HOME/bin/spark-submit" --conf "spark.driver.bindAddress=$SPARK_DRIVER_BIND_ADDRESS" --deploy-mode client "$@") + exec /sbin/tini -s -- /opt/spark/bin/spark-submit --conf spark.driver.bindAddress=172.17.0.4 --deploy-mode client --properties-file /opt/spark/conf/spark.properties --class org.apache.spark.examples.DFSReadWriteTest spark-internal /opt/spark/pv-tests/tmp1583665121256859192.txt /opt/spark/pv-tests Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties 19/03/05 01:22:23 INFO SecurityManager: Changing view acls to: 185,stavros 19/03/05 01:22:23 INFO SecurityManager: Changing modify acls to: 185,stavros 19/03/05 01:22:23 INFO SecurityManager: Changing view acls groups to: 19/03/05 01:22:23 INFO SecurityManager: Changing modify acls groups to: 19/03/05 01:22:23 INFO SecurityManager: SecurityManager: authentication enabled; ui acls disabled; users with view permissions: Set(185, stavros); groups with view permissions: Set(); users with modify permissions: Set(185, stavros); groups with modify permissions: Set() 19/03/05 01:22:24 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable Performing local word count Creating SparkSession 19/03/05 01:22:24 INFO SparkContext: Running Spark version 3.0.0-SNAPSHOT 19/03/05 01:22:24 INFO SparkContext: Submitted application: DFS Read Write Test 19/03/05 01:22:24 INFO SecurityManager: Changing view acls to: 185,stavros 19/03/05 01:22:24 INFO SecurityManager: Changing modify acls to: 185,stavros 19/03/05 01:22:24 INFO SecurityManager: Changing view acls groups to: 19/03/05 01:22:24 INFO SecurityManager: Changing modify acls groups to: 19/03/05 01:22:24 INFO SecurityManager: SecurityManager: authentication enabled; ui acls disabled; users with view permissions: Set(185, stavros); groups with view permissions: Set(); users with modify permissions: Set(185, stavros); groups with modify permissions: Set() 19/03/05 01:22:24 INFO Utils: Successfully started service 'sparkDriver' on port 7078. 19/03/05 01:22:24 INFO SparkEnv: Registering MapOutputTracker 19/03/05 01:22:24 INFO SparkEnv: Registering BlockManagerMaster 19/03/05 01:22:24 INFO BlockManagerMasterEndpoint: Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information 19/03/05 01:22:24 INFO BlockManagerMasterEndpoint: BlockManagerMasterEndpoint up 19/03/05 01:22:24 INFO DiskBlockManager: Created local directory at /var/data/spark-ddaf8ace-85e7-4bc9-8ceb-2795ab03c8c4/blockmgr-c82635ff-8ba0-47f4-b7a6-131b8944e42b 19/03/05 01:22:24 INFO MemoryStore: MemoryStore started with capacity 593.9 MiB 19/03/05 01:22:24 INFO SparkEnv: Registering OutputCommitCoordinator 19/03/05 01:22:25 INFO Utils: Successfully started service 'SparkUI' on port 4040. 19/03/05 01:22:25 INFO SparkUI: Bound SparkUI to 0.0.0.0, and started at http://spark-test-app-1551748865574-driver-svc.spark.svc:4040 19/03/05 01:22:25 INFO SparkContext: Added JAR local:///opt/spark/examples/jars/spark-examples_2.12-3.0.0-SNAPSHOT.jar at file:/opt/spark/examples/jars/spark-examples_2.12-3.0.0-SNAPSHOT.jar with timestamp 1551748945067 19/03/05 01:22:25 INFO SparkKubernetesClientFactory: Auto-configuring K8S client using current context from users K8S config file 19/03/05 01:22:26 INFO ExecutorPodsAllocator: Going to request 2 executors from Kubernetes. 19/03/05 01:22:26 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 7079. 19/03/05 01:22:26 INFO NettyBlockTransferService: Server created on spark-test-app-1551748865574-driver-svc.spark.svc:7079 19/03/05 01:22:26 INFO BlockManager: Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy 19/03/05 01:22:26 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(driver, spark-test-app-1551748865574-driver-svc.spark.svc, 7079, None) 19/03/05 01:22:26 INFO BlockManagerMasterEndpoint: Registering block manager spark-test-app-1551748865574-driver-svc.spark.svc:7079 with 593.9 MiB RAM, BlockManagerId(driver, spark-test-app-1551748865574-driver-svc.spark.svc, 7079, None) 19/03/05 01:22:26 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(driver, spark-test-app-1551748865574-driver-svc.spark.svc, 7079, None) 19/03/05 01:22:26 INFO BlockManager: Initialized BlockManager: BlockManagerId(driver, spark-test-app-1551748865574-driver-svc.spark.svc, 7079, None) 19/03/05 01:22:30 INFO KubernetesClusterSchedulerBackend$KubernetesDriverEndpoint: Registered executor NettyRpcEndpointRef(spark-client://Executor) (172.17.0.5:60034) with ID 1 19/03/05 01:22:31 INFO BlockManagerMasterEndpoint: Registering block manager 172.17.0.5:38705 with 593.9 MiB RAM, BlockManagerId(1, 172.17.0.5, 38705, None) 19/03/05 01:22:56 INFO KubernetesClusterSchedulerBackend: SchedulerBackend is ready for scheduling beginning after waiting maxRegisteredResourcesWaitingTime: 30000000000(ns) Writing local file to DFS 19/03/05 01:22:56 INFO SharedState: Setting hive.metastore.warehouse.dir ('null') to the value of spark.sql.warehouse.dir ('file:/opt/spark/work-dir/spark-warehouse'). 19/03/05 01:22:56 INFO SharedState: Warehouse path is 'file:/opt/spark/work-dir/spark-warehouse'. 19/03/05 01:22:56 INFO StateStoreCoordinatorRef: Registered StateStoreCoordinator endpoint 19/03/05 01:22:57 INFO deprecation: mapred.output.dir is deprecated. Instead, use mapreduce.output.fileoutputformat.outputdir 19/03/05 01:22:57 INFO HadoopMapRedCommitProtocol: Using output committer class org.apache.hadoop.mapred.FileOutputCommitter 19/03/05 01:22:57 INFO FileOutputCommitter: File Output Committer Algorithm version is 1 19/03/05 01:22:57 INFO SparkContext: Starting job: runJob at SparkHadoopWriter.scala:78 19/03/05 01:22:57 INFO DAGScheduler: Got job 0 (runJob at SparkHadoopWriter.scala:78) with 2 output partitions 19/03/05 01:22:57 INFO DAGScheduler: Final stage: ResultStage 0 (runJob at SparkHadoopWriter.scala:78) 19/03/05 01:22:57 INFO DAGScheduler: Parents of final stage: List() 19/03/05 01:22:57 INFO DAGScheduler: Missing parents: List() 19/03/05 01:22:57 INFO DAGScheduler: Submitting ResultStage 0 (MapPartitionsRDD[1] at saveAsTextFile at DFSReadWriteTest.scala:121), which has no missing parents 19/03/05 01:22:57 INFO MemoryStore: Block broadcast_0 stored as values in memory (estimated size 73.0 KiB, free 593.9 MiB) 19/03/05 01:22:57 INFO MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 26.1 KiB, free 593.8 MiB) 19/03/05 01:22:57 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on spark-test-app-1551748865574-driver-svc.spark.svc:7079 (size: 26.1 KiB, free: 593.9 MiB) 19/03/05 01:22:57 INFO SparkContext: Created broadcast 0 from broadcast at DAGScheduler.scala:1174 19/03/05 01:22:57 INFO DAGScheduler: Submitting 2 missing tasks from ResultStage 0 (MapPartitionsRDD[1] at saveAsTextFile at DFSReadWriteTest.scala:121) (first 15 tasks are for partitions Vector(0, 1)) 19/03/05 01:22:57 INFO TaskSchedulerImpl: Adding task set 0.0 with 2 tasks 19/03/05 01:22:57 INFO TaskSetManager: Starting task 0.0 in stage 0.0 (TID 0, 172.17.0.5, executor 1, partition 0, PROCESS_LOCAL, 7368 bytes) 19/03/05 01:22:58 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on 172.17.0.5:38705 (size: 26.1 KiB, free: 593.9 MiB) 19/03/05 01:22:59 INFO TaskSetManager: Starting task 1.0 in stage 0.0 (TID 1, 172.17.0.5, executor 1, partition 1, PROCESS_LOCAL, 7379 bytes) 19/03/05 01:22:59 INFO TaskSetManager: Finished task 0.0 in stage 0.0 (TID 0) in 2211 ms on 172.17.0.5 (executor 1) (1/2) 19/03/05 01:23:00 INFO TaskSetManager: Finished task 1.0 in stage 0.0 (TID 1) in 1062 ms on 172.17.0.5 (executor 1) (2/2) 19/03/05 01:23:00 INFO TaskSchedulerImpl: Removed TaskSet 0.0, whose tasks have all completed, from pool 19/03/05 01:23:00 INFO DAGScheduler: ResultStage 0 (runJob at SparkHadoopWriter.scala:78) finished in 3.434 s 19/03/05 01:23:00 INFO DAGScheduler: Job 0 is finished. Cancelling potential speculative or zombie tasks for this job 19/03/05 01:23:00 INFO TaskSchedulerImpl: Killing all running tasks in stage 0: Stage finished 19/03/05 01:23:00 INFO DAGScheduler: Job 0 finished: runJob at SparkHadoopWriter.scala:78, took 3.490002 s Reading file from DFS and running Word Count 19/03/05 01:23:02 INFO SparkHadoopWriter: Job job_20190305012257_0001 committed. 19/03/05 01:23:02 INFO MemoryStore: Block broadcast_1 stored as values in memory (estimated size 243.6 KiB, free 593.6 MiB) 19/03/05 01:23:02 INFO MemoryStore: Block broadcast_1_piece0 stored as bytes in memory (estimated size 23.5 KiB, free 593.6 MiB) 19/03/05 01:23:02 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on spark-test-app-1551748865574-driver-svc.spark.svc:7079 (size: 23.5 KiB, free: 593.9 MiB) 19/03/05 01:23:02 INFO SparkContext: Created broadcast 1 from textFile at DFSReadWriteTest.scala:124 19/03/05 01:23:02 INFO FileInputFormat: Total input paths to process : 2 19/03/05 01:23:02 INFO SparkContext: Starting job: countByKey at DFSReadWriteTest.scala:130 19/03/05 01:23:02 INFO DAGScheduler: Registering RDD 8 (countByKey at DFSReadWriteTest.scala:130) 19/03/05 01:23:02 INFO DAGScheduler: Got job 1 (countByKey at DFSReadWriteTest.scala:130) with 4 output partitions 19/03/05 01:23:02 INFO DAGScheduler: Final stage: ResultStage 2 (countByKey at DFSReadWriteTest.scala:130) 19/03/05 01:23:02 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 1) 19/03/05 01:23:02 INFO DAGScheduler: Missing parents: List(ShuffleMapStage 1) 19/03/05 01:23:02 INFO DAGScheduler: Submitting ShuffleMapStage 1 (MapPartitionsRDD[8] at countByKey at DFSReadWriteTest.scala:130), which has no missing parents 19/03/05 01:23:03 INFO MemoryStore: Block broadcast_2 stored as values in memory (estimated size 6.6 KiB, free 593.6 MiB) 19/03/05 01:23:03 INFO MemoryStore: Block broadcast_2_piece0 stored as bytes in memory (estimated size 3.7 KiB, free 593.6 MiB) 19/03/05 01:23:03 INFO BlockManagerInfo: Added broadcast_2_piece0 in memory on spark-test-app-1551748865574-driver-svc.spark.svc:7079 (size: 3.7 KiB, free: 593.9 MiB) 19/03/05 01:23:03 INFO SparkContext: Created broadcast 2 from broadcast at DAGScheduler.scala:1174 19/03/05 01:23:03 INFO DAGScheduler: Submitting 4 missing tasks from ShuffleMapStage 1 (MapPartitionsRDD[8] at countByKey at DFSReadWriteTest.scala:130) (first 15 tasks are for partitions Vector(0, 1, 2, 3)) 19/03/05 01:23:03 INFO TaskSchedulerImpl: Adding task set 1.0 with 4 tasks 19/03/05 01:23:03 INFO TaskSetManager: Starting task 0.0 in stage 1.0 (TID 2, 172.17.0.5, executor 1, partition 0, PROCESS_LOCAL, 7404 bytes) 19/03/05 01:23:03 INFO BlockManagerInfo: Added broadcast_2_piece0 in memory on 172.17.0.5:38705 (size: 3.7 KiB, free: 593.9 MiB) 19/03/05 01:23:03 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on 172.17.0.5:38705 (size: 23.5 KiB, free: 593.9 MiB) 19/03/05 01:23:03 INFO TaskSetManager: Starting task 1.0 in stage 1.0 (TID 3, 172.17.0.5, executor 1, partition 1, PROCESS_LOCAL, 7404 bytes) 19/03/05 01:23:03 INFO TaskSetManager: Finished task 0.0 in stage 1.0 (TID 2) in 425 ms on 172.17.0.5 (executor 1) (1/4) 19/03/05 01:23:03 INFO TaskSetManager: Starting task 2.0 in stage 1.0 (TID 4, 172.17.0.5, executor 1, partition 2, PROCESS_LOCAL, 7404 bytes) 19/03/05 01:23:03 INFO TaskSetManager: Finished task 1.0 in stage 1.0 (TID 3) in 446 ms on 172.17.0.5 (executor 1) (2/4) 19/03/05 01:23:03 INFO TaskSetManager: Starting task 3.0 in stage 1.0 (TID 5, 172.17.0.5, executor 1, partition 3, PROCESS_LOCAL, 7404 bytes) 19/03/05 01:23:03 INFO TaskSetManager: Finished task 2.0 in stage 1.0 (TID 4) in 63 ms on 172.17.0.5 (executor 1) (3/4) 19/03/05 01:23:04 INFO TaskSetManager: Finished task 3.0 in stage 1.0 (TID 5) in 55 ms on 172.17.0.5 (executor 1) (4/4) 19/03/05 01:23:04 INFO TaskSchedulerImpl: Removed TaskSet 1.0, whose tasks have all completed, from pool 19/03/05 01:23:04 INFO DAGScheduler: ShuffleMapStage 1 (countByKey at DFSReadWriteTest.scala:130) finished in 1.020 s 19/03/05 01:23:04 INFO DAGScheduler: looking for newly runnable stages 19/03/05 01:23:04 INFO DAGScheduler: running: Set() 19/03/05 01:23:04 INFO DAGScheduler: waiting: Set(ResultStage 2) 19/03/05 01:23:04 INFO DAGScheduler: failed: Set() 19/03/05 01:23:04 INFO DAGScheduler: Submitting ResultStage 2 (ShuffledRDD[9] at countByKey at DFSReadWriteTest.scala:130), which has no missing parents 19/03/05 01:23:04 INFO MemoryStore: Block broadcast_3 stored as values in memory (estimated size 4.1 KiB, free 593.6 MiB) 19/03/05 01:23:04 INFO MemoryStore: Block broadcast_3_piece0 stored as bytes in memory (estimated size 2.5 KiB, free 593.6 MiB) 19/03/05 01:23:04 INFO BlockManagerInfo: Added broadcast_3_piece0 in memory on spark-test-app-1551748865574-driver-svc.spark.svc:7079 (size: 2.5 KiB, free: 593.9 MiB) 19/03/05 01:23:04 INFO SparkContext: Created broadcast 3 from broadcast at DAGScheduler.scala:1174 19/03/05 01:23:04 INFO DAGScheduler: Submitting 4 missing tasks from ResultStage 2 (ShuffledRDD[9] at countByKey at DFSReadWriteTest.scala:130) (first 15 tasks are for partitions Vector(0, 1, 2, 3)) 19/03/05 01:23:04 INFO TaskSchedulerImpl: Adding task set 2.0 with 4 tasks 19/03/05 01:23:04 INFO TaskSetManager: Starting task 1.0 in stage 2.0 (TID 6, 172.17.0.5, executor 1, partition 1, NODE_LOCAL, 7162 bytes) 19/03/05 01:23:04 INFO BlockManagerInfo: Added broadcast_3_piece0 in memory on 172.17.0.5:38705 (size: 2.5 KiB, free: 593.9 MiB) 19/03/05 01:23:04 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 0 to 172.17.0.5:60034 19/03/05 01:23:04 INFO TaskSetManager: Starting task 2.0 in stage 2.0 (TID 7, 172.17.0.5, executor 1, partition 2, NODE_LOCAL, 7162 bytes) 19/03/05 01:23:04 INFO TaskSetManager: Finished task 1.0 in stage 2.0 (TID 6) in 108 ms on 172.17.0.5 (executor 1) (1/4) 19/03/05 01:23:04 INFO TaskSetManager: Starting task 0.0 in stage 2.0 (TID 8, 172.17.0.5, executor 1, partition 0, PROCESS_LOCAL, 7162 bytes) 19/03/05 01:23:04 INFO TaskSetManager: Finished task 2.0 in stage 2.0 (TID 7) in 23 ms on 172.17.0.5 (executor 1) (2/4) 19/03/05 01:23:04 INFO TaskSetManager: Starting task 3.0 in stage 2.0 (TID 9, 172.17.0.5, executor 1, partition 3, PROCESS_LOCAL, 7162 bytes) 19/03/05 01:23:04 INFO TaskSetManager: Finished task 0.0 in stage 2.0 (TID 8) in 12 ms on 172.17.0.5 (executor 1) (3/4) 19/03/05 01:23:04 INFO TaskSetManager: Finished task 3.0 in stage 2.0 (TID 9) in 10 ms on 172.17.0.5 (executor 1) (4/4) 19/03/05 01:23:04 INFO TaskSchedulerImpl: Removed TaskSet 2.0, whose tasks have all completed, from pool 19/03/05 01:23:04 INFO DAGScheduler: ResultStage 2 (countByKey at DFSReadWriteTest.scala:130) finished in 0.158 s 19/03/05 01:23:04 INFO DAGScheduler: Job 1 is finished. Cancelling potential speculative or zombie tasks for this job 19/03/05 01:23:04 INFO TaskSchedulerImpl: Killing all running tasks in stage 2: Stage finished 19/03/05 01:23:04 INFO DAGScheduler: Job 1 finished: countByKey at DFSReadWriteTest.scala:130, took 1.514685 s 19/03/05 01:23:04 INFO SparkUI: Stopped Spark web UI at http://spark-test-app-1551748865574-driver-svc.spark.svc:4040 19/03/05 01:23:04 INFO KubernetesClusterSchedulerBackend: Shutting down all executors 19/03/05 01:23:04 INFO KubernetesClusterSchedulerBackend$KubernetesDriverEndpoint: Asking each executor to shut down 19/03/05 01:23:04 WARN ExecutorPodsWatchSnapshotSource: Kubernetes client has been closed (this is expected if the application is shutting down.) 19/03/05 01:23:04 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped! 19/03/05 01:23:04 INFO MemoryStore: MemoryStore cleared 19/03/05 01:23:04 INFO BlockManager: BlockManager stopped 19/03/05 01:23:04 INFO BlockManagerMaster: BlockManagerMaster stopped 19/03/05 01:23:04 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped! 19/03/05 01:23:04 INFO SparkContext: Successfully stopped SparkContext Success! Local Word Count 2 and DFS Word Count 2 agree. 19/03/05 01:23:04 INFO ShutdownHookManager: Shutdown hook called 19/03/05 01:23:04 INFO ShutdownHookManager: Deleting directory /tmp/spark-35e91f25-4d01-4c19-ba92-4e2aa6940e89 19/03/05 01:23:04 INFO ShutdownHookManager: Deleting directory /var/data/spark-ddaf8ace-85e7-4bc9-8ceb-2795ab03c8c4/spark-a2668634-72b2-4dbe-badf-268388f08370 ``` Had to lower the interval to 10 ms (for eventually) because that write finishes fast but things work.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
