shouwangyw commented on PR #3157: URL: https://github.com/apache/celeborn/pull/3157#issuecomment-2764943445
> Can you provide some test results for this PR? For example, 10G terasort tests could be enough. We can compare the performance with the OSS and the local disks. I performed the following test: 1. Spark Submission Command： `spark-shell --conf spark.driver.cores=1 --conf spark.driver.memory=1G --conf spark.executor.instances=4 --conf spark.executor.cores=1 --conf spark.executor.memory=2G --conf spark.celeborn.storage.availableTypes=OSS` 2. Generating ~20GB of Shuffle Data: spark.sparkContext.parallelize(1 to 200, 200).flatMap(_ => (1 to 15000000).iterator.map(num => num)).repartition(20).count 3. Test Screenshot:  -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
