zhztheplayer commented on issue #5348:
URL:
https://github.com/apache/incubator-gluten/issues/5348#issuecomment-2066009960
I used Gluten+Uniffle bundle from newest main code to run the test but still
didn't repeat the error.
Spark config (3.2.2):
```
spark.app.id local-1713511964160
spark.app.name test-sql-context
spark.app.startTime 1713511959220
spark.driver.host 55f8ad28c275
spark.driver.port 46003
spark.executor.id driver
spark.gluten.memory.conservative.task.offHeap.size.in.bytes 268435456
spark.gluten.memory.offHeap.size.in.bytes 1073741824
spark.gluten.memory.task.offHeap.size.in.bytes 536870912
spark.gluten.sql.session.timeZone.default America/Los_Angeles
spark.gluten.ui.enabled true
spark.hadoop.fs.file.impl org.apache.spark.DebugFilesystem
spark.master local[2]
spark.memory.offHeap.enabled true
spark.memory.offHeap.size 1g
spark.plugins org.apache.gluten.GlutenPlugin
spark.rss.client.type GRPC_NETTY
spark.rss.coordinator.quorum localhost:19999
spark.rss.row.based false
spark.rss.storage.type MEMORY_LOCALFILE
spark.scheduler.mode FIFO
spark.serializer org.apache.spark.serializer.KryoSerializer
spark.shuffle.manager
org.apache.spark.shuffle.gluten.uniffle.UniffleShuffleManager
spark.shuffle.reduceLocality.enabled false
spark.shuffle.service.enabled false
spark.sql.adaptive.localShuffleReader.enabled false
spark.sql.codegen.factoryMode CODEGEN_ONLY
spark.sql.codegen.fallback false
spark.sql.extensions org.apache.gluten.GlutenSessionExtensions
spark.sql.optimizer.excludedRules
org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation
spark.sql.testkey true
spark.sql.warehouse.dir
file:/opt/gluten/spark-warehouse//org.apache.gluten.execution.TestOperator
spark.ui.enabled true
spark.unsafe.exceptionOnMemoryLeak true
```
The xxx dir after all scripts were finished execution:
```
root@55f8ad28c275:~/Workloads/xxx/bug1# tree --du -h ./
./
|-- [801M] data.txt
|-- [ 760] gen_spill_data.py
|-- [704M] parquet
| |-- [ 0] _SUCCESS
| |-- [141M]
part-00000-70f39175-7da9-41db-9f80-08e6870be919-c000.snappy.parquet
| |-- [141M]
part-00001-70f39175-7da9-41db-9f80-08e6870be919-c000.snappy.parquet
| |-- [141M]
part-00002-70f39175-7da9-41db-9f80-08e6870be919-c000.snappy.parquet
| |-- [141M]
part-00003-70f39175-7da9-41db-9f80-08e6870be919-c000.snappy.parquet
| `-- [141M]
part-00004-70f39175-7da9-41db-9f80-08e6870be919-c000.snappy.parquet
`-- [673M] parquet2
|-- [ 0] _SUCCESS
|-- [135M]
part-00000-0ddad3bd-fd52-493b-aabe-7d3f1d9a9861-c000.snappy.parquet
|-- [135M]
part-00001-0ddad3bd-fd52-493b-aabe-7d3f1d9a9861-c000.snappy.parquet
|-- [135M]
part-00002-0ddad3bd-fd52-493b-aabe-7d3f1d9a9861-c000.snappy.parquet
|-- [134M]
part-00003-0ddad3bd-fd52-493b-aabe-7d3f1d9a9861-c000.snappy.parquet
`-- [134M]
part-00004-0ddad3bd-fd52-493b-aabe-7d3f1d9a9861-c000.snappy.parquet
```
Log includes something like
```
24/04/19 15:32:50 WARN RssShuffleWriter: RssShuffle start write
taskAttemptId data8
24/04/19 15:32:50 WARN RssShuffleWriter: RssShuffle start write
taskAttemptId data9
24/04/19 15:33:39 WARN RssShuffleWriter: RssShuffle start write
taskAttemptId data10
24/04/19 15:33:43 WARN RssShuffleWriter: RssShuffle start write
taskAttemptId data11
24/04/19 15:34:31 WARN RssShuffleWriter: RssShuffle start write
taskAttemptId data12
24/04/19 15:34:38 WARN RssShuffleWriter: RssShuffle start write
taskAttemptId data13
```
So I assume uniffle was successfully enabled in my test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
