sagarlakshmipathy commented on issue #4909: URL: https://github.com/apache/incubator-gluten/issues/4909#issuecomment-1989484397
thanks @zhouyuan Can confirm this worked on ubuntu 22.04 ``` $SPARK_HOME/bin/spark-shell --jars https://github.com/oap-project/gluten/releases/download/v1.1.0/gluten-velox-bundle-spark3.3_2.12-1.1.0.jar --conf spark.plugins=io.glutenproject.GlutenPlugin --conf spark.memory.offHeap.enabled=true --conf spark.memory.offHeap.size=20g --conf spark.shuffle.manager=org.apache.spark.shuffle.sort.ColumnarShuffleManager ``` ``` scala> (0 until 10).toDF("a").write.mode("overwrite").parquet("/tmp/test") 24/03/11 20:48:10 WARN MemoryManager: Total allocation exceeds 95.00% (962,776,257 bytes) of heap memory Scaling row group sizes to 89.67% for 8 writers 24/03/11 20:48:10 WARN MemoryManager: Total allocation exceeds 95.00% (962,776,257 bytes) of heap memory Scaling row group sizes to 79.70% for 9 writers 24/03/11 20:48:10 WARN MemoryManager: Total allocation exceeds 95.00% (962,776,257 bytes) of heap memory Scaling row group sizes to 71.73% for 10 writers 24/03/11 20:48:10 WARN MemoryManager: Total allocation exceeds 95.00% (962,776,257 bytes) of heap memory Scaling row group sizes to 79.70% for 9 writers 24/03/11 20:48:10 WARN MemoryManager: Total allocation exceeds 95.00% (962,776,257 bytes) of heap memory Scaling row group sizes to 89.67% for 8 writers scala> spark.read.parquet("/tmp/test").createOrReplaceTempView("t1") scala> spark.sql("select * from t1 where a > 5").explain == Physical Plan == VeloxColumnarToRowExec +- ^(1) FilterExecTransformer (isnotnull(a#7) AND (a#7 > 5)) +- ^(1) NativeFileNativeScan parquet [a#7] Batched: true, DataFilters: [isnotnull(a#7), (a#7 > 5)], Format: Parquet, Location: InMemoryFileIndex(1 paths)[file:/tmp/test], PartitionFilters: [], PushedFilters: [IsNotNull(a), GreaterThan(a,5)], ReadSchema: struct<a:int> ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
