yew1eb opened a new issue, #1626:
URL: https://github.com/apache/auron/issues/1626
**Describe the bug**
**To Reproduce**
reproduce the issue with the following code:
```
val customSpark = SparkSession
.builder()
.master("local[2]")
.appName("checkConvertToNativeShuffleManger")
.config("spark.sql.shuffle.partitions", "4")
.config("spark.sql.autoBroadcastJoinThreshold", -1)
.config("spark.shuffle.manager",
"org.apache.spark.shuffle.sort.SortShuffleManager")
.config("spark.sql.extensions",
"org.apache.spark.sql.auron.AuronSparkSessionExtension")
.config("spark.memory.offHeap.enabled", "false")
.config("spark.auron.enable", "true")
.getOrCreate()
customSpark.sql("drop table if exists test_shuffle")
customSpark.sql(
"create table if not exists test_shuffle using parquet PARTITIONED BY
(part) " +
"as select 1 as c1, 2 as c2, 'test test' as part"
)
val df = customSpark.sql("select c1, count(1) from test_shuffle group by c1")
checkAnswer(df, Seq(Row(1, 1)))
```
generated execution plan:
```
HashAggregate(keys=[c1#10], functions=[count(1)], output=[c1#10,
count(1)#13L])
+- AQEShuffleRead coalesced
+- ShuffleQueryStage 0
+- Exchange hashpartitioning(c1#10, 4), ENSURE_REQUIREMENTS,
[plan_id=68]
+- NativeHashAggregate HashAgg, [c1#10], [partial_count(1)],
[count#17L], 0
+- NativeProject [c1#10]
+- InputAdapter [#10, #12]
+- NativeParquetScan spark_catalog.default.test_shuffle
(FileScan parquet spark_catalog.default.test_shuffle[c1#10,part#12] Batched:
true, DataFilters: [], Format: Parquet, Location: CatalogFileIndex(1
paths)[file:/Users/yew1eb/workspaces/auron-master/spark-extension-shims-spark/...,
PartitionFilters: [], PushedFilters: [], ReadSchema: struct<c1:int>)
partial agg: NativeHashAggregate
|
v
shuffle: Spark Exchange
|
v
final agg: Spark HashAggregate
```
Incorrect results:
```
== Results ==
!== Correct Answer - 1 == == Spark Answer - 1 ==
!struct<> struct<c1:int,count(1):bigint>
![1,1] [1,103079215105]
```
**Expected behavior**
**Screenshots**
**Additional context**
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
