yucai commented on issue #23821: [SPARK-26909][FOLLOWUP][SQL] use unsafeRow.hashCode() as hash value in HashAggregate URL: https://github.com/apache/spark/pull/23821#issuecomment-464627362 Generated codes: ``` /* 054 */ // generate grouping key /* 055 */ agg_mutableStateArray_0[0].reset(); /* 056 */ /* 057 */ agg_mutableStateArray_0[0].write(0, agg_expr_0_0); /* 058 */ int agg_unsafeRowHash_0 = (agg_mutableStateArray_0[0].getRow()).hashCode(); /* 059 */ if (true) { /* 060 */ // try to get the buffer from hash map /* 061 */ agg_unsafeRowAggBuffer_0 = /* 062 */ agg_hashMap_0.getAggregationBufferFromUnsafeRow((agg_mutableStateArray_0[0].getRow()), agg_unsafeRowHash_0); /* 063 */ } /* 064 */ // Can't allocate buffer from the hash map. Spill the map and fallback to sort-based /* 065 */ // aggregation after processing all input rows. /* 066 */ if (agg_unsafeRowAggBuffer_0 == null) { /* 067 */ if (agg_sorter_0 == null) { /* 068 */ agg_sorter_0 = agg_hashMap_0.destructAndCreateExternalSorter(); /* 069 */ } else { /* 070 */ agg_sorter_0.merge(agg_hashMap_0.destructAndCreateExternalSorter()); /* 071 */ } /* 072 */ /* 073 */ // the hash map had be spilled, it should have enough memory now, /* 074 */ // try to allocate buffer again. /* 075 */ agg_unsafeRowAggBuffer_0 = agg_hashMap_0.getAggregationBufferFromUnsafeRow( /* 076 */ (agg_mutableStateArray_0[0].getRow()), agg_unsafeRowHash_0); /* 077 */ if (agg_unsafeRowAggBuffer_0 == null) { /* 078 */ // failed to allocate the first page /* 079 */ throw new org.apache.spark.memory.SparkOutOfMemoryError("No enough memory for aggregation"); /* 080 */ } /* 081 */ } ```
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
