Github user nongli commented on a diff in the pull request:
https://github.com/apache/spark/pull/12345#discussion_r59649417
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/ColumnarAggMapCodeGenerator.scala
---
@@ -65,27 +69,43 @@ class ColumnarAggMapCodeGenerator(
.mkString("\n")};
""".stripMargin
+ val generatedAggBufferSchema: String =
+ s"""
+ |new org.apache.spark.sql.types.StructType()
+ |${bufferSchema.map(key =>
+ s""".add("${key.name}",
org.apache.spark.sql.types.DataTypes.${key.dataType})""")
+ .mkString("\n")};
+ """.stripMargin
+
s"""
| private org.apache.spark.sql.execution.vectorized.ColumnarBatch
batch;
+ | private org.apache.spark.sql.execution.vectorized.ColumnarBatch
aggregateBufferBatch;
| private int[] buckets;
| private int numBuckets;
| private int maxSteps;
| private int numRows = 0;
| private org.apache.spark.sql.types.StructType schema =
$generatedSchema
+ | private org.apache.spark.sql.types.StructType
aggregateBufferSchema =
+ | $generatedAggBufferSchema
|
- | public $generatedClassName(int capacity, double loadFactor, int
maxSteps) {
- | assert (capacity > 0 && ((capacity & (capacity - 1)) == 0));
- | this.maxSteps = maxSteps;
- | numBuckets = (int) (capacity / loadFactor);
+ | public $generatedClassName() {
+ | // TODO: These should be generated based on the schema
+ | int DEFAULT_CAPACITY = 1 << 16;
+ | double DEFAULT_LOAD_FACTOR = 0.25;
+ | int DEFAULT_MAX_STEPS = 2;
+ | assert (DEFAULT_CAPACITY > 0 && ((DEFAULT_CAPACITY &
(DEFAULT_CAPACITY - 1)) == 0));
+ | this.maxSteps = DEFAULT_MAX_STEPS;
+ | numBuckets = (int) (DEFAULT_CAPACITY / DEFAULT_LOAD_FACTOR);
| batch =
org.apache.spark.sql.execution.vectorized.ColumnarBatch.allocate(schema,
- | org.apache.spark.memory.MemoryMode.ON_HEAP, capacity);
+ | org.apache.spark.memory.MemoryMode.ON_HEAP,
DEFAULT_CAPACITY);
+ | aggregateBufferBatch =
org.apache.spark.sql.execution.vectorized.ColumnarBatch.allocate(
+ | aggregateBufferSchema,
org.apache.spark.memory.MemoryMode.ON_HEAP, DEFAULT_CAPACITY);
--- End diff --
Let's leave a TODO to fix this. There should be a nicer way to get a
projection of a batch instead of this.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
