cloud-fan commented on code in PR #57497:
URL: https://github.com/apache/spark/pull/57497#discussion_r3746488886
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/variant/VariantExpressionEvalUtils.scala:
##########
@@ -280,6 +280,61 @@ object VariantExpressionEvalUtils {
new VariantVal(v.getValue, v.getMetadata)
}
+ /**
+ * Build a variant object directly from a keys array and a values array,
without materializing an
+ * intermediate map. Keys must be non-null strings and the two arrays must
have equal length. A
+ * null key raises `NULL_MAP_KEY`, a duplicate key raises
`VARIANT_DUPLICATE_KEY` (matching
+ * to_variant_object), and null values are kept as variant null.
+ */
+ def variantFromArrays(keys: ArrayData, values: ArrayData, valueType:
DataType): VariantVal = {
+ if (keys.numElements() != values.numElements()) {
+ // Reuse the same error map_from_arrays raises for a keys/values length
mismatch.
+ throw
QueryExecutionErrors.mapDataKeyArrayLengthDiffersFromValueArrayLengthError()
+ }
+ val builder = new VariantBuilder(false)
+ val start = builder.getWritePos
+ val fields = new
java.util.ArrayList[VariantBuilder.FieldEntry](keys.numElements())
+ for (i <- 0 until keys.numElements()) {
Review Comment:
Please cache `numElements()` and use index-based `while` loops here and in
the matching entries loop at `VariantExpressionEvalUtils.scala:322`. These
helpers execute once per input row, so `0 until ...` adds avoidable
Range/foreach allocation in a Catalyst hot path.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
