sunchao commented on code in PR #5603:
URL: https://github.com/apache/datafusion-comet/pull/5603#discussion_r3901211316
##########
spark/src/main/scala/org/apache/comet/vector/NativeUtil.scala:
##########
@@ -322,6 +333,149 @@ class NativeUtil extends AutoCloseable {
}
object NativeUtil {
+
+ /**
+ * Create a vector whose physical struct children remain positional when the
exported Arrow
+ * schema contains duplicate names. Arrow's default struct factory indexes
children by name and
+ * collapses such fields.
+ */
+ private[comet] def createVector(field: Field, allocator: BufferAllocator):
FieldVector = {
+ val runtimeField = fieldForAllocation(field)
+ createPinnedVector(runtimeField, field, allocator)
+ }
+
+ /**
+ * Preserve Arrow's default allocation path unless a duplicate-name struct
needs positional
+ * runtime children. This is called for every imported column of every
native batch.
+ */
+ private[comet] def createVectorForImport(
+ field: Field,
+ allocator: BufferAllocator): FieldVector = {
+ val runtimeField = fieldForAllocation(field)
+ if (runtimeField eq field) {
+ field.createVector(allocator).asInstanceOf[FieldVector]
+ } else {
+ createPinnedVector(runtimeField, field, allocator)
+ }
+ }
+
+ private def createPinnedVector(
+ runtimeField: Field,
+ exportField: Field,
+ allocator: BufferAllocator): FieldVector = {
+ exportField.getType match {
+ case _: ArrowType.List | _: ArrowType.LargeList | _:
ArrowType.FixedSizeList =>
+ val vector = new RenamedListVector(runtimeField, exportField,
allocator)
+ vector.initializeChildrenFromFields(runtimeField.getChildren)
+ vector
+ case _: ArrowType.Map =>
+ val vector = new RenamedMapVector(runtimeField, exportField, allocator)
+ vector.initializeChildrenFromFields(runtimeField.getChildren)
+ vector
+ case _: ArrowType.Struct =>
+ val vector = new RenamedStructVector(runtimeField, exportField,
allocator)
+ // The Field-based StructVector constructor creates the direct
children. Initialize each
+ // child's descendants from the runtime schema without adding the
direct children twice.
+ val runtimeChildren = runtimeField.getChildren
+ var ordinal = 0
+ while (ordinal < runtimeChildren.size()) {
+ vector
+ .getChildByOrdinal(ordinal)
+ .asInstanceOf[FieldVector]
+
.initializeChildrenFromFields(runtimeChildren.get(ordinal).getChildren)
Review Comment:
[P2] Preserve case-distinct direct children before the ordinal loop
Could this also materialize every direct child for case-distinct names such
as `a INT` and `A INT`? An ordinary nonconstant `from_json` result with that
explicit `StructType` reaches this allocator when the dispatcher is enabled and
the native JSON opt-in is off. `fieldForAllocation` leaves these names
unchanged, but Arrow 18.3.0's Field-based `StructVector` constructor creates a
`NullableStructWriter` whose name cache lowercases them. The second same-type
field reuses the first writer without creating another vector. Consequently,
`getChildByOrdinal(1)` is null and this call fails before row processing.
BASE's explicit root-child initialization installed both children under its
default replacement policy. Please preserve both physical children here and add
a same-type, case-distinct regression through the ordinary dispatcher. Simply
repeating the old initialization under the new conflict-error policy is not
sufficient. This is a source-derived failure prediction, not an executed
reproduction.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
