xxlaykxx opened a new pull request, #36351:
URL: https://github.com/apache/arrow/pull/36351
Added new method rootAsMap() to ComplexWriter and implement it in
ComplexWriterImpl for supporting @map type.
Previously in dremio side:
When i trying to return map like @output ComplexWrite
with this code:
org.apache.arrow.vector.complex.writer.BaseWriter.MapWriter mapWriter =
out.rootAsList().map(false);
mapWriter.startMap();
for (java.util.Map.Entry<java.lang.Integer, java.lang.Integer> element :
map.entrySet()) {
mapWriter.startEntry();
mapWriter.key().integer().writeInt((Integer) element.getKey());
mapWriter.value().integer().writeInt((Integer) element.getValue());
mapWriter.endEntry();
}
mapWriter.endMap();
It use UnionMapWriter and generate schema like:
EXPR$0: Map(false)<$data$: Union(Sparse, [1, 39])<struct: Struct<key:
Int(32, true) not null, value: Int(32, true) not null> not null, map:
Map(false)<entries: Struct<key: Int(32, true) not null, value: Int(32, true)>
not null>>>
But in OutputDerivation impl class where i should create output Complete type
List<Field> children = Arrays.asList( CompleteType.INT.toField("key",
false), CompleteType.INT.toField("value", false));
return new CompleteType(CompleteType.MAP.getType(),
CompleteType.struct(children).toField(MapVector.DATA_VECTOR_NAME, false));
(This is only one valid case, because
MapVector.initializeChildrenFromFields())
return
EXPR$0::map<key::int32, value::int32> I found a place where it start using
union - PromotableWriter.promoteToUnion.
And in the end i have
SCHEMA_CHANGE ERROR: Schema changed during projection. Schema was
schema(EXPR$0::map<key::int32, value::int32>)
but then changed to
schema(EXPR$0::map<struct::struct<key::int32, value::int32>,
map::map<key::int32, value::int32>>)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
