snleee commented on a change in pull request #6094:
URL: https://github.com/apache/incubator-pinot/pull/6094#discussion_r501253784
##########
File path:
pinot-core/src/main/java/org/apache/pinot/core/segment/processing/utils/SegmentProcessorUtils.java
##########
@@ -66,42 +66,58 @@ public static Schema
convertPinotSchemaToAvroSchema(org.apache.pinot.spi.data.Sc
if (fieldSpec.isSingleValueField()) {
switch (dataType) {
case INT:
- fieldAssembler =
fieldAssembler.name(name).type().intType().noDefault();
+ fieldAssembler =
+
fieldAssembler.name(name).type().unionOf().intType().and().nullType().endUnion().noDefault();
Review comment:
By default, Avro doesn't allow you to have `null` value unless you use
the type of `union(int, null)`. For instance, let's assume that we added a
column to the schema while the segment that we are reading is generated before
and doesn't contain the new column. In this case, if we read the row, the new
column value will always be `null`. Without using a union type, our segment
processor will throw the exception and cannot generate the segment. I added
the comments to the code. This change is more for fixing the existing issue and
make the tests pass. I don't think that this is an important enough to be part
of PR description.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
