xiangfu0 opened a new pull request, #18970: URL: https://github.com/apache/pinot/pull/18970
## What Fixes a bug in `AvroSchemaUtil.toAvroSchemaJsonObject` flagged in https://github.com/apache/pinot/pull/18870#discussion_r3553639832 (raised as *"worth a separate bugfix"*). The method builds the Avro schema used by the controller's data recommender to generate sample Avro data from a Pinot schema. It switched on `fieldSpec.getDataType().getStoredType()`, which collapses logical types to their physical storage: - a `BOOLEAN` field (stored as INT) emitted Avro `"int"` - a `TIMESTAMP` field (stored as LONG) emitted a plain `"long"` So the generated Avro schema misrepresented the column and could not round-trip back to `BOOLEAN` / `TIMESTAMP`. ## Changes - `AvroSchemaUtil.toAvroSchemaJsonObject` now switches on the **original** `DataType`: - `BOOLEAN` → Avro `boolean` - `TIMESTAMP` → `timestamp-millis` `long` - `UUID` → `bytes` (unchanged; grouped with `BYTES` so it does not regress to a hard failure) - all other types unchanged - `AvroWriter` coerces the generator's stored `int` `0`/`1` for BOOLEAN columns to a `Boolean` so the new schema serializes. The boolean columns are resolved once per file (in the appender constructor), not per row. - Added a `///` doc on `toAvroSchemaJsonObject` recording the output contract and the **intentional** divergence from the segment-processing converters `AvroUtils.getAvroSchemaFromPinotSchema` and `SegmentProcessorAvroUtils.convertPinotSchemaToAvroSchema`, which keep the stored type on purpose because they serialize physically-stored values. Those converters are deliberately **not** changed. ## Tests - `AvroSchemaUtilTest` — asserts every supported type maps to the correct Avro type (incl. `BOOLEAN` → `boolean`, `TIMESTAMP` → `timestamp-millis` long, `UUID` → `bytes`) and that an unsupported type (`BIG_DECIMAL`) is rejected. - `AvroWriterTest` — end-to-end round-trip: generates a record set with INT/LONG/BOOLEAN/TIMESTAMP/STRING columns, writes the Avro file, reads it back, and asserts the schema types and that boolean/timestamp values deserialize as `Boolean`/`Long`; plus a focused test of the boolean coercion (`0`→false, `1`→true, pass-through for non-boolean and null). ## Compatibility Only affects the controller data-recommender's Avro output. No change to segment/ingestion Avro paths. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
