xiangfu0 commented on code in PR #20060:
URL: https://github.com/apache/pinot/pull/20060#discussion_r4384815723
##########
pinot-plugins/pinot-input-format/pinot-parquet/src/test/java/org/apache/pinot/plugin/inputformat/parquet/ParquetNativeRecordExtractorTest.java:
##########
@@ -274,6 +320,33 @@ public void testInt96ExtractedAsRawLongEpochNanosWhenRaw()
{
epochMillis * 1_000_000L);
}
+ @Test
+ public void testInt96RangeAndNanosecondBoundaries() {
+ MessageType schema = MessageTypeParser.parseMessageType("message R {
required int96 col; }");
+ for (String text : new String[]{"1970-01-01T00:00:00Z",
"1969-12-31T23:59:59.999999999Z",
+ "1677-09-21T00:12:43.145224192Z", "2262-04-11T23:47:16.854775807Z",
Review Comment:
Replaced the repeated time-unit literals with SECONDS_PER_DAY and
NANOS_PER_SECOND in 535d42f9d1538dcaa5b94607f1c85064e2371e0f. Both use TimeUnit
and preserve the existing arithmetic.
##########
pinot-plugins/pinot-input-format/pinot-parquet/src/main/java/org/apache/pinot/plugin/inputformat/parquet/ParquetNativeRecordExtractor.java:
##########
@@ -149,6 +148,11 @@ private Object extractValue(Group from, int fieldIndex,
Type fieldType, int inde
return intValue;
case INT64:
long longValue = from.getLong(fieldIndex, index);
+ if (logicalTypeAnnotation instanceof
LogicalTypeAnnotation.IntLogicalTypeAnnotation integerType
+ && integerType.getBitWidth() == 64 && !integerType.isSigned() &&
longValue < 0) {
+ throw new IllegalArgumentException("Parquet UINT64 field " +
fieldType.getName() + " value "
+ + Long.toUnsignedString(longValue) + " exceeds the signed Long
range");
Review Comment:
The native extractor intentionally rejects these values at every nesting
level. Ingestion aborts with continueOnError=false (default); callers
configured with continueOnError=true can skip rejected rows. I clarified the
description and fixed consumed-row accounting in
2dd474800d2dd281e54a877e1586311b258e88e0 so extraction failures cannot leave
hasNext() stuck at a row-group boundary. A real-file rejected/valid/rejected
regression fails before that fix and passes afterward; all 50 focused tests and
formatting/checkstyle/license checks pass.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
