rohityadav1993 commented on code in PR #19067:
URL: https://github.com/apache/pinot/pull/19067#discussion_r3680717956
##########
pinot-connectors/pinot-spark-3-connector/src/main/scala/org/apache/pinot/connector/spark/v3/datasource/PinotDataWriter.scala:
##########
@@ -79,8 +80,10 @@ class PinotDataWriter[InternalRow](
override def write(record: catalyst.InternalRow): Unit = {
bufferedRecordReader.write(internalRowToGenericRow(record))
- // Tracking startTime and endTime for segment name generation purposes
- if (timeColumnIndex > -1 && isTimeColumnNumeric) {
+ // Tracking startTime and endTime for segment name generation purposes.
Skip null values so a
+ // null time column does not silently pull `startTime` to 0 and produce a
malformed segment
+ // name -- `record.getLong` returns 0 for nulls without an `isNullAt`
guard.
+ if (timeColumnIndex > -1 && isTimeColumnNumeric &&
!record.isNullAt(timeColumnIndex)) {
Review Comment:
`!record.isNullAt(timeColumnIndex)`
This would regress the behaviour in the scenario of all null time values.
Let's add a check when generting segment name for `startTime ==
Long.MaxValue` and still fallback to `0` to keep the segment name behaviour
unchanged and emit warn logs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
