zhengxiang378928908-code commented on issue #350: URL: https://github.com/apache/doris-spark-connector/issues/350#issuecomment-6322886042
This looks fixed in 26.1.0 by #366. In 25.1.0, `RowBatch.longToLocalDateTime(long)` infers the unit of the Arrow timestamp from its value: anything below `10_000_000_000` is treated as seconds. Every value before 1970 is negative, so a pre-1970 timestamp sent in microseconds is read as seconds. For `0203-04-30 00:05:43` that is far outside the `Instant` range, which gives `Instant exceeds minimum or maximum instant`. Values closer to 1970 stay in range but come out wrong: running that helper on the microseconds of `1960-01-01 00:00:00` (UTC) returns a year around `-9999605`. #366 removes the guess. Timestamps without a time zone use the vector's own `LocalDateTime`, and timestamps with one are converted with the unit declared in the Arrow type. @doublewaters could you retry with `spark-doris-connector-spark-3.3-26.1.0` and share the result and your Doris version? I only checked the conversion code, not the full read path for your data. If you read pre-1970 `DATETIME` values with an older connector before, it's worth checking those results too. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
