1fanwang opened a new pull request, #29962:
URL: https://github.com/apache/flink/pull/29962

   ## What is the purpose of the change
   
   [FLINK-41306](https://issues.apache.org/jira/browse/FLINK-41306) reports 
pre-1970 timestamps that make Flink reads fail. Floor division selects the 
correct Julian day, and floor modulo keeps nanos-of-day within one day in both 
timezone modes.
   
   ## Brief change log
   
   - Normalize pre-epoch INT96 values.
   - Add raw encoding, reader, SQL, and INT64 regression coverage.
   
   ## Verifying this change
   
   ```bash
   export 
JAVA_HOME=/opt/homebrew/opt/openjdk@17/libexec/openjdk.jdk/Contents/Home
   ./mvnw -pl flink-formats/flink-parquet -Dtest=ParquetRowDataWriterTest test
   ./mvnw -pl flink-formats/flink-parquet 
-Dtest=ParquetFileSystemITCase#testPreEpochTimestamp -DfailIfNoTests=false test
   ```
   
   ```text
   UTC=false: -28798750000000L (expected 0..86399999999999)
   UTC=true: -43199750000000L (expected 0..86399999999999)
   Caused by: java.lang.IllegalArgumentException: nanos > 999999999 or < 0
   ```
   
   After the fix:
   
   ```bash
   ./mvnw -pl flink-formats/flink-parquet -Dtest=ParquetRowDataWriterTest test
   ./mvnw -pl flink-formats/flink-parquet 
-Dtest=ParquetFileSystemITCase#testPreEpochTimestamp -DfailIfNoTests=false test
   ```
   
   ```text
   UTC=false: julianDay=2437101 nanosOfDay=57600250000000
   UTC=true: julianDay=2437101 nanosOfDay=43200250000000
   Rows: [+I[1, 1960-06-15T12:00], +I[2, 1960-06-15T12:00:00.250], +I[3, 
1980-06-15T12:00:00.250]]
   ```
   
   <details>
   <summary>Reproducer source: ParquetFileSystemITCase.java</summary>
   
   ```sql
   CREATE TABLE t (id INT, ts TIMESTAMP(3))
   WITH ('connector'='filesystem', 'path'='<temporary path>', 
'format'='parquet');
   INSERT INTO t VALUES (1, TIMESTAMP '1960-06-15 12:00:00.000'),
     (2, TIMESTAMP '1960-06-15 12:00:00.250'),
     (3, TIMESTAMP '1980-06-15 12:00:00.250');
   SELECT * FROM t;
   ```
   
   </details>
   
   ## Does this pull request potentially affect one of the following parts:
   
     - Dependencies (does it add or upgrade a dependency): no
     - The public API, i.e., is any changed class annotated with 
@Public(Evolving): no
     - The serializers: yes
     - The runtime per-record code paths (performance sensitive): yes
     - Anything that affects deployment or recovery: no
     - The S3 file system connector: no
   
   ## Documentation
   
     - Does this pull request introduce a new feature? no
     - If yes, how is the feature documented? not applicable
   
   ---
   
   ##### Was generative AI tooling used to co-author this PR?
   
   - [X] Yes (please specify the tool below)
   
   Generated-by: GitHub Copilot CLI 1.0.110 (Auto)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to