SteNicholas opened a new issue, #540:
URL: https://github.com/apache/paimon-cpp/issues/540

   ### Search before asking
   
   - [x] I searched in the 
[issues](https://github.com/apache/paimon-cpp/issues) and found no existing 
issue tracking the remaining end-to-end TIME support.
   
   Related completed work: #292 / #378 added Parquet TIME reading, and #477 / 
#478 added Lance TIME support. This issue tracks the remaining gaps.
   
   ### Motivation
   
   Sub-issue of #519: **Core Feature Alignment — Complete TIME type support**.
   
   Paimon C++ already parses `TIME(p)` and supports TIME in parts of the 
append-table and file-format paths. The current [data type 
documentation](https://github.com/apache/paimon-cpp/blob/main/docs/source/user_guide/data_types.rst)
 still lists important restrictions:
   
   - Primary-key tables cannot use TIME as either a key or a value column.
   - TIME cannot be a partition or bucket key.
   - Predicates cannot reference TIME columns.
   - ORC and Avro TIME support, and end-to-end read/write/statistics coverage 
across formats, remain incomplete.
   
   These gaps prevent Java-written TIME schemas from being used consistently 
through the C++ table APIs. The roadmap item should be completed through 
focused changes with explicit compatibility checks.
   
   ### Solution
   
   Preserve the existing Java-compatible representation: `TIME(p)`, for `p = 
0..9`, uses Arrow `time32[ms]` and an integer number of milliseconds since 
midnight. Preserve declared precision through `paimon.time.precision` metadata, 
including the default `TIME(0)`. Higher declared precision must continue to 
follow Paimon Java's millisecond storage semantics.
   
   Suggested implementation order:
   
   - [ ] **Complete scalar and row plumbing.** Add the missing public/internal 
type mappings and literal conversions; support TIME in internal row/array 
access, binary serialization, comparisons, Arrow row conversion, statistics, 
and the casting/default-value paths needed for Java compatibility. Preserve 
existing public enum values and serialization compatibility when extending type 
handling.
   - [ ] **Enable predicates.** Support equality, ordering, `IN` / `NOT IN`, 
and null checks through predicate validation, row evaluation, Arrow batch 
evaluation, and statistics pruning. Validate that pruning never removes 
matching rows.
   - [ ] **Enable primary-key tables.** Support TIME value columns and TIME 
primary keys through write, commit, read, upsert, delete, and compaction, 
covering the existing applicable merge and deletion-vector paths.
   - [ ] **Enable partition and bucket keys.** Align TIME partition string 
conversion and bucket hashing with Java, and validate routing and pruning using 
Java-generated tables.
   - [ ] **Complete file-format support.** Audit the existing TIME paths and 
close read/write/statistics gaps, including Parquet table writes and ORC/Avro 
mappings. Preserve working Lance/Vortex paths and cover other enabled formats 
where their Java counterparts support TIME.
   - [ ] **Update documentation.** Replace the current restrictions with the 
verified support matrix and document any remaining format-specific limitations.
   
   Useful implementation entry points:
   
   - [Public field 
types](https://github.com/apache/paimon-cpp/blob/main/include/paimon/defs.h) 
and [field type 
mapping](https://github.com/apache/paimon-cpp/blob/main/src/paimon/common/utils/field_type_utils.h).
   - [Literal 
conversion](https://github.com/apache/paimon-cpp/blob/main/src/paimon/common/predicate/literal_converter.cpp)
 and [predicate 
validation](https://github.com/apache/paimon-cpp/blob/main/src/paimon/common/predicate/predicate_validator.h).
   - [Internal row 
access](https://github.com/apache/paimon-cpp/blob/main/src/paimon/common/data/internal_row.cpp),
 binary serializers, and [Arrow row 
conversion](https://github.com/apache/paimon-cpp/blob/main/src/paimon/core/io/row_to_arrow_array_converter.h).
   - [Schema 
validation](https://github.com/apache/paimon-cpp/blob/main/src/paimon/core/schema/schema_validation.cpp),
 partition conversion, bucket calculation, and format-specific 
adapters/statistics.
   
   ### Anything else?
   
   Acceptance and validation:
   
   - [ ] Add focused unit and integration tests for the implemented paths.
   - [ ] Cover `NULL`, midnight, `00:00:00.123`, `23:59:59.999`, declared 
precisions `0/3/6/9`, precision-metadata round trips, and invalid 
precision/unit inputs.
   - [ ] Cover TIME predicates with and without pruning, primary-key 
upserts/deletes and compaction, partition/bucket routing, and format read/write 
finalization.
   - [ ] Validate Java-written data from C++ and C++-written data from Java for 
supported formats and table modes; include key hashing, partition values, 
nulls, and declared precision.
   - [ ] Keep existing compatibility tests passing and update the documented 
support matrix.
   
   This issue may be implemented in multiple focused PRs, starting with 
scalar/predicate support and primary-key value columns. Keep the parent roadmap 
item open until the full agreed scope and compatibility validation are complete.
   
   ### Are you willing to submit a PR?
   
   - [ ] I'm willing to submit a PR!
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to