github-actions[bot] commented on code in PR #69536:
URL: https://github.com/apache/doris/pull/69536#discussion_r4369032165
##########
fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanPlanProvider.java:
##########
@@ -2327,8 +2343,77 @@ private static List<Schema> metadataSchemaHistory(Table
table) {
return new ArrayList<>(table.schemas().values());
}
- private static void addHistoricalEqualityFields(
- List<NestedField> fields, Set<Integer> missingFieldIds, Schema
historicalSchema) {
+ /**
+ * A writer commits with the schema that is current at commit, so a key
dropped before its delete commits loses
+ * the schema-only changes made before the drop from every snapshot. Field
IDs are never reused and a field
+ * only relaxes compatibly, by promotion or from required to optional, so
each field of the selected definition
+ * takes the widest promotion of its type and the optionality it has in
any retained schema; neither depends on
+ * the order of the schemas. A field required in every retained schema
stays required; a rebuilt field keeps its
+ * initial default in the promoted type ({@link #promotedDefault}) and
drops its write default, which no reader
+ * uses and whose stale value may not fit the promoted type.
+ */
+ private static List<NestedField> withRetainedRelaxations(List<NestedField>
fields, List<Schema> schemas) {
+ List<NestedField> relaxed = new ArrayList<>(fields.size());
+ for (NestedField field : fields) {
+ Type type = field.type();
+ boolean optional = field.isOptional();
+ for (Schema schema : schemas) {
+ NestedField retained = schema.findField(field.fieldId());
+ if (retained == null) {
+ continue;
+ }
+ optional |= retained.isOptional();
+ if (type.isPrimitiveType() && retained.type().isPrimitiveType()
+ && isWideningPromotion(type.asPrimitiveType(),
retained.type().asPrimitiveType())) {
Review Comment:
[P1] Resolve competing temporal targets before choosing the carrier type
This single pass can keep an abandoned TIMESTAMP definition and reject the
TIMESTAMP_NS definition actually used by the delete. On an unpartitioned v3
table, append S1 with optional `k DATE = 2024-01-02`; publish a schema-only
DATE→TIMESTAMP promotion; restore the original DATE schema with
`TableMetadata.Builder.setCurrentSchema(existingId)` before any new snapshot or
data is written; then publish DATE→TIMESTAMP_NS. Both forward promotions are
allowed by the [v3 promotion
table](https://github.com/apache/iceberg/blob/apache-iceberg-1.12.0/format/spec.md#schema-evolution),
and
[setCurrentSchema](https://github.com/apache/iceberg/blob/apache-iceberg-1.12.0/core/src/main/java/org/apache/iceberg/TableMetadata.java#L1019-L1051)
accepts the retained ID. Prepare a nanos equality delete for
`2024-01-02T00:00:00.000000001`, drop k, and commit the pending delete. Omit
all column metrics so the delete reaches BE.
The lineage supplies DATE and retained order
DATE/TIMESTAMP/TIMESTAMP_NS/dropped first selects TIMESTAMP.
TIMESTAMP→TIMESTAMP_NS is not a recognized promotion, so the carrier stays
DATETIMEV2(6). V2 `_resolve_equality_delete_fields` uses that carrier type, and
`_load_equality_delete_file` casts the nanos tombstone to it;
`round_timestamp_ns_fraction` rounds 1ns to zero. The original midnight row is
incorrectly deleted. The base reverse lookup selected TIMESTAMP_NS and kept it.
Handle competing retained temporal domains without a lossy comparison (or
fail explicitly), and cover this rollback history. This differs from the
existing DATE thread: both DATE promotion edges are implemented here, but the
first retained alternative blocks the other; it is also a current, in-range
wrong-result case rather than the existing time-travel range error.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
