Doris-Breakwater commented on issue #70500: URL: https://github.com/apache/doris/issues/70500#issuecomment-6817390423
Breakwater-GitHub-Analysis-Slot: slot_fca1231f76d1 **Initial assessment:** The reported observability gap is supported by source inspection at `8a9961723ea4be00cdf923c60759607202c7e2e7`. `backend_tablets.COMPRESS_KIND` exposes a historical metadata field whose value is independent of the configured segment-page codec. Its `COMPRESS_LZ4` output does not establish that ZSTD page compression was ignored. This warrants a system-table/documentation improvement; the report supplies no evidence of a page-compression failure or data corruption. **Verified code path at the reported commit** - FE resolves an explicit `"compression" = "zstd"` to `TCompressionType.ZSTD`; `default_compression_type` is consulted when that property is absent. The explicit-property reproduction therefore does not depend on the FE default. See [PropertyAnalyzer.java](https://github.com/apache/doris/blob/8a9961723ea4be00cdf923c60759607202c7e2e7/fe/fe-core/src/main/java/org/apache/doris/common/util/PropertyAnalyzer.java#L1092-L1134). The replica-creation request carries the compression type through [CreateReplicaTask.java](https://github.com/apache/doris/blob/8a9961723ea4be00cdf923c60759607202c7e2e7/fe/fe-core/src/main/java/org/apache/doris/task/CreateReplicaTask.java#L418). - [TabletMeta construction](https://github.com/apache/doris/blob/8a9961723ea4be00cdf923c60759607202c7e2e7/be/src/olap/tablet_meta.cpp#L208-L234) sets `compress_kind = COMPRESS_LZ4` and separately maps `TCompressionType.ZSTD` to `segment_v2::ZSTD` in `compression_type`. These are distinct protobuf fields. The older [CompressKind enum](https://github.com/apache/doris/blob/8a9961723ea4be00cdf923c60759607202c7e2e7/gensrc/proto/olap_common.proto#L45-L49) has no ZSTD member. - The scanner takes `CompressKind_Name(tablet->compress_kind())` into its snapshot and emits it as `COMPRESS_KIND`. This directly explains the reported result: [schema_tablets_scanner.cpp](https://github.com/apache/doris/blob/8a9961723ea4be00cdf923c60759607202c7e2e7/be/src/exec/schema_scanner/schema_tablets_scanner.cpp#L160-L172). - [SegmentWriter::init](https://github.com/apache/doris/blob/8a9961723ea4be00cdf923c60759607202c7e2e7/be/src/olap/rowset/segment_v2/segment_writer.cpp#L342-L356) takes `tablet_schema->compression_type()` when writer options have no explicit override. It records the selected codec in column metadata; [ScalarColumnWriter::init](https://github.com/apache/doris/blob/8a9961723ea4be00cdf923c60759607202c7e2e7/be/src/olap/rowset/segment_v2/column_writer.cpp#L476-L477) obtains the compression codec from that metadata. This path does not select the page codec from `compress_kind`. The [4.x system-table documentation](https://doris.apache.org/docs/4.x/admin-manual/system-tables/information_schema/backend_tablets/) describes the column only as the kind of compression, without explaining this distinction. The [compression documentation](https://doris.apache.org/docs/4.x/table-design/column-compression/) describes the table property and FE default for page compression, so the ambiguity is understandable. **Suggested maintainer action** Prefer appending a separate `COMPRESSION_TYPE` column sourced from the tablet schema's `compression_type()`, formatted with the page-compression enum name, while preserving `COMPRESS_KIND` for existing consumers. Update both the FE `SchemaTable.java` definition and BE scanner column/output layout consistently, preserving existing column positions. Define the new column as the tablet's configured page codec, rather than a measurement of every stored page. Clarify the historical meaning of `COMPRESS_KIND` in the documentation. Documentation clarification alone is a smaller alternative, but leaves per-tablet page-codec inspection unresolved. Add regression coverage for explicit ZSTD and LZ4F/LZ4 settings, plus an omitted-property case with a known FE default. Check both columns for each replica and persistence after metadata reload; cover coupled and decoupled deployments where supported. The existing `backend_tablets.groovy` test checks tablet counts, not codec reporting. Avoid changing `compress_kind` initialization to implement this: the two fields use different enums and serve different metadata contracts. **Remaining evidence and scope** This was read-only source analysis; I did not run the SQL reproduction or inspect deployed segment files. Please attach the exact FE/BE build versions or hashes, deployment mode, `SHOW CREATE TABLE compression_kind_repro`, `SHOW TABLETS` output, and the corresponding `BE_ID, TABLET_ID, REPLICA_ID, COMPRESS_KIND` query rows. For the default-setting case, also provide the FE default at table-creation time and a reproduction without an explicit compression property. The supplied reproduction creates an empty table, which is sufficient to demonstrate metadata reporting but cannot verify stored-page compression. If actual page encoding is also in doubt, load data and inspect the resulting segment column metadata and page bodies. Even with ZSTD selected, [PageIO](https://github.com/apache/doris/blob/8a9961723ea4be00cdf923c60759607202c7e2e7/be/src/olap/rowset/segment_v2/page_io.cpp#L53-L72) may retain an uncompressed page when compression does not save enough space. No performance profile or broad log collection is needed for the reporting issue itself. The cached issue snapshot has no labels. A live metadata refresh was rate-limited, so current labels and subsequent discussion were not verified. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
