hongzhi-gao commented on PR #18726: URL: https://github.com/apache/iotdb/pull/18726#issuecomment-6791175800
Since this PR enables compaction replacement by default for newly created user Pipes, could we add an enabled/disabled performance comparison before merging? The queue tests and parameter IT provide useful functional coverage, but the end-to-end benefit and cost of enabling the feature are still unmeasured. I suggest covering these three workloads: - **Little or no backlog:** check synchronization throughput/lag and sender CPU for regressions. - **Backlog with eligible compaction sources:** compare catch-up time, transferred bytes/file count, and receiver loading cost. Please also report the replacement hit rate. - **A large queue with mostly ineligible replacements:** measure the queue-scanning CPU/allocation cost when the complete-source and oldest-prefix conditions rarely hold. Please keep the dataset, topology and other settings identical between runs, document the environment/configuration, and repeat the measurements to show variability. Since polling may race with replacement and deliver both source and target files, transferred bytes should include that duplicate-transfer overhead rather than assuming fewer queued events means less network traffic. I would also suggest a focused end-to-end IT with concurrent sending and compaction. After the Pipe catches up, verify that the receiver's query results match the source and that replication progress advances correctly, including when a source event is polled during replacement. The accepted duplicate-delivery behavior should not cause missing data or incorrect final values. A small reproducible comparison plus that targeted correctness IT would help justify the default-on behavior. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
