namtran1812 opened a new pull request, #51400: URL: https://github.com/apache/arrow/pull/51400
<html><head></head><body><h3>Rationale for this change</h3><p><code inline="">SchemaDescriptor::BuildTree</code> currently constructs each leaf's dot-separated path through <code inline="">node->path()->ToDotString()</code>. For wide schemas, this repeatedly walks parent links and creates temporary path and string objects for every leaf during schema initialization.</p><p>This change reduces that overhead. It is relevant to the wide-schema metadata costs discussed in #38149, although it does not eliminate full-schema metadata deserialization or fully resolve selective column reads.</p><h3>What changes are included in this PR?</h3><ul><li><p>Build the dot-separated column path incrementally while traversing the schema tree instead of reconstructing it independently for every leaf.</p></li><li><p>Insert the constructed path directly into <code inline="">leaf_to_idx_</code>.</p></li><li><p>Preserve existing nested-path and duplicate-path lookup semantics.</p></li><li><p>Add a r egression test verifying that duplicate paths are still disambiguated by node identity.</p></li><li><p>Extend <code inline="">ReadFileMetadata</code> benchmarks with 5K, 10K, 20K, and 50K-column single-row-group cases.</p></li></ul><p>Local benchmark results (median wall time):</p> Columns | Before | After | Improvement -- | -- | -- | -- 1,000 | 0.907 ms | 0.771 ms | 15.1% 5,000 | 4.619 ms | 3.936 ms | 14.8% 10,000 | 9.369 ms | 8.017 ms | 14.4% 20,000 | 18.951 ms | 16.213 ms | 14.4% 50,000 | 47.894 ms | 42.542 ms | 11.2% <p>The 50K-column case showed higher variance than the smaller cases. Across these measurements, metadata-read time improved by approximately 11–15%.</p><h3>Are these changes tested?</h3><p>Yes.</p><ul><li><p><code inline="">parquet-schema-test</code>: 44/44 tests passed.</p></li><li><p>Added <code inline="">TestSchemaDescriptor.ColumnIndexDuplicatePath</code> to cover duplicate-path identity lookup.</p></li><li><p><code inline="">git diff --check</code> passes.</p></li><li><p>The updated metadata benchmark was built and run locally across 1K–50K-column schemas.</p></li></ul><h3>Are there any user-facing changes?</h3><p>No. This is an internal Parquet schema initialization optimization with no intended public API or behavioral changes.</p><h3>Was AI used for this PR?</h3><p>In accordance to the AI generation guidelines, AI was used to assist with implementation analysis, review, testing strategy, and drafting the PR description. The changes and benchmark/test results were revi ewed and validated locally before submission.</p><p><strong>PR code and description written by:</strong></p><ul class="contains-task-list"><li class="task-list-item"><p><input type="checkbox" checked="" disabled=""> Human</p></li><li class="task-list-item"><p><input type="checkbox" checked="" disabled=""> AI</p></li></ul><p><strong>Reviewed before submission by:</strong></p><ul class="contains-task-list"><li class="task-list-item"><p><input type="checkbox" checked="" disabled=""> Human</p></li><li class="task-list-item"><p><input type="checkbox" checked="" disabled=""> AI</p></li><li class="task-list-item"><p><input type="checkbox" disabled=""> Not reviewed</p></li></ul></body></html> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
