AmanBoronbaev commented on PR #55641:
URL: https://github.com/apache/arrow/pull/55641#issuecomment-6961313451

   Follow-up on initialization overhead: updated in 
ddb215160c40192deb14a6a6cab25ae1a9b00d2e to reuse the lookup already performed 
by `AddField`. A unique field gets the next index; a duplicate returns an 
existing index. Comparing that return value with the previous field count 
detects duplicates without a second hash-table lookup. The object-builder 
pointer is obtained after recursive builder construction.
   
   I also measured parser construction/destruction with fixed-length unique 
field names and schema construction outside the timed loop. Local results below 
are medians of seven batches after one warm-up batch, with `100000 / 
field_count` initializations per batch:
   
   | Fields | Baseline (us) | Original PR (us) | Updated PR (us) |
   | --- | --- | --- | --- |
   | 10 | 93.56 | 100.59 | 94.48 |
   | 100 | 724.47 | 814.91 | 716.59 |
   | 1000 | 6221.55 | 7041.70 | 6422.56 |
   | 10000 | 64663.62 | 66689.00 | 63723.14 |
   
   Environment: Intel i5-12450H, Clang 22.1.8, Arrow Debug library (`-O0 -g0`), 
C++20; the standalone driver was built with `-O2`. Baseline is the parent of 
the original PR commit; “Original PR” is 
7e7a876d27b1d02601ae42d9b488284bca70f959. Variants ran sequentially. These are 
local microbenchmark results, not release-build throughput estimates or a 
statistically controlled speedup claim. They include parser destruction and 
allocations and show approximately linear scaling over the tested widths; they 
do not establish a worst-case complexity bound.
   
   Validation after the update: all 254 C++ JSON tests passed, including 
duplicate names at root/nested/list levels and valid schemas; clang-format, 
cpplint and the `-Werror` build passed.
   
   <details>
   <summary>Standalone reproduction driver</summary>
   
   ```cpp
   #include <chrono>
   #include <iomanip>
   #include <iostream>
   #include <memory>
   #include <string>
   #include "arrow/json/options.h"
   #include "arrow/json/parser.h"
   #include "arrow/type.h"
   
   int main() {
     std::cout << "fields,repeat,iterations,ns_per_init,ns_per_field\n";
     for (int count : {10, 100, 1000, 10000}) {
       arrow::FieldVector fields;
       for (int i = 0; i < count; ++i) {
         // Fixed-length names keep hashing cost comparable across schema 
widths.
         auto suffix = std::to_string(i);
         fields.push_back(arrow::field("field_" + std::string(8 - 
suffix.size(), '0') + suffix,
                                       arrow::int64()));
       }
       auto options = arrow::json::ParseOptions::Defaults();
       options.explicit_schema = arrow::schema(fields);
       options.unexpected_field_behavior = 
arrow::json::UnexpectedFieldBehavior::Error;
       int iterations = 100000 / count;
       for (int repeat = -1; repeat < 7; ++repeat) {
         auto start = std::chrono::steady_clock::now();
         for (int i = 0; i < iterations; ++i) {
           std::unique_ptr<arrow::json::BlockParser> parser;
           auto status = arrow::json::BlockParser::Make(options, &parser);
           if (!status.ok()) { std::cerr << status << '\n'; return 1; }
         }
         double ns = std::chrono::duration<double, std::nano>(
             std::chrono::steady_clock::now() - start).count() / iterations;
         if (repeat >= 0) std::cout << count << ',' << repeat << ',' << 
iterations << ','
                                   << std::fixed << std::setprecision(1) << ns 
<< ',' << ns/count << '\n';
       }
     }
   }
   
   ```
   
   Build the Arrow JSON shared library at each revision with the same 
configuration. Compile this driver against that checkout's `cpp/src` and 
build's `src` include paths, link `-larrow`, and set the library path to that 
build. Rebuild only the parser/library between variants; construct the schema 
outside the timed loop as above.
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to