leaves12138 commented on code in PR #9: URL: https://github.com/apache/paimon-mosaic/pull/9#discussion_r3265045523
########## docs/cpp-api.html: ########## @@ -0,0 +1,399 @@ +<!-- + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. +--> + +<!DOCTYPE html> +<html lang="en"> +<head> + <meta charset="UTF-8"> + <meta name="viewport" content="width=device-width, initial-scale=1.0"> + <title>C++ API - Mosaic</title> + <link rel="stylesheet" href="css/style.css"> + <script src="js/main.js"></script> +</head> +<body> + <button class="menu-toggle" aria-label="Menu">☰</button> + <div class="overlay"></div> + + <aside class="sidebar"> + <div class="sidebar-header"> + <h2>Mosaic</h2> + <p>Columnar-bucket hybrid format</p> + </div> + <nav> + <ul> + <li><a href="index.html">Home</a></li> + <li><a href="design.html">Design</a></li> + <li><a href="rust-api.html">Rust API</a></li> + <li><a href="java-api.html">Java API</a></li> + <li><a href="python-api.html">Python API</a></li> + <li><a href="cpp-api.html">C++ API</a></li> + </ul> + </nav> + <div class="sidebar-footer"> + <button class="theme-toggle">Dark Mode</button> + </div> + </aside> + + <main class="main"> + <div class="content"> + <h1>C++ API</h1> + <p class="subtitle">Use Mosaic from C or C++ via the FFI bindings.</p> + + <h2>Overview</h2> + <p> + The <code>ffi/</code> crate generates a shared library (<code>libmosaic_ffi</code>) and a + C header (<code>mosaic.h</code>) via <a href="https://github.com/mozilla/cbindgen";>cbindgen</a>. + The C++ header (<code>mosaic.hpp</code>) is a hand-written RAII wrapper on top of the C API. + </p> + + <h2>Building</h2> +<pre><code><span class="cmt"># Build the FFI shared library</span> +cargo build --release -p mosaic-ffi + +<span class="cmt"># C header generated at include/mosaic.h</span> +<span class="cmt"># C++ RAII wrapper: include/mosaic.hpp (checked in, not generated)</span></code></pre> + + <h2>Linking</h2> + <p>Link against the shared library and include the appropriate header:</p> +<pre><code><span class="cmt"># macOS</span> +g++ -std=c++17 -I include/ example.cpp \ + -L target/release -lmosaic_ffi -o example + +<span class="cmt"># Linux</span> +g++ -std=c++17 -I include/ example.cpp \ + -L target/release -lmosaic_ffi -Wl,-rpath,target/release -o example</code></pre> + + <h2>Writing (C++)</h2> + <p> + Data is written as Arrow RecordBatches via the + <a href="https://arrow.apache.org/docs/format/CDataInterface.html";>Arrow C Data Interface</a>. + Build your data as Arrow arrays, export via <code>ArrowArray</code> / <code>ArrowSchema</code>, + then pass to the writer: + </p> +<pre><code><span class="kw">#include</span> <span class="str">"mosaic.hpp"</span> +<span class="kw">#include</span> <arrow/api.h> +<span class="kw">#include</span> <arrow/c/bridge.h> + +<span class="kw">int</span> <span class="fn">main</span>() { + <span class="kw">try</span> { + <span class="cmt">// 1. Set up output stream callbacks</span> + <span class="kw">auto</span>* fp = std::fopen(<span class="str">"output.mosaic"</span>, <span class="str">"wb"</span>); + <span class="kw">auto</span> file = std::shared_ptr<FILE>(fp, [](<span class="ty">FILE</span>* f) { std::fclose(f); }); + <span class="kw">int64_t</span> pos = <span class="num">0</span>; + + <span class="ty">mosaic</span>::<span class="ty">OutputFile</span> cbs; + cbs.write_fn = [file, pos](<span class="kw">const uint8_t</span>* data, <span class="kw">size_t</span> len) <span class="kw">mutable</span> -> <span class="kw">int</span> { + <span class="kw">size_t</span> written = std::fwrite(data, <span class="num">1</span>, len, file.get()); + pos += <span class="kw">static_cast</span><<span class="kw">int64_t</span>>(written); + <span class="kw">return</span> (written == len) ? <span class="num">0</span> : <span class="num">-1</span>; + }; + cbs.flush_fn = [file]() -> <span class="kw">int</span> { <span class="kw">return</span> std::fflush(file.get()); }; + cbs.get_pos_fn = [file, pos]() <span class="kw">mutable</span> -> <span class="kw">int64_t</span> { <span class="kw">return</span> pos; }; Review Comment: The two lambdas capture separate copies of `pos`, so `get_pos_fn` always returns its own unchanged copy (0). The writer relies on `get_pos_fn` for file offsets, so this example can produce an invalid file. The tested code uses shared buffer state (`buf.pos`) for both callbacks; this snippet should do the same or use `ftell`/shared state. ########## docs/rust-api.html: ########## @@ -0,0 +1,254 @@ +<!-- + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. +--> + +<!DOCTYPE html> +<html lang="en"> +<head> + <meta charset="UTF-8"> + <meta name="viewport" content="width=device-width, initial-scale=1.0"> + <title>Rust API - Mosaic</title> + <link rel="stylesheet" href="css/style.css"> + <script src="js/main.js"></script> +</head> +<body> + <button class="menu-toggle" aria-label="Menu">☰</button> + <div class="overlay"></div> + + <aside class="sidebar"> + <div class="sidebar-header"> + <h2>Mosaic</h2> + <p>Columnar-bucket hybrid format</p> + </div> + <nav> + <ul> + <li><a href="index.html">Home</a></li> + <li><a href="design.html">Design</a></li> + <li><a href="rust-api.html" class="active">Rust API</a></li> + <li><a href="java-api.html">Java API</a></li> + <li><a href="python-api.html">Python API</a></li> + <li><a href="cpp-api.html">C++ API</a></li> + </ul> + </nav> + <div class="sidebar-footer"> + <button class="theme-toggle">Dark Mode</button> + </div> + </aside> + + <main class="main"> + <div class="content"> + <h1>Rust API</h1> + <p class="subtitle">Build from source and write your first Mosaic file in Rust.</p> + + <h2>Building from Source</h2> +<pre><code>git clone <repo-url> +cd mosaic +cargo build</code></pre> + + <p>Run the test suite to verify everything works:</p> +<pre><code>cargo test</code></pre> + + <h2>Rust: Writing a File</h2> + <p> + Add <code>mosaic-core</code> as a dependency (path dependency within the workspace), + then use <code>MosaicWriter</code> to create a file. Data is written as Arrow + <code>RecordBatch</code> objects: + </p> +<pre><code><span class="kw">use</span> std::sync::Arc; +<span class="kw">use</span> arrow::datatypes::{DataType, Field, Schema}; +<span class="kw">use</span> arrow::array::*; +<span class="kw">use</span> arrow::record_batch::RecordBatch; +<span class="kw">use</span> mosaic_core::writer::{MosaicWriter, WriterOptions}; +<span class="kw">use</span> mosaic_core::spec::*; + +<span class="cmt">// 1. Define an Arrow Schema</span> +<span class="kw">let</span> arrow_schema = Schema::new(<span class="kw">vec!</span>[ + Field::new(<span class="str">"age"</span>, DataType::Int32, <span class="kw">true</span>), + Field::new(<span class="str">"name"</span>, DataType::Utf8, <span class="kw">true</span>), + Field::new(<span class="str">"score"</span>, DataType::Float64, <span class="kw">true</span>), +]); + +<span class="cmt">// 2. Create writer (writes to any OutputFile implementation)</span> +<span class="kw">let</span> <span class="kw">mut</span> writer = MosaicWriter::new(output, &arrow_schema, WriterOptions { + num_buckets: <span class="num">2</span>, + compression: COMPRESSION_ZSTD, + ..Default::default() +})?; + +<span class="cmt">// 3. Build an Arrow RecordBatch and write it</span> +<span class="kw">let</span> ages = Int32Array::from((<span class="num">0</span>..<span class="num">1000</span>).map(|i| <span class="num">20</span> + (i % <span class="num">50</span>)).collect::<Vec<_>>()); +<span class="kw">let</span> names = StringArray::from((<span class="num">0</span>..<span class="num">1000</span>).map(|i| <span class="kw">format!</span>(<span class="str">"user_{}"</span>, i)).collect::<Vec<_>>()); +<span class="kw">let</span> scores = Float64Array::from((<span class="num">0</span>..<span class="num">1000</span>).map(|i| i <span class="kw">as</span> <span class="ty">f64</span> * <span class="num">1.5</span>).collect::<Vec<_>>()); +<span class="kw">let</span> batch = RecordBatch::try_new( + Arc::new(Schema::new(<span class="kw">vec!</span>[ + Field::new(<span class="str">"age"</span>, DataType::Int32, <span class="kw">true</span>), + Field::new(<span class="str">"name"</span>, DataType::Utf8, <span class="kw">true</span>), + Field::new(<span class="str">"score"</span>, DataType::Float64, <span class="kw">true</span>), + ])), + <span class="kw">vec!</span>[Arc::new(ages), Arc::new(names), Arc::new(scores)], +).unwrap(); +writer.write_batch(&batch).unwrap(); + +<span class="cmt">// 4. Finalize</span> +writer.close().unwrap(); + +<span class="cmt">// Estimated file size (for file rolling decisions)</span> +<span class="kw">let</span> est = writer.estimated_file_size();</code></pre> + + <h3>Writer Methods</h3> + <table> + <thead> + <tr><th>Method</th><th>Return</th><th>Description</th></tr> + </thead> + <tbody> + <tr><td><code>write_batch(&RecordBatch)</code></td><td><code>io::Result<()></code></td><td>Write an Arrow RecordBatch</td></tr> + <tr><td><code>estimated_file_size()</code></td><td><code>u64</code></td><td>Estimated output file size in bytes (for file rolling)</td></tr> + <tr><td><code>close()</code></td><td><code>io::Result<()></code></td><td>Flush remaining data and write footer</td></tr> + </tbody> + </table> + + <h2>Rust: Reading a File</h2> + <p> + <code>MosaicReader</code> is generic over any <code>InputFile</code> implementation. + Implement the <code>InputFile</code> trait for your data source to read from + local files, memory-mapped buffers, remote storage, etc.: + </p> +<pre><code><span class="kw">use</span> mosaic_core::reader::InputFile; + +<span class="kw">pub trait</span> <span class="ty">InputFile</span> { + <span class="kw">fn</span> <span class="fn">read_at</span>(&<span class="kw">self</span>, offset: <span class="ty">u64</span>, buf: &<span class="kw">mut</span> [<span class="ty">u8</span>]) -> io::Result<()>; +}</code></pre> + +<h3>1. Open and Inspect the Schema</h3> + <p> + Pass your <code>InputFile</code> implementation and the file length directly + to <code>MosaicReader::new</code>: + </p> +<pre><code><span class="kw">use</span> mosaic_core::reader::{MosaicReader, InputFile, ReaderAccess}; + +<span class="kw">let</span> reader = MosaicReader::new(my_input_file, file_len).unwrap(); + +<span class="cmt">// Iterate over all columns</span> +<span class="kw">for</span> col <span class="kw">in</span> &reader.schema().columns { + println!(<span class="str">"name={} type={:?} nullable={}"</span>, + col.name, col.data_type, col.nullable); +}</code></pre> + + <h3>ColumnMeta Fields</h3> + <table> + <thead> + <tr><th>Field</th><th>Type</th><th>Description</th></tr> + </thead> + <tbody> + <tr><td><code>name</code></td><td><code>String</code></td><td>Column name</td></tr> + <tr><td><code>data_type</code></td><td><code>arrow::datatypes::DataType</code></td><td>Arrow DataType (Int32, Utf8, Decimal128, Timestamp, etc.)</td></tr> + <tr><td><code>nullable</code></td><td><code>bool</code></td><td>Whether column allows nulls</td></tr> + </tbody> + </table> + + <h3>2. Read Row Groups as Arrow RecordBatch</h3> + <p> + Each row group is read as an Arrow <code>RecordBatch</code> via + <code>read_columns()</code>. This returns fully-constructed Arrow arrays with + proper null handling and type mapping. + </p> +<pre><code><span class="kw">use</span> arrow::array::*; + +<span class="kw">for</span> rg_idx <span class="kw">in</span> <span class="num">0</span>..reader.num_row_groups() { + <span class="kw">let</span> <span class="kw">mut</span> rg = reader.row_group_reader(rg_idx).unwrap(); + <span class="kw">let</span> batch = rg.read_columns().unwrap(); + + println!(<span class="str">"row group {} has {} rows, {} cols"</span>, + rg_idx, batch.num_rows(), batch.num_columns()); + + <span class="cmt">// Access columns by name</span> + <span class="kw">let</span> ages = batch.column_by_name(<span class="str">"age"</span>).unwrap() + .as_any().downcast_ref::<Int32Array>().unwrap(); + <span class="kw">let</span> names = batch.column_by_name(<span class="str">"name"</span>).unwrap() + .as_any().downcast_ref::<StringArray>().unwrap(); + + <span class="kw">for</span> i <span class="kw">in</span> <span class="num">0</span>..batch.num_rows() { + <span class="kw">if</span> !ages.is_null(i) { + println!(<span class="str">"age={} name={}"</span>, ages.value(i), names.value(i)); + } + } +}</code></pre> + + <h3>Projection Pushdown</h3> + <p> + Use <code>row_group_reader_projected</code> to read only specific columns. + Only the buckets containing the projected columns are decompressed — + reading 1 column out of 10,000 only touches 1 bucket instead of all 100. + The returned batch contains only the projected columns. + </p> +<pre><code><span class="kw">let</span> name_col = reader.schema().columns.iter() + .position(|c| c.name == <span class="str">"name"</span>).unwrap(); +<span class="kw">let</span> score_col = reader.schema().columns.iter() + .position(|c| c.name == <span class="str">"score"</span>).unwrap(); + +<span class="kw">let</span> <span class="kw">mut</span> rg = reader.row_group_reader_projected(rg_idx, &[name_col, score_col]).unwrap(); +<span class="kw">let</span> batch = rg.read_columns().unwrap(); +<span class="cmt">// batch contains only 2 columns: "name" and "score"</span> +<span class="cmt">// Use batch.schema().field(i).name() to identify columns</span></code></pre> + + <div class="note"> + <strong>Column ordering</strong> + The reader preserves the original schema column order. + Use <code>reader.schema().columns[i].name</code> or positional lookup to access columns. + </div> + + + <h2>Column Statistics (Filter Pushdown)</h2> + <p> + Enable per-column min/max statistics to allow query engines to skip entire row groups + whose value range does not match a filter predicate. + </p> + + <h3>Writing with Stats</h3> +<pre><code><span class="kw">let</span> <span class="kw">mut</span> writer = MosaicWriter::new(output, &arrow_schema, WriterOptions { + stats_columns: <span class="kw">vec!</span>[<span class="num">0</span>, <span class="num">2</span>], <span class="cmt">// build stats for columns 0 and 2</span> + ..Default::default() +})?;</code></pre> + + <h3>Reading Stats</h3> +<pre><code><span class="kw">for</span> rg_idx <span class="kw">in</span> <span class="num">0</span>..reader.num_row_groups() { + <span class="kw">let</span> stats = reader.row_group_stats(rg_idx); Review Comment: `row_group_stats` returns `io::Result<&[ColumnStats]>` in `ReaderAccess`, so this snippet does not compile as written. It should unwrap/propagate the result before iterating, e.g. `let stats = reader.row_group_stats(rg_idx)?;` or `.unwrap()` in a sample. ########## docs/cpp-api.html: ########## @@ -0,0 +1,399 @@ +<!-- + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. +--> + +<!DOCTYPE html> +<html lang="en"> +<head> + <meta charset="UTF-8"> + <meta name="viewport" content="width=device-width, initial-scale=1.0"> + <title>C++ API - Mosaic</title> + <link rel="stylesheet" href="css/style.css"> + <script src="js/main.js"></script> +</head> +<body> + <button class="menu-toggle" aria-label="Menu">☰</button> + <div class="overlay"></div> + + <aside class="sidebar"> + <div class="sidebar-header"> + <h2>Mosaic</h2> + <p>Columnar-bucket hybrid format</p> + </div> + <nav> + <ul> + <li><a href="index.html">Home</a></li> + <li><a href="design.html">Design</a></li> + <li><a href="rust-api.html">Rust API</a></li> + <li><a href="java-api.html">Java API</a></li> + <li><a href="python-api.html">Python API</a></li> + <li><a href="cpp-api.html">C++ API</a></li> + </ul> + </nav> + <div class="sidebar-footer"> + <button class="theme-toggle">Dark Mode</button> + </div> + </aside> + + <main class="main"> + <div class="content"> + <h1>C++ API</h1> + <p class="subtitle">Use Mosaic from C or C++ via the FFI bindings.</p> + + <h2>Overview</h2> + <p> + The <code>ffi/</code> crate generates a shared library (<code>libmosaic_ffi</code>) and a + C header (<code>mosaic.h</code>) via <a href="https://github.com/mozilla/cbindgen";>cbindgen</a>. + The C++ header (<code>mosaic.hpp</code>) is a hand-written RAII wrapper on top of the C API. + </p> + + <h2>Building</h2> +<pre><code><span class="cmt"># Build the FFI shared library</span> +cargo build --release -p mosaic-ffi + +<span class="cmt"># C header generated at include/mosaic.h</span> +<span class="cmt"># C++ RAII wrapper: include/mosaic.hpp (checked in, not generated)</span></code></pre> + + <h2>Linking</h2> + <p>Link against the shared library and include the appropriate header:</p> +<pre><code><span class="cmt"># macOS</span> +g++ -std=c++17 -I include/ example.cpp \ + -L target/release -lmosaic_ffi -o example + +<span class="cmt"># Linux</span> +g++ -std=c++17 -I include/ example.cpp \ + -L target/release -lmosaic_ffi -Wl,-rpath,target/release -o example</code></pre> + + <h2>Writing (C++)</h2> + <p> + Data is written as Arrow RecordBatches via the + <a href="https://arrow.apache.org/docs/format/CDataInterface.html";>Arrow C Data Interface</a>. + Build your data as Arrow arrays, export via <code>ArrowArray</code> / <code>ArrowSchema</code>, + then pass to the writer: + </p> +<pre><code><span class="kw">#include</span> <span class="str">"mosaic.hpp"</span> +<span class="kw">#include</span> <arrow/api.h> +<span class="kw">#include</span> <arrow/c/bridge.h> + +<span class="kw">int</span> <span class="fn">main</span>() { + <span class="kw">try</span> { + <span class="cmt">// 1. Set up output stream callbacks</span> + <span class="kw">auto</span>* fp = std::fopen(<span class="str">"output.mosaic"</span>, <span class="str">"wb"</span>); + <span class="kw">auto</span> file = std::shared_ptr<FILE>(fp, [](<span class="ty">FILE</span>* f) { std::fclose(f); }); + <span class="kw">int64_t</span> pos = <span class="num">0</span>; + + <span class="ty">mosaic</span>::<span class="ty">OutputFile</span> cbs; + cbs.write_fn = [file, pos](<span class="kw">const uint8_t</span>* data, <span class="kw">size_t</span> len) <span class="kw">mutable</span> -> <span class="kw">int</span> { + <span class="kw">size_t</span> written = std::fwrite(data, <span class="num">1</span>, len, file.get()); + pos += <span class="kw">static_cast</span><<span class="kw">int64_t</span>>(written); + <span class="kw">return</span> (written == len) ? <span class="num">0</span> : <span class="num">-1</span>; + }; + cbs.flush_fn = [file]() -> <span class="kw">int</span> { <span class="kw">return</span> std::fflush(file.get()); }; + cbs.get_pos_fn = [file, pos]() <span class="kw">mutable</span> -> <span class="kw">int64_t</span> { <span class="kw">return</span> pos; }; + + <span class="cmt">// 2. Build an Arrow RecordBatch and write it</span> + arrow::Int32Builder age_builder; + arrow::StringBuilder name_builder; + arrow::DoubleBuilder score_builder; + <span class="kw">for</span> (<span class="kw">int</span> i = <span class="num">0</span>; i < <span class="num">10000</span>; i++) { + name_builder.Append(<span class="str">"user_"</span> + std::to_string(i)); + age_builder.Append(<span class="num">20</span> + (i % <span class="num">50</span>)); + score_builder.Append(i * <span class="num">1.5</span>); + } + <span class="kw">auto</span> batch = arrow::RecordBatch::Make( + arrow::schema({ + arrow::field(<span class="str">"name"</span>, arrow::utf8()), + arrow::field(<span class="str">"age"</span>, arrow::int32()), + arrow::field(<span class="str">"score"</span>, arrow::float64()), + }), + <span class="num">10000</span>, + {name_builder.Finish().ValueOrDie(), + age_builder.Finish().ValueOrDie(), + score_builder.Finish().ValueOrDie()}); + + <span class="cmt">// 3. Export via Arrow C Data Interface, create writer, and write</span> + ArrowArray ffi_array; + ArrowSchema ffi_schema; + arrow::ExportRecordBatch(*batch, &ffi_array, &ffi_schema); + + <span class="ty">mosaic</span>::<span class="ty">Writer</span> writer(std::move(cbs), &ffi_schema, { + .num_buckets = <span class="num">2</span>, + .compression = <span class="num">1</span>, <span class="cmt">// ZSTD</span> + .zstd_level = <span class="num">1</span>, + }); + writer.write(&ffi_array, &ffi_schema); + + <span class="cmt">// 4. Close (also happens on destructor)</span> + writer.close(); + + } <span class="kw">catch</span> (<span class="kw">const</span> <span class="ty">mosaic</span>::<span class="ty">Error</span>& e) { + fprintf(stderr, <span class="str">"Error: %s\n"</span>, e.what()); + <span class="kw">return</span> <span class="num">1</span>; + } + <span class="kw">return</span> <span class="num">0</span>; +}</code></pre> + + <h2>C++ API Reference</h2> + + <h3>Writer Options</h3> + <table> + <thead> + <tr><th>Field</th><th>Type</th><th>Default</th><th>Description</th></tr> + </thead> + <tbody> + <tr><td><code>num_buckets</code></td><td>uint32_t</td><td>0</td><td>Number of buckets (0 = auto)</td></tr> + <tr><td><code>compression</code></td><td>uint8_t</td><td>1</td><td>0=none, 1=zstd</td></tr> + <tr><td><code>zstd_level</code></td><td>int32_t</td><td>1</td><td>Zstd compression level</td></tr> + <tr><td><code>row_group_max_size</code></td><td>uint64_t</td><td>256 MB</td><td>Max row group size</td></tr> + <tr><td><code>max_dict_total_bytes</code></td><td>uint32_t</td><td>32 KB</td><td>Max dict size per column</td></tr> + <tr><td><code>max_dict_entries</code></td><td>uint32_t</td><td>255</td><td>Max dict entries per column</td></tr> + <tr><td><code>stats_columns</code></td><td>const uint32_t*</td><td>NULL</td><td>Column indices to build min/max stats for</td></tr> + <tr><td><code>num_stats_columns</code></td><td>uint32_t</td><td>0</td><td>Length of stats_columns array</td></tr> + <tr><td><code>page_size_threshold</code></td><td>uint32_t</td><td>32 KB</td><td>Min avg column page size to enable paged mode</td></tr> + </tbody> + </table> + + <h3>Writer Methods</h3> + <table> + <thead> + <tr><th>Method</th><th>Return</th><th>Description</th></tr> + </thead> + <tbody> + <tr><td><code>write(&ffi_array, &ffi_schema)</code></td><td><code>void</code></td><td>Write an Arrow RecordBatch via C Data Interface</td></tr> + <tr><td><code>estimated_file_size()</code></td><td><code>int64_t</code></td><td>Estimated output file size in bytes (for file rolling)</td></tr> + <tr><td><code>close()</code></td><td><code>void</code></td><td>Flush remaining data and write footer</td></tr> + </tbody> + </table> + + <h2>Reading a File</h2> + + <h3>1. Open the Reader</h3> + <p> + Construct a <code>Reader</code> by providing an <code>InputFile</code> + with your I/O implementation: + </p> +<pre><code><span class="cmt">// Example: memory-mapped reader</span> +<span class="ty">mosaic</span>::<span class="ty">InputFile</span> input; +input.read_at_fn = [data](<span class="kw">uint64_t</span> offset, <span class="kw">uint8_t</span>* buf, <span class="kw">size_t</span> len) -> <span class="kw">int</span> { + std::memcpy(buf, data + offset, len); + <span class="kw">return</span> <span class="num">0</span>; +}; + +<span class="kw">auto</span> reader = <span class="ty">mosaic</span>::make_reader(std::move(input), file_size);</code></pre> + + <h3>2. Inspect the Schema</h3> + <p> + Export the schema via the + <a href="https://arrow.apache.org/docs/format/CDataInterface.html";>Arrow C Data Interface</a> + and import into Arrow C++: + </p> +<pre><code>ArrowSchema ffi_schema; +reader.export_schema(&ffi_schema); +<span class="kw">auto</span> schema = arrow::ImportSchema(&ffi_schema).ValueOrDie();</code></pre> + + <h3>Reader Methods</h3> + <table> + <thead> + <tr><th>Method</th><th>Return</th><th>Description</th></tr> + </thead> + <tbody> + <tr><td><code>num_row_groups()</code></td><td><code>uint32_t</code></td><td>Row group count</td></tr> + <tr><td><code>export_schema(&ffi_schema)</code></td><td><code>void</code></td><td>Export schema via Arrow C Data Interface</td></tr> + <tr><td><code>read_row_group(rg, &array, &schema)</code></td><td><code>void</code></td><td>Read all columns of a row group</td></tr> + <tr><td><code>read_row_group(rg, cols, n, &array, &schema)</code></td><td><code>void</code></td><td>Read projected columns of a row group</td></tr> + <tr><td><code>get_row_group_statistics(rg)</code></td><td><code>vector<ColumnStatistics></code></td><td>Column statistics for a row group</td></tr> + </tbody> + </table> + + <h3>3. Read Row Groups as Arrow RecordBatch</h3> + <p> + Each row group is read directly via <code>read_row_group()</code>, which exports + via the <a href="https://arrow.apache.org/docs/format/CDataInterface.html";>Arrow C Data Interface</a> + for zero-copy import into Arrow C++: + </p> +<pre><code><span class="kw">#include</span> <arrow/c/bridge.h> +<span class="kw">#include</span> <span class="str">"mosaic.hpp"</span> + +<span class="kw">for</span> (<span class="kw">uint32_t</span> rg = <span class="num">0</span>; rg < reader.num_row_groups(); rg++) { + ArrowArray ffi_array; + ArrowSchema ffi_schema; + reader.read_row_group(rg, &ffi_array, &ffi_schema); + + <span class="cmt">// Import into Arrow C++</span> + <span class="kw">auto</span> batch = arrow::ImportRecordBatch(&ffi_array, &ffi_schema).ValueOrDie(); + printf(<span class="str">"row group %u: %lld rows, %d cols\n"</span>, + rg, batch->num_rows(), batch->num_columns()); + + <span class="cmt">// Access columns via Arrow C++ API</span> + <span class="kw">auto</span> ages = std::static_pointer_cast<arrow::Int32Array>( + batch->GetColumnByName(<span class="str">"age"</span>)); + <span class="kw">for</span> (<span class="kw">int64_t</span> i = <span class="num">0</span>; i < ages->length(); i++) { + <span class="kw">if</span> (!ages->IsNull(i)) { + printf(<span class="str">"age=%d\n"</span>, ages->Value(i)); + } + } +}</code></pre> + + <h3>Projection Pushdown</h3> + <p> + Use <code>read_row_group</code> with column indices to read only specific columns. + Only the buckets containing the projected columns are decompressed, reducing + I/O and memory for wide tables. The returned batch contains only the projected columns. + </p> +<pre><code><span class="kw">uint32_t</span> projected[] = { <span class="num">0</span>, <span class="num">2</span> }; +ArrowArray ffi_array; +ArrowSchema ffi_schema; +reader.read_row_group(<span class="num">0</span>, projected, <span class="num">2</span>, &ffi_array, &ffi_schema); +<span class="kw">auto</span> batch = arrow::ImportRecordBatch(&ffi_array, &ffi_schema).ValueOrDie(); +<span class="cmt">// batch contains only the projected columns</span></code></pre> + + <h3>Column Statistics (Filter Pushdown)</h3> + <p> + When stats columns are configured during writing, the reader can access per-row-group + min/max statistics to skip row groups that don't match a filter predicate: + </p> +<pre><code><span class="cmt">// Writing with stats (arrow_schema is an ArrowSchema* from C Data Interface)</span> +<span class="ty">mosaic</span>::<span class="ty">Writer</span> writer(std::move(cbs), arrow_schema, { + .compression = <span class="num">1</span>, Review Comment: This says "Writing with stats" but it only sets `compression`; `stats_columns` and `num_stats_columns` remain unset, so `get_row_group_statistics` will return an empty vector. Please include a `uint32_t stats_cols[] = { ... }` and set both `opts.stats_columns` and `opts.num_stats_columns`. ########## docs/design.html: ########## @@ -0,0 +1,781 @@ +<!-- + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. +--> + +<!DOCTYPE html> +<html lang="en"> +<head> + <meta charset="UTF-8"> + <meta name="viewport" content="width=device-width, initial-scale=1.0"> + <title>Design & Format Specification - Mosaic</title> + <link rel="stylesheet" href="css/style.css"> + <script src="js/main.js"></script> +</head> +<body> + <button class="menu-toggle" aria-label="Menu">☰</button> + <div class="overlay"></div> + + <aside class="sidebar"> + <div class="sidebar-header"> + <h2>Mosaic</h2> + <p>Columnar-bucket hybrid format</p> + </div> + <nav> + <ul> + <li><a href="index.html">Home</a></li> + <li><a href="design.html" class="active">Design</a></li> + <li><a href="rust-api.html">Rust API</a></li> + <li><a href="java-api.html">Java API</a></li> + <li><a href="python-api.html">Python API</a></li> + <li><a href="cpp-api.html">C++ API</a></li> + </ul> + </nav> + <div class="sidebar-footer"> + <button class="theme-toggle">Dark Mode</button> + </div> + </aside> + + <main class="main"> + <div class="content"> + <h1>Design & Format Specification</h1> + <p class="subtitle">Architecture, binary layout, and internal data structures of Mosaic v1.</p> + + <!-- ============================================================ --> + <h2>File Format Layout</h2> + <p>A Mosaic file consists of four sections, written sequentially:</p> +<svg class="arch-svg" viewBox="0 0 700 470" xmlns="http://www.w3.org/2000/svg";> + <!-- Bucket Data Section --> + <rect x="15" y="8" width="525" height="210" rx="8" + fill="var(--bg-secondary)" stroke="var(--border)" stroke-width="1.2"/> + <rect x="15" y="8" width="5" height="210" rx="2.5" fill="#818cf8"/> + <text x="38" y="32" fill="var(--text)" font-size="13" font-weight="600" + font-family="system-ui, sans-serif">Bucket Data</text> + + <!-- Row Group 0 --> + <text x="38" y="54" fill="var(--text-secondary)" font-size="10.5" + font-family="system-ui, sans-serif">Row Group 0</text> + <rect x="38" y="60" width="488" height="52" rx="5" + fill="var(--bg)" stroke="var(--border)" stroke-width="0.8" stroke-dasharray="4,2"/> + <rect x="50" y="68" width="82" height="30" rx="4" fill="var(--accent-light)" stroke="var(--accent)" stroke-width="1"/> + <text x="91" y="87" fill="var(--accent)" font-size="10" text-anchor="middle" font-family="system-ui, sans-serif">Bucket 0</text> + <rect x="142" y="68" width="82" height="30" rx="4" fill="var(--accent-light)" stroke="var(--accent)" stroke-width="1"/> + <text x="183" y="87" fill="var(--accent)" font-size="10" text-anchor="middle" font-family="system-ui, sans-serif">Bucket 1</text> + <rect x="234" y="68" width="82" height="30" rx="4" fill="var(--accent-light)" stroke="var(--accent)" stroke-width="1"/> + <text x="275" y="87" fill="var(--accent)" font-size="10" text-anchor="middle" font-family="system-ui, sans-serif">Bucket 2</text> + <text x="332" y="88" fill="var(--text-secondary)" font-size="14" font-family="system-ui, sans-serif">...</text> + <rect x="362" y="68" width="100" height="30" rx="4" fill="var(--accent-light)" stroke="var(--accent)" stroke-width="1"/> + <text x="412" y="87" fill="var(--accent)" font-size="10" text-anchor="middle" font-family="system-ui, sans-serif">Bucket N-1</text> + + <!-- Row Group 1 --> + <text x="38" y="132" fill="var(--text-secondary)" font-size="10.5" + font-family="system-ui, sans-serif">Row Group 1</text> + <rect x="38" y="138" width="488" height="52" rx="5" + fill="var(--bg)" stroke="var(--border)" stroke-width="0.8" stroke-dasharray="4,2"/> + <rect x="50" y="146" width="82" height="30" rx="4" fill="var(--accent-light)" stroke="var(--accent)" stroke-width="1"/> + <text x="91" y="165" fill="var(--accent)" font-size="10" text-anchor="middle" font-family="system-ui, sans-serif">Bucket 0</text> + <rect x="142" y="146" width="82" height="30" rx="4" fill="var(--accent-light)" stroke="var(--accent)" stroke-width="1"/> + <text x="183" y="165" fill="var(--accent)" font-size="10" text-anchor="middle" font-family="system-ui, sans-serif">Bucket 1</text> + <rect x="234" y="146" width="82" height="30" rx="4" fill="var(--accent-light)" stroke="var(--accent)" stroke-width="1"/> + <text x="275" y="165" fill="var(--accent)" font-size="10" text-anchor="middle" font-family="system-ui, sans-serif">Bucket 2</text> + <text x="332" y="166" fill="var(--text-secondary)" font-size="14" font-family="system-ui, sans-serif">...</text> + <rect x="362" y="146" width="100" height="30" rx="4" fill="var(--accent-light)" stroke="var(--accent)" stroke-width="1"/> + <text x="412" y="165" fill="var(--accent)" font-size="10" text-anchor="middle" font-family="system-ui, sans-serif">Bucket N-1</text> + + <text x="270" y="208" fill="var(--text-secondary)" font-size="14" text-anchor="middle" + font-family="system-ui, sans-serif">⋮</text> + + <!-- Schema Block --> + <rect x="15" y="232" width="525" height="52" rx="8" + fill="var(--bg-secondary)" stroke="var(--border)" stroke-width="1.2"/> + <rect x="15" y="232" width="5" height="52" rx="2.5" fill="#60a5fa"/> + <text x="38" y="255" fill="var(--text)" font-size="13" font-weight="600" + font-family="system-ui, sans-serif">Schema Block</text> + <text x="38" y="273" fill="var(--text-secondary)" font-size="10.5" + font-family="system-ui, sans-serif">4B uncompressed size | compressed schema bytes</text> + + <!-- Row Group Index --> + <rect x="15" y="298" width="525" height="52" rx="8" + fill="var(--bg-secondary)" stroke="var(--border)" stroke-width="1.2"/> + <rect x="15" y="298" width="5" height="52" rx="2.5" fill="#4ade80"/> + <text x="38" y="321" fill="var(--text)" font-size="13" font-weight="600" + font-family="system-ui, sans-serif">Row Group Index</text> + <text x="38" y="339" fill="var(--text-secondary)" font-size="10.5" + font-family="system-ui, sans-serif">numRows | nonEmpty | [bucketId, offset, compSize, uncompSize] ... | columnStats</text> + + <!-- Footer --> + <rect x="15" y="364" width="525" height="86" rx="8" + fill="var(--bg-secondary)" stroke="var(--border)" stroke-width="1.2"/> + <rect x="15" y="364" width="5" height="86" rx="2.5" fill="#fbbf24"/> + <text x="38" y="387" fill="var(--text)" font-size="13" font-weight="600" + font-family="system-ui, sans-serif">Footer (32 bytes)</text> + <text x="38" y="405" fill="var(--text-secondary)" font-size="10.5" + font-family="system-ui, sans-serif">indexOffset(8) | schemaOffset(8) | numBuckets(4) | numRowGroups(4)</text> + <text x="38" y="423" fill="var(--text-secondary)" font-size="10.5" + font-family="system-ui, sans-serif">compression(1) | version(1) | reserved(2) | magic "MOSA"(4)</text> + + <!-- Offset annotations (right side) --> + <line x1="575" y1="418" x2="575" y2="250" + stroke="var(--text-secondary)" stroke-width="0.8" stroke-dasharray="3,3" opacity="0.4"/> + <circle cx="575" cy="418" r="3.5" fill="#fbbf24"/> + <text x="584" y="422" fill="#fbbf24" font-size="9.5" font-family="system-ui, sans-serif">Footer</text> + + <!-- Arrow to Schema Block --> + <line x1="575" y1="258" x2="544" y2="258" stroke="#60a5fa" stroke-width="1.5"/> + <polygon points="544,255 536,258 544,261" fill="#60a5fa"/> + <text x="584" y="254" fill="#60a5fa" font-size="9.5" font-family="system-ui, sans-serif">schema</text> + <text x="584" y="265" fill="#60a5fa" font-size="9.5" font-family="system-ui, sans-serif">Offset</text> + + <!-- Arrow to Row Group Index --> + <line x1="575" y1="324" x2="544" y2="324" stroke="#4ade80" stroke-width="1.5"/> + <polygon points="544,321 536,324 544,327" fill="#4ade80"/> + <text x="584" y="320" fill="#4ade80" font-size="9.5" font-family="system-ui, sans-serif">index</text> + <text x="584" y="331" fill="#4ade80" font-size="9.5" font-family="system-ui, sans-serif">Offset</text> +</svg> + + <p>Reading starts from the footer (last 32 bytes), which provides absolute offsets to locate the schema block and row group index.</p> + + <!-- ============================================================ --> + <h2>Columnar-Bucket Hybrid</h2> + <p> + Mosaic is a columnar-bucket hybrid format. Columns are sorted by name and evenly distributed + into buckets using range-based assignment: + </p> +<pre><code><span class="fn">bucket_id</span> = sorted_position * num_buckets / num_columns</code></pre> + +<svg class="arch-svg" viewBox="0 0 680 190" xmlns="http://www.w3.org/2000/svg";> + <!-- Top label --> + <text x="340" y="14" fill="var(--text-secondary)" font-size="10.5" text-anchor="middle" + font-family="system-ui, sans-serif">Columns (sorted by name)</text> + + <!-- Column boxes --> + <rect x="24" y="24" width="72" height="26" rx="4" fill="var(--bg-secondary)" stroke="#818cf8" stroke-width="1.2"/> + <text x="60" y="41" fill="var(--text)" font-size="9" text-anchor="middle" font-family="SF Mono, Consolas, monospace">amount</text> + <rect x="104" y="24" width="72" height="26" rx="4" fill="var(--bg-secondary)" stroke="#818cf8" stroke-width="1.2"/> + <text x="140" y="41" fill="var(--text)" font-size="9" text-anchor="middle" font-family="SF Mono, Consolas, monospace">city</text> + <rect x="184" y="24" width="72" height="26" rx="4" fill="var(--bg-secondary)" stroke="#818cf8" stroke-width="1.2"/> + <text x="220" y="41" fill="var(--text)" font-size="9" text-anchor="middle" font-family="SF Mono, Consolas, monospace">email</text> + + <rect x="264" y="24" width="72" height="26" rx="4" fill="var(--bg-secondary)" stroke="#60a5fa" stroke-width="1.2"/> + <text x="300" y="41" fill="var(--text)" font-size="9" text-anchor="middle" font-family="SF Mono, Consolas, monospace">id</text> + <rect x="344" y="24" width="72" height="26" rx="4" fill="var(--bg-secondary)" stroke="#60a5fa" stroke-width="1.2"/> + <text x="380" y="41" fill="var(--text)" font-size="9" text-anchor="middle" font-family="SF Mono, Consolas, monospace">name</text> + <rect x="424" y="24" width="72" height="26" rx="4" fill="var(--bg-secondary)" stroke="#60a5fa" stroke-width="1.2"/> + <text x="460" y="41" fill="var(--text)" font-size="9" text-anchor="middle" font-family="SF Mono, Consolas, monospace">phone</text> + + <rect x="504" y="24" width="72" height="26" rx="4" fill="var(--bg-secondary)" stroke="#4ade80" stroke-width="1.2"/> + <text x="540" y="41" fill="var(--text)" font-size="9" text-anchor="middle" font-family="SF Mono, Consolas, monospace">score</text> + <rect x="584" y="24" width="72" height="26" rx="4" fill="var(--bg-secondary)" stroke="#4ade80" stroke-width="1.2"/> + <text x="620" y="41" fill="var(--text)" font-size="9" text-anchor="middle" font-family="SF Mono, Consolas, monospace">zip</text> + + <!-- Connecting lines --> + <line x1="60" y1="50" x2="133" y2="120" stroke="#818cf8" stroke-width="1" opacity="0.45"/> + <line x1="140" y1="50" x2="133" y2="120" stroke="#818cf8" stroke-width="1" opacity="0.45"/> + <line x1="220" y1="50" x2="133" y2="120" stroke="#818cf8" stroke-width="1" opacity="0.45"/> + <line x1="300" y1="50" x2="340" y2="120" stroke="#60a5fa" stroke-width="1" opacity="0.45"/> + <line x1="380" y1="50" x2="340" y2="120" stroke="#60a5fa" stroke-width="1" opacity="0.45"/> + <line x1="460" y1="50" x2="340" y2="120" stroke="#60a5fa" stroke-width="1" opacity="0.45"/> + <line x1="540" y1="50" x2="547" y2="120" stroke="#4ade80" stroke-width="1" opacity="0.45"/> + <line x1="620" y1="50" x2="547" y2="120" stroke="#4ade80" stroke-width="1" opacity="0.45"/> + + <!-- Bucket boxes --> + <rect x="40" y="120" width="186" height="48" rx="6" fill="var(--bg-secondary)" stroke="#818cf8" stroke-width="1.5"/> + <text x="133" y="140" fill="#818cf8" font-size="11" font-weight="600" text-anchor="middle" + font-family="system-ui, sans-serif">Bucket 0</text> + <text x="133" y="157" fill="var(--text-secondary)" font-size="9" text-anchor="middle" + font-family="SF Mono, Consolas, monospace">amount, city, email</text> + + <rect x="247" y="120" width="186" height="48" rx="6" fill="var(--bg-secondary)" stroke="#60a5fa" stroke-width="1.5"/> + <text x="340" y="140" fill="#60a5fa" font-size="11" font-weight="600" text-anchor="middle" + font-family="system-ui, sans-serif">Bucket 1</text> + <text x="340" y="157" fill="var(--text-secondary)" font-size="9" text-anchor="middle" + font-family="SF Mono, Consolas, monospace">id, name, phone</text> + + <rect x="454" y="120" width="186" height="48" rx="6" fill="var(--bg-secondary)" stroke="#4ade80" stroke-width="1.5"/> + <text x="547" y="140" fill="#4ade80" font-size="11" font-weight="600" text-anchor="middle" + font-family="system-ui, sans-serif">Bucket 2</text> + <text x="547" y="157" fill="var(--text-secondary)" font-size="9" text-anchor="middle" + font-family="SF Mono, Consolas, monospace">score, zip</text> + + <!-- Bottom label --> + <text x="340" y="186" fill="var(--text-secondary)" font-size="9.5" text-anchor="middle" font-style="italic" + font-family="system-ui, sans-serif">Example: 8 columns, 3 buckets</text> +</svg> + + <p> + Within each bucket, data is stored column-oriented and independently compressed. + This design enables efficient projection pushdown at bucket granularity — + reading 10 columns out of 10,000 only decompresses the buckets that contain those 10 columns. + </p> + <p> + Range-based assignment ensures that columns with similar name prefixes + (e.g., <code>sensor_temp_1</code>, <code>sensor_temp_2</code>) land in the same bucket, + improving both compression ratio and projection locality. + </p> + <p> + The default is <strong>100 buckets</strong>, automatically clamped to <code>min(num_columns, 100)</code>. + The bucket assignment is deterministic and derived from the sorted column order — + it is not stored in the file. + </p> + + <!-- ============================================================ --> + <h2>Encoding Strategy</h2> + <p>Each column within a bucket is independently encoded. The writer selects the most compact encoding for each column:</p> + <table> + <thead> + <tr><th>Encoding</th><th>Tag</th><th>When Used</th><th>Storage</th></tr> + </thead> + <tbody> + <tr> + <td><strong>PLAIN</strong></td><td>0</td> + <td>Fallback for everything else</td> + <td>Raw values (fixed-width or varint-prefixed) + null bitmap</td> + </tr> + <tr> + <td><strong>CONST</strong></td><td>1</td> + <td>All non-null values are identical</td> + <td>One value + null bitmap</td> + </tr> + <tr> + <td><strong>DICT</strong></td><td>2</td> + <td>Number of distinct values ≤ 255 and total dict size ≤ 32 KB</td> + <td>Dictionary + bit-packed indices + null bitmap</td> + </tr> + <tr> + <td><strong>ALL_NULL</strong></td><td>3</td> + <td>Every value in the column is null</td> + <td>Zero bytes (no data, no bitmap)</td> + </tr> + </tbody> + </table> + + <h3>Column Encoding Selection</h3> + <p>The encoding for each column is chosen automatically during writing based on value distribution and cost:</p> + <ul> + <li><strong>ALL_NULL</strong>: 0 non-null values</li> + <li><strong>CONST</strong>: exactly 1 distinct non-null value (any number of nulls allowed)</li> + <li><strong>DICT</strong>: 2–255 distinct non-null values, <strong>and</strong> the + dictionary-encoded size is smaller than plain — the writer compares + <code>varint(numEntries) + sum(entryBytes) + ceil(nonNullCount * bitWidth / 8)</code> + against the raw value buffer size</li> + <li><strong>PLAIN</strong>: 256+ distinct values, dict tracking was abandoned, or dict encoding + would be larger than plain</li> + </ul> + <p> + CONST detection is independent of dictionary tracking — it uses a lightweight byte comparison + against the first non-null value, so it works for all types and value sizes (including long strings). + </p> + <p> + Dictionary encoding works for all data types including variable-width types (VARCHAR, VARBINARY, DECIMAL). + Variable-width dictionary tracking is bounded by a configurable cumulative byte + budget (default 32 KB) and abandoned when cardinality exceeds 255 or total dictionary entry bytes + exceed the budget. + </p> + + <h3>Bit-packed Dictionary Indices</h3> + <p> + Dictionary indices are bit-packed using <code>bitWidth = ceil(log2(numEntries))</code> bits per + non-null cell, packed LSB-first within each byte. The reader derives <code>bitWidth</code> from + <code>numEntries</code> (already stored in dict metadata). + </p> + <p>Examples: 2 distinct values → 1 bit/cell, 4 → 2 bits, 16 → 4 bits, 256 → 8 bits.</p> + + <div class="note"> + <strong>Note</strong> + Null rows do not consume any bits in the bit-packed index array. + Only non-null rows have corresponding dictionary indices. + </div> + + <!-- ============================================================ --> + <h2>Bucket Internal Structure</h2> + <p> + Each bucket stores column data in one of two modes, chosen automatically based on the + uncompressed data size. The mode determines how compression is applied. + </p> + + <h3>Monolithic Mode</h3> + <p> + When the average column page size is <strong>smaller than 32 KB</strong> (configurable via + <code>page_size_threshold</code>), the entire bucket is compressed as a single zstd block. + Individual column pages that are too small yield poor zstd compression ratios, + so monolithic compression is more efficient in this case. + </p> + +<!-- Monolithic Bucket SVG --> +<svg class="arch-svg" viewBox="0 0 680 260" xmlns="http://www.w3.org/2000/svg";> + <!-- Outer bucket frame --> + <rect x="30" y="8" width="620" height="240" rx="8" + fill="var(--bg-secondary)" stroke="var(--border)" stroke-width="1.2"/> + <rect x="30" y="8" width="5" height="240" rx="2.5" fill="#818cf8"/> + <text x="54" y="30" fill="var(--text)" font-size="13" font-weight="600" + font-family="system-ui, sans-serif">Monolithic Bucket (single zstd block)</text> + + <!-- Encoding Flags --> + <rect x="54" y="42" width="580" height="28" rx="4" fill="var(--bg)" stroke="var(--border)" stroke-width="0.8"/> + <text x="344" y="60" fill="var(--text-secondary)" font-size="10" text-anchor="middle" + font-family="system-ui, sans-serif">Encoding Flags (2 bits/column)</text> + + <!-- Has-Nulls Flags --> + <rect x="54" y="76" width="580" height="28" rx="4" fill="var(--bg)" stroke="var(--border)" stroke-width="0.8"/> + <text x="344" y="94" fill="var(--text-secondary)" font-size="10" text-anchor="middle" + font-family="system-ui, sans-serif">Has-Nulls Flags (1 bit/column)</text> + + <!-- CONST metadata --> + <rect x="54" y="110" width="280" height="28" rx="4" fill="var(--bg)" stroke="#fbbf24" stroke-width="0.8"/> + <text x="194" y="128" fill="#fbbf24" font-size="10" text-anchor="middle" + font-family="system-ui, sans-serif">CONST Metadata</text> + + <!-- DICT metadata --> + <rect x="354" y="110" width="280" height="28" rx="4" fill="var(--bg)" stroke="#f97316" stroke-width="0.8"/> + <text x="494" y="128" fill="#f97316" font-size="10" text-anchor="middle" + font-family="system-ui, sans-serif">DICT Metadata (entries per column)</text> + + <!-- Null Bitmaps --> + <rect x="54" y="144" width="580" height="28" rx="4" fill="var(--bg)" stroke="#60a5fa" stroke-width="0.8"/> + <text x="344" y="162" fill="#60a5fa" font-size="10" text-anchor="middle" + font-family="system-ui, sans-serif">Null Bitmaps (columns with nulls, excluding ALL_NULL)</text> + + <!-- Column Data --> + <rect x="54" y="178" width="185" height="28" rx="4" fill="var(--accent-light)" stroke="var(--accent)" stroke-width="0.8"/> + <text x="146" y="196" fill="var(--accent)" font-size="10" text-anchor="middle" + font-family="system-ui, sans-serif">Col 0 data (PLAIN)</text> + <rect x="249" y="178" width="185" height="28" rx="4" fill="var(--accent-light)" stroke="var(--accent)" stroke-width="0.8"/> + <text x="341" y="196" fill="var(--accent)" font-size="10" text-anchor="middle" + font-family="system-ui, sans-serif">Col 1 data (DICT)</text> + <rect x="444" y="178" width="190" height="28" rx="4" fill="var(--accent-light)" stroke="var(--accent)" stroke-width="0.8"/> + <text x="539" y="196" fill="var(--accent)" font-size="10" text-anchor="middle" + font-family="system-ui, sans-serif">Col 2 data (PLAIN)</text> + + <!-- Compression arrow --> + <text x="344" y="230" fill="var(--text-secondary)" font-size="10" text-anchor="middle" font-style="italic" + font-family="system-ui, sans-serif">All of the above compressed together as one zstd block</text> +</svg> + + <h3>Paged Mode</h3> + <p> + When the average column page size is <strong>≥ 32 KB</strong>, the bucket switches to paged mode. + The bucket begins with a <strong>fixed-length page directory</strong> followed by <strong>self-describing, + independently compressed column slots</strong>. The directory size is deterministic from the schema + (<code>num_columns_in_bucket × 4</code> bytes), enabling projection queries to read only + the target columns' data with exactly 2 range-read operations on remote storage. + </p> + +<!-- Paged Bucket SVG --> +<svg class="arch-svg" viewBox="0 0 680 380" xmlns="http://www.w3.org/2000/svg";> + <!-- Outer bucket frame --> + <rect x="30" y="8" width="620" height="360" rx="8" + fill="var(--bg-secondary)" stroke="var(--border)" stroke-width="1.2"/> + <rect x="30" y="8" width="5" height="360" rx="2.5" fill="#818cf8"/> + <text x="54" y="30" fill="var(--text)" font-size="13" font-weight="600" + font-family="system-ui, sans-serif">Paged Bucket</text> + + <!-- Directory section --> + <rect x="54" y="42" width="580" height="70" rx="6" + fill="var(--bg)" stroke="#fbbf24" stroke-width="1.2" stroke-dasharray="4,2"/> + <text x="64" y="60" fill="#fbbf24" font-size="11" font-weight="600" + font-family="system-ui, sans-serif">Page Directory (fixed-length, uncompressed)</text> + + <rect x="68" y="68" width="120" height="28" rx="3" fill="var(--bg-secondary)" stroke="var(--border)" stroke-width="0.6"/> + <text x="128" y="86" fill="var(--text-secondary)" font-size="9" text-anchor="middle" + font-family="system-ui, sans-serif">Col 0: size (u32 LE)</text> + + <rect x="198" y="68" width="120" height="28" rx="3" fill="var(--bg-secondary)" stroke="var(--border)" stroke-width="0.6"/> + <text x="258" y="86" fill="var(--text-secondary)" font-size="9" text-anchor="middle" + font-family="system-ui, sans-serif">Col 1: size (u32 LE)</text> + + <rect x="328" y="68" width="120" height="28" rx="3" fill="var(--bg-secondary)" stroke="var(--border)" stroke-width="0.6"/> + <text x="388" y="86" fill="var(--text-secondary)" font-size="9" text-anchor="middle" + font-family="system-ui, sans-serif">Col 2: 0 (ALL_NULL)</text> + + <rect x="458" y="68" width="120" height="28" rx="3" fill="var(--bg-secondary)" stroke="var(--border)" stroke-width="0.6"/> + <text x="518" y="86" fill="var(--text-secondary)" font-size="9" text-anchor="middle" + font-family="system-ui, sans-serif">Col 3: size (u32 LE)</text> + + <!-- Column slots section --> + <text x="64" y="140" fill="var(--text)" font-size="11" font-weight="600" + font-family="system-ui, sans-serif">Column Slots (each self-describing, independently zstd compressed)</text> + + <!-- Slot 0 --> + <rect x="54" y="155" width="185" height="80" rx="5" + fill="var(--bg)" stroke="#818cf8" stroke-width="1.2"/> + <text x="146" y="173" fill="#818cf8" font-size="10" font-weight="600" text-anchor="middle" + font-family="system-ui, sans-serif">Slot 0 (Col A - PLAIN)</text> + <text x="146" y="190" fill="var(--text-secondary)" font-size="8.5" text-anchor="middle" + font-family="system-ui, sans-serif">uncompressed_size (varint)</text> + <text x="146" y="205" fill="var(--text-secondary)" font-size="8.5" text-anchor="middle" + font-family="system-ui, sans-serif">+ zstd(encoding | flags</text> + <text x="146" y="220" fill="var(--text-secondary)" font-size="8.5" text-anchor="middle" + font-family="system-ui, sans-serif">| bitmap | data)</text> + + <!-- Slot 1 --> + <rect x="249" y="155" width="185" height="80" rx="5" + fill="var(--bg)" stroke="#60a5fa" stroke-width="1.2"/> + <text x="341" y="173" fill="#60a5fa" font-size="10" font-weight="600" text-anchor="middle" + font-family="system-ui, sans-serif">Slot 1 (Col B - DICT)</text> + <text x="341" y="190" fill="var(--text-secondary)" font-size="8.5" text-anchor="middle" + font-family="system-ui, sans-serif">uncompressed_size (varint)</text> + <text x="341" y="205" fill="var(--text-secondary)" font-size="8.5" text-anchor="middle" + font-family="system-ui, sans-serif">+ zstd(encoding | flags</text> + <text x="341" y="220" fill="var(--text-secondary)" font-size="8.5" text-anchor="middle" + font-family="system-ui, sans-serif">| dict | bitmap | indices)</text> + + <!-- Slot 2 = ALL_NULL, no data --> + <text x="539" y="175" fill="var(--text-secondary)" font-size="9" text-anchor="middle" font-style="italic" + font-family="system-ui, sans-serif">Col C (ALL_NULL)</text> + <text x="539" y="192" fill="var(--text-secondary)" font-size="9" text-anchor="middle" font-style="italic" + font-family="system-ui, sans-serif">size=0 in directory</text> + <text x="539" y="209" fill="var(--text-secondary)" font-size="9" text-anchor="middle" font-style="italic" + font-family="system-ui, sans-serif">no on-disk slot</text> + + <!-- Slot 3 --> + <rect x="444" y="240" width="190" height="80" rx="5" + fill="var(--bg)" stroke="#4ade80" stroke-width="1.2"/> + <text x="539" y="258" fill="#4ade80" font-size="10" font-weight="600" text-anchor="middle" + font-family="system-ui, sans-serif">Slot 3 (Col D - CONST)</text> + <text x="539" y="275" fill="var(--text-secondary)" font-size="8.5" text-anchor="middle" + font-family="system-ui, sans-serif">uncompressed_size (varint)</text> + <text x="539" y="290" fill="var(--text-secondary)" font-size="8.5" text-anchor="middle" + font-family="system-ui, sans-serif">+ zstd(encoding | flags</text> + <text x="539" y="305" fill="var(--text-secondary)" font-size="8.5" text-anchor="middle" + font-family="system-ui, sans-serif">| const_value | bitmap)</text> + + <!-- Projection annotation --> + <rect x="54" y="330" width="580" height="30" rx="5" + fill="var(--accent-light)" stroke="var(--accent)" stroke-width="0.8" stroke-dasharray="4,2"/> + <text x="344" y="345" fill="var(--accent)" font-size="10" font-weight="600" text-anchor="middle" + font-family="system-ui, sans-serif">Projection: SELECT col_A, col_D → read directory (fixed) + only Slot 0 & Slot 3</text> + <text x="344" y="358" fill="var(--text-secondary)" font-size="9" text-anchor="middle" + font-family="system-ui, sans-serif">2 range-reads on remote storage — skip all other columns entirely</text> +</svg> + + <h4>Page Directory</h4> + <p> + The directory is an array of <code>num_columns_in_bucket</code> entries, each a 4-byte <code>u32</code> + (little-endian) representing the total on-disk slot size for that column. A value of <code>0</code> + means the column is ALL_NULL and has no on-disk data. The directory size is deterministic: + <code>num_columns_in_bucket × 4</code> bytes, computable from the schema alone. + </p> + + <h4>Column Slot Format</h4> + <p>Each non-ALL_NULL column has a slot on disk immediately after the directory:</p> +<pre><code>On-disk slot: + uncompressed_size (varint, uncompressed prefix) + compressed_data (zstd compressed page_content) + +page_content (after decompression): + encoding (1 byte: PLAIN=0, CONST=1, DICT=2) + flags (1 byte: bit 0 = has_nulls) + [meta] (encoding-specific, see below) + [data] (null bitmap if has_nulls, then column data)</code></pre> + + <h4>Page Content by Encoding</h4> + <table> + <thead> + <tr><th>Encoding</th><th>On-Disk Slot?</th><th>page_content layout</th></tr> + </thead> + <tbody> + <tr><td>ALL_NULL</td><td>No (size=0)</td><td>—</td></tr> + <tr><td>CONST (no nulls)</td><td>Yes (tiny)</td><td>encoding + flags + const_value</td></tr> + <tr><td>CONST (has nulls)</td><td>Yes</td><td>encoding + flags + const_value + null_bitmap</td></tr> + <tr><td>DICT</td><td>Yes</td><td>encoding + flags + dict_table + [null_bitmap] + bit-packed indices</td></tr> + <tr><td>PLAIN</td><td>Yes</td><td>encoding + flags + [null_bitmap] + raw column data</td></tr> + </tbody> + </table> + + <h4>Projected Read Path</h4> + <ol> + <li>Compute <code>dir_size = num_columns_in_bucket × 4</code> (known from schema)</li> + <li>Range-read the directory from <code>bucket_offset</code></li> + <li>For each projected column, compute slot offset via prefix-sum of directory entries</li> + <li>Range-read only the projected columns' slots (merge adjacent slots into a single IO)</li> + <li>For each slot: parse <code>uncompressed_size</code> varint, then <code>zstd::decompress</code></li> + <li>Parse <code>page_content</code>: encoding, flags, meta, data → build column reader</li> + </ol> + + <h4>Monolithic vs Paged Signaling</h4> + <p> + Each bucket in the row group index is described by a pair + <code>(compressed_size, bulk_decompress_size)</code>. + This pair encodes three layout variants with zero additional bytes: + </p> + <table> + <thead> + <tr> + <th>Condition</th> + <th>Layout</th> + <th>Meaning</th> + </tr> + </thead> + <tbody> + <tr> + <td><code>compressed_size == 0</code></td> + <td>Empty</td> + <td>No data on disk for this bucket; skip entirely.</td> + </tr> + <tr> + <td><code>compressed_size > 0 && bulk_decompress_size > 0</code></td> + <td>Monolithic</td> + <td> + The on-disk blob is a single compressed block. + <code>bulk_decompress_size</code> is the decompressed size + (used to allocate the output buffer before decompression). + </td> + </tr> + <tr> + <td><code>compressed_size > 0 && bulk_decompress_size == 0</code></td> + <td>Paged</td> + <td> + The on-disk content is + <code>[directory (num_cols × u32le slot sizes)]</code> + followed by per-column compressed slots. + Each slot is independently decompressible. + </td> + </tr> + </tbody> + </table> + <p> + This encoding is unambiguous: a non-empty monolithic bucket always has + <code>bulk_decompress_size > 0</code> because a decompressed payload + cannot be zero bytes. The combination + <code>compressed_size == 0 && bulk_decompress_size != 0</code> + is invalid and must be rejected by the reader. + </p> + <h5>Validation Invariants</h5> + <ul> + <li>Paged buckets require <code>compression == ZSTD</code>.</li> + <li>The paged directory size is <code>num_cols × 4</code> bytes; + <code>dir_size + sum(slot_sizes) == compressed_size</code> must hold exactly.</li> + <li>All varint-encoded sizes (<code>compressed_size</code>, + <code>bulk_decompress_size</code>) and u32 LE slot sizes must fit in + <code>u32</code>; values exceeding <code>u32::MAX</code> are rejected + at write time.</li> + </ul> + + <!-- ============================================================ --> + <h2>Compression</h2> + <p>Both bucket data and the schema block support compression:</p> + <table> + <thead> + <tr><th>ID</th><th>Name</th><th>Description</th></tr> + </thead> + <tbody> + <tr><td>0</td><td>None</td><td>No compression</td></tr> + <tr><td>1</td><td>Zstd</td><td>Zstandard compression (default level 1)</td></tr> + </tbody> + </table> + <p> + In monolithic mode, compression is applied to the entire bucket as one block. + In paged mode, the page directory is uncompressed (fixed-length, enabling direct offset computation), + while each column slot is independently zstd-compressed. + Paged mode is only used when the compression method is Zstd. + </p> + + <!-- ============================================================ --> + <h2>Row Groups</h2> + <p> + Large files are split into row groups to bound memory usage during writing. + Each row group contains up to <code>row_group_max_size</code> bytes of uncompressed bucket data + (default: 256 MB). The row group index in the file footer records offsets and sizes for each + bucket in each row group, enabling random access to any row group. + </p> + + <!-- ============================================================ --> + <h2>Footer (32 bytes, big-endian)</h2> + <table> + <thead> + <tr><th>Offset</th><th>Size</th><th>Field</th><th>Description</th></tr> + </thead> + <tbody> + <tr><td>0</td><td>8</td><td><code>indexOffset</code></td><td>Absolute offset of Row Group Index</td></tr> + <tr><td>8</td><td>8</td><td><code>schemaBlockOffset</code></td><td>Absolute offset of Schema Block</td></tr> + <tr><td>16</td><td>4</td><td><code>numBuckets</code></td><td>Total number of buckets</td></tr> + <tr><td>20</td><td>4</td><td><code>numRowGroups</code></td><td>Total number of row groups</td></tr> + <tr><td>24</td><td>1</td><td><code>compression</code></td><td>0 = none, 1 = zstd</td></tr> + <tr><td>25</td><td>1</td><td><code>version</code></td><td>Format version (currently 1)</td></tr> + <tr><td>26</td><td>2</td><td><em>(reserved)</em></td><td>Padding, set to 0</td></tr> + <tr><td>28</td><td>4</td><td><code>magic</code></td><td><code>MOSA</code> (0x4D4F5341)</td></tr> + </tbody> + </table> + + <!-- ============================================================ --> + <h2>Row Group Index</h2> + <p>Varint-encoded, only non-empty buckets are stored. For each row group:</p> +<pre><code>varint numRows +varint nonEmptyCount +repeated nonEmptyCount times: + varint bucketId + 8 bytes bucketOffset (big-endian, absolute file offset) + varint compressedSize (total bytes: monolithic blob or directory + column slots) + varint bulkDecompressSize (> 0 = monolithic, = 0 = paged) + +--- Column Statistics (appended after bucket entries) --- +varint numStats (0 if no stats configured) +repeated numStats times: + varint columnIndex (global column index) + varint nullCount + [if nullCount < numRows]: + value minValue (serialized using standard value encoding) + value maxValue (serialized using standard value encoding)</code></pre> + <p>Empty buckets (no data) are omitted entirely, saving space for sparse schemas.</p> + + <!-- ============================================================ --> + <h2>Column Statistics</h2> + <p> + Mosaic supports optional per-column min/max statistics at row group granularity, enabling + filter pushdown: query engines can skip entire row groups whose value range does not overlap + with a filter predicate. + </p> + <ul> + <li><strong>Opt-in</strong>: Statistics are only collected for columns specified in + <code>WriterOptions.stats_columns</code>. By default, no stats are built.</li> + <li><strong>Zero overhead when disabled</strong>: When no stats columns are configured, + each row group adds only 1 byte (a varint <code>0</code>) to the row group index.</li> + <li><strong>Supported types</strong>: All orderable types — numeric (BOOLEAN through DOUBLE), + DATE, TIME, TIMESTAMP, compact DECIMAL, and string types (CHAR, VARCHAR, STRING).</li> + <li><strong>Storage</strong>: Stats are stored inline in the row group index after each + row group's bucket entries.</li> + </ul> + + <h3>Filter Pushdown</h3> + <p> + Query engines can use column statistics to skip entire row groups whose min/max range does not + overlap with a filter predicate. For example, a filter <code>age > 50</code> can skip any + row group where <code>max(age) ≤ 50</code>. + </p> + + <!-- ============================================================ --> + <h2>Schema Block</h2> + <p> + Prefixed with a 4-byte big-endian int (uncompressed size), followed by the schema data + (compressed with the file's compression method). + </p> + <p> + Columns are serialized in <strong>name-sorted order</strong>. Column names are compressed using + one of two encodings, chosen dynamically by the writer based on which produces smaller output: + </p> + <ul> + <li><strong>Front coding</strong> (mode 0): Each name shares a prefix with the previous name; + only the suffix is stored.</li> + <li><strong>BPE + front coding</strong> (mode 1): Byte Pair Encoding is applied first to + compress repeated substrings across column names (e.g., <code>_status</code>, + <code>_value</code>), then front coding is applied to the + BPE-encoded names. BPE uses token bytes 0x80–0xFF (up to 128 merge rules), and is + only applicable when all column names are ASCII-only.</li> + </ul> + + <h3>Schema Block Layout</h3> +<pre><code>varint numColumns +varint numBuckets +1 byte nameEncoding (0 = front coding, 1 = BPE + front coding) + +--- if nameEncoding == 1 (BPE) --- +varint numRules +repeated numRules times: + 1 byte left (left token of merge rule) + 1 byte right (right token of merge rule) + +--- per column (repeated numColumns times, name-sorted order) --- +varint sharedPrefixLen (bytes shared with previous column name) +varint suffixLen (bytes of new suffix) +bytes suffix (suffixLen bytes, raw or BPE-encoded) +TypeDescriptor</code></pre> Review Comment: This schema layout is missing the serialized logical/global column index. The implementation writes `global_idx` between `suffix` and `TypeDescriptor` (`core/src/schema.rs`), and the reader decodes it before `deserialize_field`. Without documenting that varint, an independent reader/writer based on this spec would be off by one field and could not preserve the original schema column order. ########## docs/java-api.html: ########## @@ -0,0 +1,386 @@ +<!-- + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. +--> + +<!DOCTYPE html> +<html lang="en"> +<head> + <meta charset="UTF-8"> + <meta name="viewport" content="width=device-width, initial-scale=1.0"> + <title>Java API - Mosaic</title> + <link rel="stylesheet" href="css/style.css"> + <script src="js/main.js"></script> +</head> +<body> + <button class="menu-toggle" aria-label="Menu">☰</button> + <div class="overlay"></div> + + <aside class="sidebar"> + <div class="sidebar-header"> + <h2>Mosaic</h2> + <p>Columnar-bucket hybrid format</p> + </div> + <nav> + <ul> + <li><a href="index.html">Home</a></li> + <li><a href="design.html">Design</a></li> + <li><a href="rust-api.html">Rust API</a></li> + <li><a href="java-api.html" class="active">Java API</a></li> + <li><a href="python-api.html">Python API</a></li> + <li><a href="cpp-api.html">C++ API</a></li> + </ul> + </nav> + <div class="sidebar-footer"> + <button class="theme-toggle">Dark Mode</button> + </div> + </aside> + + <main class="main"> + <div class="content"> + <h1>Java API</h1> + <p class="subtitle">Write and read Mosaic files from Java using Arrow Java (<code>VectorSchemaRoot</code>).</p> + + <h2>Setup</h2> + <p> + The Java API lives in the <code>java/</code> directory and depends on the + <code>mosaic_jni</code> native library. Build the native library first: + </p> +<pre><code>cargo build --release -p mosaic-jni</code></pre> + <p> + Ensure <code>libmosaic_jni.dylib</code> (macOS) or <code>libmosaic_jni.so</code> (Linux) + is on <code>java.library.path</code>. The library is loaded automatically via: + </p> +<pre><code>System.loadLibrary(<span class="str">"mosaic_jni"</span>);</code></pre> + <p> + Add the Arrow Java dependencies to your <code>pom.xml</code>: + </p> +<pre><code><span class="cmt"><!-- Arrow BOM for version management --></span> +<dependencyManagement> + <dependencies> + <dependency> + <groupId>org.apache.arrow</groupId> + <artifactId>arrow-bom</artifactId> + <version>18.1.0</version> + <type>pom</type> + <scope>import</scope> + </dependency> + </dependencies> +</dependencyManagement> + +<dependencies> + <dependency> + <groupId>org.apache.arrow</groupId> + <artifactId>arrow-vector</artifactId> + </dependency> + <dependency> + <groupId>org.apache.arrow</groupId> + <artifactId>arrow-memory-netty</artifactId> + <scope>runtime</scope> + </dependency> + <dependency> + <groupId>org.apache.arrow</groupId> + <artifactId>arrow-c-data</artifactId> + </dependency> +</dependencies></code></pre> + + <h2>Writing a File</h2> + + <h3>1. Define an Arrow Schema</h3> +<pre><code><span class="kw">import</span> org.apache.arrow.vector.types.pojo.*; +<span class="kw">import</span> org.apache.arrow.vector.types.*; + +<span class="ty">Schema</span> arrowSchema = <span class="kw">new</span> <span class="ty">Schema</span>(<span class="ty">Arrays</span>.asList( + <span class="ty">Field</span>.notNullable(<span class="str">"id"</span>, <span class="kw">new</span> <span class="ty">ArrowType.Int</span>(<span class="num">32</span>, <span class="kw">true</span>)), + <span class="ty">Field</span>.nullable(<span class="str">"name"</span>, <span class="ty">ArrowType.Utf8</span>.INSTANCE), + <span class="ty">Field</span>.nullable(<span class="str">"score"</span>, <span class="kw">new</span> <span class="ty">ArrowType.FloatingPoint</span>(DOUBLE)), Review Comment: The Java snippets use `new ArrowType.FloatingPoint(DOUBLE)`, but `DOUBLE` is not in scope with the shown imports. The implementation/tests use `FloatingPointPrecision.DOUBLE` (or a static import would be required). Please update all Java snippets that use `DOUBLE` so users can copy/paste them. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
