This is an automated email from the ASF dual-hosted git repository.
asf-gitbox-commits pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/hop-website.git
The following commit(s) were added to refs/heads/asf-site by this push:
new ab017a3280 Website updated to 45a48e9
ab017a3280 is described below
commit ab017a32807c409de3c754d023090c0014e99b5c
Author: jenkins <[email protected]>
AuthorDate: Wed Sep 23 07:40:26 2026 +0000
Website updated to 45a48e9
---
.../pipeline/transforms/parquet-file-output.html | 49 ++++++++++++++++------
1 file changed, 37 insertions(+), 12 deletions(-)
diff --git a/manual/next/pipeline/transforms/parquet-file-output.html
b/manual/next/pipeline/transforms/parquet-file-output.html
index f5abcb52bd..5c90ee897c 100644
--- a/manual/next/pipeline/transforms/parquet-file-output.html
+++ b/manual/next/pipeline/transforms/parquet-file-output.html
@@ -2869,15 +2869,14 @@
<p>The date optionally referenced in the output file name(s) will be the start
of the pipeline execution.</p>
</li>
<li>
-<p>Hop Date types are serialized as EPOC: milliseconds since <code>1970-01-01
00:00:00.000</code></p>
+<p>Hop Date and Timestamp types are serialized as milliseconds since
<code>1970-01-01 00:00:00.000</code> UTC, with the Parquet
<code>timestamp-millis</code> logical type.</p>
</li>
<li>
-<p>Strings are written as binary in UTF-8</p>
+<p>Strings, BigNumbers, JSON and UUID values are written as UTF-8 strings.</p>
</li>
<li>
-<p>Compression of data into columnar format is being done in memory.
-This happens when all rows are written.
-To not run out of memory make sure to specify a split size.</p>
+<p>Rows are buffered in memory until a row group is full (see the <strong>Row
group size</strong> option), then encoded, compressed and written.
+Memory use is therefore bounded by the row group size (times the number of
copies and, when partitioning, the number of open partitions), not by the split
size.</p>
</li>
</ul>
</div>
@@ -2935,7 +2934,7 @@ The copy number is formatted with mask
<code>00</code></p></td>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">Split into
parts and include number?</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">Enable
this option if you want to split the output into multiple parts.
-Specify a split size larger than 0 and this is then the number of rows per
file.
+Specify a split size larger than 0 and this is then the number of
<strong>rows</strong> per file.
The file part (split) number will be included in the filename to make sure
that the same file is not being overwritten.
The split number is formatted with mask <code>0000</code></p></td>
</tr>
@@ -2979,19 +2978,45 @@ The default is UNCOMPRESSED.</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">Choose the
protocol version of Parquet (1.0 or 2.0)</p></td>
</tr>
<tr>
-<td class="tableblock halign-left valign-top"><p class="tableblock">Row group
size</p></td>
-<td class="tableblock halign-left valign-top"><p class="tableblock">The amount
of rows in a group</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Row group
size (bytes)</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">The size
of a row group in <strong>bytes</strong>, not rows.
+A row group is the unit of work for a reader: the transform buffers rows in
memory until the row group is full, then encodes, compresses and writes it.
+The default is <code>268435456</code> (256 MB); Parquet itself defaults to 128
MB, and the Parquet project recommends 512 MB to 1 GB for large files.
+See <a href="#row-group-size">Row group size and the file footer</a> below
before lowering this value.</p></td>
</tr>
<tr>
-<td class="tableblock halign-left valign-top"><p class="tableblock">Data page
size</p></td>
-<td class="tableblock halign-left valign-top"><p class="tableblock">The data
page size on a 1kB boundary (default is 1048576)</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Data page
size (bytes)</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">The size
of a data page in bytes, the unit of encoding and compression inside a column
chunk.
+The default is <code>8192</code> (8 kB), which the Parquet project recommends
for good read performance; Parquet itself defaults to <code>1048576</code> (1
MB).</p></td>
</tr>
<tr>
-<td class="tableblock halign-left valign-top"><p class="tableblock">Dictionary
page size</p></td>
-<td class="tableblock halign-left valign-top"><p class="tableblock">The data
dictionary page size on a 1kB boundary (default is 1048576)</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Dictionary
page size (bytes)</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">The
maximum size of a dictionary page in bytes.
+When the dictionary of a column grows past this size the writer falls back to
plain encoding for that column chunk.
+The default is <code>1048576</code> (1 MB).</p></td>
</tr>
</tbody>
</table>
+<div class="paragraph">
+<p>The three size options accept human-friendly numbers such as
<code>256m</code> (256,000,000), <code>1.5m</code> or <code>1e6</code>, and
variables.</p>
+</div>
+<div class="sect3">
+<h4 id="row-group-size"><a class="link" href="#row-group-size">Row group size
and the file footer</a></h4>
+<div class="paragraph">
+<p>Every row group adds an entry per column to the file footer: offsets,
sizes, encodings and min/max statistics.
+The footer therefore grows with the number of row groups times the number of
columns, and with the size of the string values in the statistics.
+Readers such as Dremio and Apache Drill refuse files whose footer is larger
than 16 MB (<code>Max supported footer size is 16777216</code>).</p>
+</div>
+<div class="paragraph">
+<p>Small row groups are the usual cause of an oversized footer.
+Hop 2.3 and 2.4 saved a row group size of <code>20000</code> into every new
Parquet file output transform, so pipelines created with those versions write a
row group every 20 kB and can produce footers of tens of megabytes.
+Open such transforms and set the row group size back to the default, or at
least to a few megabytes.
+The transform logs a warning at startup when the row group size is below 1
MB.</p>
+</div>
+<div class="paragraph">
+<p>With detailed logging enabled the transform logs every file it closes with
its size, its number of rows and row groups and the size of its footer, so you
can see the effect of the settings on your data without inspecting the
files.</p>
+</div>
+</div>
</div>
<div class="sect2">
<h3 id="_partitioning"><a class="link"
href="#_partitioning">Partitioning</a></h3>
