This is an automated email from the ASF dual-hosted git repository.
asf-gitbox-commits pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/hop-website.git
The following commit(s) were added to refs/heads/asf-site by this push:
new 1021102bbb Website updated to 45a48e9
1021102bbb is described below
commit 1021102bbbc7cfcd7d03a5ca5c5d13b484e75053
Author: jenkins <[email protected]>
AuthorDate: Wed Sep 23 10:44:45 2026 +0000
Website updated to 45a48e9
---
.../pipeline/transforms/parquet-file-input.html | 118 +++++++++++++++++++--
.../pipeline/transforms/parquet-file-output.html | 77 ++++++++++++--
2 files changed, 181 insertions(+), 14 deletions(-)
diff --git a/manual/next/pipeline/transforms/parquet-file-input.html
b/manual/next/pipeline/transforms/parquet-file-input.html
index 25c641aa3a..d6f4034e23 100644
--- a/manual/next/pipeline/transforms/parquet-file-input.html
+++ b/manual/next/pipeline/transforms/parquet-file-input.html
@@ -2863,20 +2863,18 @@
<div class="ulist">
<ul>
<li>
-<p>To support reading from any location through Apache VFS each file is loaded
into memory (one at a time).
-Make sure to allocate enough memory to allow this.</p>
+<p>Files are read in place through Apache VFS: local files directly, other
locations (S3, Azure, …​) by seeking in the file.</p>
</li>
<li>
-<p>Long values can be de-serialized to Dates if they are EPOC: milliseconds
since <code>1970-01-01 00:00:00.000</code></p>
+<p>All input values are passed to the output.</p>
</li>
<li>
-<p>Parquet Binary fields are considered to be Hop Strings but you can read
them as Hop Binary.</p>
+<p>A column must have the same type in every file read by the transform, or be
read into a Hop type both of its types convert to (see the table below).
+A value which can’t be converted stops the transform with an error
naming the Parquet column, its type and the field.</p>
</li>
<li>
-<p>All input values are passed to the output</p>
-</li>
-<li>
-<p>INT96 is converted to the Hop Binary data type.</p>
+<p>Timestamps annotated as adjusted to UTC are instants.
+The ones which aren’t hold the date and time fields of a local time,
taken in the time zone of the Hop JVM.</p>
</li>
</ul>
</div>
@@ -2924,6 +2922,110 @@ This row can then be used to extract metadata with the
<a href="metastructure.ht
</tbody>
</table>
</div>
+</div>
+<div class="sect1">
+<h2 id="_parquet_to_hop_types"><a class="link"
href="#_parquet_to_hop_types">Parquet to Hop types</a></h2>
+<div class="sectionbody">
+<div class="paragraph">
+<p>The <strong>Get fields</strong> button proposes a Hop type for every
column, based on its <a
href="https://parquet.apache.org/docs/file-format/types/logicaltypes/";
rel="noopener" target="_blank">logical type</a> or, without one, its physical
type.
+You can pick another Hop type from the last column instead.</p>
+</div>
+<table class="tableblock frame-all grid-all stretch">
+<colgroup>
+<col style="width: 33.3333%;">
+<col style="width: 33.3333%;">
+<col style="width: 33.3334%;">
+</colgroup>
+<thead>
+<tr>
+<th class="tableblock halign-left valign-top">Parquet column</th>
+<th class="tableblock halign-left valign-top">Get fields proposes</th>
+<th class="tableblock halign-left valign-top">Can also be read as</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td class="tableblock halign-left valign-top"><p
class="tableblock">BOOLEAN</p></td>
+<td class="tableblock halign-left valign-top"><p
class="tableblock">Boolean</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">String,
Integer (1 or 0)</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">INT32,
INT64, INT(8/16/32/64, signed), INT(8/16/32, unsigned)</p></td>
+<td class="tableblock halign-left valign-top"><p
class="tableblock">Integer</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">String,
Number, BigNumber</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">INT(64,
unsigned)</p></td>
+<td class="tableblock halign-left valign-top"><p
class="tableblock">BigNumber</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">String,
Number, Integer (up to 9223372036854775807)</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">DECIMAL
(on INT32, INT64, FIXED_LEN_BYTE_ARRAY or BINARY)</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">BigNumber,
with the precision as length and the scale as precision</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">String,
Number, Integer (the fraction is dropped), Binary (the stored bytes)</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">FLOAT,
DOUBLE</p></td>
+<td class="tableblock halign-left valign-top"><p
class="tableblock">Number</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">String,
BigNumber, Integer (rounded)</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p
class="tableblock">FLOAT16</p></td>
+<td class="tableblock halign-left valign-top"><p
class="tableblock">Number</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">String,
BigNumber, Integer (rounded), Binary (the stored bytes)</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p
class="tableblock">DATE</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Date,
midnight in the local time zone</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Timestamp.
String, Integer, Number and BigNumber get the stored number of days.</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">TIME
(MILLIS, MICROS, NANOS)</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Timestamp,
the time on 1970-01-01</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Date.
String, Integer, Number and BigNumber get the stored number.</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">TIMESTAMP
(MILLIS, MICROS, NANOS)</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Timestamp,
with its full precision</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Date.
String, Integer, Number and BigNumber get the stored number.</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p
class="tableblock">INT96</p></td>
+<td class="tableblock halign-left valign-top"><p
class="tableblock">Timestamp</p></td>
+<td class="tableblock halign-left valign-top"><p
class="tableblock">Date</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">STRING,
ENUM</p></td>
+<td class="tableblock halign-left valign-top"><p
class="tableblock">String</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">BigNumber
(text holding a number), JSON (text holding JSON), Binary</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p
class="tableblock">JSON</p></td>
+<td class="tableblock halign-left valign-top"><p
class="tableblock">JSON</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">String,
BigNumber (text holding a number), Binary</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p
class="tableblock">UUID</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">UUID, or
String when the UUID value type isn’t installed</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">String,
Binary</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">BINARY,
FIXED_LEN_BYTE_ARRAY without a logical type</p></td>
+<td class="tableblock halign-left valign-top"><p
class="tableblock">Binary</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">String
(UTF-8 text), JSON, BigNumber (text holding a number, or else the unscaled
bytes of a decimal with the length and precision of the field)</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">BSON,
INTERVAL, GEOMETRY, GEOGRAPHY</p></td>
+<td class="tableblock halign-left valign-top"><p
class="tableblock">Binary</p></td>
+<td class="tableblock halign-left valign-top"></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">LIST, MAP,
VARIANT and other nested columns</p></td>
+<td class="tableblock halign-left valign-top" colspan="2"><p
class="tableblock">Not supported</p></td>
+</tr>
+</tbody>
+</table>
+</div>
</div>
</article>
<nav class="pagination" data-pagefind-ignore aria-label="Page">
diff --git a/manual/next/pipeline/transforms/parquet-file-output.html
b/manual/next/pipeline/transforms/parquet-file-output.html
index 5c90ee897c..461119ac12 100644
--- a/manual/next/pipeline/transforms/parquet-file-output.html
+++ b/manual/next/pipeline/transforms/parquet-file-output.html
@@ -2869,12 +2869,6 @@
<p>The date optionally referenced in the output file name(s) will be the start
of the pipeline execution.</p>
</li>
<li>
-<p>Hop Date and Timestamp types are serialized as milliseconds since
<code>1970-01-01 00:00:00.000</code> UTC, with the Parquet
<code>timestamp-millis</code> logical type.</p>
-</li>
-<li>
-<p>Strings, BigNumbers, JSON and UUID values are written as UTF-8 strings.</p>
-</li>
-<li>
<p>Rows are buffered in memory until a row group is full (see the <strong>Row
group size</strong> option), then encoded, compressed and written.
Memory use is therefore bounded by the row group size (times the number of
copies and, when partitioning, the number of open partitions), not by the split
size.</p>
</li>
@@ -3087,6 +3081,77 @@ Leave empty to output all input fields.</p></td>
</table>
</div>
</div>
+</div>
+<div class="sect1">
+<h2 id="_hop_to_parquet_types"><a class="link"
href="#_hop_to_parquet_types">Hop to Parquet types</a></h2>
+<div class="sectionbody">
+<div class="paragraph">
+<p>Every field is written as an optional column of the <a
href="https://parquet.apache.org/docs/file-format/types/logicaltypes/";
rel="noopener" target="_blank">Parquet type</a> below.</p>
+</div>
+<table class="tableblock frame-all grid-all stretch">
+<colgroup>
+<col style="width: 50%;">
+<col style="width: 50%;">
+</colgroup>
+<thead>
+<tr>
+<th class="tableblock halign-left valign-top">Hop type</th>
+<th class="tableblock halign-left valign-top">Parquet column</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td class="tableblock halign-left valign-top"><p
class="tableblock">Integer</p></td>
+<td class="tableblock halign-left valign-top"><p
class="tableblock">INT64</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p
class="tableblock">Number</p></td>
+<td class="tableblock halign-left valign-top"><p
class="tableblock">DOUBLE</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">BigNumber
with a length of 1 to 38</p></td>
+<td class="tableblock halign-left valign-top"><p
class="tableblock">DECIMAL(length, precision), rounded with the rounding type
of the field.
+A value with more digits before the decimal point than the column holds stops
the transform with an error.</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">BigNumber
without a length, or longer than 38</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">STRING,
the number as text</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p
class="tableblock">String</p></td>
+<td class="tableblock halign-left valign-top"><p
class="tableblock">STRING</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p
class="tableblock">Boolean</p></td>
+<td class="tableblock halign-left valign-top"><p
class="tableblock">BOOLEAN</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p
class="tableblock">Date</p></td>
+<td class="tableblock halign-left valign-top"><p
class="tableblock">TIMESTAMP(MILLIS), adjusted to UTC</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p
class="tableblock">Timestamp</p></td>
+<td class="tableblock halign-left valign-top"><p
class="tableblock">TIMESTAMP(MICROS), adjusted to UTC.
+The nanoseconds beyond the microsecond are dropped.</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p
class="tableblock">Binary</p></td>
+<td class="tableblock halign-left valign-top"><p
class="tableblock">BINARY</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p
class="tableblock">JSON</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">JSON, the
JSON text</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p
class="tableblock">UUID</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">UUID, the
16 bytes of the UUID</p></td>
+</tr>
+</tbody>
+</table>
+<div class="paragraph">
+<p>Other Hop types can’t be written to Parquet.</p>
+</div>
+</div>
</div>
</article>
<nav class="pagination" data-pagefind-ignore aria-label="Page">
