vbhanuchander-lang opened a new pull request, #8254:
URL: https://github.com/apache/hop/pull/8254
Closes #8080.
Brings Hive-style partitioned writes to the standard **Parquet File Output**
transform, so the
layout that Native Spark mode can already produce no longer requires the
Spark engine.
### What it does
Configure one or more incoming fields as partition fields and the transform
writes
```
/datalake/sales/year=2026/month=08/part-00-0000.parquet
/datalake/sales/year=2026/month=09/part-00-0001.parquet
```
instead of a single file set. With no partition fields configured the
transform behaves exactly as
before — same single file, same name, same columns.
### Design decisions worth a reviewer's attention
**Partition columns are not written into the files.** This matches
`DataFrameWriter.partitionBy`,
which is what the issue asks for parity with, and it is what a reader
expects when it recovers the
column from the folder name. Configuring *every* field as a partition field
is rejected rather than
producing empty files.
**Two things surfaced while testing that are worth calling out, because
neither is obvious.**
*HopVfs percent-decodes the path it is given.* Writing `region=EU%2FWest`
produced
`region=EU/West/` on disk — two folder levels — because VFS resolves the
path as a URI. The Hive
convention still wants `%2F` in the folder name, so the partition key keeps
its Hive-style form and
only the string handed to VFS has its `%` doubled to `%25`; VFS decodes that
back to a literal `%`.
A literal `%` in a value is escaped as `%25` for the same round-trip reason.
*A per-run token in the file name.* Without it, two runs in Append mode both
wrote
`part-00-0000.parquet` and the second silently replaced the first — which is
the opposite of
appending. Each run now contributes eight hex characters to the name.
**Nulls and awkward values follow the Hive convention.** A null or empty
partition value becomes
`__HIVE_DEFAULT_PARTITION__`, the same sentinel Hive and Spark use, so the
output stays readable by
them. Values are percent-escaped for the characters that would otherwise
break the layout — most
importantly `/`, so `region=EU/West` produces one folder named
`region=EU%2FWest` rather than two
levels.
**Write modes** (`Existing data` in the dialog) decide what happens to data
already in a partition
folder, and only apply while partitioning:
| Mode | Behaviour |
|---|---|
| Append *(default)* | Leave existing files, add new ones. Same as the
pre-partitioning behaviour. |
| Overwrite partitions | Empty a partition folder before this run's first
file goes into it. Untouched partitions are left alone. |
| Fail if exists | Refuse to write if the partition folder already exists. |
| Overwrite all | Empty the whole base folder once, before the first file is
written. |
The subtle one is *Overwrite partitions*: it must clear a folder **once per
run**, not once per
file. A partition that gets closed and reopened (see below) would otherwise
delete the rows the same
run had just written to it. There is a test for exactly that.
**Bounded memory: `Maximum open partitions` (default 10).** Every open
Parquet writer buffers up to
a full row group — the row group size defaults to 256 MB — so one writer per
distinct partition
value would run a pipeline out of memory on a wide key. Spark avoids this by
sorting; a streaming
transform cannot, so instead the transform keeps at most N writers open and
closes the
least-recently-written one when it needs another. Reopening a partition
starts a new `part-...` file,
which is why a partition can end up with several files. The row-count split
(`Split into parts`)
applies per partition file as well.
### Structure
The partitioned path is additive: `processRow` branches on whether partition
fields are configured,
and the existing single-file code is untouched on the other side of that
branch. Building the Avro
and Parquet schema was extracted out of `openNewFile()` into `buildSchema()`
so both paths share it
without the partitioned path rebuilding it per file.
### Tests
`ParquetOutputPartitionTest`, 16 tests (the module is 47 in total, all
passing), covering the path building and escaping, the field
exclusion and its three rejection cases, and end-to-end writes that assert
the folder layout on disk
and read the Parquet files back:
- one folder per distinct value, and the partition column absent from the
file's schema while both
matching rows are present
- a value containing `/` stays one folder level
- each write mode, including *Overwrite partitions* leaving a sibling
partition untouched, and not
deleting what the same run just wrote (forced by setting the
open-partition limit to 1)
- six distinct values with a limit of two open partitions still writes every
row
- with no partition fields, still a single file with all three columns
`./mvnw -pl plugins/tech/parquet -am test` is green, and `spotless:check`
passes.
### One unrelated one-character fix
`messages_en_US.properties` had `ParquetOutputDialog.RowGroupSize.Label Row
group size` with no `=`,
so that dialog label never resolved. Fixed while adding keys to the same
file — happy to split it
out if you would rather it went separately.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
