kotwal-itpro opened a new pull request, #8859:
URL: https://github.com/apache/hop/pull/8859
PR 4 of 4 for #5524. With #8802 and #8819 merged, Lake table input and Lake
table output run on the Hop engine as well as on the native Spark engine. This
PR updates the docs and names to match and adds a local engine integration
test. No code changes.
## Docs
- **Lake table input** and **Lake table output** lose "Spark" from their
names (display names and doc titles). The pages explain how the transforms work
on each engine:
- on the Hop engine: Iceberg only, through Hop VFS; PATH and TABLE tables
(`hadoop` and `rest` catalogs); deletes; type mapping; time zones; parallel
copies;
- for output: one commit per pipeline run, nothing committed when the
pipeline fails, and the save modes;
- **concurrent writers on PATH tables:** safe on a local file system with
hard links. HDFS, object stores such as S3, and local file systems without hard
links need one writer at a time, or a REST catalog.
- **Spark Catalog** becomes **Lakehouse catalog**, since both engines use
it. The metadata key stays `spark-catalog`, so existing catalogs keep working.
The page lists what the Hop engine reads from a catalog definition, and says
Spark-style keys are accepted, so one definition serves both engines.
- Lake table merge and Lake table maintenance still run on Spark only and
keep their names.
## Integration test
`integration-tests/iceberg/main-0004-local-input-output.hwf` runs the same
scenario as the existing Spark round trip (`0001`), but on the local engine:
- write a PATH table with Lake table output;
- read it back with Lake table input;
- check the rows with a unit test.
## Testing
- I built a client from this branch, which is the current main plus the
docs, and ran `./integration-tests/scripts/run-tests.sh iceberg`. All 4 pass:
`0001-input-output`, `0002-overwrite-timetravel` and `0003-merge` on Spark
4.1.3 (`local[2]`), and the new `0004-local-input-output` on the local engine.
- `hop-tech-lakehouse`: 91 tests pass. This PR changes no code; the message
bundles only get the new display names.
- `apache-rat:check` is clean.
I used an AI coding assistant while writing this change (noted in the commit
with `Generated-by`); I reviewed and tested all of it.
addresses #5524. This completes the planned four PRs for Iceberg on the Hop
engine; Delta Lake reads on the Hop engine (#3941) can follow as a separate
proposal.
- [x] I hereby declare this contribution to be licensed under the [Apache
License Version 2.0, January 2004](http://www.apache.org/licenses/LICENSE-2.0)
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
