hansva opened a new issue, #9597:
URL: https://github.com/apache/hop/issues/9597
### What would you like to happen?
Some transforms do setup work for the whole run, such as truncating a table,
creating it, or adding columns. Today they do it on the first row that copy 0
processes. That causes trouble whenever more than one copy runs the transform:
- **Local engine, several copies:** copy 1 can insert rows before copy 0 has
truncated or changed the table. Those rows are then truncated away, or written
into a table whose structure is still being changed.
- **Native Spark and Beam engines:** every partition (or bundle) runs as
copy 0, so every partition does the setup:
- partitions truncate each other's rows;
- two partitions create the same table;
- adding or dropping columns deadlocks.
Transforms affected:
- Table output: "Truncate table" and "Automatically update table structure".
- PGVector upsert: creating the table, the HNSW index and the mapped columns.
- Neo4j Output and Graph Output: creating indexes and constraints.
Partitions deadlock on the same statements.
### Proposal
Give transforms a hook for work that runs once per run, before any copy,
partition or bundle reads a row:
- `ITransformMeta.needsPrepareOnce()`: the transform says it has such work.
- `ITransform.prepareOnce()`: does the work. The input row metadata is set
to what the previous transforms declare.
- `TransformPrepareOnce` runs it:
- **Local engine:** once per transform, on copy 0, after init and before
the threads start. A transactional pipeline (one that uses a connection group)
leaves it to the first row, so the work stays inside the transaction.
- **Native Spark and Beam:** on the driver, before the job, for the
transforms an enabled hop connects. The names of the prepared transforms go to
the partitions or bundles in `Internal.Pipeline.PreparedOnceTransforms`, so the
partitions don't do the work again.
- A transform flagged as prepared (`isPreparedOnce()`) skips the work. A
transform that doesn't use the hook behaves exactly as before.
Transforms that use the hook:
| Transform | Work done once per run |
|---|---|
| Table output | "Truncate table", "Automatically update table structure".
"Truncate on first row" still waits for the first row. |
| PGVector upsert | Creates the table, the HNSW index and the mapped
columns, with autocommit on as before. |
| Graph Output | Creates the indexes of the model's nodes. |
| Neo4j Output | Creates the indexes of its fixed labels. Labels that come
from a field are only known on the first row, so those indexes are still
created there. |
Neo4j index and constraint statements also run in a write transaction where
the database allows schema changes in one. The driver then retries a statement
that deadlocks with the same statement from another copy or partition.
### Tests
- Unit tests for `TransformPrepareOnce` on the local engine and on native
Spark.
- Unit tests for Table output, PGVector upsert, Neo4j Output and Graph
Output.
- On native Spark, these database integration tests now pass: 0006, 0020 and
0033–0036. The two Neo4j tests that deadlocked pass too.
- The docs for Table output, PGVector upsert, Neo4j Output and Graph Output
describe when the setup work happens.
### Issue Priority
Priority: 2
### Issue Component
Component: API
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]