bamaer commented on PR #8680:
URL: https://github.com/apache/hop/pull/8680#issuecomment-5932287221
**Should fix**
- **"Fetch on every use" re-clones on every file operation, not once per
run.** Every `GitFileObject` operation (`doGetType`, `doIsReadable`,
`doGetInputStream`, …) resolves through `GitFileSystem.resolveToFile` →
`GitCheckout.getBasePath()` → `getWorkingCopy()`. With `alwaysFetch`,
`needsFetch()` (`GitCheckout.java:267`) is always true, so each call does a
full clone into staging, deletes the live checkout and moves the new one in.
Browsing a folder or loading a workflow with a few sub-pipelines triggers one
clone per file. All of them run behind the global `CHECKOUT_LOCK`, and a reader
can have its files deleted underneath it. The docs promise a fetch "the first
time the connection is used in a run" (`git-vfs-connection.adoc:97`).
Suggested fix: fetch at most once per JVM for each checkout folder (for
example, a static set of refreshed folders), and leave periodic refreshes in
long-running Hop GUI or Hop Server processes to *Maximum checkout age*. Add a
test for `alwaysFetch`.
- **Two Hop processes sharing the cache folder can break each other's
checkout.** `CHECKOUT_LOCK` (`GitCheckout.java:88`) only locks within one JVM.
Two `hop-run` processes starting at the same time on a cold cache (the default
`${java.io.tmpdir}/hop-git-vfs`, or a shared volume in containers) can make the
following happen in `materialize()`:
- one process's `deleteQuietly(checkout)` removes the checkout the other
is reading, or
- the second `Files.move` fails because the target already exists, and the
run fails with "Unable to check out".
Suggested fix: hold a file lock (`FileChannel.lock` on `<checkout>.lock`)
around the check and the materialize step. If a complete checkout appears in
the meantime, treat losing the move as success.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
