cstamas commented on issue #836: URL: https://github.com/apache/maven-mvnd/issues/836#issuecomment-1706509873
Locking in Mavem aims to solve two things: * parallel build coordination (within one Maven parallel building process) * shared local repository across several Maven (ST or MT, parallel) processes For brevity we call them * ST, single threaded, the "usual" way of Maven working, * MT, multi threaded, or "parallel", either using `-T` with or without "smart builder"/mvnd and * MP, multi process case, when local repository is shared across multiple Maven ST/MT processes (on single or multiple hosts). HOW locking is set up (especially for which scenario, MT/MP) is left to user, some examples: * the default locking (since 3.9.0) the `rwlock-local` as name says is (JVM) local, so handles only MT builds * the `file-lock` can handle MP cases (on single host, or on properly set up NFS volume) * for larger scale, usually CI use case with shared local repositories, we recommend the Hazelcast/Redisson solutions By the way, for non-mvnd users I'd personally recommend to NOT use the "vanilla" `-T` builder (that comes with Maven), but rather use the superior Takari Smart Builder (see [README](https://github.com/takari/takari-smart-builder/blob/master/README.md)). This will make Maven parallel builds behave similarly as mvnd (also uses Smart builder) sans the persistent daemon and logging feature. I would really like to see a reproducer project that produces this kind of error, but let me explain what happens here... By default, the parameter `aether.syncContext.named.time` (30 seconds) is the amount of time a synchronization context shall wait to obtain a needed lock. In latest Maven/mvnd releases the message is actually improved to clearly show that (as can be seen above in comments, but not in original issue description: "Could not acquire write lock for $lockName in 30 SECONDS"). So, there is no cause per se, the cause IS the timeout. Sadly, way before NamedLocks (that are actually used as "low level fine grained locking implementation") the [`SyncContext`](https://github.com/apache/maven-resolver/blob/maven-resolver-1.9.15/maven-resolver-api/src/main/java/org/eclipse/aether/SyncContext.java) was defined way earlier and I "inherited" it. Major problem with this API is "coarseness" of it. My current assumption/focus is "hot artifacts": artifacts that are highly referenced (for example like slf4j-api might be in a project, or some "api" module that has zillion downstream plugin modules building against given api module, etc -- usually "star shaped" projects at least when dependencies observed) due `SyncContext` above could cause that parallel resolved module dependencies became serially resolved, and -- especially on larger projects -- the "looser" threads end up congested, waiting more then 30 seconds, that will cause the timeout. Current (bad) workaround for this is raising the time limit using `-Daether.syncContext.named.time=40` (default time unit is seconds) and "experiment", as this value theoretically depends on project layout and size. But is bad, as you want fast builds, but this actually slows it down even more. --- An example of perf test against Apache Camel is here: https://cstamas.github.io/camel-perftest/ Sadly, I could not make it perform these errors. --- So any (hopefully OSS) "reproducer project" is welcome, to sort out last bits of Maven locking. TIA! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
