Re: MNG-6843 Parallel build fails due to missing JAR artifacts in compilePath
Thanks, will hook it up with our internal build On Sun, Jul 11, 2021 at 6:59 PM Olivier Lamy wrote: > done > > On Sun, 11 Jul 2021 at 07:44, Dan Tran wrote: > > > possible to push the latest snapshot to > > > > > https://repository.apache.org/content/groups/snapshots/org/apache/maven/apache-maven > > ? > > > > Thanks > > > > -D > > > > On Sat, Jul 10, 2021 at 2:33 PM Dan Tran wrote: > > > > > Thanks, I will test out the latest 3.8 > > > > > > -D > > > > > > On Sat, Jul 10, 2021 at 12:29 PM Falko Modler > wrote: > > > > > >> Done already: https://github.com/apache/maven/pull/482 > > >> > > >> Cheers, > > >> Falko > > >> > > >> - > > >> To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org > > >> For additional commands, e-mail: dev-h...@maven.apache.org > > >> > > >> > > > > > -- > Olivier Lamy > http://twitter.com/olamy | http://linkedin.com/in/olamy >
Re: MNG-6843 Parallel build fails due to missing JAR artifacts in compilePath
done On Sun, 11 Jul 2021 at 07:44, Dan Tran wrote: > possible to push the latest snapshot to > > https://repository.apache.org/content/groups/snapshots/org/apache/maven/apache-maven > ? > > Thanks > > -D > > On Sat, Jul 10, 2021 at 2:33 PM Dan Tran wrote: > > > Thanks, I will test out the latest 3.8 > > > > -D > > > > On Sat, Jul 10, 2021 at 12:29 PM Falko Modler wrote: > > > >> Done already: https://github.com/apache/maven/pull/482 > >> > >> Cheers, > >> Falko > >> > >> - > >> To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org > >> For additional commands, e-mail: dev-h...@maven.apache.org > >> > >> > -- Olivier Lamy http://twitter.com/olamy | http://linkedin.com/in/olamy
Re: MNG-6843 Parallel build fails due to missing JAR artifacts in compilePath
possible to push the latest snapshot to https://repository.apache.org/content/groups/snapshots/org/apache/maven/apache-maven ? Thanks -D On Sat, Jul 10, 2021 at 2:33 PM Dan Tran wrote: > Thanks, I will test out the latest 3.8 > > -D > > On Sat, Jul 10, 2021 at 12:29 PM Falko Modler wrote: > >> Done already: https://github.com/apache/maven/pull/482 >> >> Cheers, >> Falko >> >> - >> To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org >> For additional commands, e-mail: dev-h...@maven.apache.org >> >>
Re: MNG-6843 Parallel build fails due to missing JAR artifacts in compilePath
Thanks, I will test out the latest 3.8 -D On Sat, Jul 10, 2021 at 12:29 PM Falko Modler wrote: > Done already: https://github.com/apache/maven/pull/482 > > Cheers, > Falko > > - > To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org > For additional commands, e-mail: dev-h...@maven.apache.org > >
Re: MNG-6843 Parallel build fails due to missing JAR artifacts in compilePath
Done already: https://github.com/apache/maven/pull/482 Cheers, Falko - To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org For additional commands, e-mail: dev-h...@maven.apache.org
Re: MNG-6843 Parallel build fails due to missing JAR artifacts in compilePath
I think this is the one https://github.com/apache/maven/pull/413 -D On Sat, Jul 10, 2021 at 2:56 AM Michael Osipov wrote: > Am 2021-07-10 um 00:52 schrieb Dan Tran: > > ah sorry, the proposed PR merged into master. Can we back port it to > 3.8? > > Which PR exactly? > > - > To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org > For additional commands, e-mail: dev-h...@maven.apache.org > >
Re: MNG-6843 Parallel build fails due to missing JAR artifacts in compilePath
Am 2021-07-10 um 00:52 schrieb Dan Tran: ah sorry, the proposed PR merged into master. Can we back port it to 3.8? Which PR exactly? - To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org For additional commands, e-mail: dev-h...@maven.apache.org
Re: MNG-6843 Parallel build fails due to missing JAR artifacts in compilePath
ah sorry, the proposed PR merged into master. Can we back port it to 3.8? -D On Fri, Jul 9, 2021 at 3:47 PM Dan Tran wrote: > ping again :-) to keep discussion going :-) > > -D > > On Tue, May 18, 2021 at 6:20 AM Guillaume Nodet wrote: > >> Maybe not that many people are using parallel builds... >> Imho, the PR should be merged. I've created >> https://issues.apache.org/jira/browse/MNG-7157 to provide a better API >> and deprecate the getArtifacts() method which is flawed (see discussion on >> the PR https://github.com/apache/maven/pull/413#issuecomment-841720517). >> >> Le mar. 4 mai 2021 à 15:15, Michael Osipov a écrit >> : >> >> > Am 2021-05-03 um 22:15 schrieb Falko Modler: >> > > Earlier today Michael Osipov added this note to the ticket: >> > > >> > > I confirm that this still happens on Maven master with Resolver >> > > 1.7.0-SNAPSHOT. This isn't Resolver related because no dependency >> > > resolution happens in the build. >> > >> > I am still confused why only a few suffer from this. Is this related to >> > dependency inheritance? >> > >> > - >> > To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org >> > For additional commands, e-mail: dev-h...@maven.apache.org >> > >> > >> >> -- >> >> Guillaume Nodet >> >
Re: MNG-6843 Parallel build fails due to missing JAR artifacts in compilePath
ping again :-) to keep discussion going :-) -D On Tue, May 18, 2021 at 6:20 AM Guillaume Nodet wrote: > Maybe not that many people are using parallel builds... > Imho, the PR should be merged. I've created > https://issues.apache.org/jira/browse/MNG-7157 to provide a better API > and deprecate the getArtifacts() method which is flawed (see discussion on > the PR https://github.com/apache/maven/pull/413#issuecomment-841720517). > > Le mar. 4 mai 2021 à 15:15, Michael Osipov a écrit : > > > Am 2021-05-03 um 22:15 schrieb Falko Modler: > > > Earlier today Michael Osipov added this note to the ticket: > > > > > > I confirm that this still happens on Maven master with Resolver > > > 1.7.0-SNAPSHOT. This isn't Resolver related because no dependency > > > resolution happens in the build. > > > > I am still confused why only a few suffer from this. Is this related to > > dependency inheritance? > > > > - > > To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org > > For additional commands, e-mail: dev-h...@maven.apache.org > > > > > > -- > > Guillaume Nodet >
Re: MNG-6843 Parallel build fails due to missing JAR artifacts in compilePath
Maybe not that many people are using parallel builds... Imho, the PR should be merged. I've created https://issues.apache.org/jira/browse/MNG-7157 to provide a better API and deprecate the getArtifacts() method which is flawed (see discussion on the PR https://github.com/apache/maven/pull/413#issuecomment-841720517). Le mar. 4 mai 2021 à 15:15, Michael Osipov a écrit : > Am 2021-05-03 um 22:15 schrieb Falko Modler: > > Earlier today Michael Osipov added this note to the ticket: > > > > I confirm that this still happens on Maven master with Resolver > > 1.7.0-SNAPSHOT. This isn't Resolver related because no dependency > > resolution happens in the build. > > I am still confused why only a few suffer from this. Is this related to > dependency inheritance? > > - > To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org > For additional commands, e-mail: dev-h...@maven.apache.org > > -- Guillaume Nodet
Re: MNG-6843 Parallel build fails due to missing JAR artifacts in compilePath
Am 2021-05-03 um 22:15 schrieb Falko Modler: Earlier today Michael Osipov added this note to the ticket: I confirm that this still happens on Maven master with Resolver 1.7.0-SNAPSHOT. This isn't Resolver related because no dependency resolution happens in the build. I am still confused why only a few suffer from this. Is this related to dependency inheritance? - To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org For additional commands, e-mail: dev-h...@maven.apache.org
Re: MNG-6843 Parallel build fails due to missing JAR artifacts in compilePath
Earlier today Michael Osipov added this note to the ticket: I confirm that this still happens on Maven master with Resolver 1.7.0-SNAPSHOT. This isn't Resolver related because no dependency resolution happens in the build. Cheers, Falko Am 03.05.2021 um 02:25 schrieb Tibor Digana: We have already moved ahead and the new Resolver should solve this issue. This looks similar to the issue in Surefire which was closed as a bug of Resoler. Michael is currently working on a new Resolver and he was participating in the analysis of the bug too. So, maybe it is still the same root cause for this one. T On Sun, May 2, 2021 at 10:24 PM Dan Tran wrote: we are in pain with this issue :-) -D On Sun, Jan 24, 2021 at 2:56 PM Tibor Digana wrote: Hi Falco, This is not the first time I have been talking about these principles in our team. Seven years ago and then in 2019. But sorry I cannot force the people to do it and they have to start by themself. We have to do it together. All I can do is to provide some training and elaborate a problem but I am convinced that we will meet again after the next seven years if we don't understand the JMM. Thus we should prevent from redoing the same bad and understand the process on how to make the code better in the early beginning. Cheers Tibor17 On Sun, Jan 24, 2021 at 10:51 PM Falko Modler wrote: Hi Tibor, thanks for this very elaborate answer and I always appreciate your feedback, but to me it kind of misses the point a bit...? may not necessarily have to do with concurrent access. But it does in this special case. Please see the issue and the linked explanations. The solution with ThreadLocal would eat too much memory. Is that so? Are you sure about this? How much is "too much"? Are there any predefined profiling tests I can run? I mean: yes, it is a workaround and immutable core classes that are _designed_ for concurrent access would be much better, but who is going to do such a massive refactoring (without breaking Maven extensions that are today mutating MavenProject etc.)? TBH, this is one of the, IMHO, critical bugs that should have been fixed before Maven 4. Cheers, Falko Am 21.01.2021 um 02:13 schrieb Tibor Digana: I commented on one issue regarding the NULL JAR file in Artifact a few days ago. The thing that some data is "missing" in some large object structures in the environment with multiple threads may not necessarily have to do with concurrent access. There may not be any writes to MavenProject or MavenSession causing "missed" data, and the answer why this happens is Memory Model. It's the fact that non-concurrent or non-immutable objects may lose some references very easily! This has all to do with JMM and not the happens-before relationship. Suppose that we have thread T1 creating ArrayList and adding elements into this collection. artifacts = new ArrayList(); artifacts.add(new DefaultArtifact(...)); Suppose thread T2 reads the artifacts from the collection right after "artifacts.add()". Artifact a = artifacts.get(0); In practice the following happens: artifacts.size() returns 1 but artifacts.get(0) returns NULL Let;s explain why it happens. The implementation of ArrayList is not native. It is a pure Java implementation which has two variables inside: + count:int + array:Object[] These two variables always appear in a critical section and they do not have proper treatments in ArrayList. Technically, the things are complicated on the CPU level and more complicated than happens-before theorems. T1 contains pointers and data in CPU registers or CPU cache. No Thread has a direct access to a stack of another Thread, and of course it does not operate on main memory. The CPU uses memory barriers (assembler instructions) and a cache to operate with RAM and memory coherency. These instructions are used via Java keywords: final, volatile and synchronized. T2 may not see all elements completely from the ArrayList because there are no safety mechanisms in the implementation of ArrayList to make this happen. Thus the T2 may see the values in the Java variable "count" *but it may not see the values in* "array", or vice versa. The results are NPE, or missing JAR artifacts or the issues with Maven Resolver, as we can see in https://github.com/apache/maven/pull/310 The solution with ThreadLocal would eat too much memory. Reimplementing the POJO classes in Maven and making them thread safe would solve many issues in the Core and Resolver. Considering my examples with ArrayList, the thread safety should continue deeper with the implementation of DefaultArtifact, etc. In my experience, it's worth using the collection which appears in the package "java.util.concurrent". For instance, I use ConcurrentLinkedDequeue for simple iterators with small amounts of elements. Alternatively use COWAL for large data and reordering of elements by adding or removing them somewhere inside. Cheers Tibor17 On Sat,
Re: MNG-6843 Parallel build fails due to missing JAR artifacts in compilePath
We have already moved ahead and the new Resolver should solve this issue. This looks similar to the issue in Surefire which was closed as a bug of Resoler. Michael is currently working on a new Resolver and he was participating in the analysis of the bug too. So, maybe it is still the same root cause for this one. T On Sun, May 2, 2021 at 10:24 PM Dan Tran wrote: > we are in pain with this issue :-) > > -D > > On Sun, Jan 24, 2021 at 2:56 PM Tibor Digana > wrote: > > > Hi Falco, > > > > This is not the first time I have been talking about these principles in > > our team. > > Seven years ago and then in 2019. But sorry I cannot force the people to > do > > it and they have to start by themself. We have to do it together. > > All I can do is to provide some training and elaborate a problem but I am > > convinced that we will meet again after the next seven years if we > > don't understand the JMM. Thus we should prevent from redoing the same > bad > > and understand the process on how to make the code better in the early > > beginning. > > > > Cheers > > Tibor17 > > > > > > On Sun, Jan 24, 2021 at 10:51 PM Falko Modler wrote: > > > > > Hi Tibor, > > > > > > thanks for this very elaborate answer and I always appreciate your > > > feedback, but to me it kind of misses the point a bit...? > > > > > > > may not necessarily have to do with concurrent access. > > > But it does in this special case. Please see the issue and the linked > > > explanations. > > > > > > > The solution with ThreadLocal would eat too much memory. > > > Is that so? Are you sure about this? How much is "too much"? > > > Are there any predefined profiling tests I can run? > > > > > > I mean: yes, it is a workaround and immutable core classes that are > > > _designed_ for concurrent access would be much better, > > > but who is going to do such a massive refactoring (without breaking > > > Maven extensions that are today mutating MavenProject etc.)? > > > > > > TBH, this is one of the, IMHO, critical bugs that should have been > fixed > > > before Maven 4. > > > > > > Cheers, > > > Falko > > > > > > Am 21.01.2021 um 02:13 schrieb Tibor Digana: > > > > I commented on one issue regarding the NULL JAR file in Artifact a > few > > > days > > > > ago. > > > > The thing that some data is "missing" in some large object structures > > in > > > > the environment with multiple threads may not necessarily have to do > > with > > > > concurrent access. > > > > There may not be any writes to MavenProject or MavenSession causing > > > > "missed" data, and the answer why this happens is Memory Model. > > > > > > > > It's the fact that non-concurrent or non-immutable objects may lose > > some > > > > references very easily! > > > > This has all to do with JMM and not the happens-before relationship. > > > > > > > > Suppose that we have thread T1 creating ArrayList and adding elements > > > into > > > > this collection. > > > > artifacts = new ArrayList(); > > > > artifacts.add(new DefaultArtifact(...)); > > > > > > > > Suppose thread T2 reads the artifacts from the collection right after > > > > "artifacts.add()". > > > > Artifact a = artifacts.get(0); > > > > > > > > In practice the following happens: > > > > artifacts.size() returns 1 > > > > but artifacts.get(0) returns NULL > > > > > > > > Let;s explain why it happens. > > > > The implementation of ArrayList is not native. It is a pure Java > > > > implementation which has two variables inside: > > > > + count:int > > > > + array:Object[] > > > > These two variables always appear in a critical section and they do > not > > > > have proper treatments in ArrayList. > > > > Technically, the things are complicated on the CPU level and more > > > > complicated than happens-before theorems. > > > > T1 contains pointers and data in CPU registers or CPU cache. No > Thread > > > has > > > > a direct access to a stack of another Thread, and of course it does > not > > > > operate on main memory. > > > > The CPU uses memory barriers (assembler instructions) and a cache to > > > > operate with RAM and memory coherency. > > > > These instructions are used via Java keywords: final, volatile and > > > > synchronized. > > > > T2 may not see all elements completely from the ArrayList because > there > > > are > > > > no safety mechanisms in the implementation of ArrayList to make this > > > happen. > > > > Thus the T2 may see the values in the Java variable "count" *but it > may > > > not > > > > see the values in* "array", or vice versa. > > > > > > > > The results are NPE, or missing JAR artifacts or the issues with > Maven > > > > Resolver, as we can see in https://github.com/apache/maven/pull/310 > > > > > > > > The solution with ThreadLocal would eat too much memory. > > > > Reimplementing the POJO classes in Maven and making them thread safe > > > would > > > > solve many issues in the Core and Resolver. > > > > Considering my examples with ArrayList, the thread safety should > > continue > > > > deeper
Re: MNG-6843 Parallel build fails due to missing JAR artifacts in compilePath
we are in pain with this issue :-) -D On Sun, Jan 24, 2021 at 2:56 PM Tibor Digana wrote: > Hi Falco, > > This is not the first time I have been talking about these principles in > our team. > Seven years ago and then in 2019. But sorry I cannot force the people to do > it and they have to start by themself. We have to do it together. > All I can do is to provide some training and elaborate a problem but I am > convinced that we will meet again after the next seven years if we > don't understand the JMM. Thus we should prevent from redoing the same bad > and understand the process on how to make the code better in the early > beginning. > > Cheers > Tibor17 > > > On Sun, Jan 24, 2021 at 10:51 PM Falko Modler wrote: > > > Hi Tibor, > > > > thanks for this very elaborate answer and I always appreciate your > > feedback, but to me it kind of misses the point a bit...? > > > > > may not necessarily have to do with concurrent access. > > But it does in this special case. Please see the issue and the linked > > explanations. > > > > > The solution with ThreadLocal would eat too much memory. > > Is that so? Are you sure about this? How much is "too much"? > > Are there any predefined profiling tests I can run? > > > > I mean: yes, it is a workaround and immutable core classes that are > > _designed_ for concurrent access would be much better, > > but who is going to do such a massive refactoring (without breaking > > Maven extensions that are today mutating MavenProject etc.)? > > > > TBH, this is one of the, IMHO, critical bugs that should have been fixed > > before Maven 4. > > > > Cheers, > > Falko > > > > Am 21.01.2021 um 02:13 schrieb Tibor Digana: > > > I commented on one issue regarding the NULL JAR file in Artifact a few > > days > > > ago. > > > The thing that some data is "missing" in some large object structures > in > > > the environment with multiple threads may not necessarily have to do > with > > > concurrent access. > > > There may not be any writes to MavenProject or MavenSession causing > > > "missed" data, and the answer why this happens is Memory Model. > > > > > > It's the fact that non-concurrent or non-immutable objects may lose > some > > > references very easily! > > > This has all to do with JMM and not the happens-before relationship. > > > > > > Suppose that we have thread T1 creating ArrayList and adding elements > > into > > > this collection. > > > artifacts = new ArrayList(); > > > artifacts.add(new DefaultArtifact(...)); > > > > > > Suppose thread T2 reads the artifacts from the collection right after > > > "artifacts.add()". > > > Artifact a = artifacts.get(0); > > > > > > In practice the following happens: > > > artifacts.size() returns 1 > > > but artifacts.get(0) returns NULL > > > > > > Let;s explain why it happens. > > > The implementation of ArrayList is not native. It is a pure Java > > > implementation which has two variables inside: > > > + count:int > > > + array:Object[] > > > These two variables always appear in a critical section and they do not > > > have proper treatments in ArrayList. > > > Technically, the things are complicated on the CPU level and more > > > complicated than happens-before theorems. > > > T1 contains pointers and data in CPU registers or CPU cache. No Thread > > has > > > a direct access to a stack of another Thread, and of course it does not > > > operate on main memory. > > > The CPU uses memory barriers (assembler instructions) and a cache to > > > operate with RAM and memory coherency. > > > These instructions are used via Java keywords: final, volatile and > > > synchronized. > > > T2 may not see all elements completely from the ArrayList because there > > are > > > no safety mechanisms in the implementation of ArrayList to make this > > happen. > > > Thus the T2 may see the values in the Java variable "count" *but it may > > not > > > see the values in* "array", or vice versa. > > > > > > The results are NPE, or missing JAR artifacts or the issues with Maven > > > Resolver, as we can see in https://github.com/apache/maven/pull/310 > > > > > > The solution with ThreadLocal would eat too much memory. > > > Reimplementing the POJO classes in Maven and making them thread safe > > would > > > solve many issues in the Core and Resolver. > > > Considering my examples with ArrayList, the thread safety should > continue > > > deeper with the implementation of DefaultArtifact, etc. > > > In my experience, it's worth using the collection which appears in the > > > package "java.util.concurrent". > > > For instance, I use ConcurrentLinkedDequeue for simple iterators with > > small > > > amounts of elements. Alternatively use COWAL for large data and > > reordering > > > of elements by adding or removing them somewhere inside. > > > > > > Cheers > > > Tibor17 > > > > > > > > > > > > On Sat, Jan 16, 2021 at 10:21 PM Dan Tran wrote: > > > > > >> we are facing the same issue at work (300+ modules), classpath > > >> empty randomly empty > > >> > >
Re: MNG-6843 Parallel build fails due to missing JAR artifacts in compilePath
Hi Falco, This is not the first time I have been talking about these principles in our team. Seven years ago and then in 2019. But sorry I cannot force the people to do it and they have to start by themself. We have to do it together. All I can do is to provide some training and elaborate a problem but I am convinced that we will meet again after the next seven years if we don't understand the JMM. Thus we should prevent from redoing the same bad and understand the process on how to make the code better in the early beginning. Cheers Tibor17 On Sun, Jan 24, 2021 at 10:51 PM Falko Modler wrote: > Hi Tibor, > > thanks for this very elaborate answer and I always appreciate your > feedback, but to me it kind of misses the point a bit...? > > > may not necessarily have to do with concurrent access. > But it does in this special case. Please see the issue and the linked > explanations. > > > The solution with ThreadLocal would eat too much memory. > Is that so? Are you sure about this? How much is "too much"? > Are there any predefined profiling tests I can run? > > I mean: yes, it is a workaround and immutable core classes that are > _designed_ for concurrent access would be much better, > but who is going to do such a massive refactoring (without breaking > Maven extensions that are today mutating MavenProject etc.)? > > TBH, this is one of the, IMHO, critical bugs that should have been fixed > before Maven 4. > > Cheers, > Falko > > Am 21.01.2021 um 02:13 schrieb Tibor Digana: > > I commented on one issue regarding the NULL JAR file in Artifact a few > days > > ago. > > The thing that some data is "missing" in some large object structures in > > the environment with multiple threads may not necessarily have to do with > > concurrent access. > > There may not be any writes to MavenProject or MavenSession causing > > "missed" data, and the answer why this happens is Memory Model. > > > > It's the fact that non-concurrent or non-immutable objects may lose some > > references very easily! > > This has all to do with JMM and not the happens-before relationship. > > > > Suppose that we have thread T1 creating ArrayList and adding elements > into > > this collection. > > artifacts = new ArrayList(); > > artifacts.add(new DefaultArtifact(...)); > > > > Suppose thread T2 reads the artifacts from the collection right after > > "artifacts.add()". > > Artifact a = artifacts.get(0); > > > > In practice the following happens: > > artifacts.size() returns 1 > > but artifacts.get(0) returns NULL > > > > Let;s explain why it happens. > > The implementation of ArrayList is not native. It is a pure Java > > implementation which has two variables inside: > > + count:int > > + array:Object[] > > These two variables always appear in a critical section and they do not > > have proper treatments in ArrayList. > > Technically, the things are complicated on the CPU level and more > > complicated than happens-before theorems. > > T1 contains pointers and data in CPU registers or CPU cache. No Thread > has > > a direct access to a stack of another Thread, and of course it does not > > operate on main memory. > > The CPU uses memory barriers (assembler instructions) and a cache to > > operate with RAM and memory coherency. > > These instructions are used via Java keywords: final, volatile and > > synchronized. > > T2 may not see all elements completely from the ArrayList because there > are > > no safety mechanisms in the implementation of ArrayList to make this > happen. > > Thus the T2 may see the values in the Java variable "count" *but it may > not > > see the values in* "array", or vice versa. > > > > The results are NPE, or missing JAR artifacts or the issues with Maven > > Resolver, as we can see in https://github.com/apache/maven/pull/310 > > > > The solution with ThreadLocal would eat too much memory. > > Reimplementing the POJO classes in Maven and making them thread safe > would > > solve many issues in the Core and Resolver. > > Considering my examples with ArrayList, the thread safety should continue > > deeper with the implementation of DefaultArtifact, etc. > > In my experience, it's worth using the collection which appears in the > > package "java.util.concurrent". > > For instance, I use ConcurrentLinkedDequeue for simple iterators with > small > > amounts of elements. Alternatively use COWAL for large data and > reordering > > of elements by adding or removing them somewhere inside. > > > > Cheers > > Tibor17 > > > > > > > > On Sat, Jan 16, 2021 at 10:21 PM Dan Tran wrote: > > > >> we are facing the same issue at work (300+ modules), classpath > >> empty randomly empty > >> > >> Love to see some resolution, will help to test it > >> > >> Thanks > >> > >> -D > >> > >> On Fri, Jan 15, 2021 at 1:51 PM Falko Modler wrote: > >> > >>> Hi everyone, > >>> > >>> I'd like to raise awareness for the MavenProject concurrency problem > >>> that is causing MNG-6843 "Parallel build fails due to missing JAR > >>> artifacts in
Re: MNG-6843 Parallel build fails due to missing JAR artifacts in compilePath
Hi Tibor, thanks for this very elaborate answer and I always appreciate your feedback, but to me it kind of misses the point a bit...? may not necessarily have to do with concurrent access. But it does in this special case. Please see the issue and the linked explanations. The solution with ThreadLocal would eat too much memory. Is that so? Are you sure about this? How much is "too much"? Are there any predefined profiling tests I can run? I mean: yes, it is a workaround and immutable core classes that are _designed_ for concurrent access would be much better, but who is going to do such a massive refactoring (without breaking Maven extensions that are today mutating MavenProject etc.)? TBH, this is one of the, IMHO, critical bugs that should have been fixed before Maven 4. Cheers, Falko Am 21.01.2021 um 02:13 schrieb Tibor Digana: I commented on one issue regarding the NULL JAR file in Artifact a few days ago. The thing that some data is "missing" in some large object structures in the environment with multiple threads may not necessarily have to do with concurrent access. There may not be any writes to MavenProject or MavenSession causing "missed" data, and the answer why this happens is Memory Model. It's the fact that non-concurrent or non-immutable objects may lose some references very easily! This has all to do with JMM and not the happens-before relationship. Suppose that we have thread T1 creating ArrayList and adding elements into this collection. artifacts = new ArrayList(); artifacts.add(new DefaultArtifact(...)); Suppose thread T2 reads the artifacts from the collection right after "artifacts.add()". Artifact a = artifacts.get(0); In practice the following happens: artifacts.size() returns 1 but artifacts.get(0) returns NULL Let;s explain why it happens. The implementation of ArrayList is not native. It is a pure Java implementation which has two variables inside: + count:int + array:Object[] These two variables always appear in a critical section and they do not have proper treatments in ArrayList. Technically, the things are complicated on the CPU level and more complicated than happens-before theorems. T1 contains pointers and data in CPU registers or CPU cache. No Thread has a direct access to a stack of another Thread, and of course it does not operate on main memory. The CPU uses memory barriers (assembler instructions) and a cache to operate with RAM and memory coherency. These instructions are used via Java keywords: final, volatile and synchronized. T2 may not see all elements completely from the ArrayList because there are no safety mechanisms in the implementation of ArrayList to make this happen. Thus the T2 may see the values in the Java variable "count" *but it may not see the values in* "array", or vice versa. The results are NPE, or missing JAR artifacts or the issues with Maven Resolver, as we can see in https://github.com/apache/maven/pull/310 The solution with ThreadLocal would eat too much memory. Reimplementing the POJO classes in Maven and making them thread safe would solve many issues in the Core and Resolver. Considering my examples with ArrayList, the thread safety should continue deeper with the implementation of DefaultArtifact, etc. In my experience, it's worth using the collection which appears in the package "java.util.concurrent". For instance, I use ConcurrentLinkedDequeue for simple iterators with small amounts of elements. Alternatively use COWAL for large data and reordering of elements by adding or removing them somewhere inside. Cheers Tibor17 On Sat, Jan 16, 2021 at 10:21 PM Dan Tran wrote: we are facing the same issue at work (300+ modules), classpath empty randomly empty Love to see some resolution, will help to test it Thanks -D On Fri, Jan 15, 2021 at 1:51 PM Falko Modler wrote: Hi everyone, I'd like to raise awareness for the MavenProject concurrency problem that is causing MNG-6843 "Parallel build fails due to missing JAR artifacts in compilePath" [1] and probably others [2] [3] [4]. Almost a month ago, I created a ThreadLocal-based fix for this [5] (after another, older cloning-based approach had raised some concerns by Robert Scholte [6]). Michael Osipov was the only one so far having a look (thanks!) and he suggested that more Maven team members should review this. So, before I take a stab at the not so trivial integration test that Michael proposed [7], I'd like to get an approval for the general aproach (or a declination in case someone has a better idea). Thanks for your attention and feedback! Cheers, Falko [1] https://issues.apache.org/jira/browse/MNG-6843 [2] https://issues.apache.org/jira/browse/MNG-4996 [3] https://issues.apache.org/jira/browse/MNG-5750 [4] https://issues.apache.org/jira/browse/MNG-5960 [5] https://github.com/apache/maven/pull/413 [6] https://github.com/apache/maven/pull/310#issuecomment-571317501 [7] https://github.com/apache/maven/pull/413#issuecomment-754661032
Re: MNG-6843 Parallel build fails due to missing JAR artifacts in compilePath
I commented on one issue regarding the NULL JAR file in Artifact a few days ago. The thing that some data is "missing" in some large object structures in the environment with multiple threads may not necessarily have to do with concurrent access. There may not be any writes to MavenProject or MavenSession causing "missed" data, and the answer why this happens is Memory Model. It's the fact that non-concurrent or non-immutable objects may lose some references very easily! This has all to do with JMM and not the happens-before relationship. Suppose that we have thread T1 creating ArrayList and adding elements into this collection. artifacts = new ArrayList(); artifacts.add(new DefaultArtifact(...)); Suppose thread T2 reads the artifacts from the collection right after "artifacts.add()". Artifact a = artifacts.get(0); In practice the following happens: artifacts.size() returns 1 but artifacts.get(0) returns NULL Let;s explain why it happens. The implementation of ArrayList is not native. It is a pure Java implementation which has two variables inside: + count:int + array:Object[] These two variables always appear in a critical section and they do not have proper treatments in ArrayList. Technically, the things are complicated on the CPU level and more complicated than happens-before theorems. T1 contains pointers and data in CPU registers or CPU cache. No Thread has a direct access to a stack of another Thread, and of course it does not operate on main memory. The CPU uses memory barriers (assembler instructions) and a cache to operate with RAM and memory coherency. These instructions are used via Java keywords: final, volatile and synchronized. T2 may not see all elements completely from the ArrayList because there are no safety mechanisms in the implementation of ArrayList to make this happen. Thus the T2 may see the values in the Java variable "count" *but it may not see the values in* "array", or vice versa. The results are NPE, or missing JAR artifacts or the issues with Maven Resolver, as we can see in https://github.com/apache/maven/pull/310 The solution with ThreadLocal would eat too much memory. Reimplementing the POJO classes in Maven and making them thread safe would solve many issues in the Core and Resolver. Considering my examples with ArrayList, the thread safety should continue deeper with the implementation of DefaultArtifact, etc. In my experience, it's worth using the collection which appears in the package "java.util.concurrent". For instance, I use ConcurrentLinkedDequeue for simple iterators with small amounts of elements. Alternatively use COWAL for large data and reordering of elements by adding or removing them somewhere inside. Cheers Tibor17 On Sat, Jan 16, 2021 at 10:21 PM Dan Tran wrote: > we are facing the same issue at work (300+ modules), classpath > empty randomly empty > > Love to see some resolution, will help to test it > > Thanks > > -D > > On Fri, Jan 15, 2021 at 1:51 PM Falko Modler wrote: > > > Hi everyone, > > > > I'd like to raise awareness for the MavenProject concurrency problem > > that is causing MNG-6843 "Parallel build fails due to missing JAR > > artifacts in compilePath" [1] and probably others [2] [3] [4]. > > > > Almost a month ago, I created a ThreadLocal-based fix for this [5] > > (after another, older cloning-based approach had raised some concerns by > > Robert Scholte [6]). > > > > Michael Osipov was the only one so far having a look (thanks!) and he > > suggested that more Maven team members should review this. > > > > So, before I take a stab at the not so trivial integration test that > > Michael proposed [7], I'd like to get an approval for the general > > aproach (or a declination in case someone has a better idea). > > > > Thanks for your attention and feedback! > > > > Cheers, > > > > Falko > > > > > > [1] https://issues.apache.org/jira/browse/MNG-6843 > > > > [2] https://issues.apache.org/jira/browse/MNG-4996 > > > > [3] https://issues.apache.org/jira/browse/MNG-5750 > > > > [4] https://issues.apache.org/jira/browse/MNG-5960 > > > > [5] https://github.com/apache/maven/pull/413 > > > > [6] https://github.com/apache/maven/pull/310#issuecomment-571317501 > > > > [7] https://github.com/apache/maven/pull/413#issuecomment-754661032 > > > > > > - > > To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org > > For additional commands, e-mail: dev-h...@maven.apache.org > > > > >
Re: MNG-6843 Parallel build fails due to missing JAR artifacts in compilePath
we are facing the same issue at work (300+ modules), classpath empty randomly empty Love to see some resolution, will help to test it Thanks -D On Fri, Jan 15, 2021 at 1:51 PM Falko Modler wrote: > Hi everyone, > > I'd like to raise awareness for the MavenProject concurrency problem > that is causing MNG-6843 "Parallel build fails due to missing JAR > artifacts in compilePath" [1] and probably others [2] [3] [4]. > > Almost a month ago, I created a ThreadLocal-based fix for this [5] > (after another, older cloning-based approach had raised some concerns by > Robert Scholte [6]). > > Michael Osipov was the only one so far having a look (thanks!) and he > suggested that more Maven team members should review this. > > So, before I take a stab at the not so trivial integration test that > Michael proposed [7], I'd like to get an approval for the general > aproach (or a declination in case someone has a better idea). > > Thanks for your attention and feedback! > > Cheers, > > Falko > > > [1] https://issues.apache.org/jira/browse/MNG-6843 > > [2] https://issues.apache.org/jira/browse/MNG-4996 > > [3] https://issues.apache.org/jira/browse/MNG-5750 > > [4] https://issues.apache.org/jira/browse/MNG-5960 > > [5] https://github.com/apache/maven/pull/413 > > [6] https://github.com/apache/maven/pull/310#issuecomment-571317501 > > [7] https://github.com/apache/maven/pull/413#issuecomment-754661032 > > > - > To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org > For additional commands, e-mail: dev-h...@maven.apache.org > >
