Re: exclusive execution scheduling
Hi. It should already be possible for you to build a dependency graph or execution order the way you need for multi-threaded builds. Example: A and B are required executions for C and D can then be executed after all, A, B and C. So just package every execution into a separate Maven submodule and declare dependecies from D to C and from C to A and B. In this case A and B might be executed at the same time, but C waits for A and B and D waits for C. pe 5. helmik. 2021 klo 19.18 Delany (delany.middle...@gmail.com) kirjoitti: > I completely get what you're saying, in fact it hasn't been 48hrs since > I've had to reboot an NTFS machine to unlock a file. > But here the problem as it manifests is with msbuild itself (if you're > interested https://github.com/dotnet/sdk/issues/9585) > > That might be where we leave it. But I think something else can be done, > and without much effort. The exec-maven-plugin claims to be thread-safe - > whatever that means (internally thread-safe?) But if it's going to call > some ad-hoc binary, it's an empty claim. > > Making a simple configuration available to the model could simply and > easily prevent this situation, i.e. maintain a list of active executions in > the session, and delay the start of an execution if it was configured to > run exclusively. Does anyone else think it's worth a ticket? > > Thanks, > Delany > > On Fri, 5 Feb 2021 at 12:56, Tommy Svensson wrote: > > > I would say this is a windows problem rather than a maven problem. I have > > had > > similar problems trying to build simpler things where only solution is to > > reboot > > windows to unlock NTFS (NoT a File System). > > > > I'm very seriously considering getting VMWare and installing a Linux Mint > > 20 and > > work in Linux instead. It seems like the only serious way to develop Java > > on a > > Windows 10 machine. Working on Windows directly costs to much time in > > constant reboots to resolve locks. > > > > I'm having a very good 4K display, which the free virtualbox does not > > support > > full size, thereby VmWare for me. If you don't have my display > > requirement, test > > with VirtualBox. > > > > Do note however that a virtual machine takes more memory! Personally I > > have > > 32 GB memory in my machine, but if you are at 16 GB or lower it can be > > problematic. > > > > And no, this wasn't exactly what you were asking. But since I'm having > NTFS > > locks commonly even without multi threaded builds I don't believe the > > problem > > lies in maven. > > > > I have to admit that Windows 10 is far better than previous windows > > versions. It however seem to still be using same old, NTFS. > > > > Cheers, > > Tommy > > > > > > Från: Delany > > Svara: Maven Users List > > > Datum: 4 februari 2021 at 19:01:18 > > Till: Maven Users List > > Ämne: exclusive execution scheduling > > > > We use the exec-maven-plugin (a thread-safe plugin) to call msbuild for > > dotnet builds within Maven. > > We have 8 dotnet projects with 3 of them depending on shared libraries in > > another directory in the repo (a total of 500 projects). > > > > With multi-threading on (4 threads), these dotnet projects always fail > > because the msbuild instances are all trying to lock the files in the > > shared libraries and can't open them in the other instances. This is on > > Windows 10 with NTFS. With only one thread, they always build > > successfully. > > > > Note that I don't use the multi-threading capabilities of msbuild because > > that would be silly. > > > > To get this reactor working I can either > > > > - package up the shared libraries as part of the build, and unpack them > > into every dotnet project that needs them (using > > https://maven.apache.org/plugins/maven-remote-resources-plugin/ ) > > > > Or ensure that no 2 instances of msbuild ever run at the same time, by > > either > > > > - chaining the dotnet projects together as one sequence of dependent > > projects (ridiculous) > > - building the dotnet projects separately with their own maven invocation > > - turning off multi-threading for the whole reactor > > > > Are there other options? > > > > Would it be feasible to consider adding a further thread-safe check in > the > > plugin architecture whereby a configuration could be set to enforce that > > no > > two executions with the same id will ever run simultaneously? > > > > Thanks, > > Delany > > > > >
Re: exclusive execution scheduling
I completely get what you're saying, in fact it hasn't been 48hrs since I've had to reboot an NTFS machine to unlock a file. But here the problem as it manifests is with msbuild itself (if you're interested https://github.com/dotnet/sdk/issues/9585) That might be where we leave it. But I think something else can be done, and without much effort. The exec-maven-plugin claims to be thread-safe - whatever that means (internally thread-safe?) But if it's going to call some ad-hoc binary, it's an empty claim. Making a simple configuration available to the model could simply and easily prevent this situation, i.e. maintain a list of active executions in the session, and delay the start of an execution if it was configured to run exclusively. Does anyone else think it's worth a ticket? Thanks, Delany On Fri, 5 Feb 2021 at 12:56, Tommy Svensson wrote: > I would say this is a windows problem rather than a maven problem. I have > had > similar problems trying to build simpler things where only solution is to > reboot > windows to unlock NTFS (NoT a File System). > > I'm very seriously considering getting VMWare and installing a Linux Mint > 20 and > work in Linux instead. It seems like the only serious way to develop Java > on a > Windows 10 machine. Working on Windows directly costs to much time in > constant reboots to resolve locks. > > I'm having a very good 4K display, which the free virtualbox does not > support > full size, thereby VmWare for me. If you don't have my display > requirement, test > with VirtualBox. > > Do note however that a virtual machine takes more memory! Personally I > have > 32 GB memory in my machine, but if you are at 16 GB or lower it can be > problematic. > > And no, this wasn't exactly what you were asking. But since I'm having NTFS > locks commonly even without multi threaded builds I don't believe the > problem > lies in maven. > > I have to admit that Windows 10 is far better than previous windows > versions. It however seem to still be using same old, NTFS. > > Cheers, > Tommy > > > Från: Delany > Svara: Maven Users List > Datum: 4 februari 2021 at 19:01:18 > Till: Maven Users List > Ämne: exclusive execution scheduling > > We use the exec-maven-plugin (a thread-safe plugin) to call msbuild for > dotnet builds within Maven. > We have 8 dotnet projects with 3 of them depending on shared libraries in > another directory in the repo (a total of 500 projects). > > With multi-threading on (4 threads), these dotnet projects always fail > because the msbuild instances are all trying to lock the files in the > shared libraries and can't open them in the other instances. This is on > Windows 10 with NTFS. With only one thread, they always build > successfully. > > Note that I don't use the multi-threading capabilities of msbuild because > that would be silly. > > To get this reactor working I can either > > - package up the shared libraries as part of the build, and unpack them > into every dotnet project that needs them (using > https://maven.apache.org/plugins/maven-remote-resources-plugin/ ) > > Or ensure that no 2 instances of msbuild ever run at the same time, by > either > > - chaining the dotnet projects together as one sequence of dependent > projects (ridiculous) > - building the dotnet projects separately with their own maven invocation > - turning off multi-threading for the whole reactor > > Are there other options? > > Would it be feasible to consider adding a further thread-safe check in the > plugin architecture whereby a configuration could be set to enforce that > no > two executions with the same id will ever run simultaneously? > > Thanks, > Delany > >
Re: exclusive execution scheduling
I would say this is a windows problem rather than a maven problem. I have had similar problems trying to build simpler things where only solution is to reboot windows to unlock NTFS (NoT a File System). I'm very seriously considering getting VMWare and installing a Linux Mint 20 and work in Linux instead. It seems like the only serious way to develop Java on a Windows 10 machine. Working on Windows directly costs to much time in constant reboots to resolve locks. I'm having a very good 4K display, which the free virtualbox does not support full size, thereby VmWare for me. If you don't have my display requirement, test with VirtualBox. Do note however that a virtual machine takes more memory! Personally I have 32 GB memory in my machine, but if you are at 16 GB or lower it can be problematic. And no, this wasn't exactly what you were asking. But since I'm having NTFS locks commonly even without multi threaded builds I don't believe the problem lies in maven. I have to admit that Windows 10 is far better than previous windows versions. It however seem to still be using same old, NTFS. Cheers, Tommy Från: Delany Svara: Maven Users List Datum: 4 februari 2021 at 19:01:18 Till: Maven Users List Ämne: exclusive execution scheduling We use the exec-maven-plugin (a thread-safe plugin) to call msbuild for dotnet builds within Maven. We have 8 dotnet projects with 3 of them depending on shared libraries in another directory in the repo (a total of 500 projects). With multi-threading on (4 threads), these dotnet projects always fail because the msbuild instances are all trying to lock the files in the shared libraries and can't open them in the other instances. This is on Windows 10 with NTFS. With only one thread, they always build successfully. Note that I don't use the multi-threading capabilities of msbuild because that would be silly. To get this reactor working I can either - package up the shared libraries as part of the build, and unpack them into every dotnet project that needs them (using https://maven.apache.org/plugins/maven-remote-resources-plugin/ ) Or ensure that no 2 instances of msbuild ever run at the same time, by either - chaining the dotnet projects together as one sequence of dependent projects (ridiculous) - building the dotnet projects separately with their own maven invocation - turning off multi-threading for the whole reactor Are there other options? Would it be feasible to consider adding a further thread-safe check in the plugin architecture whereby a configuration could be set to enforce that no two executions with the same id will ever run simultaneously? Thanks, Delany
