[jira] [Commented] (KARAF-6454) FeaturesServiceImpl downloadThreads does not improve throughput
[ https://issues.apache.org/jira/browse/KARAF-6454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16952496#comment-16952496 ] Jean-Baptiste Onofré commented on KARAF-6454: - Only `prerequisite` should be "blocker", the other features installation should be performed in parallel. Or maybe, we don't talk about the same thing ? > FeaturesServiceImpl downloadThreads does not improve throughput > --- > > Key: KARAF-6454 > URL: https://issues.apache.org/jira/browse/KARAF-6454 > Project: Karaf > Issue Type: Bug >Reporter: Fabian Lange >Assignee: Jean-Baptiste Onofré >Priority: Major > Fix For: 4.3.0, 4.2.8 > > Attachments: Screenshot 2019-10-15 at 16.53.00.png > > > I am looking into a slow Karaf startup and I stumbled onto this problem > FeaturesServiceImpl does allow for parallelism thanks to the downloadThreads > configuration > It does respect this value and createDownloadManager() will use it to size > the threadpool. > However adding a large amount of features at the same time results in one by > one downloads across these threads. > I am currently looking for any pointers where the features install could be > sequenced. > Attached is a JProfiler screenshot illustrating this problem. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (KARAF-6454) FeaturesServiceImpl downloadThreads does not improve throughput
[ https://issues.apache.org/jira/browse/KARAF-6454?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jean-Baptiste Onofré reassigned KARAF-6454: --- Assignee: Jean-Baptiste Onofré > FeaturesServiceImpl downloadThreads does not improve throughput > --- > > Key: KARAF-6454 > URL: https://issues.apache.org/jira/browse/KARAF-6454 > Project: Karaf > Issue Type: Bug >Reporter: Fabian Lange >Assignee: Jean-Baptiste Onofré >Priority: Major > Attachments: Screenshot 2019-10-15 at 16.53.00.png > > > I am looking into a slow Karaf startup and I stumbled onto this problem > FeaturesServiceImpl does allow for parallelism thanks to the downloadThreads > configuration > It does respect this value and createDownloadManager() will use it to size > the threadpool. > However adding a large amount of features at the same time results in one by > one downloads across these threads. > I am currently looking for any pointers where the features install could be > sequenced. > Attached is a JProfiler screenshot illustrating this problem. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KARAF-6454) FeaturesServiceImpl downloadThreads does not improve throughput
[ https://issues.apache.org/jira/browse/KARAF-6454?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jean-Baptiste Onofré updated KARAF-6454: Fix Version/s: 4.2.8 4.3.0 > FeaturesServiceImpl downloadThreads does not improve throughput > --- > > Key: KARAF-6454 > URL: https://issues.apache.org/jira/browse/KARAF-6454 > Project: Karaf > Issue Type: Bug >Reporter: Fabian Lange >Assignee: Jean-Baptiste Onofré >Priority: Major > Fix For: 4.3.0, 4.2.8 > > Attachments: Screenshot 2019-10-15 at 16.53.00.png > > > I am looking into a slow Karaf startup and I stumbled onto this problem > FeaturesServiceImpl does allow for parallelism thanks to the downloadThreads > configuration > It does respect this value and createDownloadManager() will use it to size > the threadpool. > However adding a large amount of features at the same time results in one by > one downloads across these threads. > I am currently looking for any pointers where the features install could be > sequenced. > Attached is a JProfiler screenshot illustrating this problem. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KARAF-6451) Upgrade to Jackson 2.10.0
[ https://issues.apache.org/jira/browse/KARAF-6451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16952247#comment-16952247 ] ASF GitHub Bot commented on KARAF-6451: --- jbonofre commented on pull request #956: [KARAF-6451] Upgrade to Jackson 2.10.0 URL: https://github.com/apache/karaf/pull/956 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Upgrade to Jackson 2.10.0 > - > > Key: KARAF-6451 > URL: https://issues.apache.org/jira/browse/KARAF-6451 > Project: Karaf > Issue Type: Bug > Components: cave, decanter, karaf >Reporter: Jean-Baptiste Onofré >Assignee: Jean-Baptiste Onofré >Priority: Major > Fix For: 4.3.0, cave-4.2.0, decanter-2.3.0, 4.2.8 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KARAF-6454) FeaturesServiceImpl downloadThreads does not improve throughput
[
https://issues.apache.org/jira/browse/KARAF-6454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16952030#comment-16952030
]
Fabian Lange commented on KARAF-6454:
-
Yes that seems to be the case, it waits for a feature to install before moving
to the next one.
{{* Thread group "main":
Thread "features-3-thread-1":
at java.lang.Object.wait(long)
at java.lang.Object.wait() (line: 502)
at
org.apache.karaf.features.internal.download.impl.MavenDownloadManager$MavenDownloader.await()
(line: 101)
at
org.apache.karaf.features.internal.region.Subsystem.downloadBundles(org.apache.karaf.features.internal.download.DownloadManager,
java.lang.String,
org.apache.karaf.features.FeaturesService$ServiceRequirementsBehavior,
org.apache.karaf.features.internal.region.RepositoryManager,
org.apache.karaf.features.internal.region.SubsystemResolverCallback) (line: 537)
at
org.apache.karaf.features.internal.region.Subsystem.downloadBundles(org.apache.karaf.features.internal.download.DownloadManager,
java.lang.String,
org.apache.karaf.features.FeaturesService$ServiceRequirementsBehavior,
org.apache.karaf.features.internal.region.RepositoryManager,
org.apache.karaf.features.internal.region.SubsystemResolverCallback) (line: 452)
at
org.apache.karaf.features.internal.region.SubsystemResolver.resolve(java.lang.String,
org.apache.karaf.features.FeaturesService$ServiceRequirementsBehavior,
org.osgi.service.repository.Repository, java.lang.String) (line: 224)
at
org.apache.karaf.features.internal.service.Deployer.deploy(org.apache.karaf.features.internal.service.Deployer$DeploymentState,
org.apache.karaf.features.internal.service.Deployer$DeploymentRequest) (line:
393)
at
org.apache.karaf.features.internal.service.FeaturesServiceImpl.doProvision(java.util.Map,
java.util.Map, org.apache.karaf.features.internal.service.State,
java.util.Map, java.util.EnumSet, java.lang.String) (line: 1062)
at
org.apache.karaf.features.internal.service.FeaturesServiceImpl.lambda$doProvisionInThread$13(java.util.Map,
java.util.Map, org.apache.karaf.features.internal.service.State,
java.util.Map, java.util.EnumSet, java.lang.String) (line: 998)
at
org.apache.karaf.features.internal.service.FeaturesServiceImpl$$Lambda$78.672983579.call()
at java.util.concurrent.FutureTask.run() (line: 266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker)
(line: 1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run() (line: 624)
at java.lang.Thread.run() (line: 748)}}
Not sure if this is intentional, as the graph resolution was done before so
this should be possible to parallelise
> FeaturesServiceImpl downloadThreads does not improve throughput
> ---
>
> Key: KARAF-6454
> URL: https://issues.apache.org/jira/browse/KARAF-6454
> Project: Karaf
> Issue Type: Bug
>Reporter: Fabian Lange
>Priority: Major
> Attachments: Screenshot 2019-10-15 at 16.53.00.png
>
>
> I am looking into a slow Karaf startup and I stumbled onto this problem
> FeaturesServiceImpl does allow for parallelism thanks to the downloadThreads
> configuration
> It does respect this value and createDownloadManager() will use it to size
> the threadpool.
> However adding a large amount of features at the same time results in one by
> one downloads across these threads.
> I am currently looking for any pointers where the features install could be
> sequenced.
> Attached is a JProfiler screenshot illustrating this problem.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (KARAF-6454) FeaturesServiceImpl downloadThreads does not improve throughput
[ https://issues.apache.org/jira/browse/KARAF-6454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16952015#comment-16952015 ] Fabian Lange commented on KARAF-6454: - Looks to me like Subsystem#deploy() is invoked for each feature, and it does do a downloader.await() Since this is a single deployment request, this is running on the threadpool, but it is not forking out the work for the subsystems. > FeaturesServiceImpl downloadThreads does not improve throughput > --- > > Key: KARAF-6454 > URL: https://issues.apache.org/jira/browse/KARAF-6454 > Project: Karaf > Issue Type: Bug >Reporter: Fabian Lange >Priority: Major > Attachments: Screenshot 2019-10-15 at 16.53.00.png > > > I am looking into a slow Karaf startup and I stumbled onto this problem > FeaturesServiceImpl does allow for parallelism thanks to the downloadThreads > configuration > It does respect this value and createDownloadManager() will use it to size > the threadpool. > However adding a large amount of features at the same time results in one by > one downloads across these threads. > I am currently looking for any pointers where the features install could be > sequenced. > Attached is a JProfiler screenshot illustrating this problem. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (KARAF-6454) FeaturesServiceImpl downloadThreads does not improve throughput
Fabian Lange created KARAF-6454: --- Summary: FeaturesServiceImpl downloadThreads does not improve throughput Key: KARAF-6454 URL: https://issues.apache.org/jira/browse/KARAF-6454 Project: Karaf Issue Type: Bug Reporter: Fabian Lange Attachments: Screenshot 2019-10-15 at 16.53.00.png I am looking into a slow Karaf startup and I stumbled onto this problem FeaturesServiceImpl does allow for parallelism thanks to the downloadThreads configuration It does respect this value and createDownloadManager() will use it to size the threadpool. However adding a large amount of features at the same time results in one by one downloads across these threads. I am currently looking for any pointers where the features install could be sequenced. Attached is a JProfiler screenshot illustrating this problem. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KARAF-6203) ensure karaf can build and run with JDK12
[ https://issues.apache.org/jira/browse/KARAF-6203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16951939#comment-16951939 ] Jean-Baptiste Onofré commented on KARAF-6203: - [~ffang] cool, I'm doing a local run/try and then I will update on Jenkins. > ensure karaf can build and run with JDK12 > - > > Key: KARAF-6203 > URL: https://issues.apache.org/jira/browse/KARAF-6203 > Project: Karaf > Issue Type: Task > Components: karaf >Reporter: Freeman Yue Fang >Assignee: Freeman Yue Fang >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KARAF-6203) ensure karaf can build and run with JDK12
[ https://issues.apache.org/jira/browse/KARAF-6203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16951937#comment-16951937 ] Freeman Yue Fang commented on KARAF-6203: - Hi [~jbonofre]. Feel free to pick it up, please note I pushed jdk12 and jdk13 branches with my commits separately. If you upgrade to latest Aries, I don't expect too many errors. Anyway, thanks for double check this. Freeman > ensure karaf can build and run with JDK12 > - > > Key: KARAF-6203 > URL: https://issues.apache.org/jira/browse/KARAF-6203 > Project: Karaf > Issue Type: Task > Components: karaf >Reporter: Freeman Yue Fang >Assignee: Freeman Yue Fang >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KARAF-6453) Registering a FeaturesListener re-loads all repositories
[ https://issues.apache.org/jira/browse/KARAF-6453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16951918#comment-16951918 ] Fabian Lange commented on KARAF-6453: - Well I have not yet figured out if it is intentional ;) It doesn't look like however. > Registering a FeaturesListener re-loads all repositories > > > Key: KARAF-6453 > URL: https://issues.apache.org/jira/browse/KARAF-6453 > Project: Karaf > Issue Type: Bug > Components: karaf >Affects Versions: 4.2.0 >Reporter: Fabian Lange >Assignee: Jean-Baptiste Onofré >Priority: Major > Labels: features > Fix For: 4.3.0, 4.2.8 > > > This is a regression from 4.2, just discovered. > It was caused in 2017 by this commit: > https://github.com/apache/karaf/commit/a75fabaf071b8274c7f266affa7ce7cdd87a2a22 > https://github.com/apache/karaf/commit/a75fabaf071b8274c7f266affa7ce7cdd87a2a22#diff-29d341aa18575466b113be9168a6e7adR264 > Here the new RepositoryCacheImpl is used to create the repository. But > despite its name, it does not use something from the cache, but it creates a > new Repository. > This now will always "load" the repository, potentially refreshing it over > http. > https://github.com/apache/karaf/commit/a75fabaf071b8274c7f266affa7ce7cdd87a2a22#diff-a3bb3d36b86e3a4c291627077972097aR47 > previously it only returned a Repository that was unloaded. > Registering many FeaturesListeners wastes a lot of CPU. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KARAF-6453) Registering a FeaturesListener re-loads all repositories
[ https://issues.apache.org/jira/browse/KARAF-6453?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jean-Baptiste Onofré updated KARAF-6453: Fix Version/s: 4.2.8 4.3.0 > Registering a FeaturesListener re-loads all repositories > > > Key: KARAF-6453 > URL: https://issues.apache.org/jira/browse/KARAF-6453 > Project: Karaf > Issue Type: Bug > Components: karaf >Affects Versions: 4.2.0 >Reporter: Fabian Lange >Assignee: Jean-Baptiste Onofré >Priority: Major > Fix For: 4.3.0, 4.2.8 > > > This is a regression from 4.2, just discovered. > It was caused in 2017 by this commit: > https://github.com/apache/karaf/commit/a75fabaf071b8274c7f266affa7ce7cdd87a2a22 > https://github.com/apache/karaf/commit/a75fabaf071b8274c7f266affa7ce7cdd87a2a22#diff-29d341aa18575466b113be9168a6e7adR264 > Here the new RepositoryCacheImpl is used to create the repository. But > despite its name, it does not use something from the cache, but it creates a > new Repository. > This now will always "load" the repository, potentially refreshing it over > http. > https://github.com/apache/karaf/commit/a75fabaf071b8274c7f266affa7ce7cdd87a2a22#diff-a3bb3d36b86e3a4c291627077972097aR47 > previously it only returned a Repository that was unloaded. > Registering many FeaturesListeners wastes a lot of CPU. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KARAF-6453) Registering a FeaturesListener re-loads all repositories
[ https://issues.apache.org/jira/browse/KARAF-6453?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jean-Baptiste Onofré updated KARAF-6453: Labels: features (was: ) > Registering a FeaturesListener re-loads all repositories > > > Key: KARAF-6453 > URL: https://issues.apache.org/jira/browse/KARAF-6453 > Project: Karaf > Issue Type: Bug > Components: karaf >Affects Versions: 4.2.0 >Reporter: Fabian Lange >Assignee: Jean-Baptiste Onofré >Priority: Major > Labels: features > Fix For: 4.3.0, 4.2.8 > > > This is a regression from 4.2, just discovered. > It was caused in 2017 by this commit: > https://github.com/apache/karaf/commit/a75fabaf071b8274c7f266affa7ce7cdd87a2a22 > https://github.com/apache/karaf/commit/a75fabaf071b8274c7f266affa7ce7cdd87a2a22#diff-29d341aa18575466b113be9168a6e7adR264 > Here the new RepositoryCacheImpl is used to create the repository. But > despite its name, it does not use something from the cache, but it creates a > new Repository. > This now will always "load" the repository, potentially refreshing it over > http. > https://github.com/apache/karaf/commit/a75fabaf071b8274c7f266affa7ce7cdd87a2a22#diff-a3bb3d36b86e3a4c291627077972097aR47 > previously it only returned a Repository that was unloaded. > Registering many FeaturesListeners wastes a lot of CPU. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KARAF-6453) Registering a FeaturesListener re-loads all repositories
[ https://issues.apache.org/jira/browse/KARAF-6453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16951914#comment-16951914 ] Jean-Baptiste Onofré commented on KARAF-6453: - Good catch ! > Registering a FeaturesListener re-loads all repositories > > > Key: KARAF-6453 > URL: https://issues.apache.org/jira/browse/KARAF-6453 > Project: Karaf > Issue Type: Bug > Components: karaf >Affects Versions: 4.2.0 >Reporter: Fabian Lange >Assignee: Jean-Baptiste Onofré >Priority: Major > Labels: features > Fix For: 4.3.0, 4.2.8 > > > This is a regression from 4.2, just discovered. > It was caused in 2017 by this commit: > https://github.com/apache/karaf/commit/a75fabaf071b8274c7f266affa7ce7cdd87a2a22 > https://github.com/apache/karaf/commit/a75fabaf071b8274c7f266affa7ce7cdd87a2a22#diff-29d341aa18575466b113be9168a6e7adR264 > Here the new RepositoryCacheImpl is used to create the repository. But > despite its name, it does not use something from the cache, but it creates a > new Repository. > This now will always "load" the repository, potentially refreshing it over > http. > https://github.com/apache/karaf/commit/a75fabaf071b8274c7f266affa7ce7cdd87a2a22#diff-a3bb3d36b86e3a4c291627077972097aR47 > previously it only returned a Repository that was unloaded. > Registering many FeaturesListeners wastes a lot of CPU. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (KARAF-6453) Registering a FeaturesListener re-loads all repositories
[ https://issues.apache.org/jira/browse/KARAF-6453?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jean-Baptiste Onofré reassigned KARAF-6453: --- Assignee: Jean-Baptiste Onofré > Registering a FeaturesListener re-loads all repositories > > > Key: KARAF-6453 > URL: https://issues.apache.org/jira/browse/KARAF-6453 > Project: Karaf > Issue Type: Bug > Components: karaf >Affects Versions: 4.2.0 >Reporter: Fabian Lange >Assignee: Jean-Baptiste Onofré >Priority: Major > > This is a regression from 4.2, just discovered. > It was caused in 2017 by this commit: > https://github.com/apache/karaf/commit/a75fabaf071b8274c7f266affa7ce7cdd87a2a22 > https://github.com/apache/karaf/commit/a75fabaf071b8274c7f266affa7ce7cdd87a2a22#diff-29d341aa18575466b113be9168a6e7adR264 > Here the new RepositoryCacheImpl is used to create the repository. But > despite its name, it does not use something from the cache, but it creates a > new Repository. > This now will always "load" the repository, potentially refreshing it over > http. > https://github.com/apache/karaf/commit/a75fabaf071b8274c7f266affa7ce7cdd87a2a22#diff-a3bb3d36b86e3a4c291627077972097aR47 > previously it only returned a Repository that was unloaded. > Registering many FeaturesListeners wastes a lot of CPU. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (KARAF-6453) Registering a FeaturesListener re-loads all repositories
Fabian Lange created KARAF-6453: --- Summary: Registering a FeaturesListener re-loads all repositories Key: KARAF-6453 URL: https://issues.apache.org/jira/browse/KARAF-6453 Project: Karaf Issue Type: Bug Components: karaf Affects Versions: 4.2.0 Reporter: Fabian Lange This is a regression from 4.2, just discovered. It was caused in 2017 by this commit: https://github.com/apache/karaf/commit/a75fabaf071b8274c7f266affa7ce7cdd87a2a22 https://github.com/apache/karaf/commit/a75fabaf071b8274c7f266affa7ce7cdd87a2a22#diff-29d341aa18575466b113be9168a6e7adR264 Here the new RepositoryCacheImpl is used to create the repository. But despite its name, it does not use something from the cache, but it creates a new Repository. This now will always "load" the repository, potentially refreshing it over http. https://github.com/apache/karaf/commit/a75fabaf071b8274c7f266affa7ce7cdd87a2a22#diff-a3bb3d36b86e3a4c291627077972097aR47 previously it only returned a Repository that was unloaded. Registering many FeaturesListeners wastes a lot of CPU. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KARAF-6203) ensure karaf can build and run with JDK12
[ https://issues.apache.org/jira/browse/KARAF-6203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16951881#comment-16951881 ] Jean-Baptiste Onofré commented on KARAF-6203: - [~ffang] did you test JDK12/JDK13 recently ? I will do a pass. Mind if I assign the Jira to me ? > ensure karaf can build and run with JDK12 > - > > Key: KARAF-6203 > URL: https://issues.apache.org/jira/browse/KARAF-6203 > Project: Karaf > Issue Type: Task > Components: karaf >Reporter: Freeman Yue Fang >Assignee: Freeman Yue Fang >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
