[jira] [Commented] (KARAF-5314) The performance of profile builder used by karaf maven plugin has reduced significantly in 4.1 compared to 4.0
[ https://issues.apache.org/jira/browse/KARAF-5314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16134437#comment-16134437 ] ASF subversion and git services commented on KARAF-5314: Commit 5a8133dbbfe7ede7366bcb943e2bd04937d8eff5 in karaf's branch refs/heads/KARAF-5314-2 from [~ch...@die-schneider.net] [ https://git-wip-us.apache.org/repos/asf?p=karaf.git;h=5a8133d ] [KARAF-5314] Only recurse into dependencies once per feature > The performance of profile builder used by karaf maven plugin has reduced > significantly in 4.1 compared to 4.0 > -- > > Key: KARAF-5314 > URL: https://issues.apache.org/jira/browse/KARAF-5314 > Project: Karaf > Issue Type: Bug >Reporter: Vinay Shankar > Fix For: 4.1.3 > > > The performance of profile builder used by karaf maven plugin has reduced > significantly in 4.1 compared to 4.0. > The java streams API is being used in 4.1 to filter our the required features > from the set of all features present in the repositories that are part of the > profile. This is done in the "addFeatures" method in the "Builder.java" class > in the "org.apache.karaf.profile.core" bundle. This change (from 4.0) has > drastically reduced the performance. For a profile with ~900 features in all > the repositories in the profile and ~300 required features from a highly > complex feature dependency tree, this function is taking around 13min to > complete. The same execution took around 3-5min in 4.0 (where simple for > loops were being used). -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (KARAF-5314) The performance of profile builder used by karaf maven plugin has reduced significantly in 4.1 compared to 4.0
[ https://issues.apache.org/jira/browse/KARAF-5314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16134430#comment-16134430 ] Christian Schneider commented on KARAF-5314: I just found a possible change that should speed up the code quite a lot. Currently we always recurse into every feature dependency. I think we we can make this a lot faster by only recursing into the dependencies of a feature once. > The performance of profile builder used by karaf maven plugin has reduced > significantly in 4.1 compared to 4.0 > -- > > Key: KARAF-5314 > URL: https://issues.apache.org/jira/browse/KARAF-5314 > Project: Karaf > Issue Type: Bug >Reporter: Vinay Shankar > Fix For: 4.1.3 > > > The performance of profile builder used by karaf maven plugin has reduced > significantly in 4.1 compared to 4.0. > The java streams API is being used in 4.1 to filter our the required features > from the set of all features present in the repositories that are part of the > profile. This is done in the "addFeatures" method in the "Builder.java" class > in the "org.apache.karaf.profile.core" bundle. This change (from 4.0) has > drastically reduced the performance. For a profile with ~900 features in all > the repositories in the profile and ~300 required features from a highly > complex feature dependency tree, this function is taking around 13min to > complete. The same execution took around 3-5min in 4.0 (where simple for > loops were being used). -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (KARAF-5314) The performance of profile builder used by karaf maven plugin has reduced significantly in 4.1 compared to 4.0
[ https://issues.apache.org/jira/browse/KARAF-5314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16134429#comment-16134429 ] Christian Schneider commented on KARAF-5314: In this part of the code we are only going through the features and their feature dependencies. So we are traversing at max 900 features and recurse through their dependencies. With the caching on feature name we only need to traverse the different versions of a feature. So I think this should be way below the level where parallel execution would help. After my extraction of the FeatureSelector class it should be quite easy to do unit tests on the performance. So I will setup some test cases. Then we can easily try the performance impact of changes. > The performance of profile builder used by karaf maven plugin has reduced > significantly in 4.1 compared to 4.0 > -- > > Key: KARAF-5314 > URL: https://issues.apache.org/jira/browse/KARAF-5314 > Project: Karaf > Issue Type: Bug >Reporter: Vinay Shankar > Fix For: 4.1.3 > > > The performance of profile builder used by karaf maven plugin has reduced > significantly in 4.1 compared to 4.0. > The java streams API is being used in 4.1 to filter our the required features > from the set of all features present in the repositories that are part of the > profile. This is done in the "addFeatures" method in the "Builder.java" class > in the "org.apache.karaf.profile.core" bundle. This change (from 4.0) has > drastically reduced the performance. For a profile with ~900 features in all > the repositories in the profile and ~300 required features from a highly > complex feature dependency tree, this function is taking around 13min to > complete. The same execution took around 3-5min in 4.0 (where simple for > loops were being used). -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (KARAF-5314) The performance of profile builder used by karaf maven plugin has reduced significantly in 4.1 compared to 4.0
[ https://issues.apache.org/jira/browse/KARAF-5314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16134406#comment-16134406 ] Achim Nierbeck commented on KARAF-5314: --- Do we have a "high load test" available? In that case I'd try if parallel streams might help to increase performance when doing the streaming. > The performance of profile builder used by karaf maven plugin has reduced > significantly in 4.1 compared to 4.0 > -- > > Key: KARAF-5314 > URL: https://issues.apache.org/jira/browse/KARAF-5314 > Project: Karaf > Issue Type: Bug >Reporter: Vinay Shankar > Fix For: 4.1.3 > > > The performance of profile builder used by karaf maven plugin has reduced > significantly in 4.1 compared to 4.0. > The java streams API is being used in 4.1 to filter our the required features > from the set of all features present in the repositories that are part of the > profile. This is done in the "addFeatures" method in the "Builder.java" class > in the "org.apache.karaf.profile.core" bundle. This change (from 4.0) has > drastically reduced the performance. For a profile with ~900 features in all > the repositories in the profile and ~300 required features from a highly > complex feature dependency tree, this function is taking around 13min to > complete. The same execution took around 3-5min in 4.0 (where simple for > loops were being used). -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (KARAF-5314) The performance of profile builder used by karaf maven plugin has reduced significantly in 4.1 compared to 4.0
[ https://issues.apache.org/jira/browse/KARAF-5314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16134368#comment-16134368 ] Christian Schneider commented on KARAF-5314: I now extracted the caching logic into its own class. Please see the branch KARAF-5314-2. Compared to your code I removed the caching of the version -> feature for now. We also need to support ranges like spring/[3.2,4) and It think this is not covered by the caching you implemented. I also reintroduced the stream for selecting the matching versions. I am not sure if this is slower than a for loop. Can you test this with a loop and with a stream. If the for loop is performing noticeably better then we will use a for loop. I think the main performance gain you saw came from the caching. In my branch this will be a little slower as I only cache by name. The question is how big the difference really is. We can create a caching using name, range if needed but I am not sure if it makes a big difference. I expect this to only matter when there is a high number of versions for a feature name. So if you think this should make a difference in your case we can do this. > The performance of profile builder used by karaf maven plugin has reduced > significantly in 4.1 compared to 4.0 > -- > > Key: KARAF-5314 > URL: https://issues.apache.org/jira/browse/KARAF-5314 > Project: Karaf > Issue Type: Bug >Reporter: Vinay Shankar > Fix For: 4.1.3 > > > The performance of profile builder used by karaf maven plugin has reduced > significantly in 4.1 compared to 4.0. > The java streams API is being used in 4.1 to filter our the required features > from the set of all features present in the repositories that are part of the > profile. This is done in the "addFeatures" method in the "Builder.java" class > in the "org.apache.karaf.profile.core" bundle. This change (from 4.0) has > drastically reduced the performance. For a profile with ~900 features in all > the repositories in the profile and ~300 required features from a highly > complex feature dependency tree, this function is taking around 13min to > complete. The same execution took around 3-5min in 4.0 (where simple for > loops were being used). -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (KARAF-5314) The performance of profile builder used by karaf maven plugin has reduced significantly in 4.1 compared to 4.0
[ https://issues.apache.org/jira/browse/KARAF-5314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16134366#comment-16134366 ] ASF subversion and git services commented on KARAF-5314: Commit 0ed141dafd5ffb321bc851886e88457b616c1c06 in karaf's branch refs/heads/KARAF-5314-2 from [~ch...@die-schneider.net] [ https://git-wip-us.apache.org/repos/asf?p=karaf.git;h=0ed141d ] [KARAF-5314] Extract addFeatures into class FeatureSelector > The performance of profile builder used by karaf maven plugin has reduced > significantly in 4.1 compared to 4.0 > -- > > Key: KARAF-5314 > URL: https://issues.apache.org/jira/browse/KARAF-5314 > Project: Karaf > Issue Type: Bug >Reporter: Vinay Shankar > Fix For: 4.1.3 > > > The performance of profile builder used by karaf maven plugin has reduced > significantly in 4.1 compared to 4.0. > The java streams API is being used in 4.1 to filter our the required features > from the set of all features present in the repositories that are part of the > profile. This is done in the "addFeatures" method in the "Builder.java" class > in the "org.apache.karaf.profile.core" bundle. This change (from 4.0) has > drastically reduced the performance. For a profile with ~900 features in all > the repositories in the profile and ~300 required features from a highly > complex feature dependency tree, this function is taking around 13min to > complete. The same execution took around 3-5min in 4.0 (where simple for > loops were being used). -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (KARAF-5314) The performance of profile builder used by karaf maven plugin has reduced significantly in 4.1 compared to 4.0
[ https://issues.apache.org/jira/browse/KARAF-5314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16134365#comment-16134365 ] ASF subversion and git services commented on KARAF-5314: Commit 3cbd810ea37e837950f1f8427f793c878423ba7a in karaf's branch refs/heads/KARAF-5314-2 from Vinay Shankar (e24113) [ https://git-wip-us.apache.org/repos/asf?p=karaf.git;h=3cbd810 ] KARAF-5314 Added a features cache that increases the performance of the filtering of the required features from the available features. The performance of the filtering logic decresed when the java streams API was used in 4.1 compared to for loops in 4.0. I reverted the filtering logic to use for loops and also introduced a features cache. This increased the performance by a huge margin, almost 20X faster when there are ~900 available features and ~300 required features from a highly complex and huge feature dependency tree. > The performance of profile builder used by karaf maven plugin has reduced > significantly in 4.1 compared to 4.0 > -- > > Key: KARAF-5314 > URL: https://issues.apache.org/jira/browse/KARAF-5314 > Project: Karaf > Issue Type: Bug >Reporter: Vinay Shankar > Fix For: 4.1.3 > > > The performance of profile builder used by karaf maven plugin has reduced > significantly in 4.1 compared to 4.0. > The java streams API is being used in 4.1 to filter our the required features > from the set of all features present in the repositories that are part of the > profile. This is done in the "addFeatures" method in the "Builder.java" class > in the "org.apache.karaf.profile.core" bundle. This change (from 4.0) has > drastically reduced the performance. For a profile with ~900 features in all > the repositories in the profile and ~300 required features from a highly > complex feature dependency tree, this function is taking around 13min to > complete. The same execution took around 3-5min in 4.0 (where simple for > loops were being used). -- This message was sent by Atlassian JIRA (v6.4.14#64029)
