[jira] [Commented] (PHOENIX-4718) Decrease overhead of tracking aggregate heap size
[ https://issues.apache.org/jira/browse/PHOENIX-4718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16460318#comment-16460318 ] Hudson commented on PHOENIX-4718: - FAILURE: Integrated in Jenkins build PreCommit-PHOENIX-Build #1849 (See [https://builds.apache.org/job/PreCommit-PHOENIX-Build/1849/]) PHOENIX-4718 Decrease overhead of tracking aggregate heap size (jtaylor: rev 4096fc4893adeca871e97d5d5d60d2f332b572c4) * (edit) phoenix-core/src/main/java/org/apache/phoenix/query/QueryServices.java * (edit) phoenix-core/src/main/java/org/apache/phoenix/expression/aggregator/DistinctValueWithCountServerAggregator.java * (edit) phoenix-core/src/main/java/org/apache/phoenix/expression/aggregator/ServerAggregators.java * (edit) phoenix-core/src/it/java/org/apache/phoenix/end2end/SpillableGroupByIT.java * (edit) phoenix-core/src/main/java/org/apache/phoenix/coprocessor/UngroupedAggregateRegionObserver.java * (add) phoenix-core/src/main/java/org/apache/phoenix/expression/aggregator/SizeTrackingServerAggregators.java * (edit) phoenix-core/src/main/java/org/apache/phoenix/expression/aggregator/Aggregator.java * (edit) phoenix-core/src/main/java/org/apache/phoenix/coprocessor/GroupedAggregateRegionObserver.java * (edit) phoenix-core/src/test/java/org/apache/phoenix/compile/QueryCompilerTest.java * (edit) phoenix-core/src/main/java/org/apache/phoenix/expression/aggregator/BaseAggregator.java * (edit) phoenix-core/src/test/java/org/apache/phoenix/query/QueryServicesTestImpl.java * (edit) phoenix-core/src/main/java/org/apache/phoenix/execute/ClientAggregatePlan.java * (edit) phoenix-core/src/main/java/org/apache/phoenix/query/QueryServicesOptions.java * (edit) phoenix-core/src/main/java/org/apache/phoenix/expression/aggregator/Aggregators.java * (edit) phoenix-core/src/main/java/org/apache/phoenix/expression/aggregator/ClientAggregators.java * (add) phoenix-core/src/main/java/org/apache/phoenix/expression/aggregator/NonSizeTrackingServerAggregators.java > Decrease overhead of tracking aggregate heap size > - > > Key: PHOENIX-4718 > URL: https://issues.apache.org/jira/browse/PHOENIX-4718 > Project: Phoenix > Issue Type: Bug >Reporter: James Taylor >Assignee: James Taylor >Priority: Major > Fix For: 4.14.0, 5.0.0 > > Attachments: PHOENIX-4718-4.x-HBase-0.98.patch, PHOENIX-4718.patch, > PHOENIX-4718_v2.patch, PHOENIX-4718_v3.patch > > > Since PHOENIX-4148, we track the heap size while aggregation is occurring. > This decreased performance of aggregation by ~20%. We really only need to > track this for the DistinctValueWithCountServerAggregator (used by DISTINCT > COUNT, DISTINCT, PERCENTILE functions, and STDDEV functions). By > conditionally tracking, we should be able to bring perf back to what it was > before. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (PHOENIX-4718) Decrease overhead of tracking aggregate heap size
[ https://issues.apache.org/jira/browse/PHOENIX-4718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16460271#comment-16460271 ] Hudson commented on PHOENIX-4718: - SUCCESS: Integrated in Jenkins build Phoenix-4.x-HBase-0.98 #1876 (See [https://builds.apache.org/job/Phoenix-4.x-HBase-0.98/1876/]) PHOENIX-4718 Decrease overhead of tracking aggregate heap size (jtaylor: rev dcbaeac5613aa1c91adf070e51e9a3e96ccf7c0a) * (edit) phoenix-core/src/main/java/org/apache/phoenix/expression/aggregator/ServerAggregators.java * (edit) phoenix-core/src/it/java/org/apache/phoenix/end2end/SpillableGroupByIT.java * (edit) phoenix-core/src/main/java/org/apache/phoenix/coprocessor/UngroupedAggregateRegionObserver.java * (edit) phoenix-core/src/main/java/org/apache/phoenix/query/QueryServices.java * (edit) phoenix-core/src/main/java/org/apache/phoenix/expression/aggregator/ClientAggregators.java * (edit) phoenix-core/src/main/java/org/apache/phoenix/expression/aggregator/Aggregator.java * (edit) phoenix-core/src/main/java/org/apache/phoenix/execute/ClientAggregatePlan.java * (edit) phoenix-core/src/main/java/org/apache/phoenix/coprocessor/GroupedAggregateRegionObserver.java * (edit) phoenix-core/src/main/java/org/apache/phoenix/expression/aggregator/BaseAggregator.java * (edit) phoenix-core/src/test/java/org/apache/phoenix/query/QueryServicesTestImpl.java * (edit) phoenix-core/src/test/java/org/apache/phoenix/compile/QueryCompilerTest.java * (edit) phoenix-core/src/main/java/org/apache/phoenix/expression/aggregator/DistinctValueWithCountServerAggregator.java * (edit) phoenix-core/src/main/java/org/apache/phoenix/query/QueryServicesOptions.java * (edit) phoenix-core/src/main/java/org/apache/phoenix/expression/aggregator/Aggregators.java * (add) phoenix-core/src/main/java/org/apache/phoenix/expression/aggregator/NonSizeTrackingServerAggregators.java * (add) phoenix-core/src/main/java/org/apache/phoenix/expression/aggregator/SizeTrackingServerAggregators.java > Decrease overhead of tracking aggregate heap size > - > > Key: PHOENIX-4718 > URL: https://issues.apache.org/jira/browse/PHOENIX-4718 > Project: Phoenix > Issue Type: Bug >Reporter: James Taylor >Assignee: James Taylor >Priority: Major > Fix For: 4.14.0, 5.0.0 > > Attachments: PHOENIX-4718-4.x-HBase-0.98.patch, PHOENIX-4718.patch, > PHOENIX-4718_v2.patch, PHOENIX-4718_v3.patch > > > Since PHOENIX-4148, we track the heap size while aggregation is occurring. > This decreased performance of aggregation by ~20%. We really only need to > track this for the DistinctValueWithCountServerAggregator (used by DISTINCT > COUNT, DISTINCT, PERCENTILE functions, and STDDEV functions). By > conditionally tracking, we should be able to bring perf back to what it was > before. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (PHOENIX-4718) Decrease overhead of tracking aggregate heap size
[ https://issues.apache.org/jira/browse/PHOENIX-4718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16460202#comment-16460202 ] Hudson commented on PHOENIX-4718: - FAILURE: Integrated in Jenkins build Phoenix-4.x-HBase-1.3 #112 (See [https://builds.apache.org/job/Phoenix-4.x-HBase-1.3/112/]) PHOENIX-4718 Decrease overhead of tracking aggregate heap size (jtaylor: rev dec9f2897a62709d0b9b73670ea73c8438997b03) * (edit) phoenix-core/src/main/java/org/apache/phoenix/expression/aggregator/BaseAggregator.java * (edit) phoenix-core/src/test/java/org/apache/phoenix/query/QueryServicesTestImpl.java * (edit) phoenix-core/src/main/java/org/apache/phoenix/expression/aggregator/DistinctValueWithCountServerAggregator.java * (edit) phoenix-core/src/main/java/org/apache/phoenix/coprocessor/UngroupedAggregateRegionObserver.java * (edit) phoenix-core/src/main/java/org/apache/phoenix/expression/aggregator/ClientAggregators.java * (edit) phoenix-core/src/main/java/org/apache/phoenix/coprocessor/GroupedAggregateRegionObserver.java * (edit) phoenix-core/src/it/java/org/apache/phoenix/end2end/SpillableGroupByIT.java * (add) phoenix-core/src/main/java/org/apache/phoenix/expression/aggregator/NonSizeTrackingServerAggregators.java * (edit) phoenix-core/src/main/java/org/apache/phoenix/expression/aggregator/ServerAggregators.java * (edit) phoenix-core/src/test/java/org/apache/phoenix/compile/QueryCompilerTest.java * (add) phoenix-core/src/main/java/org/apache/phoenix/expression/aggregator/SizeTrackingServerAggregators.java * (edit) phoenix-core/src/main/java/org/apache/phoenix/execute/ClientAggregatePlan.java * (edit) phoenix-core/src/main/java/org/apache/phoenix/expression/aggregator/Aggregators.java * (edit) phoenix-core/src/main/java/org/apache/phoenix/expression/aggregator/Aggregator.java * (edit) phoenix-core/src/main/java/org/apache/phoenix/query/QueryServices.java * (edit) phoenix-core/src/main/java/org/apache/phoenix/query/QueryServicesOptions.java > Decrease overhead of tracking aggregate heap size > - > > Key: PHOENIX-4718 > URL: https://issues.apache.org/jira/browse/PHOENIX-4718 > Project: Phoenix > Issue Type: Bug >Reporter: James Taylor >Assignee: James Taylor >Priority: Major > Fix For: 4.14.0, 5.0.0 > > Attachments: PHOENIX-4718-4.x-HBase-0.98.patch, PHOENIX-4718.patch, > PHOENIX-4718_v2.patch, PHOENIX-4718_v3.patch > > > Since PHOENIX-4148, we track the heap size while aggregation is occurring. > This decreased performance of aggregation by ~20%. We really only need to > track this for the DistinctValueWithCountServerAggregator (used by DISTINCT > COUNT, DISTINCT, PERCENTILE functions, and STDDEV functions). By > conditionally tracking, we should be able to bring perf back to what it was > before. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (PHOENIX-4718) Decrease overhead of tracking aggregate heap size
[ https://issues.apache.org/jira/browse/PHOENIX-4718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16459971#comment-16459971 ] Thomas D'Silva commented on PHOENIX-4718: - +1 > Decrease overhead of tracking aggregate heap size > - > > Key: PHOENIX-4718 > URL: https://issues.apache.org/jira/browse/PHOENIX-4718 > Project: Phoenix > Issue Type: Bug >Reporter: James Taylor >Assignee: James Taylor >Priority: Major > Fix For: 4.14.0, 5.0.0 > > Attachments: PHOENIX-4718-4.x-HBase-0.98.patch, PHOENIX-4718.patch, > PHOENIX-4718_v2.patch, PHOENIX-4718_v3.patch > > > Since PHOENIX-4148, we track the heap size while aggregation is occurring. > This decreased performance of aggregation by ~20%. We really only need to > track this for the DistinctValueWithCountServerAggregator (used by DISTINCT > COUNT, DISTINCT, PERCENTILE functions, and STDDEV functions). By > conditionally tracking, we should be able to bring perf back to what it was > before. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (PHOENIX-4718) Decrease overhead of tracking aggregate heap size
[ https://issues.apache.org/jira/browse/PHOENIX-4718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16459840#comment-16459840 ] Mujtaba Chohan commented on PHOENIX-4718: - [~jamestaylor] verified that performance is inline with previous releases with the patch applied. > Decrease overhead of tracking aggregate heap size > - > > Key: PHOENIX-4718 > URL: https://issues.apache.org/jira/browse/PHOENIX-4718 > Project: Phoenix > Issue Type: Bug >Reporter: James Taylor >Assignee: James Taylor >Priority: Major > Fix For: 4.14.0, 5.0.0 > > Attachments: PHOENIX-4718-4.x-HBase-0.98.patch, PHOENIX-4718.patch, > PHOENIX-4718_v2.patch, PHOENIX-4718_v3.patch > > > Since PHOENIX-4148, we track the heap size while aggregation is occurring. > This decreased performance of aggregation by ~20%. We really only need to > track this for the DistinctValueWithCountServerAggregator (used by DISTINCT > COUNT, DISTINCT, PERCENTILE functions, and STDDEV functions). By > conditionally tracking, we should be able to bring perf back to what it was > before. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (PHOENIX-4718) Decrease overhead of tracking aggregate heap size
[ https://issues.apache.org/jira/browse/PHOENIX-4718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16459449#comment-16459449 ] James Taylor commented on PHOENIX-4718: --- Thanks, [~tdsilva]. I attached a cleaned up v3 patch that: * Makes the size increase configurable * Uses separate ServerAggregators classes for the tracking versus non tracking case > Decrease overhead of tracking aggregate heap size > - > > Key: PHOENIX-4718 > URL: https://issues.apache.org/jira/browse/PHOENIX-4718 > Project: Phoenix > Issue Type: Bug >Reporter: James Taylor >Assignee: James Taylor >Priority: Major > Fix For: 4.14.0, 5.0.0 > > Attachments: PHOENIX-4718-4.x-HBase-0.98.patch, PHOENIX-4718.patch, > PHOENIX-4718_v2.patch, PHOENIX-4718_v3.patch > > > Since PHOENIX-4148, we track the heap size while aggregation is occurring. > This decreased performance of aggregation by ~20%. We really only need to > track this for the DistinctValueWithCountServerAggregator (used by DISTINCT > COUNT, DISTINCT, PERCENTILE functions, and STDDEV functions). By > conditionally tracking, we should be able to bring perf back to what it was > before. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (PHOENIX-4718) Decrease overhead of tracking aggregate heap size
[ https://issues.apache.org/jira/browse/PHOENIX-4718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16459431#comment-16459431 ] Thomas D'Silva commented on PHOENIX-4718: - LGTM > Decrease overhead of tracking aggregate heap size > - > > Key: PHOENIX-4718 > URL: https://issues.apache.org/jira/browse/PHOENIX-4718 > Project: Phoenix > Issue Type: Bug >Reporter: James Taylor >Assignee: James Taylor >Priority: Major > Attachments: PHOENIX-4718-4.x-HBase-0.98.patch, PHOENIX-4718.patch, > PHOENIX-4718_v2.patch > > > Since PHOENIX-4148, we track the heap size while aggregation is occurring. > This decreased performance of aggregation by ~20%. We really only need to > track this for the DistinctValueWithCountServerAggregator (used by DISTINCT > COUNT, DISTINCT, PERCENTILE functions, and STDDEV functions). By > conditionally tracking, we should be able to bring perf back to what it was > before. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (PHOENIX-4718) Decrease overhead of tracking aggregate heap size
[ https://issues.apache.org/jira/browse/PHOENIX-4718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16459261#comment-16459261 ] Mujtaba Chohan commented on PHOENIX-4718: - I'll check that [~jamestaylor] > Decrease overhead of tracking aggregate heap size > - > > Key: PHOENIX-4718 > URL: https://issues.apache.org/jira/browse/PHOENIX-4718 > Project: Phoenix > Issue Type: Bug >Reporter: James Taylor >Assignee: James Taylor >Priority: Major > Attachments: PHOENIX-4718-4.x-HBase-0.98.patch, PHOENIX-4718.patch, > PHOENIX-4718_v2.patch > > > Since PHOENIX-4148, we track the heap size while aggregation is occurring. > This decreased performance of aggregation by ~20%. We really only need to > track this for the DistinctValueWithCountServerAggregator (used by DISTINCT > COUNT, DISTINCT, PERCENTILE functions, and STDDEV functions). By > conditionally tracking, we should be able to bring perf back to what it was > before. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (PHOENIX-4718) Decrease overhead of tracking aggregate heap size
[ https://issues.apache.org/jira/browse/PHOENIX-4718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16459255#comment-16459255 ] James Taylor commented on PHOENIX-4718: --- Here's the 0.98 version, [~mujtabachohan] > Decrease overhead of tracking aggregate heap size > - > > Key: PHOENIX-4718 > URL: https://issues.apache.org/jira/browse/PHOENIX-4718 > Project: Phoenix > Issue Type: Bug >Reporter: James Taylor >Assignee: James Taylor >Priority: Major > Attachments: PHOENIX-4718-4.x-HBase-0.98.patch, PHOENIX-4718.patch > > > Since PHOENIX-4148, we track the heap size while aggregation is occurring. > This decreased performance of aggregation by ~20%. We really only need to > track this for the DistinctValueWithCountServerAggregator (used by DISTINCT > COUNT, DISTINCT, PERCENTILE functions, and STDDEV functions). By > conditionally tracking, we should be able to bring perf back to what it was > before. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (PHOENIX-4718) Decrease overhead of tracking aggregate heap size
[ https://issues.apache.org/jira/browse/PHOENIX-4718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16459211#comment-16459211 ] James Taylor commented on PHOENIX-4718: --- Please review, [~tdsilva]. Instead of always tracking memory used by an Aggregator, we only do it if we're using an Aggregator in which trackSize() returns true. [~mujtabachohan] - can you try with this patch and see if perf is better again? > Decrease overhead of tracking aggregate heap size > - > > Key: PHOENIX-4718 > URL: https://issues.apache.org/jira/browse/PHOENIX-4718 > Project: Phoenix > Issue Type: Bug >Reporter: James Taylor >Assignee: James Taylor >Priority: Major > Attachments: PHOENIX-4718.patch > > > Since PHOENIX-4148, we track the heap size while aggregation is occurring. > This decreased performance of aggregation by ~20%. We really only need to > track this for the DistinctValueWithCountServerAggregator (used by DISTINCT > COUNT, DISTINCT, PERCENTILE functions, and STDDEV functions). By > conditionally tracking, we should be able to bring perf back to what it was > before. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
