[jira] [Commented] (PHOENIX-3361) Collect stats correctly for local indexes
[ https://issues.apache.org/jira/browse/PHOENIX-3361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15563847#comment-15563847 ] Hudson commented on PHOENIX-3361: - FAILURE: Integrated in Jenkins build Phoenix-master #1434 (See [https://builds.apache.org/job/Phoenix-master/1434/]) PHOENIX-3361 Collect stats correct for local indexes (jamestaylor: rev eaf99f23da47b86e7577837c9c73ad47ad9ef0d1) * (edit) phoenix-core/src/it/java/org/apache/phoenix/end2end/index/ViewIndexIT.java * (edit) phoenix-core/src/main/java/org/apache/phoenix/query/ConnectionQueryServices.java * (edit) phoenix-core/src/main/java/org/apache/phoenix/schema/MetaDataClient.java * (edit) phoenix-core/src/main/java/org/apache/phoenix/schema/stats/GuidePostsInfo.java * (edit) phoenix-core/src/main/java/org/apache/phoenix/schema/stats/StatisticsUtil.java * (edit) phoenix-core/src/main/java/org/apache/phoenix/query/ConnectionQueryServicesImpl.java * (edit) phoenix-core/src/main/java/org/apache/phoenix/schema/stats/NoOpStatisticsCollector.java * (edit) phoenix-core/src/it/java/org/apache/phoenix/end2end/StatsCollectorIT.java * (edit) phoenix-core/src/main/java/org/apache/phoenix/query/QueryConstants.java * (edit) phoenix-core/src/test/java/org/apache/phoenix/query/PhoenixStatsCacheRemovalListenerTest.java * (edit) phoenix-core/src/main/java/org/apache/phoenix/iterate/BaseResultIterators.java * (edit) phoenix-core/src/it/java/org/apache/phoenix/end2end/RoundFloorCeilFuncIT.java * (edit) phoenix-core/src/main/java/org/apache/phoenix/jdbc/PhoenixDatabaseMetaData.java * (edit) phoenix-core/src/main/java/org/apache/phoenix/schema/stats/StatisticsCollector.java * (delete) phoenix-core/src/main/java/org/apache/phoenix/schema/stats/PTableStatsImpl.java * (edit) phoenix-core/src/it/java/org/apache/phoenix/end2end/index/MutableIndexIT.java * (edit) phoenix-core/src/main/java/org/apache/phoenix/coprocessor/UngroupedAggregateRegionObserver.java * (edit) phoenix-core/src/test/java/org/apache/phoenix/filter/SkipScanBigFilterTest.java * (delete) phoenix-core/src/main/java/org/apache/phoenix/query/TableStatsCache.java * (edit) phoenix-core/src/test/java/org/apache/phoenix/util/TestUtil.java * (edit) phoenix-core/src/main/java/org/apache/phoenix/schema/stats/GuidePostsInfoBuilder.java * (add) phoenix-core/src/main/java/org/apache/phoenix/schema/stats/GuidePostsKey.java * (edit) phoenix-core/src/main/java/org/apache/phoenix/query/ConnectionlessQueryServicesImpl.java * (edit) phoenix-core/src/it/java/org/apache/phoenix/end2end/IndexExtendedIT.java * (edit) phoenix-core/src/main/java/org/apache/phoenix/util/MetaDataUtil.java * (delete) phoenix-core/src/main/java/org/apache/phoenix/schema/stats/PTableStats.java * (add) phoenix-core/src/main/java/org/apache/phoenix/query/GuidePostsCache.java * (edit) phoenix-core/src/main/java/org/apache/phoenix/schema/stats/DefaultStatisticsCollector.java * (edit) phoenix-core/src/it/java/org/apache/phoenix/end2end/index/MutableIndexFailureIT.java * (edit) phoenix-core/src/main/java/org/apache/phoenix/query/DelegateConnectionQueryServices.java * (edit) phoenix-core/src/main/java/org/apache/phoenix/schema/stats/StatisticsWriter.java * (edit) phoenix-core/src/it/java/org/apache/phoenix/end2end/GroupByCaseIT.java > Collect stats correctly for local indexes > - > > Key: PHOENIX-3361 > URL: https://issues.apache.org/jira/browse/PHOENIX-3361 > Project: Phoenix > Issue Type: Bug >Reporter: James Taylor >Assignee: James Taylor > Fix For: 4.9.0 > > Attachments: PHOENIX-3361.patch, PHOENIX-3361_wip.patch, > PHOENIX-3361_wip2.patch > > > Stats are not being correctly collected for local indexes and tables with > local indexes, because the logic that deletes the old stats is based on all > store families of a table. Thus, when stats are collected for a table with > local indexes, the local index stats would be deleted and when the stats for > a local index are collected, the stats for the table will be deleted. > Instead, if we cache stats per column family instead of across entire table, > plus detect if we're collecting stats for a local index versus the data > table, we can fix this issue. Caching stats per column family makes more > sense anyway, because depending on the query, we may only use the stats > associated with the default column family. Also, we wouldn't want to have to > pull over stats for local index with the data table stats. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (PHOENIX-3361) Collect stats correctly for local indexes
[ https://issues.apache.org/jira/browse/PHOENIX-3361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15563281#comment-15563281 ] James Taylor commented on PHOENIX-3361: --- That's true about TestUtil#dumpTable, but I keep having to write that again when I need it, so I thought I'd just leave it in TestUtil instead. > Collect stats correctly for local indexes > - > > Key: PHOENIX-3361 > URL: https://issues.apache.org/jira/browse/PHOENIX-3361 > Project: Phoenix > Issue Type: Bug >Reporter: James Taylor >Assignee: James Taylor > Fix For: 4.9.0 > > Attachments: PHOENIX-3361.patch, PHOENIX-3361_wip.patch, > PHOENIX-3361_wip2.patch > > > Stats are not being correctly collected for local indexes and tables with > local indexes, because the logic that deletes the old stats is based on all > store families of a table. Thus, when stats are collected for a table with > local indexes, the local index stats would be deleted and when the stats for > a local index are collected, the stats for the table will be deleted. > Instead, if we cache stats per column family instead of across entire table, > plus detect if we're collecting stats for a local index versus the data > table, we can fix this issue. Caching stats per column family makes more > sense anyway, because depending on the query, we may only use the stats > associated with the default column family. Also, we wouldn't want to have to > pull over stats for local index with the data table stats. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (PHOENIX-3361) Collect stats correctly for local indexes
[ https://issues.apache.org/jira/browse/PHOENIX-3361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15563264#comment-15563264 ] Samarth Jain commented on PHOENIX-3361: --- +1, patch looks great, [~jamestaylor]. Just one minor nit - I don't see the TestUtil#dumpTable method used anywhere. > Collect stats correctly for local indexes > - > > Key: PHOENIX-3361 > URL: https://issues.apache.org/jira/browse/PHOENIX-3361 > Project: Phoenix > Issue Type: Bug >Reporter: James Taylor >Assignee: James Taylor > Fix For: 4.9.0 > > Attachments: PHOENIX-3361.patch, PHOENIX-3361_wip.patch, > PHOENIX-3361_wip2.patch > > > Stats are not being correctly collected for local indexes and tables with > local indexes, because the logic that deletes the old stats is based on all > store families of a table. Thus, when stats are collected for a table with > local indexes, the local index stats would be deleted and when the stats for > a local index are collected, the stats for the table will be deleted. > Instead, if we cache stats per column family instead of across entire table, > plus detect if we're collecting stats for a local index versus the data > table, we can fix this issue. Caching stats per column family makes more > sense anyway, because depending on the query, we may only use the stats > associated with the default column family. Also, we wouldn't want to have to > pull over stats for local index with the data table stats. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (PHOENIX-3361) Collect stats correctly for local indexes
[ https://issues.apache.org/jira/browse/PHOENIX-3361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15562869#comment-15562869 ] James Taylor commented on PHOENIX-3361: --- [~samarthjain] or [~tdsilva]? Please review - I'd like to check this in today. > Collect stats correctly for local indexes > - > > Key: PHOENIX-3361 > URL: https://issues.apache.org/jira/browse/PHOENIX-3361 > Project: Phoenix > Issue Type: Bug >Reporter: James Taylor >Assignee: James Taylor > Fix For: 4.9.0 > > Attachments: PHOENIX-3361.patch, PHOENIX-3361_wip.patch, > PHOENIX-3361_wip2.patch > > > Stats are not being correctly collected for local indexes and tables with > local indexes, because the logic that deletes the old stats is based on all > store families of a table. Thus, when stats are collected for a table with > local indexes, the local index stats would be deleted and when the stats for > a local index are collected, the stats for the table will be deleted. > Instead, if we cache stats per column family instead of across entire table, > plus detect if we're collecting stats for a local index versus the data > table, we can fix this issue. Caching stats per column family makes more > sense anyway, because depending on the query, we may only use the stats > associated with the default column family. Also, we wouldn't want to have to > pull over stats for local index with the data table stats. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (PHOENIX-3361) Collect stats correctly for local indexes
[
https://issues.apache.org/jira/browse/PHOENIX-3361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15556815#comment-15556815
]
Hadoop QA commented on PHOENIX-3361:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12832226/PHOENIX-3361.patch
against master branch at commit ad99952792b8b119085499d8d0d734c3d2dc053b.
ATTACHMENT ID: 12832226
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 10 new
or modified tests.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-PHOENIX-Build/619//console
This message is automatically generated.
> Collect stats correctly for local indexes
> -
>
> Key: PHOENIX-3361
> URL: https://issues.apache.org/jira/browse/PHOENIX-3361
> Project: Phoenix
> Issue Type: Bug
>Reporter: James Taylor
>Assignee: James Taylor
> Fix For: 4.9.0
>
> Attachments: PHOENIX-3361.patch, PHOENIX-3361_wip.patch,
> PHOENIX-3361_wip2.patch
>
>
> Stats are not being correctly collected for local indexes and tables with
> local indexes, because the logic that deletes the old stats is based on all
> store families of a table. Thus, when stats are collected for a table with
> local indexes, the local index stats would be deleted and when the stats for
> a local index are collected, the stats for the table will be deleted.
> Instead, if we cache stats per column family instead of across entire table,
> plus detect if we're collecting stats for a local index versus the data
> table, we can fix this issue. Caching stats per column family makes more
> sense anyway, because depending on the query, we may only use the stats
> associated with the default column family. Also, we wouldn't want to have to
> pull over stats for local index with the data table stats.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
