[jira] [Commented] (PHOENIX-5651) IndexScrutiny does not handle TTL/row-expiry
[ https://issues.apache.org/jira/browse/PHOENIX-5651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17015680#comment-17015680 ] Hudson commented on PHOENIX-5651: - FAILURE: Integrated in Jenkins build PreCommit-PHOENIX-Build #3303 (See [https://builds.apache.org/job/PreCommit-PHOENIX-Build/3303/]) PHOENIX-5644 and PHOENIX-5651 addendum patch (s.kadam: rev beb6457086c8e51fc8be5eceb468a1edf8a358b7) * (edit) phoenix-core/src/it/java/org/apache/phoenix/end2end/ParameterizedIndexUpgradeToolIT.java * (edit) phoenix-core/src/main/java/org/apache/phoenix/mapreduce/index/IndexUpgradeTool.java * (edit) phoenix-core/src/main/java/org/apache/phoenix/mapreduce/index/IndexScrutinyMapperForTest.java * (add) phoenix-core/src/it/java/org/apache/phoenix/end2end/IndexUpgradeToolIT.java > IndexScrutiny does not handle TTL/row-expiry > > > Key: PHOENIX-5651 > URL: https://issues.apache.org/jira/browse/PHOENIX-5651 > Project: Phoenix > Issue Type: Bug >Affects Versions: 4.15.1, 4.14.3 >Reporter: Priyank Porwal >Assignee: Swaroopa Kadam >Priority: Major > Fix For: 4.15.1, 4.16.0 > > Attachments: PHOENIX-5651.4.x-HBase-1.3.patch, > PHOENIX-5651.4.x-HBase-1.3.v1.patch, PHOENIX-5651.4.x-HBase-1.3.v2.patch, > PHOENIX-5651.master.v1.patch, PHOENIX-5651.master.v2.patch > > Time Spent: 3h 50m > Remaining Estimate: 0h > > If a data-table has TTL on it, it's indexes inherit the TTL too. Hence when > we run IndexScrutiny on such tables and it's indexes, scrutiny's attempts to > find matching index rows for near-expiry data rows results in no-matches > since the index row gets expired before the read from data-region mapper. The > same happens in the MR job for the other direction Index->Data. > This does not impact correctness of indexing design, but makes it very > inconvenient to get a clean scrutiny run. All reported invalid rows have to > be matched against the table TTL, which is non-trivial exercise. > IndexScrutiny itself could detect such expired rows when the matching pair is > not found and not report them as INVALID_ROWS. Perhaps a new counter for > EXPIRED_ROWS should be added as well for better visibility. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (PHOENIX-5651) IndexScrutiny does not handle TTL/row-expiry
[ https://issues.apache.org/jira/browse/PHOENIX-5651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17015569#comment-17015569 ] Hudson commented on PHOENIX-5651: - ABORTED: Integrated in Jenkins build PreCommit-PHOENIX-Build #3298 (See [https://builds.apache.org/job/PreCommit-PHOENIX-Build/3298/]) PHOENIX-5651: IndexScrutiny does not handle TTL/row-expiry (#681) (github: rev 208509b2ee18699d18bc623d2f09d38b7b298a09) * (edit) phoenix-core/src/it/java/org/apache/phoenix/end2end/IndexScrutinyToolBaseIT.java * (edit) phoenix-core/src/it/java/org/apache/phoenix/end2end/IndexScrutinyToolIT.java * (add) phoenix-core/src/it/java/org/apache/phoenix/end2end/NonParameterizedIndexScrutinyToolIT.java * (edit) phoenix-core/src/main/java/org/apache/phoenix/mapreduce/index/PhoenixScrutinyJobCounters.java * (edit) phoenix-core/src/it/java/org/apache/phoenix/end2end/IndexScrutinyToolForTenantIT.java * (edit) phoenix-core/src/main/java/org/apache/phoenix/mapreduce/index/IndexScrutinyMapper.java * (add) phoenix-core/src/main/java/org/apache/phoenix/mapreduce/index/IndexScrutinyMapperForTest.java * (edit) phoenix-core/src/main/java/org/apache/phoenix/mapreduce/index/IndexScrutinyTool.java > IndexScrutiny does not handle TTL/row-expiry > > > Key: PHOENIX-5651 > URL: https://issues.apache.org/jira/browse/PHOENIX-5651 > Project: Phoenix > Issue Type: Bug >Affects Versions: 4.15.1, 4.14.3 >Reporter: Priyank Porwal >Assignee: Swaroopa Kadam >Priority: Major > Fix For: 4.15.1, 4.16.0 > > Attachments: PHOENIX-5651.4.x-HBase-1.3.patch, > PHOENIX-5651.4.x-HBase-1.3.v1.patch, PHOENIX-5651.4.x-HBase-1.3.v2.patch, > PHOENIX-5651.master.v1.patch, PHOENIX-5651.master.v2.patch > > Time Spent: 3h 50m > Remaining Estimate: 0h > > If a data-table has TTL on it, it's indexes inherit the TTL too. Hence when > we run IndexScrutiny on such tables and it's indexes, scrutiny's attempts to > find matching index rows for near-expiry data rows results in no-matches > since the index row gets expired before the read from data-region mapper. The > same happens in the MR job for the other direction Index->Data. > This does not impact correctness of indexing design, but makes it very > inconvenient to get a clean scrutiny run. All reported invalid rows have to > be matched against the table TTL, which is non-trivial exercise. > IndexScrutiny itself could detect such expired rows when the matching pair is > not found and not report them as INVALID_ROWS. Perhaps a new counter for > EXPIRED_ROWS should be added as well for better visibility. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (PHOENIX-5651) IndexScrutiny does not handle TTL/row-expiry
[ https://issues.apache.org/jira/browse/PHOENIX-5651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17015418#comment-17015418 ] Hudson commented on PHOENIX-5651: - FAILURE: Integrated in Jenkins build Phoenix-4.x-HBase-1.3 #649 (See [https://builds.apache.org/job/Phoenix-4.x-HBase-1.3/649/]) PHOENIX-5644 and PHOENIX-5651 addendum patch (s.kadam: rev fac60df981afceacbfbcfb5d04a190fcf349e498) * (edit) phoenix-core/src/main/java/org/apache/phoenix/mapreduce/index/IndexScrutinyMapperForTest.java * (edit) phoenix-core/src/main/java/org/apache/phoenix/mapreduce/index/IndexUpgradeTool.java * (add) phoenix-core/src/it/java/org/apache/phoenix/end2end/IndexUpgradeToolIT.java * (edit) phoenix-core/src/it/java/org/apache/phoenix/end2end/ParameterizedIndexUpgradeToolIT.java > IndexScrutiny does not handle TTL/row-expiry > > > Key: PHOENIX-5651 > URL: https://issues.apache.org/jira/browse/PHOENIX-5651 > Project: Phoenix > Issue Type: Bug >Affects Versions: 4.15.1, 4.14.3 >Reporter: Priyank Porwal >Assignee: Swaroopa Kadam >Priority: Major > Fix For: 4.15.1, 4.16.0 > > Attachments: PHOENIX-5651.4.x-HBase-1.3.patch, > PHOENIX-5651.4.x-HBase-1.3.v1.patch, PHOENIX-5651.4.x-HBase-1.3.v2.patch, > PHOENIX-5651.master.v1.patch, PHOENIX-5651.master.v2.patch > > Time Spent: 3h 40m > Remaining Estimate: 0h > > If a data-table has TTL on it, it's indexes inherit the TTL too. Hence when > we run IndexScrutiny on such tables and it's indexes, scrutiny's attempts to > find matching index rows for near-expiry data rows results in no-matches > since the index row gets expired before the read from data-region mapper. The > same happens in the MR job for the other direction Index->Data. > This does not impact correctness of indexing design, but makes it very > inconvenient to get a clean scrutiny run. All reported invalid rows have to > be matched against the table TTL, which is non-trivial exercise. > IndexScrutiny itself could detect such expired rows when the matching pair is > not found and not report them as INVALID_ROWS. Perhaps a new counter for > EXPIRED_ROWS should be added as well for better visibility. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (PHOENIX-5651) IndexScrutiny does not handle TTL/row-expiry
[ https://issues.apache.org/jira/browse/PHOENIX-5651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17015416#comment-17015416 ] Hudson commented on PHOENIX-5651: - FAILURE: Integrated in Jenkins build Phoenix-4.x-HBase-1.4 #365 (See [https://builds.apache.org/job/Phoenix-4.x-HBase-1.4/365/]) PHOENIX-5644 and PHOENIX-5651 addendum patch (s.kadam: rev aa0b3dca19b99d47d4553c6776f31e9cee26c68f) * (add) phoenix-core/src/it/java/org/apache/phoenix/end2end/IndexUpgradeToolIT.java * (edit) phoenix-core/src/main/java/org/apache/phoenix/mapreduce/index/IndexScrutinyMapperForTest.java * (edit) phoenix-core/src/it/java/org/apache/phoenix/end2end/ParameterizedIndexUpgradeToolIT.java * (edit) phoenix-core/src/main/java/org/apache/phoenix/mapreduce/index/IndexUpgradeTool.java > IndexScrutiny does not handle TTL/row-expiry > > > Key: PHOENIX-5651 > URL: https://issues.apache.org/jira/browse/PHOENIX-5651 > Project: Phoenix > Issue Type: Bug >Affects Versions: 4.15.1, 4.14.3 >Reporter: Priyank Porwal >Assignee: Swaroopa Kadam >Priority: Major > Fix For: 4.15.1, 4.16.0 > > Attachments: PHOENIX-5651.4.x-HBase-1.3.patch, > PHOENIX-5651.4.x-HBase-1.3.v1.patch, PHOENIX-5651.4.x-HBase-1.3.v2.patch, > PHOENIX-5651.master.v1.patch, PHOENIX-5651.master.v2.patch > > Time Spent: 3h 40m > Remaining Estimate: 0h > > If a data-table has TTL on it, it's indexes inherit the TTL too. Hence when > we run IndexScrutiny on such tables and it's indexes, scrutiny's attempts to > find matching index rows for near-expiry data rows results in no-matches > since the index row gets expired before the read from data-region mapper. The > same happens in the MR job for the other direction Index->Data. > This does not impact correctness of indexing design, but makes it very > inconvenient to get a clean scrutiny run. All reported invalid rows have to > be matched against the table TTL, which is non-trivial exercise. > IndexScrutiny itself could detect such expired rows when the matching pair is > not found and not report them as INVALID_ROWS. Perhaps a new counter for > EXPIRED_ROWS should be added as well for better visibility. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (PHOENIX-5651) IndexScrutiny does not handle TTL/row-expiry
[ https://issues.apache.org/jira/browse/PHOENIX-5651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17015412#comment-17015412 ] Hudson commented on PHOENIX-5651: - FAILURE: Integrated in Jenkins build Phoenix-4.x-HBase-1.5 #245 (See [https://builds.apache.org/job/Phoenix-4.x-HBase-1.5/245/]) PHOENIX-5644 and PHOENIX-5651 addendum patch (s.kadam: rev 2aa5616e460fca477e8a213ebc0e5025077230f5) * (add) phoenix-core/src/it/java/org/apache/phoenix/end2end/IndexUpgradeToolIT.java * (edit) phoenix-core/src/main/java/org/apache/phoenix/mapreduce/index/IndexScrutinyMapperForTest.java * (edit) phoenix-core/src/it/java/org/apache/phoenix/end2end/ParameterizedIndexUpgradeToolIT.java * (edit) phoenix-core/src/main/java/org/apache/phoenix/mapreduce/index/IndexUpgradeTool.java > IndexScrutiny does not handle TTL/row-expiry > > > Key: PHOENIX-5651 > URL: https://issues.apache.org/jira/browse/PHOENIX-5651 > Project: Phoenix > Issue Type: Bug >Affects Versions: 4.15.1, 4.14.3 >Reporter: Priyank Porwal >Assignee: Swaroopa Kadam >Priority: Major > Fix For: 4.15.1, 4.16.0 > > Attachments: PHOENIX-5651.4.x-HBase-1.3.patch, > PHOENIX-5651.4.x-HBase-1.3.v1.patch, PHOENIX-5651.4.x-HBase-1.3.v2.patch, > PHOENIX-5651.master.v1.patch, PHOENIX-5651.master.v2.patch > > Time Spent: 3h 40m > Remaining Estimate: 0h > > If a data-table has TTL on it, it's indexes inherit the TTL too. Hence when > we run IndexScrutiny on such tables and it's indexes, scrutiny's attempts to > find matching index rows for near-expiry data rows results in no-matches > since the index row gets expired before the read from data-region mapper. The > same happens in the MR job for the other direction Index->Data. > This does not impact correctness of indexing design, but makes it very > inconvenient to get a clean scrutiny run. All reported invalid rows have to > be matched against the table TTL, which is non-trivial exercise. > IndexScrutiny itself could detect such expired rows when the matching pair is > not found and not report them as INVALID_ROWS. Perhaps a new counter for > EXPIRED_ROWS should be added as well for better visibility. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (PHOENIX-5651) IndexScrutiny does not handle TTL/row-expiry
[
https://issues.apache.org/jira/browse/PHOENIX-5651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17015354#comment-17015354
]
Hadoop QA commented on PHOENIX-5651:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12990908/PHOENIX-5651.master.v1.patch
against master branch at commit 6405839b60c410292c0e2c8cbbdee55d09343bab.
ATTACHMENT ID: 12990908
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:red}-1 javac{color}. The patch appears to cause mvn compile goal to
fail .
Compilation errors resume:
[ERROR] COMPILATION ERROR :
[ERROR]
/home/jenkins/jenkins-slave/workspace/PreCommit-PHOENIX-Build/phoenix-core/src/it/java/org/apache/phoenix/end2end/IndexScrutinyToolIT.java:[426,33]
cannot find symbol
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-compiler-plugin:3.7.0:testCompile
(default-testCompile) on project phoenix-core: Compilation failure
[ERROR]
/home/jenkins/jenkins-slave/workspace/PreCommit-PHOENIX-Build/phoenix-core/src/it/java/org/apache/phoenix/end2end/IndexScrutinyToolIT.java:[426,33]
cannot find symbol
[ERROR] symbol: variable Sets
[ERROR] location: class org.apache.phoenix.end2end.IndexScrutinyToolIT
[ERROR]
[ERROR] -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e
switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please
read the following articles:
[ERROR] [Help 1]
http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR] mvn -rf :phoenix-core
Console output:
https://builds.apache.org/job/PreCommit-PHOENIX-Build/3291//console
This message is automatically generated.
> IndexScrutiny does not handle TTL/row-expiry
>
>
> Key: PHOENIX-5651
> URL: https://issues.apache.org/jira/browse/PHOENIX-5651
> Project: Phoenix
> Issue Type: Bug
>Affects Versions: 4.15.1, 4.14.3
>Reporter: Priyank Porwal
>Assignee: Swaroopa Kadam
>Priority: Major
> Fix For: 4.15.1, 4.16.0
>
> Attachments: PHOENIX-5651.4.x-HBase-1.3.patch,
> PHOENIX-5651.4.x-HBase-1.3.v1.patch, PHOENIX-5651.4.x-HBase-1.3.v2.patch,
> PHOENIX-5651.master.v1.patch
>
> Time Spent: 3h 40m
> Remaining Estimate: 0h
>
> If a data-table has TTL on it, it's indexes inherit the TTL too. Hence when
> we run IndexScrutiny on such tables and it's indexes, scrutiny's attempts to
> find matching index rows for near-expiry data rows results in no-matches
> since the index row gets expired before the read from data-region mapper. The
> same happens in the MR job for the other direction Index->Data.
> This does not impact correctness of indexing design, but makes it very
> inconvenient to get a clean scrutiny run. All reported invalid rows have to
> be matched against the table TTL, which is non-trivial exercise.
> IndexScrutiny itself could detect such expired rows when the matching pair is
> not found and not report them as INVALID_ROWS. Perhaps a new counter for
> EXPIRED_ROWS should be added as well for better visibility.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (PHOENIX-5651) IndexScrutiny does not handle TTL/row-expiry
[ https://issues.apache.org/jira/browse/PHOENIX-5651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17014214#comment-17014214 ] Hudson commented on PHOENIX-5651: - ABORTED: Integrated in Jenkins build Phoenix-4.x-HBase-1.4 #362 (See [https://builds.apache.org/job/Phoenix-4.x-HBase-1.4/362/]) PHOENIX-5651: IndexScrutiny does not handle TTL/row-expiry (s.kadam: rev bd4f0b33fa7ac2ee99a9e160e2778ce52e69eaa7) * (add) phoenix-core/src/it/java/org/apache/phoenix/end2end/NonParameterizedIndexScrutinyToolIT.java * (edit) phoenix-core/src/main/java/org/apache/phoenix/mapreduce/index/IndexScrutinyMapper.java * (edit) phoenix-core/src/it/java/org/apache/phoenix/end2end/IndexScrutinyToolIT.java * (edit) phoenix-core/src/main/java/org/apache/phoenix/mapreduce/index/IndexScrutinyTool.java * (edit) phoenix-core/src/it/java/org/apache/phoenix/end2end/IndexScrutinyToolForTenantIT.java * (add) phoenix-core/src/main/java/org/apache/phoenix/mapreduce/index/IndexScrutinyMapperForTest.java * (edit) phoenix-core/src/main/java/org/apache/phoenix/mapreduce/index/PhoenixScrutinyJobCounters.java * (edit) phoenix-core/src/it/java/org/apache/phoenix/end2end/IndexScrutinyToolBaseIT.java > IndexScrutiny does not handle TTL/row-expiry > > > Key: PHOENIX-5651 > URL: https://issues.apache.org/jira/browse/PHOENIX-5651 > Project: Phoenix > Issue Type: Bug >Affects Versions: 4.15.1, 4.14.3 >Reporter: Priyank Porwal >Assignee: Swaroopa Kadam >Priority: Major > Fix For: 4.15.1, 4.16.0 > > Attachments: PHOENIX-5651.4.x-HBase-1.3.patch, > PHOENIX-5651.4.x-HBase-1.3.v1.patch, PHOENIX-5651.4.x-HBase-1.3.v2.patch > > Time Spent: 3h 40m > Remaining Estimate: 0h > > If a data-table has TTL on it, it's indexes inherit the TTL too. Hence when > we run IndexScrutiny on such tables and it's indexes, scrutiny's attempts to > find matching index rows for near-expiry data rows results in no-matches > since the index row gets expired before the read from data-region mapper. The > same happens in the MR job for the other direction Index->Data. > This does not impact correctness of indexing design, but makes it very > inconvenient to get a clean scrutiny run. All reported invalid rows have to > be matched against the table TTL, which is non-trivial exercise. > IndexScrutiny itself could detect such expired rows when the matching pair is > not found and not report them as INVALID_ROWS. Perhaps a new counter for > EXPIRED_ROWS should be added as well for better visibility. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (PHOENIX-5651) IndexScrutiny does not handle TTL/row-expiry
[ https://issues.apache.org/jira/browse/PHOENIX-5651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17014205#comment-17014205 ] Hudson commented on PHOENIX-5651: - ABORTED: Integrated in Jenkins build Phoenix-4.x-HBase-1.5 #242 (See [https://builds.apache.org/job/Phoenix-4.x-HBase-1.5/242/]) PHOENIX-5651: IndexScrutiny does not handle TTL/row-expiry (s.kadam: rev 3784d2c5e76ddbdae37642a9911b2a9b8f36a4f2) * (edit) phoenix-core/src/it/java/org/apache/phoenix/end2end/IndexScrutinyToolBaseIT.java * (edit) phoenix-core/src/main/java/org/apache/phoenix/mapreduce/index/IndexScrutinyMapper.java * (edit) phoenix-core/src/main/java/org/apache/phoenix/mapreduce/index/PhoenixScrutinyJobCounters.java * (add) phoenix-core/src/it/java/org/apache/phoenix/end2end/NonParameterizedIndexScrutinyToolIT.java * (edit) phoenix-core/src/main/java/org/apache/phoenix/mapreduce/index/IndexScrutinyTool.java * (edit) phoenix-core/src/it/java/org/apache/phoenix/end2end/IndexScrutinyToolIT.java * (edit) phoenix-core/src/it/java/org/apache/phoenix/end2end/IndexScrutinyToolForTenantIT.java * (add) phoenix-core/src/main/java/org/apache/phoenix/mapreduce/index/IndexScrutinyMapperForTest.java > IndexScrutiny does not handle TTL/row-expiry > > > Key: PHOENIX-5651 > URL: https://issues.apache.org/jira/browse/PHOENIX-5651 > Project: Phoenix > Issue Type: Bug >Affects Versions: 4.15.1, 4.14.3 >Reporter: Priyank Porwal >Assignee: Swaroopa Kadam >Priority: Major > Fix For: 4.15.1, 4.16.0 > > Attachments: PHOENIX-5651.4.x-HBase-1.3.patch, > PHOENIX-5651.4.x-HBase-1.3.v1.patch, PHOENIX-5651.4.x-HBase-1.3.v2.patch > > Time Spent: 3h 40m > Remaining Estimate: 0h > > If a data-table has TTL on it, it's indexes inherit the TTL too. Hence when > we run IndexScrutiny on such tables and it's indexes, scrutiny's attempts to > find matching index rows for near-expiry data rows results in no-matches > since the index row gets expired before the read from data-region mapper. The > same happens in the MR job for the other direction Index->Data. > This does not impact correctness of indexing design, but makes it very > inconvenient to get a clean scrutiny run. All reported invalid rows have to > be matched against the table TTL, which is non-trivial exercise. > IndexScrutiny itself could detect such expired rows when the matching pair is > not found and not report them as INVALID_ROWS. Perhaps a new counter for > EXPIRED_ROWS should be added as well for better visibility. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (PHOENIX-5651) IndexScrutiny does not handle TTL/row-expiry
[ https://issues.apache.org/jira/browse/PHOENIX-5651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17013968#comment-17013968 ] Hudson commented on PHOENIX-5651: - FAILURE: Integrated in Jenkins build Phoenix-4.x-HBase-1.3 #645 (See [https://builds.apache.org/job/Phoenix-4.x-HBase-1.3/645/]) PHOENIX-5651: IndexScrutiny does not handle TTL/row-expiry (s.kadam: rev 4a905b93ca04b36988a906dd7e946aa638e5b2ac) * (edit) phoenix-core/src/it/java/org/apache/phoenix/end2end/IndexScrutinyToolIT.java * (edit) phoenix-core/src/main/java/org/apache/phoenix/mapreduce/index/IndexScrutinyTool.java * (add) phoenix-core/src/main/java/org/apache/phoenix/mapreduce/index/IndexScrutinyMapperForTest.java * (edit) phoenix-core/src/main/java/org/apache/phoenix/mapreduce/index/PhoenixScrutinyJobCounters.java * (edit) phoenix-core/src/it/java/org/apache/phoenix/end2end/IndexScrutinyToolBaseIT.java * (add) phoenix-core/src/it/java/org/apache/phoenix/end2end/NonParameterizedIndexScrutinyToolIT.java * (edit) phoenix-core/src/main/java/org/apache/phoenix/mapreduce/index/IndexScrutinyMapper.java * (edit) phoenix-core/src/it/java/org/apache/phoenix/end2end/IndexScrutinyToolForTenantIT.java > IndexScrutiny does not handle TTL/row-expiry > > > Key: PHOENIX-5651 > URL: https://issues.apache.org/jira/browse/PHOENIX-5651 > Project: Phoenix > Issue Type: Bug >Affects Versions: 4.15.1, 4.14.3 >Reporter: Priyank Porwal >Assignee: Swaroopa Kadam >Priority: Major > Fix For: 4.15.1, 4.16.0 > > Attachments: PHOENIX-5651.4.x-HBase-1.3.patch, > PHOENIX-5651.4.x-HBase-1.3.v1.patch, PHOENIX-5651.4.x-HBase-1.3.v2.patch > > Time Spent: 3h 40m > Remaining Estimate: 0h > > If a data-table has TTL on it, it's indexes inherit the TTL too. Hence when > we run IndexScrutiny on such tables and it's indexes, scrutiny's attempts to > find matching index rows for near-expiry data rows results in no-matches > since the index row gets expired before the read from data-region mapper. The > same happens in the MR job for the other direction Index->Data. > This does not impact correctness of indexing design, but makes it very > inconvenient to get a clean scrutiny run. All reported invalid rows have to > be matched against the table TTL, which is non-trivial exercise. > IndexScrutiny itself could detect such expired rows when the matching pair is > not found and not report them as INVALID_ROWS. Perhaps a new counter for > EXPIRED_ROWS should be added as well for better visibility. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (PHOENIX-5651) IndexScrutiny does not handle TTL/row-expiry
[
https://issues.apache.org/jira/browse/PHOENIX-5651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17013342#comment-17013342
]
Hadoop QA commented on PHOENIX-5651:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12990555/PHOENIX-5651.4.x-HBase-1.3.v2.patch
against 4.x-HBase-1.3 branch at commit
2ba00157878e56707a85bbbeb13ced12148c5652.
ATTACHMENT ID: 12990555
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:red}-1 release audit{color}. The applied patch generated 1 release
audit warnings (more than the master's current 0 warnings).
{color:red}-1 lineLengths{color}. The patch introduces the following lines
longer than 100:
+protected Map>>
buildTargetStatement(PreparedStatement targetStatement)
+protected void writeToOutputTable(Context context, List
sourceValues, List targetValues, long sourceTS, long targetTS)
+String basePath, long outputMaxRows, String tenantId,
Class mapperClass) {
+SourceTable sourceTable, Class
mapperClass) throws Exception {
+private Job configureSubmittableJob(Job job, Path outputPath,
Class mapperClass) throws Exception {
+outputInvalidRows, outputFormat, basePath,
outputMaxRows, tenantId, mapperClass);
{color:red}-1 core tests{color}. The patch failed these unit tests:
./phoenix-core/target/failsafe-reports/TEST-org.apache.phoenix.end2end.UpgradeIT
./phoenix-core/target/failsafe-reports/TEST-org.apache.phoenix.end2end.IndexExtendedIT
./phoenix-core/target/failsafe-reports/TEST-org.apache.phoenix.end2end.SpillableGroupByIT
./phoenix-core/target/failsafe-reports/TEST-org.apache.phoenix.end2end.FlappingLocalIndexIT
./phoenix-core/target/failsafe-reports/TEST-org.apache.phoenix.end2end.IndexRebuildTaskIT
./phoenix-core/target/failsafe-reports/TEST-org.apache.phoenix.end2end.AggregateIT
Test results:
https://builds.apache.org/job/PreCommit-PHOENIX-Build/3271//testReport/
Release audit warnings:
https://builds.apache.org/job/PreCommit-PHOENIX-Build/3271//artifact/patchprocess/patchReleaseAuditWarnings.txt
Console output:
https://builds.apache.org/job/PreCommit-PHOENIX-Build/3271//console
This message is automatically generated.
> IndexScrutiny does not handle TTL/row-expiry
>
>
> Key: PHOENIX-5651
> URL: https://issues.apache.org/jira/browse/PHOENIX-5651
> Project: Phoenix
> Issue Type: Bug
>Affects Versions: 4.15.1, 4.14.3
>Reporter: Priyank Porwal
>Assignee: Swaroopa Kadam
>Priority: Major
> Fix For: 4.15.1, 4.16.0
>
> Attachments: PHOENIX-5651.4.x-HBase-1.3.patch,
> PHOENIX-5651.4.x-HBase-1.3.v1.patch, PHOENIX-5651.4.x-HBase-1.3.v2.patch
>
> Time Spent: 3h 40m
> Remaining Estimate: 0h
>
> If a data-table has TTL on it, it's indexes inherit the TTL too. Hence when
> we run IndexScrutiny on such tables and it's indexes, scrutiny's attempts to
> find matching index rows for near-expiry data rows results in no-matches
> since the index row gets expired before the read from data-region mapper. The
> same happens in the MR job for the other direction Index->Data.
> This does not impact correctness of indexing design, but makes it very
> inconvenient to get a clean scrutiny run. All reported invalid rows have to
> be matched against the table TTL, which is non-trivial exercise.
> IndexScrutiny itself could detect such expired rows when the matching pair is
> not found and not report them as INVALID_ROWS. Perhaps a new counter for
> EXPIRED_ROWS should be added as well for better visibility.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (PHOENIX-5651) IndexScrutiny does not handle TTL/row-expiry
[
https://issues.apache.org/jira/browse/PHOENIX-5651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17009338#comment-17009338
]
Hadoop QA commented on PHOENIX-5651:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12990053/PHOENIX-5651.4.x-HBase-1.3.v1.patch
against 4.x-HBase-1.3 branch at commit
7bcc9bc89bbe86eedbf8223d101dadb3cf57cc4c.
ATTACHMENT ID: 12990053
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:red}-1 release audit{color}. The applied patch generated 1 release
audit warnings (more than the master's current 0 warnings).
{color:red}-1 lineLengths{color}. The patch introduces the following lines
longer than 100:
+Set>>> entrySet =
targetPkToSourceValues.entrySet();
+String fullTableName =
SchemaUtil.getQualifiedTableName(psourceTable.getSchemaName().toString(),
+long ttl =
tableDesc.getFamily(SchemaUtil.getEmptyColumnFamily(psourceTable)).getTimeToLive();
+protected Map>>
buildTargetStatement(PreparedStatement targetStatement)
+protected void writeToOutputTable(Context context, List
sourceValues, List targetValues, long sourceTS, long targetTS)
+String basePath, long outputMaxRows, String tenantId,
Class mapperClass) {
+SourceTable sourceTable, Class
mapperClass) throws Exception {
+private Job configureSubmittableJob(Job job, Path outputPath,
Class mapperClass) throws Exception {
+outputInvalidRows, outputFormat, basePath,
outputMaxRows, tenantId, mapperClass);
{color:red}-1 core tests{color}. The patch failed these unit tests:
./phoenix-core/target/failsafe-reports/TEST-org.apache.phoenix.rpc.PhoenixClientRpcIT
./phoenix-core/target/failsafe-reports/TEST-org.apache.phoenix.end2end.IndexScrutinyToolIT
./phoenix-core/target/failsafe-reports/TEST-org.apache.phoenix.end2end.IndexScrutinyToolForTenantIT
Test results:
https://builds.apache.org/job/PreCommit-PHOENIX-Build/3238//testReport/
Release audit warnings:
https://builds.apache.org/job/PreCommit-PHOENIX-Build/3238//artifact/patchprocess/patchReleaseAuditWarnings.txt
Console output:
https://builds.apache.org/job/PreCommit-PHOENIX-Build/3238//console
This message is automatically generated.
> IndexScrutiny does not handle TTL/row-expiry
>
>
> Key: PHOENIX-5651
> URL: https://issues.apache.org/jira/browse/PHOENIX-5651
> Project: Phoenix
> Issue Type: Bug
>Affects Versions: 4.15.1, 4.14.3
>Reporter: Priyank Porwal
>Assignee: Swaroopa Kadam
>Priority: Major
> Attachments: PHOENIX-5651.4.x-HBase-1.3.patch,
> PHOENIX-5651.4.x-HBase-1.3.v1.patch
>
> Time Spent: 3h
> Remaining Estimate: 0h
>
> If a data-table has TTL on it, it's indexes inherit the TTL too. Hence when
> we run IndexScrutiny on such tables and it's indexes, scrutiny's attempts to
> find matching index rows for near-expiry data rows results in no-matches
> since the index row gets expired before the read from data-region mapper. The
> same happens in the MR job for the other direction Index->Data.
> This does not impact correctness of indexing design, but makes it very
> inconvenient to get a clean scrutiny run. All reported invalid rows have to
> be matched against the table TTL, which is non-trivial exercise.
> IndexScrutiny itself could detect such expired rows when the matching pair is
> not found and not report them as INVALID_ROWS. Perhaps a new counter for
> EXPIRED_ROWS should be added as well for better visibility.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (PHOENIX-5651) IndexScrutiny does not handle TTL/row-expiry
[
https://issues.apache.org/jira/browse/PHOENIX-5651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17009159#comment-17009159
]
Hadoop QA commented on PHOENIX-5651:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12990029/PHOENIX-5651.4.x-HBase-1.3.patch
against 4.x-HBase-1.3 branch at commit
356f4cd7d43bccb9538a5a2b94863b1c52cd9aad.
ATTACHMENT ID: 12990029
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:red}-1 release audit{color}. The applied patch generated 1 release
audit warnings (more than the master's current 0 warnings).
{color:red}-1 lineLengths{color}. The patch introduces the following lines
longer than 100:
+String dataTableDDL = "CREATE TABLE %s (ID INTEGER NOT NULL
PRIMARY KEY, NAME VARCHAR, VB VARBINARY)";
+conn.createStatement().execute(String.format(indexTableDDL,
indexTableName, dataTableName));
+PreparedStatement upsertDataStmt =
conn.prepareStatement(String.format(upsertData, dataTableName));
+PreparedStatement upsertIndexStmt =
conn.prepareStatement(String.format(upsertIndex, indexTableName));
+conn.createStatement().execute(String.format(indexTableDDL,
indexTableName, dataTableName));
+upsertDataStmt =
conn.prepareStatement(String.format(upsertData, dataTableName));
+private void upsertRow(PreparedStatement stmt, int id, String name, int
zip) throws SQLException {
+Set>>> entrySet =
targetPkToSourceValues.entrySet();
+String fullTableName =
SchemaUtil.getQualifiedTableName(psourceTable.getSchemaName().toString(),
+long ttl =
tableDesc.getFamily(SchemaUtil.getEmptyColumnFamily(psourceTable)).getTimeToLive();
{color:red}-1 core tests{color}. The patch failed these unit tests:
./phoenix-core/target/failsafe-reports/TEST-org.apache.phoenix.end2end.IndexScrutinyToolIT
./phoenix-core/target/failsafe-reports/TEST-org.apache.phoenix.end2end.IndexScrutinyToolForTenantIT
./phoenix-core/target/failsafe-reports/TEST-org.apache.phoenix.end2end.IndexRebuildTaskIT
Test results:
https://builds.apache.org/job/PreCommit-PHOENIX-Build/3233//testReport/
Release audit warnings:
https://builds.apache.org/job/PreCommit-PHOENIX-Build/3233//artifact/patchprocess/patchReleaseAuditWarnings.txt
Console output:
https://builds.apache.org/job/PreCommit-PHOENIX-Build/3233//console
This message is automatically generated.
> IndexScrutiny does not handle TTL/row-expiry
>
>
> Key: PHOENIX-5651
> URL: https://issues.apache.org/jira/browse/PHOENIX-5651
> Project: Phoenix
> Issue Type: Bug
>Affects Versions: 4.15.1, 4.14.3
>Reporter: Priyank Porwal
>Assignee: Swaroopa Kadam
>Priority: Major
> Attachments: PHOENIX-5651.4.x-HBase-1.3.patch
>
> Time Spent: 2h 10m
> Remaining Estimate: 0h
>
> If a data-table has TTL on it, it's indexes inherit the TTL too. Hence when
> we run IndexScrutiny on such tables and it's indexes, scrutiny's attempts to
> find matching index rows for near-expiry data rows results in no-matches
> since the index row gets expired before the read from data-region mapper. The
> same happens in the MR job for the other direction Index->Data.
> This does not impact correctness of indexing design, but makes it very
> inconvenient to get a clean scrutiny run. All reported invalid rows have to
> be matched against the table TTL, which is non-trivial exercise.
> IndexScrutiny itself could detect such expired rows when the matching pair is
> not found and not report them as INVALID_ROWS. Perhaps a new counter for
> EXPIRED_ROWS should be added as well for better visibility.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
