[jira] [Commented] (LENS-1384) Update error message for data completeness check
[ https://issues.apache.org/jira/browse/LENS-1384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15842301#comment-15842301 ] Narayan Periwal commented on LENS-1384: --- Thanks [~amareshwari] > Update error message for data completeness check > > > Key: LENS-1384 > URL: https://issues.apache.org/jira/browse/LENS-1384 > Project: Apache Lens > Issue Type: Task > Components: cube > Reporter: Narayan Periwal > Assignee: Narayan Periwal >Priority: Minor > Fix For: 2.7 > > Attachments: LENS-1384-v3.patch > > > We can make the error message for the data completeness check less verbose. > New message can be something like - Requested metrics are only partially > complete. Partially complete metrics: . Try > again later or remove metric and re-run report. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (LENS-1384) Update error message for data completeness check
[ https://issues.apache.org/jira/browse/LENS-1384?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Narayan Periwal updated LENS-1384: -- Attachment: (was: LENS-1384-v3.patch) > Update error message for data completeness check > > > Key: LENS-1384 > URL: https://issues.apache.org/jira/browse/LENS-1384 > Project: Apache Lens > Issue Type: Task > Components: cube > Reporter: Narayan Periwal > Assignee: Narayan Periwal >Priority: Minor > Attachments: LENS-1384-v3.patch > > > We can make the error message for the data completeness check less verbose. > New message can be something like - Requested metrics are only partially > complete. Partially complete metrics: . Try > again later or remove metric and re-run report. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (LENS-1384) Update error message for data completeness check
[ https://issues.apache.org/jira/browse/LENS-1384?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Narayan Periwal updated LENS-1384: -- Attachment: LENS-1384-v3.patch > Update error message for data completeness check > > > Key: LENS-1384 > URL: https://issues.apache.org/jira/browse/LENS-1384 > Project: Apache Lens > Issue Type: Task > Components: cube > Reporter: Narayan Periwal > Assignee: Narayan Periwal >Priority: Minor > Attachments: LENS-1384-v3.patch > > > We can make the error message for the data completeness check less verbose. > New message can be something like - Requested metrics are only partially > complete. Partially complete metrics: . Try > again later or remove metric and re-run report. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (LENS-1384) Update error message for data completeness check
[ https://issues.apache.org/jira/browse/LENS-1384?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Narayan Periwal updated LENS-1384: -- Status: Patch Available (was: Open) > Update error message for data completeness check > > > Key: LENS-1384 > URL: https://issues.apache.org/jira/browse/LENS-1384 > Project: Apache Lens > Issue Type: Task > Components: cube > Reporter: Narayan Periwal > Assignee: Narayan Periwal >Priority: Minor > Attachments: LENS-1384-v3.patch > > > We can make the error message for the data completeness check less verbose. > New message can be something like - Requested metrics are only partially > complete. Partially complete metrics: . Try > again later or remove metric and re-run report. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (LENS-1384) Update error message for data completeness check
[ https://issues.apache.org/jira/browse/LENS-1384?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Narayan Periwal updated LENS-1384: -- Status: Open (was: Patch Available) > Update error message for data completeness check > > > Key: LENS-1384 > URL: https://issues.apache.org/jira/browse/LENS-1384 > Project: Apache Lens > Issue Type: Task > Components: cube > Reporter: Narayan Periwal > Assignee: Narayan Periwal >Priority: Minor > Attachments: LENS-1384-v3.patch > > > We can make the error message for the data completeness check less verbose. > New message can be something like - Requested metrics are only partially > complete. Partially complete metrics: . Try > again later or remove metric and re-run report. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: Review Request 55715: Update error message for data completeness check
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/55715/ --- (Updated Jan. 25, 2017, 4:01 p.m.) Review request for lens. Bugs: LENS-1384 https://issues.apache.org/jira/browse/LENS-1384 Repository: lens Description --- We can make the error message for the data completeness check less verbose. New message can be something like - Requested metrics are only partially complete. Partially complete metrics: . Try again later or remove metric and re-run report. Diffs (updated) - lens-cube/src/main/java/org/apache/lens/cube/parse/CandidateTablePruneCause.java 2ad6e20 lens-cube/src/test/java/org/apache/lens/cube/parse/TestCubeRewriter.java c9e7c29 Diff: https://reviews.apache.org/r/55715/diff/ Testing --- Thanks, Narayan Periwal
[jira] [Updated] (LENS-1384) Update error message for data completeness check
[ https://issues.apache.org/jira/browse/LENS-1384?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Narayan Periwal updated LENS-1384: -- Attachment: LENS-1384-v3.patch > Update error message for data completeness check > > > Key: LENS-1384 > URL: https://issues.apache.org/jira/browse/LENS-1384 > Project: Apache Lens > Issue Type: Task > Components: cube > Reporter: Narayan Periwal > Assignee: Narayan Periwal >Priority: Minor > Attachments: LENS-1384-v3.patch > > > We can make the error message for the data completeness check less verbose. > New message can be something like - Requested metrics are only partially > complete. Partially complete metrics: . Try > again later or remove metric and re-run report. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (LENS-1384) Update error message for data completeness check
[ https://issues.apache.org/jira/browse/LENS-1384?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Narayan Periwal updated LENS-1384: -- Status: Patch Available (was: In Progress) > Update error message for data completeness check > > > Key: LENS-1384 > URL: https://issues.apache.org/jira/browse/LENS-1384 > Project: Apache Lens > Issue Type: Task > Components: cube > Reporter: Narayan Periwal > Assignee: Narayan Periwal >Priority: Minor > Attachments: LENS-1384-v3.patch > > > We can make the error message for the data completeness check less verbose. > New message can be something like - Requested metrics are only partially > complete. Partially complete metrics: . Try > again later or remove metric and re-run report. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: Review Request 55715: Update error message for data completeness check
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/55715/ --- (Updated Jan. 25, 2017, 7:31 a.m.) Review request for lens. Bugs: LENS-1384 https://issues.apache.org/jira/browse/LENS-1384 Repository: lens Description --- We can make the error message for the data completeness check less verbose. New message can be something like - Requested metrics are only partially complete. Partially complete metrics: . Try again later or remove metric and re-run report. Diffs (updated) - lens-cube/src/main/java/org/apache/lens/cube/parse/CandidateTablePruneCause.java 2ad6e20 lens-cube/src/test/java/org/apache/lens/cube/parse/TestCubeRewriter.java c9e7c29 Diff: https://reviews.apache.org/r/55715/diff/ Testing --- Thanks, Narayan Periwal
Re: Review Request 55715: Update error message for data completeness check
> On Jan. 24, 2017, 8:32 a.m., Amareshwari Sriramadasu wrote: > > lens-cube/src/test/java/org/apache/lens/cube/parse/TestCubeRewriter.java, > > line 975 > > <https://reviews.apache.org/r/55715/diff/1/?file=1608972#file1608972line975> > > > > Can you add assert for actual messsage containing measure name as part > > of the message? Added. - Narayan --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/55715/#review162789 ------- On Jan. 25, 2017, 6:29 a.m., Narayan Periwal wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/55715/ > --- > > (Updated Jan. 25, 2017, 6:29 a.m.) > > > Review request for lens. > > > Bugs: LENS-1384 > https://issues.apache.org/jira/browse/LENS-1384 > > > Repository: lens > > > Description > --- > > We can make the error message for the data completeness check less verbose. > New message can be something like - Requested metrics are only partially > complete. Partially complete metrics: . Try > again later or remove metric and re-run report. > > > Diffs > - > > > lens-cube/src/main/java/org/apache/lens/cube/parse/CandidateTablePruneCause.java > 2ad6e20 > lens-cube/src/test/java/org/apache/lens/cube/parse/TestCubeRewriter.java > c9e7c29 > > Diff: https://reviews.apache.org/r/55715/diff/ > > > Testing > --- > > > Thanks, > > Narayan Periwal > >
Re: Review Request 55715: Update error message for data completeness check
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/55715/ --- (Updated Jan. 25, 2017, 6:29 a.m.) Review request for lens. Bugs: LENS-1384 https://issues.apache.org/jira/browse/LENS-1384 Repository: lens Description --- We can make the error message for the data completeness check less verbose. New message can be something like - Requested metrics are only partially complete. Partially complete metrics: . Try again later or remove metric and re-run report. Diffs (updated) - lens-cube/src/main/java/org/apache/lens/cube/parse/CandidateTablePruneCause.java 2ad6e20 lens-cube/src/test/java/org/apache/lens/cube/parse/TestCubeRewriter.java c9e7c29 Diff: https://reviews.apache.org/r/55715/diff/ Testing --- Thanks, Narayan Periwal
Re: Review Request 55715: Update error message for data completeness check
> On Jan. 19, 2017, 2:06 p.m., Puneet Gupta wrote: > > lens-cube/src/main/java/org/apache/lens/cube/parse/CandidateTablePruneCause.java, > > line 154 > > <https://reviews.apache.org/r/55715/diff/1/?file=1608971#file1608971line154> > > > > Are we logging the old more verbose message( metric - time - > > %completion) somewhere ? It would be required for debugging the issue or to > > answer user queries when they pour in . Yes, we are logging the entire verbose message that will help in debugging the issue, as you mentioned. - Narayan --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/55715/#review162281 ----------- On Jan. 19, 2017, 12:25 p.m., Narayan Periwal wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/55715/ > --- > > (Updated Jan. 19, 2017, 12:25 p.m.) > > > Review request for lens. > > > Bugs: LENS-1384 > https://issues.apache.org/jira/browse/LENS-1384 > > > Repository: lens > > > Description > --- > > We can make the error message for the data completeness check less verbose. > New message can be something like - Requested metrics are only partially > complete. Partially complete metrics: . Try > again later or remove metric and re-run report. > > > Diffs > - > > > lens-cube/src/main/java/org/apache/lens/cube/parse/CandidateTablePruneCause.java > 2ad6e20 > lens-cube/src/test/java/org/apache/lens/cube/parse/TestCubeRewriter.java > c9e7c29 > > Diff: https://reviews.apache.org/r/55715/diff/ > > > Testing > --- > > > Thanks, > > Narayan Periwal > >
Review Request 55715: Update error message for data completeness check
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/55715/ --- Review request for lens. Bugs: LENS-1384 https://issues.apache.org/jira/browse/LENS-1384 Repository: lens Description --- We can make the error message for the data completeness check less verbose. New message can be something like - Requested metrics are only partially complete. Partially complete metrics: . Try again later or remove metric and re-run report. Diffs - lens-cube/src/main/java/org/apache/lens/cube/parse/CandidateTablePruneCause.java 2ad6e20 lens-cube/src/test/java/org/apache/lens/cube/parse/TestCubeRewriter.java c9e7c29 Diff: https://reviews.apache.org/r/55715/diff/ Testing --- Thanks, Narayan Periwal
[jira] [Work started] (LENS-1384) Update error message for data completeness check
[ https://issues.apache.org/jira/browse/LENS-1384?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on LENS-1384 started by Narayan Periwal. - > Update error message for data completeness check > > > Key: LENS-1384 > URL: https://issues.apache.org/jira/browse/LENS-1384 > Project: Apache Lens > Issue Type: Task > Components: cube > Reporter: Narayan Periwal > Assignee: Narayan Periwal >Priority: Minor > > We can make the error message for the data completeness check less verbose. > New message can be something like - Requested metrics are only partially > complete. Partially complete metrics: . Try > again later or remove metric and re-run report. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (LENS-1384) Update error message for data completeness check
[ https://issues.apache.org/jira/browse/LENS-1384?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Narayan Periwal updated LENS-1384: -- Description: We can make the error message for the data completeness check less verbose. New message can be something like - Requested metrics are only partially complete. Partially complete metrics: . Try again later or remove metric and re-run report. was: We can make the error message for the data availability check less verbose. New message can be something like - Requested metrics are only partially available. Partially available metrics: . Try again later or remove metric and re-run report. > Update error message for data completeness check > > > Key: LENS-1384 > URL: https://issues.apache.org/jira/browse/LENS-1384 > Project: Apache Lens > Issue Type: Task > Components: cube > Reporter: Narayan Periwal > Assignee: Narayan Periwal >Priority: Minor > > We can make the error message for the data completeness check less verbose. > New message can be something like - Requested metrics are only partially > complete. Partially complete metrics: . Try > again later or remove metric and re-run report. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (LENS-1384) Update error message for data completeness check
[ https://issues.apache.org/jira/browse/LENS-1384?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Narayan Periwal updated LENS-1384: -- Summary: Update error message for data completeness check (was: Update error message for data availability check) > Update error message for data completeness check > > > Key: LENS-1384 > URL: https://issues.apache.org/jira/browse/LENS-1384 > Project: Apache Lens > Issue Type: Task > Components: cube > Reporter: Narayan Periwal > Assignee: Narayan Periwal >Priority: Minor > > We can make the error message for the data availability check less verbose. > New message can be something like - Requested metrics are only partially > available. Partially available metrics: . > Try again later or remove metric and re-run report. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (LENS-1384) Update error message for data availability check
Narayan Periwal created LENS-1384: - Summary: Update error message for data availability check Key: LENS-1384 URL: https://issues.apache.org/jira/browse/LENS-1384 Project: Apache Lens Issue Type: Task Components: cube Reporter: Narayan Periwal Assignee: Narayan Periwal Priority: Minor We can make the error message for the data availability check less verbose. New message can be something like - Requested metrics are only partially available. Partially available metrics: . Try again later or remove metric and re-run report. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (LENS-1333) Add data completeness checker
[ https://issues.apache.org/jira/browse/LENS-1333?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Narayan Periwal updated LENS-1333: -- Status: Patch Available (was: Open) > Add data completeness checker > - > > Key: LENS-1333 > URL: https://issues.apache.org/jira/browse/LENS-1333 > Project: Apache Lens > Issue Type: New Feature > Components: cube >Reporter: Amareshwari Sriramadasu > Assignee: Narayan Periwal > Attachments: LENS-1333-v1.patch, LENS-1333-v2.patch, > LENS-1333-v3.patch, LENS-1333-v4.patch, LENS-1333-v5.patch, > LENS-1333-v6.patch, LENS-1333-v7.patch, LENS-1333-v8.patch, LENS-1333-v9.patch > > > Though lens has partition registration being done whenever data is available, > there is no guarantee the partition registered is complete. There can be > different ways to know if the data is complete for partition. One option > could be to have a partition property saying whether it is complete or not. > Other could be to do a http call to another hosted service and more. > Proposal here is to add an interface for DataCompletenessChecker and do the > check while resolving partitions. > Here are some of the capabilities we would like to add in Lens : > # Lens will check partition existence first, if it exists, then check the > completeness percentage. If the completeness percentage is less than a > configured threshold (default should be 98, 99 or even 100), Lens will fail > the query. > # Lens's accept query on partial data will accept on incomplete data as well. > # Lens will also option to override the completeness percentage threshold > value at query level > # Lens will still have look ahead capability of daily being incomplete, then > it will union with hourly. > # If same measure is there in two different facts , Lens will we pick the one > with higher availability. > # In case of completeness percentage threshold missed, Lens will respond back > with available percentage. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (LENS-1333) Add data completeness checker
[ https://issues.apache.org/jira/browse/LENS-1333?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Narayan Periwal updated LENS-1333: -- Attachment: LENS-1333-v9.patch > Add data completeness checker > - > > Key: LENS-1333 > URL: https://issues.apache.org/jira/browse/LENS-1333 > Project: Apache Lens > Issue Type: New Feature > Components: cube >Reporter: Amareshwari Sriramadasu > Assignee: Narayan Periwal > Attachments: LENS-1333-v1.patch, LENS-1333-v2.patch, > LENS-1333-v3.patch, LENS-1333-v4.patch, LENS-1333-v5.patch, > LENS-1333-v6.patch, LENS-1333-v7.patch, LENS-1333-v8.patch, LENS-1333-v9.patch > > > Though lens has partition registration being done whenever data is available, > there is no guarantee the partition registered is complete. There can be > different ways to know if the data is complete for partition. One option > could be to have a partition property saying whether it is complete or not. > Other could be to do a http call to another hosted service and more. > Proposal here is to add an interface for DataCompletenessChecker and do the > check while resolving partitions. > Here are some of the capabilities we would like to add in Lens : > # Lens will check partition existence first, if it exists, then check the > completeness percentage. If the completeness percentage is less than a > configured threshold (default should be 98, 99 or even 100), Lens will fail > the query. > # Lens's accept query on partial data will accept on incomplete data as well. > # Lens will also option to override the completeness percentage threshold > value at query level > # Lens will still have look ahead capability of daily being incomplete, then > it will union with hourly. > # If same measure is there in two different facts , Lens will we pick the one > with higher availability. > # In case of completeness percentage threshold missed, Lens will respond back > with available percentage. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: Review Request 52514: Add data completeness checker
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/52514/ --- (Updated Nov. 10, 2016, 12:06 p.m.) Review request for lens. Bugs: LENS-1333 https://issues.apache.org/jira/browse/LENS-1333 Repository: lens Description --- Though lens has partition registration being done whenever data is available, there is no guarantee the partition registered is complete. There can be different ways to know if the data is complete for partition. One option could be to have a partition property saying whether it is complete or not. Other could be to do a http call to another hosted service and more. Proposal here is to add an interface for DataCompletenessChecker and do the check while resolving partitions. Here are some of the capabilities we would like to add in Lens : # Lens will check partition existence first, if it exists, then check the completeness percentage. If the completeness percentage is less than a configured threshold (default should be 98, 99 or even 100), Lens will fail the query. # Lens's accept query on partial data will accept on incomplete data as well. # Lens will also option to override the completeness percentage threshold value at query level # Lens will still have look ahead capability of daily being incomplete, then it will union with hourly. # If same measure is there in two different facts , Lens will we pick the one with higher availability. # In case of completeness percentage threshold missed, Lens will respond back with available percentage. Diffs (updated) - lens-cube/src/main/java/org/apache/lens/cube/metadata/CubeFactTable.java fb958c3 lens-cube/src/main/java/org/apache/lens/cube/metadata/CubeMetastoreClient.java e14c43f lens-cube/src/main/java/org/apache/lens/cube/metadata/MetastoreConstants.java 4585ef7 lens-cube/src/main/java/org/apache/lens/cube/parse/CandidateFact.java 5dc9dc9 lens-cube/src/main/java/org/apache/lens/cube/parse/CandidateTablePruneCause.java 78fb21d lens-cube/src/main/java/org/apache/lens/cube/parse/CubeQueryConfUtil.java 408086f lens-cube/src/main/java/org/apache/lens/cube/parse/DataCompletenessChecker.java 6a0230d lens-cube/src/main/java/org/apache/lens/cube/parse/MaxCoveringFactResolver.java 13f1aa4 lens-cube/src/main/java/org/apache/lens/cube/parse/StorageTableResolver.java db26718 lens-cube/src/main/resources/olap-query-conf.xml 0c888ca lens-cube/src/test/java/org/apache/lens/cube/parse/CubeTestSetup.java 0c43cb5 lens-cube/src/test/java/org/apache/lens/cube/parse/FieldsCannotBeQueriedTogetherTest.java 7afa32e lens-cube/src/test/java/org/apache/lens/cube/parse/MockCompletenessChecker.java PRE-CREATION lens-cube/src/test/java/org/apache/lens/cube/parse/TestBaseCubeQueries.java f6cec1b lens-cube/src/test/java/org/apache/lens/cube/parse/TestCubeRewriter.java b90d4d3 lens-cube/src/test/java/org/apache/lens/cube/parse/TestQueryRewrite.java 0aa31f4 lens-server-api/src/main/java/org/apache/lens/server/api/LensConfConstants.java cf1c233 lens-server-api/src/main/java/org/apache/lens/server/api/metastore/DataCompletenessChecker.java PRE-CREATION lens-server-api/src/main/java/org/apache/lens/server/api/metastore/DefaultChecker.java PRE-CREATION lens-server/src/main/resources/lensserver-default.xml 0ac722d lens-server/src/main/resources/lenssession-default.xml 8d9f097 src/site/apt/admin/config.apt 21b23f9 src/site/apt/admin/session-config.apt 663a605 src/site/apt/user/olap-query-conf.apt ce35ec4 Diff: https://reviews.apache.org/r/52514/diff/ Testing --- Thanks, Narayan Periwal
[jira] [Updated] (LENS-1333) Add data completeness checker
[ https://issues.apache.org/jira/browse/LENS-1333?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Narayan Periwal updated LENS-1333: -- Status: Open (was: Patch Available) > Add data completeness checker > - > > Key: LENS-1333 > URL: https://issues.apache.org/jira/browse/LENS-1333 > Project: Apache Lens > Issue Type: New Feature > Components: cube >Reporter: Amareshwari Sriramadasu > Assignee: Narayan Periwal > Attachments: LENS-1333-v1.patch, LENS-1333-v2.patch, > LENS-1333-v3.patch, LENS-1333-v4.patch, LENS-1333-v5.patch, > LENS-1333-v6.patch, LENS-1333-v7.patch, LENS-1333-v8.patch > > > Though lens has partition registration being done whenever data is available, > there is no guarantee the partition registered is complete. There can be > different ways to know if the data is complete for partition. One option > could be to have a partition property saying whether it is complete or not. > Other could be to do a http call to another hosted service and more. > Proposal here is to add an interface for DataCompletenessChecker and do the > check while resolving partitions. > Here are some of the capabilities we would like to add in Lens : > # Lens will check partition existence first, if it exists, then check the > completeness percentage. If the completeness percentage is less than a > configured threshold (default should be 98, 99 or even 100), Lens will fail > the query. > # Lens's accept query on partial data will accept on incomplete data as well. > # Lens will also option to override the completeness percentage threshold > value at query level > # Lens will still have look ahead capability of daily being incomplete, then > it will union with hourly. > # If same measure is there in two different facts , Lens will we pick the one > with higher availability. > # In case of completeness percentage threshold missed, Lens will respond back > with available percentage. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: Review Request 52514: Add data completeness checker
> On Nov. 10, 2016, 12:02 p.m., Amareshwari Sriramadasu wrote: > > lens-cube/src/main/resources/olap-query-conf.xml, line 111 > > <https://reviews.apache.org/r/52514/diff/8/?file=1560342#file1560342line111> > > > > Please leave default value as blank. Ok. - Narayan --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/52514/#review155567 --- On Nov. 10, 2016, 12:06 p.m., Narayan Periwal wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/52514/ > --- > > (Updated Nov. 10, 2016, 12:06 p.m.) > > > Review request for lens. > > > Bugs: LENS-1333 > https://issues.apache.org/jira/browse/LENS-1333 > > > Repository: lens > > > Description > --- > > Though lens has partition registration being done whenever data is available, > there is no guarantee the partition registered is complete. There can be > different ways to know if the data is complete for partition. One option > could be to have a partition property saying whether it is complete or not. > Other could be to do a http call to another hosted service and more. > > Proposal here is to add an interface for DataCompletenessChecker and do the > check while resolving partitions. > > Here are some of the capabilities we would like to add in Lens : > # Lens will check partition existence first, if it exists, then check the > completeness percentage. If the completeness percentage is less than a > configured threshold (default should be 98, 99 or even 100), Lens will fail > the query. > # Lens's accept query on partial data will accept on incomplete data as well. > # Lens will also option to override the completeness percentage threshold > value at query level > # Lens will still have look ahead capability of daily being incomplete, then > it will union with hourly. > # If same measure is there in two different facts , Lens will we pick the one > with higher availability. > # In case of completeness percentage threshold missed, Lens will respond back > with available percentage. > > > Diffs > - > > lens-cube/src/main/java/org/apache/lens/cube/metadata/CubeFactTable.java > fb958c3 > > lens-cube/src/main/java/org/apache/lens/cube/metadata/CubeMetastoreClient.java > e14c43f > > lens-cube/src/main/java/org/apache/lens/cube/metadata/MetastoreConstants.java > 4585ef7 > lens-cube/src/main/java/org/apache/lens/cube/parse/CandidateFact.java > 5dc9dc9 > > lens-cube/src/main/java/org/apache/lens/cube/parse/CandidateTablePruneCause.java > 78fb21d > lens-cube/src/main/java/org/apache/lens/cube/parse/CubeQueryConfUtil.java > 408086f > > lens-cube/src/main/java/org/apache/lens/cube/parse/DataCompletenessChecker.java > 6a0230d > > lens-cube/src/main/java/org/apache/lens/cube/parse/MaxCoveringFactResolver.java > 13f1aa4 > > lens-cube/src/main/java/org/apache/lens/cube/parse/StorageTableResolver.java > db26718 > lens-cube/src/main/resources/olap-query-conf.xml 0c888ca > lens-cube/src/test/java/org/apache/lens/cube/parse/CubeTestSetup.java > 0c43cb5 > > lens-cube/src/test/java/org/apache/lens/cube/parse/FieldsCannotBeQueriedTogetherTest.java > 7afa32e > > lens-cube/src/test/java/org/apache/lens/cube/parse/MockCompletenessChecker.java > PRE-CREATION > lens-cube/src/test/java/org/apache/lens/cube/parse/TestBaseCubeQueries.java > f6cec1b > lens-cube/src/test/java/org/apache/lens/cube/parse/TestCubeRewriter.java > b90d4d3 > lens-cube/src/test/java/org/apache/lens/cube/parse/TestQueryRewrite.java > 0aa31f4 > > lens-server-api/src/main/java/org/apache/lens/server/api/LensConfConstants.java > cf1c233 > > lens-server-api/src/main/java/org/apache/lens/server/api/metastore/DataCompletenessChecker.java > PRE-CREATION > > lens-server-api/src/main/java/org/apache/lens/server/api/metastore/DefaultChecker.java > PRE-CREATION > lens-server/src/main/resources/lensserver-default.xml 0ac722d > lens-server/src/main/resources/lenssession-default.xml 8d9f097 > src/site/apt/admin/config.apt 21b23f9 > src/site/apt/admin/session-config.apt 663a605 > src/site/apt/user/olap-query-conf.apt ce35ec4 > > Diff: https://reviews.apache.org/r/52514/diff/ > > > Testing > --- > > > Thanks, > > Narayan Periwal > >
[jira] [Updated] (LENS-1333) Add data completeness checker
[ https://issues.apache.org/jira/browse/LENS-1333?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Narayan Periwal updated LENS-1333: -- Status: Patch Available (was: Open) > Add data completeness checker > - > > Key: LENS-1333 > URL: https://issues.apache.org/jira/browse/LENS-1333 > Project: Apache Lens > Issue Type: New Feature > Components: cube >Reporter: Amareshwari Sriramadasu > Assignee: Narayan Periwal > Attachments: LENS-1333-v1.patch, LENS-1333-v2.patch, > LENS-1333-v3.patch, LENS-1333-v4.patch, LENS-1333-v5.patch, > LENS-1333-v6.patch, LENS-1333-v7.patch, LENS-1333-v8.patch > > > Though lens has partition registration being done whenever data is available, > there is no guarantee the partition registered is complete. There can be > different ways to know if the data is complete for partition. One option > could be to have a partition property saying whether it is complete or not. > Other could be to do a http call to another hosted service and more. > Proposal here is to add an interface for DataCompletenessChecker and do the > check while resolving partitions. > Here are some of the capabilities we would like to add in Lens : > # Lens will check partition existence first, if it exists, then check the > completeness percentage. If the completeness percentage is less than a > configured threshold (default should be 98, 99 or even 100), Lens will fail > the query. > # Lens's accept query on partial data will accept on incomplete data as well. > # Lens will also option to override the completeness percentage threshold > value at query level > # Lens will still have look ahead capability of daily being incomplete, then > it will union with hourly. > # If same measure is there in two different facts , Lens will we pick the one > with higher availability. > # In case of completeness percentage threshold missed, Lens will respond back > with available percentage. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: Review Request 52514: Add data completeness checker
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/52514/ --- (Updated Nov. 10, 2016, 10:39 a.m.) Review request for lens. Bugs: LENS-1333 https://issues.apache.org/jira/browse/LENS-1333 Repository: lens Description --- Though lens has partition registration being done whenever data is available, there is no guarantee the partition registered is complete. There can be different ways to know if the data is complete for partition. One option could be to have a partition property saying whether it is complete or not. Other could be to do a http call to another hosted service and more. Proposal here is to add an interface for DataCompletenessChecker and do the check while resolving partitions. Here are some of the capabilities we would like to add in Lens : # Lens will check partition existence first, if it exists, then check the completeness percentage. If the completeness percentage is less than a configured threshold (default should be 98, 99 or even 100), Lens will fail the query. # Lens's accept query on partial data will accept on incomplete data as well. # Lens will also option to override the completeness percentage threshold value at query level # Lens will still have look ahead capability of daily being incomplete, then it will union with hourly. # If same measure is there in two different facts , Lens will we pick the one with higher availability. # In case of completeness percentage threshold missed, Lens will respond back with available percentage. Diffs (updated) - lens-cube/src/main/java/org/apache/lens/cube/metadata/CubeFactTable.java fb958c3 lens-cube/src/main/java/org/apache/lens/cube/metadata/CubeMetastoreClient.java e14c43f lens-cube/src/main/java/org/apache/lens/cube/metadata/MetastoreConstants.java 4585ef7 lens-cube/src/main/java/org/apache/lens/cube/parse/CandidateFact.java 5dc9dc9 lens-cube/src/main/java/org/apache/lens/cube/parse/CandidateTablePruneCause.java 78fb21d lens-cube/src/main/java/org/apache/lens/cube/parse/CubeQueryConfUtil.java 408086f lens-cube/src/main/java/org/apache/lens/cube/parse/DataCompletenessChecker.java 6a0230d lens-cube/src/main/java/org/apache/lens/cube/parse/MaxCoveringFactResolver.java 13f1aa4 lens-cube/src/main/java/org/apache/lens/cube/parse/StorageTableResolver.java db26718 lens-cube/src/main/resources/olap-query-conf.xml 0c888ca lens-cube/src/test/java/org/apache/lens/cube/parse/CubeTestSetup.java 0c43cb5 lens-cube/src/test/java/org/apache/lens/cube/parse/FieldsCannotBeQueriedTogetherTest.java 7afa32e lens-cube/src/test/java/org/apache/lens/cube/parse/MockCompletenessChecker.java PRE-CREATION lens-cube/src/test/java/org/apache/lens/cube/parse/TestBaseCubeQueries.java f6cec1b lens-cube/src/test/java/org/apache/lens/cube/parse/TestCubeRewriter.java b90d4d3 lens-cube/src/test/java/org/apache/lens/cube/parse/TestQueryRewrite.java 0aa31f4 lens-server-api/src/main/java/org/apache/lens/server/api/LensConfConstants.java cf1c233 lens-server-api/src/main/java/org/apache/lens/server/api/metastore/DataCompletenessChecker.java PRE-CREATION lens-server-api/src/main/java/org/apache/lens/server/api/metastore/DefaultChecker.java PRE-CREATION lens-server/src/main/resources/lensserver-default.xml 0ac722d lens-server/src/main/resources/lenssession-default.xml 8d9f097 src/site/apt/admin/config.apt 21b23f9 src/site/apt/admin/session-config.apt 663a605 src/site/apt/user/olap-query-conf.apt ce35ec4 Diff: https://reviews.apache.org/r/52514/diff/ Testing --- Thanks, Narayan Periwal
[jira] [Updated] (LENS-1333) Add data completeness checker
[ https://issues.apache.org/jira/browse/LENS-1333?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Narayan Periwal updated LENS-1333: -- Attachment: LENS-1333-v8.patch > Add data completeness checker > - > > Key: LENS-1333 > URL: https://issues.apache.org/jira/browse/LENS-1333 > Project: Apache Lens > Issue Type: New Feature > Components: cube >Reporter: Amareshwari Sriramadasu > Assignee: Narayan Periwal > Attachments: LENS-1333-v1.patch, LENS-1333-v2.patch, > LENS-1333-v3.patch, LENS-1333-v4.patch, LENS-1333-v5.patch, > LENS-1333-v6.patch, LENS-1333-v7.patch, LENS-1333-v8.patch > > > Though lens has partition registration being done whenever data is available, > there is no guarantee the partition registered is complete. There can be > different ways to know if the data is complete for partition. One option > could be to have a partition property saying whether it is complete or not. > Other could be to do a http call to another hosted service and more. > Proposal here is to add an interface for DataCompletenessChecker and do the > check while resolving partitions. > Here are some of the capabilities we would like to add in Lens : > # Lens will check partition existence first, if it exists, then check the > completeness percentage. If the completeness percentage is less than a > configured threshold (default should be 98, 99 or even 100), Lens will fail > the query. > # Lens's accept query on partial data will accept on incomplete data as well. > # Lens will also option to override the completeness percentage threshold > value at query level > # Lens will still have look ahead capability of daily being incomplete, then > it will union with hourly. > # If same measure is there in two different facts , Lens will we pick the one > with higher availability. > # In case of completeness percentage threshold missed, Lens will respond back > with available percentage. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (LENS-1333) Add data completeness checker
[ https://issues.apache.org/jira/browse/LENS-1333?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Narayan Periwal updated LENS-1333: -- Status: Open (was: Patch Available) > Add data completeness checker > - > > Key: LENS-1333 > URL: https://issues.apache.org/jira/browse/LENS-1333 > Project: Apache Lens > Issue Type: New Feature > Components: cube >Reporter: Amareshwari Sriramadasu > Assignee: Narayan Periwal > Attachments: LENS-1333-v1.patch, LENS-1333-v2.patch, > LENS-1333-v3.patch, LENS-1333-v4.patch, LENS-1333-v5.patch, > LENS-1333-v6.patch, LENS-1333-v7.patch > > > Though lens has partition registration being done whenever data is available, > there is no guarantee the partition registered is complete. There can be > different ways to know if the data is complete for partition. One option > could be to have a partition property saying whether it is complete or not. > Other could be to do a http call to another hosted service and more. > Proposal here is to add an interface for DataCompletenessChecker and do the > check while resolving partitions. > Here are some of the capabilities we would like to add in Lens : > # Lens will check partition existence first, if it exists, then check the > completeness percentage. If the completeness percentage is less than a > configured threshold (default should be 98, 99 or even 100), Lens will fail > the query. > # Lens's accept query on partial data will accept on incomplete data as well. > # Lens will also option to override the completeness percentage threshold > value at query level > # Lens will still have look ahead capability of daily being incomplete, then > it will union with hourly. > # If same measure is there in two different facts , Lens will we pick the one > with higher availability. > # In case of completeness percentage threshold missed, Lens will respond back > with available percentage. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: Review Request 52514: Add data completeness checker
> On Nov. 10, 2016, 7:26 a.m., Amareshwari Sriramadasu wrote: > > lens-cube/src/main/java/org/apache/lens/cube/metadata/CubeMetastoreClient.java, > > line 99 > > <https://reviews.apache.org/r/52514/diff/7/?file=1560222#file1560222line99> > > > > call it isDataCompletenessCheckEnabled or dataCompletenessChecksEnabled > > ? > > > > is and checks in the does not look good. Sure..Will correct it. > On Nov. 10, 2016, 7:26 a.m., Amareshwari Sriramadasu wrote: > > lens-cube/src/main/java/org/apache/lens/cube/parse/CubeQueryConfUtil.java, > > line 49 > > <https://reviews.apache.org/r/52514/diff/7/?file=1560226#file1560226line49> > > > > Add entry in olap-query-conf.xml with description. Agreed. - Narayan --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/52514/#review155554 ------- On Nov. 10, 2016, 7:09 a.m., Narayan Periwal wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/52514/ > --- > > (Updated Nov. 10, 2016, 7:09 a.m.) > > > Review request for lens. > > > Bugs: LENS-1333 > https://issues.apache.org/jira/browse/LENS-1333 > > > Repository: lens > > > Description > --- > > Though lens has partition registration being done whenever data is available, > there is no guarantee the partition registered is complete. There can be > different ways to know if the data is complete for partition. One option > could be to have a partition property saying whether it is complete or not. > Other could be to do a http call to another hosted service and more. > > Proposal here is to add an interface for DataCompletenessChecker and do the > check while resolving partitions. > > Here are some of the capabilities we would like to add in Lens : > # Lens will check partition existence first, if it exists, then check the > completeness percentage. If the completeness percentage is less than a > configured threshold (default should be 98, 99 or even 100), Lens will fail > the query. > # Lens's accept query on partial data will accept on incomplete data as well. > # Lens will also option to override the completeness percentage threshold > value at query level > # Lens will still have look ahead capability of daily being incomplete, then > it will union with hourly. > # If same measure is there in two different facts , Lens will we pick the one > with higher availability. > # In case of completeness percentage threshold missed, Lens will respond back > with available percentage. > > > Diffs > - > > lens-cube/src/main/java/org/apache/lens/cube/metadata/CubeFactTable.java > fb958c3 > > lens-cube/src/main/java/org/apache/lens/cube/metadata/CubeMetastoreClient.java > e14c43f > > lens-cube/src/main/java/org/apache/lens/cube/metadata/MetastoreConstants.java > 4585ef7 > lens-cube/src/main/java/org/apache/lens/cube/parse/CandidateFact.java > 5dc9dc9 > > lens-cube/src/main/java/org/apache/lens/cube/parse/CandidateTablePruneCause.java > 78fb21d > lens-cube/src/main/java/org/apache/lens/cube/parse/CubeQueryConfUtil.java > 408086f > lens-cube/src/main/java/org/apache/lens/cube/parse/DefaultChecker.java > PRE-CREATION > > lens-cube/src/main/java/org/apache/lens/cube/parse/MaxCoveringFactResolver.java > 13f1aa4 > > lens-cube/src/main/java/org/apache/lens/cube/parse/StorageTableResolver.java > db26718 > lens-cube/src/main/resources/olap-query-conf.xml 0c888ca > lens-cube/src/test/java/org/apache/lens/cube/parse/CubeTestSetup.java > 0c43cb5 > > lens-cube/src/test/java/org/apache/lens/cube/parse/FieldsCannotBeQueriedTogetherTest.java > 7afa32e > > lens-cube/src/test/java/org/apache/lens/cube/parse/MockCompletenessChecker.java > PRE-CREATION > lens-cube/src/test/java/org/apache/lens/cube/parse/TestBaseCubeQueries.java > f6cec1b > lens-cube/src/test/java/org/apache/lens/cube/parse/TestCubeRewriter.java > b90d4d3 > lens-cube/src/test/java/org/apache/lens/cube/parse/TestQueryRewrite.java > 0aa31f4 > lens-server/src/main/resources/lenssession-default.xml 8d9f097 > src/site/apt/admin/session-config.apt 663a605 > src/site/apt/user/olap-query-conf.apt ce35ec4 > > Diff: https://reviews.apache.org/r/52514/diff/ > > > Testing > --- > > > Thanks, > > Narayan Periwal > >
[jira] [Updated] (LENS-1333) Add data completeness checker
[ https://issues.apache.org/jira/browse/LENS-1333?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Narayan Periwal updated LENS-1333: -- Status: Patch Available (was: Open) > Add data completeness checker > - > > Key: LENS-1333 > URL: https://issues.apache.org/jira/browse/LENS-1333 > Project: Apache Lens > Issue Type: New Feature > Components: cube >Reporter: Amareshwari Sriramadasu > Assignee: Narayan Periwal > Attachments: LENS-1333-v1.patch, LENS-1333-v2.patch, > LENS-1333-v3.patch, LENS-1333-v4.patch, LENS-1333-v5.patch, > LENS-1333-v6.patch, LENS-1333-v7.patch > > > Though lens has partition registration being done whenever data is available, > there is no guarantee the partition registered is complete. There can be > different ways to know if the data is complete for partition. One option > could be to have a partition property saying whether it is complete or not. > Other could be to do a http call to another hosted service and more. > Proposal here is to add an interface for DataCompletenessChecker and do the > check while resolving partitions. > Here are some of the capabilities we would like to add in Lens : > # Lens will check partition existence first, if it exists, then check the > completeness percentage. If the completeness percentage is less than a > configured threshold (default should be 98, 99 or even 100), Lens will fail > the query. > # Lens's accept query on partial data will accept on incomplete data as well. > # Lens will also option to override the completeness percentage threshold > value at query level > # Lens will still have look ahead capability of daily being incomplete, then > it will union with hourly. > # If same measure is there in two different facts , Lens will we pick the one > with higher availability. > # In case of completeness percentage threshold missed, Lens will respond back > with available percentage. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (LENS-1333) Add data completeness checker
[ https://issues.apache.org/jira/browse/LENS-1333?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Narayan Periwal updated LENS-1333: -- Attachment: LENS-1333-v7.patch > Add data completeness checker > - > > Key: LENS-1333 > URL: https://issues.apache.org/jira/browse/LENS-1333 > Project: Apache Lens > Issue Type: New Feature > Components: cube >Reporter: Amareshwari Sriramadasu > Assignee: Narayan Periwal > Attachments: LENS-1333-v1.patch, LENS-1333-v2.patch, > LENS-1333-v3.patch, LENS-1333-v4.patch, LENS-1333-v5.patch, > LENS-1333-v6.patch, LENS-1333-v7.patch > > > Though lens has partition registration being done whenever data is available, > there is no guarantee the partition registered is complete. There can be > different ways to know if the data is complete for partition. One option > could be to have a partition property saying whether it is complete or not. > Other could be to do a http call to another hosted service and more. > Proposal here is to add an interface for DataCompletenessChecker and do the > check while resolving partitions. > Here are some of the capabilities we would like to add in Lens : > # Lens will check partition existence first, if it exists, then check the > completeness percentage. If the completeness percentage is less than a > configured threshold (default should be 98, 99 or even 100), Lens will fail > the query. > # Lens's accept query on partial data will accept on incomplete data as well. > # Lens will also option to override the completeness percentage threshold > value at query level > # Lens will still have look ahead capability of daily being incomplete, then > it will union with hourly. > # If same measure is there in two different facts , Lens will we pick the one > with higher availability. > # In case of completeness percentage threshold missed, Lens will respond back > with available percentage. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (LENS-1333) Add data completeness checker
[ https://issues.apache.org/jira/browse/LENS-1333?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Narayan Periwal updated LENS-1333: -- Status: Open (was: Patch Available) > Add data completeness checker > - > > Key: LENS-1333 > URL: https://issues.apache.org/jira/browse/LENS-1333 > Project: Apache Lens > Issue Type: New Feature > Components: cube >Reporter: Amareshwari Sriramadasu > Assignee: Narayan Periwal > Attachments: LENS-1333-v1.patch, LENS-1333-v2.patch, > LENS-1333-v3.patch, LENS-1333-v4.patch, LENS-1333-v5.patch, LENS-1333-v6.patch > > > Though lens has partition registration being done whenever data is available, > there is no guarantee the partition registered is complete. There can be > different ways to know if the data is complete for partition. One option > could be to have a partition property saying whether it is complete or not. > Other could be to do a http call to another hosted service and more. > Proposal here is to add an interface for DataCompletenessChecker and do the > check while resolving partitions. > Here are some of the capabilities we would like to add in Lens : > # Lens will check partition existence first, if it exists, then check the > completeness percentage. If the completeness percentage is less than a > configured threshold (default should be 98, 99 or even 100), Lens will fail > the query. > # Lens's accept query on partial data will accept on incomplete data as well. > # Lens will also option to override the completeness percentage threshold > value at query level > # Lens will still have look ahead capability of daily being incomplete, then > it will union with hourly. > # If same measure is there in two different facts , Lens will we pick the one > with higher availability. > # In case of completeness percentage threshold missed, Lens will respond back > with available percentage. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: Review Request 52514: Add data completeness checker
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/52514/ --- (Updated Nov. 10, 2016, 7:09 a.m.) Review request for lens. Bugs: LENS-1333 https://issues.apache.org/jira/browse/LENS-1333 Repository: lens Description --- Though lens has partition registration being done whenever data is available, there is no guarantee the partition registered is complete. There can be different ways to know if the data is complete for partition. One option could be to have a partition property saying whether it is complete or not. Other could be to do a http call to another hosted service and more. Proposal here is to add an interface for DataCompletenessChecker and do the check while resolving partitions. Here are some of the capabilities we would like to add in Lens : # Lens will check partition existence first, if it exists, then check the completeness percentage. If the completeness percentage is less than a configured threshold (default should be 98, 99 or even 100), Lens will fail the query. # Lens's accept query on partial data will accept on incomplete data as well. # Lens will also option to override the completeness percentage threshold value at query level # Lens will still have look ahead capability of daily being incomplete, then it will union with hourly. # If same measure is there in two different facts , Lens will we pick the one with higher availability. # In case of completeness percentage threshold missed, Lens will respond back with available percentage. Diffs (updated) - lens-cube/src/main/java/org/apache/lens/cube/metadata/CubeFactTable.java fb958c3 lens-cube/src/main/java/org/apache/lens/cube/metadata/CubeMetastoreClient.java e14c43f lens-cube/src/main/java/org/apache/lens/cube/metadata/MetastoreConstants.java 4585ef7 lens-cube/src/main/java/org/apache/lens/cube/parse/CandidateFact.java 5dc9dc9 lens-cube/src/main/java/org/apache/lens/cube/parse/CandidateTablePruneCause.java 78fb21d lens-cube/src/main/java/org/apache/lens/cube/parse/CubeQueryConfUtil.java 408086f lens-cube/src/main/java/org/apache/lens/cube/parse/DefaultChecker.java PRE-CREATION lens-cube/src/main/java/org/apache/lens/cube/parse/MaxCoveringFactResolver.java 13f1aa4 lens-cube/src/main/java/org/apache/lens/cube/parse/StorageTableResolver.java db26718 lens-cube/src/main/resources/olap-query-conf.xml 0c888ca lens-cube/src/test/java/org/apache/lens/cube/parse/CubeTestSetup.java 0c43cb5 lens-cube/src/test/java/org/apache/lens/cube/parse/FieldsCannotBeQueriedTogetherTest.java 7afa32e lens-cube/src/test/java/org/apache/lens/cube/parse/MockCompletenessChecker.java PRE-CREATION lens-cube/src/test/java/org/apache/lens/cube/parse/TestBaseCubeQueries.java f6cec1b lens-cube/src/test/java/org/apache/lens/cube/parse/TestCubeRewriter.java b90d4d3 lens-cube/src/test/java/org/apache/lens/cube/parse/TestQueryRewrite.java 0aa31f4 lens-server/src/main/resources/lenssession-default.xml 8d9f097 src/site/apt/admin/session-config.apt 663a605 src/site/apt/user/olap-query-conf.apt ce35ec4 Diff: https://reviews.apache.org/r/52514/diff/ Testing --- Thanks, Narayan Periwal
Re: Review Request 52514: Add data completeness checker
> On Nov. 10, 2016, 6:01 a.m., Amareshwari Sriramadasu wrote: > > Overall changes look fine. But I'm just wondering it would be good if we > > can have a configuration to enable/disable the feature fully - > > lens.cube.metastore.enable.datacompleteness.checks - if it is set to false, > > it can fall back to the old code, and not otherwise. Agreed. It makes sense to add this to enable/disable the feature from this config. - Narayan --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/52514/#review10 --- On Nov. 9, 2016, 1:39 p.m., Narayan Periwal wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/52514/ > --- > > (Updated Nov. 9, 2016, 1:39 p.m.) > > > Review request for lens. > > > Bugs: LENS-1333 > https://issues.apache.org/jira/browse/LENS-1333 > > > Repository: lens > > > Description > --- > > Though lens has partition registration being done whenever data is available, > there is no guarantee the partition registered is complete. There can be > different ways to know if the data is complete for partition. One option > could be to have a partition property saying whether it is complete or not. > Other could be to do a http call to another hosted service and more. > > Proposal here is to add an interface for DataCompletenessChecker and do the > check while resolving partitions. > > Here are some of the capabilities we would like to add in Lens : > # Lens will check partition existence first, if it exists, then check the > completeness percentage. If the completeness percentage is less than a > configured threshold (default should be 98, 99 or even 100), Lens will fail > the query. > # Lens's accept query on partial data will accept on incomplete data as well. > # Lens will also option to override the completeness percentage threshold > value at query level > # Lens will still have look ahead capability of daily being incomplete, then > it will union with hourly. > # If same measure is there in two different facts , Lens will we pick the one > with higher availability. > # In case of completeness percentage threshold missed, Lens will respond back > with available percentage. > > > Diffs > - > > lens-cube/src/main/java/org/apache/lens/cube/metadata/CubeFactTable.java > fb958c3 > > lens-cube/src/main/java/org/apache/lens/cube/metadata/CubeMetastoreClient.java > e14c43f > > lens-cube/src/main/java/org/apache/lens/cube/metadata/MetastoreConstants.java > 4585ef7 > lens-cube/src/main/java/org/apache/lens/cube/parse/CandidateFact.java > 5dc9dc9 > > lens-cube/src/main/java/org/apache/lens/cube/parse/CandidateTablePruneCause.java > 78fb21d > lens-cube/src/main/java/org/apache/lens/cube/parse/CubeQueryConfUtil.java > f20f105 > lens-cube/src/main/java/org/apache/lens/cube/parse/DefaultChecker.java > PRE-CREATION > > lens-cube/src/main/java/org/apache/lens/cube/parse/MaxCoveringFactResolver.java > 13f1aa4 > > lens-cube/src/main/java/org/apache/lens/cube/parse/StorageTableResolver.java > db26718 > lens-cube/src/main/resources/olap-query-conf.xml 0c888ca > lens-cube/src/test/java/org/apache/lens/cube/parse/CubeTestSetup.java > 0c43cb5 > > lens-cube/src/test/java/org/apache/lens/cube/parse/FieldsCannotBeQueriedTogetherTest.java > 7afa32e > > lens-cube/src/test/java/org/apache/lens/cube/parse/MockCompletenessChecker.java > PRE-CREATION > lens-cube/src/test/java/org/apache/lens/cube/parse/TestBaseCubeQueries.java > f6cec1b > lens-cube/src/test/java/org/apache/lens/cube/parse/TestCubeRewriter.java > b90d4d3 > lens-cube/src/test/java/org/apache/lens/cube/parse/TestQueryRewrite.java > 0aa31f4 > lens-server/src/main/resources/lenssession-default.xml 9a8f9da > src/site/apt/admin/session-config.apt d480f88 > src/site/apt/user/olap-query-conf.apt ce35ec4 > > Diff: https://reviews.apache.org/r/52514/diff/ > > > Testing > --- > > > Thanks, > > Narayan Periwal > >
[jira] [Updated] (LENS-1333) Add data completeness checker
[ https://issues.apache.org/jira/browse/LENS-1333?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Narayan Periwal updated LENS-1333: -- Status: Patch Available (was: Open) > Add data completeness checker > - > > Key: LENS-1333 > URL: https://issues.apache.org/jira/browse/LENS-1333 > Project: Apache Lens > Issue Type: New Feature > Components: cube >Reporter: Amareshwari Sriramadasu > Assignee: Narayan Periwal > Attachments: LENS-1333-v1.patch, LENS-1333-v2.patch, > LENS-1333-v3.patch, LENS-1333-v4.patch, LENS-1333-v5.patch, LENS-1333-v6.patch > > > Though lens has partition registration being done whenever data is available, > there is no guarantee the partition registered is complete. There can be > different ways to know if the data is complete for partition. One option > could be to have a partition property saying whether it is complete or not. > Other could be to do a http call to another hosted service and more. > Proposal here is to add an interface for DataCompletenessChecker and do the > check while resolving partitions. > Here are some of the capabilities we would like to add in Lens : > # Lens will check partition existence first, if it exists, then check the > completeness percentage. If the completeness percentage is less than a > configured threshold (default should be 98, 99 or even 100), Lens will fail > the query. > # Lens's accept query on partial data will accept on incomplete data as well. > # Lens will also option to override the completeness percentage threshold > value at query level > # Lens will still have look ahead capability of daily being incomplete, then > it will union with hourly. > # If same measure is there in two different facts , Lens will we pick the one > with higher availability. > # In case of completeness percentage threshold missed, Lens will respond back > with available percentage. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (LENS-1333) Add data completeness checker
[ https://issues.apache.org/jira/browse/LENS-1333?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Narayan Periwal updated LENS-1333: -- Attachment: LENS-1333-v6.patch > Add data completeness checker > - > > Key: LENS-1333 > URL: https://issues.apache.org/jira/browse/LENS-1333 > Project: Apache Lens > Issue Type: New Feature > Components: cube >Reporter: Amareshwari Sriramadasu > Assignee: Narayan Periwal > Attachments: LENS-1333-v1.patch, LENS-1333-v2.patch, > LENS-1333-v3.patch, LENS-1333-v4.patch, LENS-1333-v5.patch, LENS-1333-v6.patch > > > Though lens has partition registration being done whenever data is available, > there is no guarantee the partition registered is complete. There can be > different ways to know if the data is complete for partition. One option > could be to have a partition property saying whether it is complete or not. > Other could be to do a http call to another hosted service and more. > Proposal here is to add an interface for DataCompletenessChecker and do the > check while resolving partitions. > Here are some of the capabilities we would like to add in Lens : > # Lens will check partition existence first, if it exists, then check the > completeness percentage. If the completeness percentage is less than a > configured threshold (default should be 98, 99 or even 100), Lens will fail > the query. > # Lens's accept query on partial data will accept on incomplete data as well. > # Lens will also option to override the completeness percentage threshold > value at query level > # Lens will still have look ahead capability of daily being incomplete, then > it will union with hourly. > # If same measure is there in two different facts , Lens will we pick the one > with higher availability. > # In case of completeness percentage threshold missed, Lens will respond back > with available percentage. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (LENS-1333) Add data completeness checker
[ https://issues.apache.org/jira/browse/LENS-1333?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Narayan Periwal updated LENS-1333: -- Status: Patch Available (was: In Progress) > Add data completeness checker > - > > Key: LENS-1333 > URL: https://issues.apache.org/jira/browse/LENS-1333 > Project: Apache Lens > Issue Type: New Feature > Components: cube >Reporter: Amareshwari Sriramadasu > Assignee: Narayan Periwal > Attachments: LENS-1333-v1.patch, LENS-1333-v2.patch, > LENS-1333-v3.patch, LENS-1333-v4.patch, LENS-1333-v5.patch > > > Though lens has partition registration being done whenever data is available, > there is no guarantee the partition registered is complete. There can be > different ways to know if the data is complete for partition. One option > could be to have a partition property saying whether it is complete or not. > Other could be to do a http call to another hosted service and more. > Proposal here is to add an interface for DataCompletenessChecker and do the > check while resolving partitions. > Here are some of the capabilities we would like to add in Lens : > # Lens will check partition existence first, if it exists, then check the > completeness percentage. If the completeness percentage is less than a > configured threshold (default should be 98, 99 or even 100), Lens will fail > the query. > # Lens's accept query on partial data will accept on incomplete data as well. > # Lens will also option to override the completeness percentage threshold > value at query level > # Lens will still have look ahead capability of daily being incomplete, then > it will union with hourly. > # If same measure is there in two different facts , Lens will we pick the one > with higher availability. > # In case of completeness percentage threshold missed, Lens will respond back > with available percentage. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (LENS-1333) Add data completeness checker
[ https://issues.apache.org/jira/browse/LENS-1333?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Narayan Periwal updated LENS-1333: -- Attachment: LENS-1333-v5.patch > Add data completeness checker > - > > Key: LENS-1333 > URL: https://issues.apache.org/jira/browse/LENS-1333 > Project: Apache Lens > Issue Type: New Feature > Components: cube >Reporter: Amareshwari Sriramadasu > Assignee: Narayan Periwal > Attachments: LENS-1333-v1.patch, LENS-1333-v2.patch, > LENS-1333-v3.patch, LENS-1333-v4.patch, LENS-1333-v5.patch > > > Though lens has partition registration being done whenever data is available, > there is no guarantee the partition registered is complete. There can be > different ways to know if the data is complete for partition. One option > could be to have a partition property saying whether it is complete or not. > Other could be to do a http call to another hosted service and more. > Proposal here is to add an interface for DataCompletenessChecker and do the > check while resolving partitions. > Here are some of the capabilities we would like to add in Lens : > # Lens will check partition existence first, if it exists, then check the > completeness percentage. If the completeness percentage is less than a > configured threshold (default should be 98, 99 or even 100), Lens will fail > the query. > # Lens's accept query on partial data will accept on incomplete data as well. > # Lens will also option to override the completeness percentage threshold > value at query level > # Lens will still have look ahead capability of daily being incomplete, then > it will union with hourly. > # If same measure is there in two different facts , Lens will we pick the one > with higher availability. > # In case of completeness percentage threshold missed, Lens will respond back > with available percentage. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Work started] (LENS-1333) Add data completeness checker
[ https://issues.apache.org/jira/browse/LENS-1333?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on LENS-1333 started by Narayan Periwal. - > Add data completeness checker > - > > Key: LENS-1333 > URL: https://issues.apache.org/jira/browse/LENS-1333 > Project: Apache Lens > Issue Type: New Feature > Components: cube >Reporter: Amareshwari Sriramadasu > Assignee: Narayan Periwal > Attachments: LENS-1333-v1.patch, LENS-1333-v2.patch, > LENS-1333-v3.patch, LENS-1333-v4.patch > > > Though lens has partition registration being done whenever data is available, > there is no guarantee the partition registered is complete. There can be > different ways to know if the data is complete for partition. One option > could be to have a partition property saying whether it is complete or not. > Other could be to do a http call to another hosted service and more. > Proposal here is to add an interface for DataCompletenessChecker and do the > check while resolving partitions. > Here are some of the capabilities we would like to add in Lens : > # Lens will check partition existence first, if it exists, then check the > completeness percentage. If the completeness percentage is less than a > configured threshold (default should be 98, 99 or even 100), Lens will fail > the query. > # Lens's accept query on partial data will accept on incomplete data as well. > # Lens will also option to override the completeness percentage threshold > value at query level > # Lens will still have look ahead capability of daily being incomplete, then > it will union with hourly. > # If same measure is there in two different facts , Lens will we pick the one > with higher availability. > # In case of completeness percentage threshold missed, Lens will respond back > with available percentage. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (LENS-1333) Add data completeness checker
[ https://issues.apache.org/jira/browse/LENS-1333?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Narayan Periwal updated LENS-1333: -- Status: Open (was: Patch Available) > Add data completeness checker > - > > Key: LENS-1333 > URL: https://issues.apache.org/jira/browse/LENS-1333 > Project: Apache Lens > Issue Type: New Feature > Components: cube >Reporter: Amareshwari Sriramadasu > Assignee: Narayan Periwal > Attachments: LENS-1333-v1.patch, LENS-1333-v2.patch, > LENS-1333-v3.patch, LENS-1333-v4.patch > > > Though lens has partition registration being done whenever data is available, > there is no guarantee the partition registered is complete. There can be > different ways to know if the data is complete for partition. One option > could be to have a partition property saying whether it is complete or not. > Other could be to do a http call to another hosted service and more. > Proposal here is to add an interface for DataCompletenessChecker and do the > check while resolving partitions. > Here are some of the capabilities we would like to add in Lens : > # Lens will check partition existence first, if it exists, then check the > completeness percentage. If the completeness percentage is less than a > configured threshold (default should be 98, 99 or even 100), Lens will fail > the query. > # Lens's accept query on partial data will accept on incomplete data as well. > # Lens will also option to override the completeness percentage threshold > value at query level > # Lens will still have look ahead capability of daily being incomplete, then > it will union with hourly. > # If same measure is there in two different facts , Lens will we pick the one > with higher availability. > # In case of completeness percentage threshold missed, Lens will respond back > with available percentage. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Work stopped] (LENS-1333) Add data completeness checker
[ https://issues.apache.org/jira/browse/LENS-1333?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on LENS-1333 stopped by Narayan Periwal. - > Add data completeness checker > - > > Key: LENS-1333 > URL: https://issues.apache.org/jira/browse/LENS-1333 > Project: Apache Lens > Issue Type: New Feature > Components: cube >Reporter: Amareshwari Sriramadasu > Assignee: Narayan Periwal > Attachments: LENS-1333-v1.patch, LENS-1333-v2.patch, > LENS-1333-v3.patch, LENS-1333-v4.patch > > > Though lens has partition registration being done whenever data is available, > there is no guarantee the partition registered is complete. There can be > different ways to know if the data is complete for partition. One option > could be to have a partition property saying whether it is complete or not. > Other could be to do a http call to another hosted service and more. > Proposal here is to add an interface for DataCompletenessChecker and do the > check while resolving partitions. > Here are some of the capabilities we would like to add in Lens : > # Lens will check partition existence first, if it exists, then check the > completeness percentage. If the completeness percentage is less than a > configured threshold (default should be 98, 99 or even 100), Lens will fail > the query. > # Lens's accept query on partial data will accept on incomplete data as well. > # Lens will also option to override the completeness percentage threshold > value at query level > # Lens will still have look ahead capability of daily being incomplete, then > it will union with hourly. > # If same measure is there in two different facts , Lens will we pick the one > with higher availability. > # In case of completeness percentage threshold missed, Lens will respond back > with available percentage. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (LENS-1333) Add data completeness checker
[ https://issues.apache.org/jira/browse/LENS-1333?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Narayan Periwal updated LENS-1333: -- Status: Patch Available (was: Open) > Add data completeness checker > - > > Key: LENS-1333 > URL: https://issues.apache.org/jira/browse/LENS-1333 > Project: Apache Lens > Issue Type: New Feature > Components: cube >Reporter: Amareshwari Sriramadasu > Assignee: Narayan Periwal > Attachments: LENS-1333-v1.patch, LENS-1333-v2.patch, > LENS-1333-v3.patch, LENS-1333-v4.patch > > > Though lens has partition registration being done whenever data is available, > there is no guarantee the partition registered is complete. There can be > different ways to know if the data is complete for partition. One option > could be to have a partition property saying whether it is complete or not. > Other could be to do a http call to another hosted service and more. > Proposal here is to add an interface for DataCompletenessChecker and do the > check while resolving partitions. > Here are some of the capabilities we would like to add in Lens : > # Lens will check partition existence first, if it exists, then check the > completeness percentage. If the completeness percentage is less than a > configured threshold (default should be 98, 99 or even 100), Lens will fail > the query. > # Lens's accept query on partial data will accept on incomplete data as well. > # Lens will also option to override the completeness percentage threshold > value at query level > # Lens will still have look ahead capability of daily being incomplete, then > it will union with hourly. > # If same measure is there in two different facts , Lens will we pick the one > with higher availability. > # In case of completeness percentage threshold missed, Lens will respond back > with available percentage. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (LENS-1333) Add data completeness checker
[ https://issues.apache.org/jira/browse/LENS-1333?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Narayan Periwal updated LENS-1333: -- Status: Open (was: Patch Available) > Add data completeness checker > - > > Key: LENS-1333 > URL: https://issues.apache.org/jira/browse/LENS-1333 > Project: Apache Lens > Issue Type: New Feature > Components: cube >Reporter: Amareshwari Sriramadasu > Assignee: Narayan Periwal > Attachments: LENS-1333-v1.patch, LENS-1333-v2.patch, > LENS-1333-v3.patch, LENS-1333-v4.patch > > > Though lens has partition registration being done whenever data is available, > there is no guarantee the partition registered is complete. There can be > different ways to know if the data is complete for partition. One option > could be to have a partition property saying whether it is complete or not. > Other could be to do a http call to another hosted service and more. > Proposal here is to add an interface for DataCompletenessChecker and do the > check while resolving partitions. > Here are some of the capabilities we would like to add in Lens : > # Lens will check partition existence first, if it exists, then check the > completeness percentage. If the completeness percentage is less than a > configured threshold (default should be 98, 99 or even 100), Lens will fail > the query. > # Lens's accept query on partial data will accept on incomplete data as well. > # Lens will also option to override the completeness percentage threshold > value at query level > # Lens will still have look ahead capability of daily being incomplete, then > it will union with hourly. > # If same measure is there in two different facts , Lens will we pick the one > with higher availability. > # In case of completeness percentage threshold missed, Lens will respond back > with available percentage. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: Review Request 52514: Add data completeness checker
> On Oct. 21, 2016, 9:12 a.m., Rajat Khandelwal wrote: > > lens-cube/src/main/java/org/apache/lens/cube/parse/StorageTableResolver.java, > > lines 118-121 > > <https://reviews.apache.org/r/52514/diff/3/?file=1529511#file1529511line118> > > > > StorageTableResolver is invoked in each rewrite step, which happens > > #drivers times for each query. Using reflection to create an instance this > > often will be harmful for performance. Agreed. Added this in Hive Conf. - Narayan --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/52514/#review153515 ----------- On Nov. 8, 2016, 1:11 p.m., Narayan Periwal wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/52514/ > --- > > (Updated Nov. 8, 2016, 1:11 p.m.) > > > Review request for lens. > > > Bugs: LENS-1333 > https://issues.apache.org/jira/browse/LENS-1333 > > > Repository: lens > > > Description > --- > > Though lens has partition registration being done whenever data is available, > there is no guarantee the partition registered is complete. There can be > different ways to know if the data is complete for partition. One option > could be to have a partition property saying whether it is complete or not. > Other could be to do a http call to another hosted service and more. > > Proposal here is to add an interface for DataCompletenessChecker and do the > check while resolving partitions. > > Here are some of the capabilities we would like to add in Lens : > # Lens will check partition existence first, if it exists, then check the > completeness percentage. If the completeness percentage is less than a > configured threshold (default should be 98, 99 or even 100), Lens will fail > the query. > # Lens's accept query on partial data will accept on incomplete data as well. > # Lens will also option to override the completeness percentage threshold > value at query level > # Lens will still have look ahead capability of daily being incomplete, then > it will union with hourly. > # If same measure is there in two different facts , Lens will we pick the one > with higher availability. > # In case of completeness percentage threshold missed, Lens will respond back > with available percentage. > > > Diffs > - > > lens-cube/src/main/java/org/apache/lens/cube/metadata/CubeFactTable.java > fb958c3 > > lens-cube/src/main/java/org/apache/lens/cube/metadata/CubeMetastoreClient.java > e14c43f > > lens-cube/src/main/java/org/apache/lens/cube/metadata/MetastoreConstants.java > 4585ef7 > lens-cube/src/main/java/org/apache/lens/cube/parse/CandidateFact.java > 5dc9dc9 > > lens-cube/src/main/java/org/apache/lens/cube/parse/CandidateTablePruneCause.java > 78fb21d > lens-cube/src/main/java/org/apache/lens/cube/parse/CubeQueryConfUtil.java > f20f105 > lens-cube/src/main/java/org/apache/lens/cube/parse/DefaultChecker.java > PRE-CREATION > > lens-cube/src/main/java/org/apache/lens/cube/parse/MaxCoveringFactResolver.java > 13f1aa4 > > lens-cube/src/main/java/org/apache/lens/cube/parse/StorageTableResolver.java > db26718 > lens-cube/src/main/resources/olap-query-conf.xml 0c888ca > lens-cube/src/test/java/org/apache/lens/cube/parse/CubeTestSetup.java > 0c43cb5 > > lens-cube/src/test/java/org/apache/lens/cube/parse/FieldsCannotBeQueriedTogetherTest.java > 7afa32e > > lens-cube/src/test/java/org/apache/lens/cube/parse/MockCompletenessChecker.java > PRE-CREATION > lens-cube/src/test/java/org/apache/lens/cube/parse/TestBaseCubeQueries.java > f6cec1b > lens-cube/src/test/java/org/apache/lens/cube/parse/TestCubeRewriter.java > b90d4d3 > lens-cube/src/test/java/org/apache/lens/cube/parse/TestQueryRewrite.java > 0aa31f4 > lens-server/src/main/resources/lenssession-default.xml 9a8f9da > src/site/apt/admin/session-config.apt d480f88 > src/site/apt/user/olap-query-conf.apt ce35ec4 > > Diff: https://reviews.apache.org/r/52514/diff/ > > > Testing > --- > > > Thanks, > > Narayan Periwal > >
Re: Review Request 52514: Add data completeness checker
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/52514/ --- (Updated Nov. 8, 2016, 1:11 p.m.) Review request for lens. Bugs: LENS-1333 https://issues.apache.org/jira/browse/LENS-1333 Repository: lens Description --- Though lens has partition registration being done whenever data is available, there is no guarantee the partition registered is complete. There can be different ways to know if the data is complete for partition. One option could be to have a partition property saying whether it is complete or not. Other could be to do a http call to another hosted service and more. Proposal here is to add an interface for DataCompletenessChecker and do the check while resolving partitions. Here are some of the capabilities we would like to add in Lens : # Lens will check partition existence first, if it exists, then check the completeness percentage. If the completeness percentage is less than a configured threshold (default should be 98, 99 or even 100), Lens will fail the query. # Lens's accept query on partial data will accept on incomplete data as well. # Lens will also option to override the completeness percentage threshold value at query level # Lens will still have look ahead capability of daily being incomplete, then it will union with hourly. # If same measure is there in two different facts , Lens will we pick the one with higher availability. # In case of completeness percentage threshold missed, Lens will respond back with available percentage. Diffs (updated) - lens-cube/src/main/java/org/apache/lens/cube/metadata/CubeFactTable.java fb958c3 lens-cube/src/main/java/org/apache/lens/cube/metadata/CubeMetastoreClient.java e14c43f lens-cube/src/main/java/org/apache/lens/cube/metadata/MetastoreConstants.java 4585ef7 lens-cube/src/main/java/org/apache/lens/cube/parse/CandidateFact.java 5dc9dc9 lens-cube/src/main/java/org/apache/lens/cube/parse/CandidateTablePruneCause.java 78fb21d lens-cube/src/main/java/org/apache/lens/cube/parse/CubeQueryConfUtil.java f20f105 lens-cube/src/main/java/org/apache/lens/cube/parse/DefaultChecker.java PRE-CREATION lens-cube/src/main/java/org/apache/lens/cube/parse/MaxCoveringFactResolver.java 13f1aa4 lens-cube/src/main/java/org/apache/lens/cube/parse/StorageTableResolver.java db26718 lens-cube/src/main/resources/olap-query-conf.xml 0c888ca lens-cube/src/test/java/org/apache/lens/cube/parse/CubeTestSetup.java 0c43cb5 lens-cube/src/test/java/org/apache/lens/cube/parse/FieldsCannotBeQueriedTogetherTest.java 7afa32e lens-cube/src/test/java/org/apache/lens/cube/parse/MockCompletenessChecker.java PRE-CREATION lens-cube/src/test/java/org/apache/lens/cube/parse/TestBaseCubeQueries.java f6cec1b lens-cube/src/test/java/org/apache/lens/cube/parse/TestCubeRewriter.java b90d4d3 lens-cube/src/test/java/org/apache/lens/cube/parse/TestQueryRewrite.java 0aa31f4 lens-server/src/main/resources/lenssession-default.xml 9a8f9da src/site/apt/admin/session-config.apt d480f88 src/site/apt/user/olap-query-conf.apt ce35ec4 Diff: https://reviews.apache.org/r/52514/diff/ Testing --- Thanks, Narayan Periwal
Re: Review Request 52514: Add data completeness checker
> On Nov. 4, 2016, 5:45 a.m., Amareshwari Sriramadasu wrote: > > lens-cube/src/main/java/org/apache/lens/cube/parse/StorageTableResolver.java, > > line 538 > > <https://reviews.apache.org/r/52514/diff/4/?file=1552900#file1552900line538> > > > > Can you cross check context.getQueriedMsrs() is actually resolving > > expression fields as well ? I'm not fully sure. We will have to use cubeql.getQueriedExprsWithMeasures and parse measures from here. > On Nov. 4, 2016, 5:45 a.m., Amareshwari Sriramadasu wrote: > > lens-cube/src/main/java/org/apache/lens/cube/parse/StorageTableResolver.java, > > line 599 > > <https://reviews.apache.org/r/52514/diff/4/?file=1552900#file1552900line599> > > > > Instead of measureFromTag should be we put actual queried phrase column > > here ? The measures from phrases already gets included in the cubeql.queriedMeasures(). For the expressions, we will put the expression name here instead of the measures that make up the expressions. > On Nov. 4, 2016, 5:45 a.m., Amareshwari Sriramadasu wrote: > > lens-cube/src/main/java/org/apache/lens/cube/parse/StorageTableResolver.java, > > lines 603-605 > > <https://reviews.apache.org/r/52514/diff/4/?file=1552900#file1552900line603> > > > > Should be moved to outer while loop and should be moved inside the > > check for isFactDataIncomplete flag for removing there. Agreed, Will fix it. - Narayan --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/52514/#review154865 --- On Nov. 3, 2016, 11:13 a.m., Narayan Periwal wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/52514/ > --- > > (Updated Nov. 3, 2016, 11:13 a.m.) > > > Review request for lens. > > > Bugs: LENS-1333 > https://issues.apache.org/jira/browse/LENS-1333 > > > Repository: lens > > > Description > --- > > Though lens has partition registration being done whenever data is available, > there is no guarantee the partition registered is complete. There can be > different ways to know if the data is complete for partition. One option > could be to have a partition property saying whether it is complete or not. > Other could be to do a http call to another hosted service and more. > > Proposal here is to add an interface for DataCompletenessChecker and do the > check while resolving partitions. > > Here are some of the capabilities we would like to add in Lens : > # Lens will check partition existence first, if it exists, then check the > completeness percentage. If the completeness percentage is less than a > configured threshold (default should be 98, 99 or even 100), Lens will fail > the query. > # Lens's accept query on partial data will accept on incomplete data as well. > # Lens will also option to override the completeness percentage threshold > value at query level > # Lens will still have look ahead capability of daily being incomplete, then > it will union with hourly. > # If same measure is there in two different facts , Lens will we pick the one > with higher availability. > # In case of completeness percentage threshold missed, Lens will respond back > with available percentage. > > > Diffs > - > > lens-cube/src/main/java/org/apache/lens/cube/metadata/CubeFactTable.java > fb958c3 > > lens-cube/src/main/java/org/apache/lens/cube/metadata/CubeMetastoreClient.java > e14c43f > > lens-cube/src/main/java/org/apache/lens/cube/metadata/MetastoreConstants.java > 4585ef7 > lens-cube/src/main/java/org/apache/lens/cube/parse/CandidateFact.java > 5dc9dc9 > > lens-cube/src/main/java/org/apache/lens/cube/parse/CandidateTablePruneCause.java > 78fb21d > lens-cube/src/main/java/org/apache/lens/cube/parse/CubeQueryConfUtil.java > f20f105 > lens-cube/src/main/java/org/apache/lens/cube/parse/DefaultChecker.java > PRE-CREATION > > lens-cube/src/main/java/org/apache/lens/cube/parse/MaxCoveringFactResolver.java > 13f1aa4 > > lens-cube/src/main/java/org/apache/lens/cube/parse/StorageTableResolver.java > db26718 > lens-cube/src/main/resources/olap-query-conf.xml 0c888ca > lens-cube/src/test/java/org/apache/lens/cube/parse/CubeTestSetup.java > 0c43cb5 > > lens-cube/src/test/java/org/apache/lens/cube/parse/FieldsCannotBeQueriedTogetherTest.java > 7afa32e > > lens-cube/src/test/java
Re: Review Request 52514: Add data completeness checker
> On Nov. 4, 2016, 5:45 a.m., Amareshwari Sriramadasu wrote: > > lens-cube/src/main/java/org/apache/lens/cube/parse/StorageTableResolver.java, > > line 529 > > <https://reviews.apache.org/r/52514/diff/4/?file=1552900#file1552900line529> > > > > Measures are getting added, by iterating over queried phrases below. > > But empty check is happening here. Populating measures should be moved up. With the empty check for the measures, there is also the empty check of the queriedPhrases with the AND clause. Hence, this check seems to be alright. > On Nov. 4, 2016, 5:45 a.m., Amareshwari Sriramadasu wrote: > > lens-cube/src/test/java/org/apache/lens/cube/parse/TestCubeRewriter.java, > > line 939 > > <https://reviews.apache.org/r/52514/diff/4/?file=1552906#file1552906line939> > > > > Why didn't the query fail here ? If the test is to check no failure on > > partial data, can you update testname and comment above. Yes, The test is to check no failure on partial data as the flag FAIL_QUERY_ON_PARTIAL_DATA is not set. Will update the test name and add appropriate comment. > On Nov. 4, 2016, 5:45 a.m., Amareshwari Sriramadasu wrote: > > lens-cube/src/test/java/org/apache/lens/cube/parse/TestCubeRewriter.java, > > line 973 > > <https://reviews.apache.org/r/52514/diff/4/?file=1552906#file1552906line973> > > > > Can you share the reason being shown up, as a comment? Sure...will add the comment. > On Nov. 4, 2016, 5:45 a.m., Amareshwari Sriramadasu wrote: > > lens-cube/src/main/java/org/apache/lens/cube/parse/StorageTableResolver.java, > > line 596 > > <https://reviews.apache.org/r/52514/diff/4/?file=1552900#file1552900line596> > > > > Shall we rename variable to incompletePartition, instead of > > partitionCompleteness ? Will rename it. > On Nov. 4, 2016, 5:45 a.m., Amareshwari Sriramadasu wrote: > > lens-cube/src/main/java/org/apache/lens/cube/parse/StorageTableResolver.java, > > lines 565-566 > > <https://reviews.apache.org/r/52514/diff/4/?file=1552900#file1552900line565> > > > > Why do you need this formatter? And why is formatting done upto hours? Actually we can use the Date as well. Agreed, If we use formattter, we should give formatting upto seconds. - Narayan ------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/52514/#review154865 --- On Nov. 3, 2016, 11:13 a.m., Narayan Periwal wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/52514/ > --- > > (Updated Nov. 3, 2016, 11:13 a.m.) > > > Review request for lens. > > > Bugs: LENS-1333 > https://issues.apache.org/jira/browse/LENS-1333 > > > Repository: lens > > > Description > --- > > Though lens has partition registration being done whenever data is available, > there is no guarantee the partition registered is complete. There can be > different ways to know if the data is complete for partition. One option > could be to have a partition property saying whether it is complete or not. > Other could be to do a http call to another hosted service and more. > > Proposal here is to add an interface for DataCompletenessChecker and do the > check while resolving partitions. > > Here are some of the capabilities we would like to add in Lens : > # Lens will check partition existence first, if it exists, then check the > completeness percentage. If the completeness percentage is less than a > configured threshold (default should be 98, 99 or even 100), Lens will fail > the query. > # Lens's accept query on partial data will accept on incomplete data as well. > # Lens will also option to override the completeness percentage threshold > value at query level > # Lens will still have look ahead capability of daily being incomplete, then > it will union with hourly. > # If same measure is there in two different facts , Lens will we pick the one > with higher availability. > # In case of completeness percentage threshold missed, Lens will respond back > with available percentage. > > > Diffs > - > > lens-cube/src/main/java/org/apache/lens/cube/metadata/CubeFactTable.java > fb958c3 > > lens-cube/src/main/java/org/apache/lens/cube/metadata/CubeMetastoreClient.java > e14c43f > > lens-cube/src/main/java/org/apache/lens/cube/metadata/MetastoreConstants.java > 4585ef7 > lens-cub
[jira] [Updated] (LENS-1333) Add data completeness checker
[ https://issues.apache.org/jira/browse/LENS-1333?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Narayan Periwal updated LENS-1333: -- Status: Patch Available (was: In Progress) > Add data completeness checker > - > > Key: LENS-1333 > URL: https://issues.apache.org/jira/browse/LENS-1333 > Project: Apache Lens > Issue Type: New Feature > Components: cube >Reporter: Amareshwari Sriramadasu > Assignee: Narayan Periwal > Attachments: LENS-1333-v1.patch, LENS-1333-v2.patch, > LENS-1333-v3.patch > > > Though lens has partition registration being done whenever data is available, > there is no guarantee the partition registered is complete. There can be > different ways to know if the data is complete for partition. One option > could be to have a partition property saying whether it is complete or not. > Other could be to do a http call to another hosted service and more. > Proposal here is to add an interface for DataCompletenessChecker and do the > check while resolving partitions. > Here are some of the capabilities we would like to add in Lens : > # Lens will check partition existence first, if it exists, then check the > completeness percentage. If the completeness percentage is less than a > configured threshold (default should be 98, 99 or even 100), Lens will fail > the query. > # Lens's accept query on partial data will accept on incomplete data as well. > # Lens will also option to override the completeness percentage threshold > value at query level > # Lens will still have look ahead capability of daily being incomplete, then > it will union with hourly. > # If same measure is there in two different facts , Lens will we pick the one > with higher availability. > # In case of completeness percentage threshold missed, Lens will respond back > with available percentage. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Work started] (LENS-1333) Add data completeness checker
[ https://issues.apache.org/jira/browse/LENS-1333?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on LENS-1333 started by Narayan Periwal. - > Add data completeness checker > - > > Key: LENS-1333 > URL: https://issues.apache.org/jira/browse/LENS-1333 > Project: Apache Lens > Issue Type: New Feature > Components: cube >Reporter: Amareshwari Sriramadasu > Assignee: Narayan Periwal > Attachments: LENS-1333-v1.patch, LENS-1333-v2.patch, > LENS-1333-v3.patch > > > Though lens has partition registration being done whenever data is available, > there is no guarantee the partition registered is complete. There can be > different ways to know if the data is complete for partition. One option > could be to have a partition property saying whether it is complete or not. > Other could be to do a http call to another hosted service and more. > Proposal here is to add an interface for DataCompletenessChecker and do the > check while resolving partitions. > Here are some of the capabilities we would like to add in Lens : > # Lens will check partition existence first, if it exists, then check the > completeness percentage. If the completeness percentage is less than a > configured threshold (default should be 98, 99 or even 100), Lens will fail > the query. > # Lens's accept query on partial data will accept on incomplete data as well. > # Lens will also option to override the completeness percentage threshold > value at query level > # Lens will still have look ahead capability of daily being incomplete, then > it will union with hourly. > # If same measure is there in two different facts , Lens will we pick the one > with higher availability. > # In case of completeness percentage threshold missed, Lens will respond back > with available percentage. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (LENS-1333) Add data completeness checker
[ https://issues.apache.org/jira/browse/LENS-1333?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Narayan Periwal updated LENS-1333: -- Attachment: LENS-1333-v3.patch > Add data completeness checker > - > > Key: LENS-1333 > URL: https://issues.apache.org/jira/browse/LENS-1333 > Project: Apache Lens > Issue Type: New Feature > Components: cube >Reporter: Amareshwari Sriramadasu > Assignee: Narayan Periwal > Attachments: LENS-1333-v1.patch, LENS-1333-v2.patch, > LENS-1333-v3.patch > > > Though lens has partition registration being done whenever data is available, > there is no guarantee the partition registered is complete. There can be > different ways to know if the data is complete for partition. One option > could be to have a partition property saying whether it is complete or not. > Other could be to do a http call to another hosted service and more. > Proposal here is to add an interface for DataCompletenessChecker and do the > check while resolving partitions. > Here are some of the capabilities we would like to add in Lens : > # Lens will check partition existence first, if it exists, then check the > completeness percentage. If the completeness percentage is less than a > configured threshold (default should be 98, 99 or even 100), Lens will fail > the query. > # Lens's accept query on partial data will accept on incomplete data as well. > # Lens will also option to override the completeness percentage threshold > value at query level > # Lens will still have look ahead capability of daily being incomplete, then > it will union with hourly. > # If same measure is there in two different facts , Lens will we pick the one > with higher availability. > # In case of completeness percentage threshold missed, Lens will respond back > with available percentage. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (LENS-1333) Add data completeness checker
[ https://issues.apache.org/jira/browse/LENS-1333?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Narayan Periwal updated LENS-1333: -- Status: Open (was: Patch Available) > Add data completeness checker > - > > Key: LENS-1333 > URL: https://issues.apache.org/jira/browse/LENS-1333 > Project: Apache Lens > Issue Type: New Feature > Components: cube >Reporter: Amareshwari Sriramadasu > Assignee: Narayan Periwal > Attachments: LENS-1333-v1.patch, LENS-1333-v2.patch > > > Though lens has partition registration being done whenever data is available, > there is no guarantee the partition registered is complete. There can be > different ways to know if the data is complete for partition. One option > could be to have a partition property saying whether it is complete or not. > Other could be to do a http call to another hosted service and more. > Proposal here is to add an interface for DataCompletenessChecker and do the > check while resolving partitions. > Here are some of the capabilities we would like to add in Lens : > # Lens will check partition existence first, if it exists, then check the > completeness percentage. If the completeness percentage is less than a > configured threshold (default should be 98, 99 or even 100), Lens will fail > the query. > # Lens's accept query on partial data will accept on incomplete data as well. > # Lens will also option to override the completeness percentage threshold > value at query level > # Lens will still have look ahead capability of daily being incomplete, then > it will union with hourly. > # If same measure is there in two different facts , Lens will we pick the one > with higher availability. > # In case of completeness percentage threshold missed, Lens will respond back > with available percentage. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: Review Request 52514: Add data completeness checker
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/52514/ --- (Updated Nov. 3, 2016, 11:13 a.m.) Review request for lens. Bugs: LENS-1333 https://issues.apache.org/jira/browse/LENS-1333 Repository: lens Description --- Though lens has partition registration being done whenever data is available, there is no guarantee the partition registered is complete. There can be different ways to know if the data is complete for partition. One option could be to have a partition property saying whether it is complete or not. Other could be to do a http call to another hosted service and more. Proposal here is to add an interface for DataCompletenessChecker and do the check while resolving partitions. Here are some of the capabilities we would like to add in Lens : # Lens will check partition existence first, if it exists, then check the completeness percentage. If the completeness percentage is less than a configured threshold (default should be 98, 99 or even 100), Lens will fail the query. # Lens's accept query on partial data will accept on incomplete data as well. # Lens will also option to override the completeness percentage threshold value at query level # Lens will still have look ahead capability of daily being incomplete, then it will union with hourly. # If same measure is there in two different facts , Lens will we pick the one with higher availability. # In case of completeness percentage threshold missed, Lens will respond back with available percentage. Diffs (updated) - lens-cube/src/main/java/org/apache/lens/cube/metadata/CubeFactTable.java fb958c3 lens-cube/src/main/java/org/apache/lens/cube/metadata/CubeMetastoreClient.java e14c43f lens-cube/src/main/java/org/apache/lens/cube/metadata/MetastoreConstants.java 4585ef7 lens-cube/src/main/java/org/apache/lens/cube/parse/CandidateFact.java 5dc9dc9 lens-cube/src/main/java/org/apache/lens/cube/parse/CandidateTablePruneCause.java 78fb21d lens-cube/src/main/java/org/apache/lens/cube/parse/CubeQueryConfUtil.java f20f105 lens-cube/src/main/java/org/apache/lens/cube/parse/DefaultChecker.java PRE-CREATION lens-cube/src/main/java/org/apache/lens/cube/parse/MaxCoveringFactResolver.java 13f1aa4 lens-cube/src/main/java/org/apache/lens/cube/parse/StorageTableResolver.java db26718 lens-cube/src/main/resources/olap-query-conf.xml 0c888ca lens-cube/src/test/java/org/apache/lens/cube/parse/CubeTestSetup.java 0c43cb5 lens-cube/src/test/java/org/apache/lens/cube/parse/FieldsCannotBeQueriedTogetherTest.java 7afa32e lens-cube/src/test/java/org/apache/lens/cube/parse/MockCompletenessChecker.java PRE-CREATION lens-cube/src/test/java/org/apache/lens/cube/parse/TestBaseCubeQueries.java f6cec1b lens-cube/src/test/java/org/apache/lens/cube/parse/TestCubeRewriter.java b90d4d3 lens-cube/src/test/java/org/apache/lens/cube/parse/TestQueryRewrite.java 0aa31f4 lens-server/src/main/resources/lenssession-default.xml 9a8f9da src/site/apt/admin/session-config.apt d480f88 src/site/apt/user/olap-query-conf.apt ce35ec4 Diff: https://reviews.apache.org/r/52514/diff/ Testing --- Thanks, Narayan Periwal
Re: Review Request 52514: Add data completeness checker
> On Oct. 21, 2016, 9:06 a.m., Amareshwari Sriramadasu wrote: > > lens-cube/src/test/java/org/apache/lens/cube/parse/DefaultCompletenessChecker.java, > > line 25 > > <https://reviews.apache.org/r/52514/diff/3/?file=1529514#file1529514line25> > > > > Instead of calling it DefaultCompletenessChecker,please call Mock or > > something else. Will rename it as MockCompletenessChecker > On Oct. 21, 2016, 9:06 a.m., Amareshwari Sriramadasu wrote: > > lens-cube/src/test/java/org/apache/lens/cube/parse/TestCubeRewriter.java, > > line 939 > > <https://reviews.apache.org/r/52514/diff/3/?file=1529517#file1529517line939> > > > > Please add appropriate asserts for the test. Will add the asserts for the hql query and the expected query. > On Oct. 21, 2016, 9:06 a.m., Amareshwari Sriramadasu wrote: > > lens-cube/src/test/java/org/apache/lens/cube/parse/TestCubeRewriter.java, > > line 950 > > <https://reviews.apache.org/r/52514/diff/3/?file=1529517#file1529517line950> > > > > Please add appropriate asserts for the test. Will add the asserts for the hql query and the expected query. > On Oct. 21, 2016, 9:06 a.m., Amareshwari Sriramadasu wrote: > > lens-cube/src/test/java/org/apache/lens/cube/parse/TestCubeRewriter.java, > > line 969 > > <https://reviews.apache.org/r/52514/diff/3/?file=1529517#file1529517line969> > > > > Not clear whats the expected and actual value here. Here we are expecting the prune cause to be the Incomplete Data. Hence to assert this, we do a substring of INCOMPLETE_PARTITION.errorFormat > On Oct. 21, 2016, 9:06 a.m., Amareshwari Sriramadasu wrote: > > lens-cube/src/main/resources/olap-query-conf.xml, line 111 > > <https://reviews.apache.org/r/52514/diff/3/?file=1529512#file1529512line111> > > > > Should we make default to 100% ? Agreed. We can give default value as 100%. > On Oct. 21, 2016, 9:06 a.m., Amareshwari Sriramadasu wrote: > > lens-cube/src/main/java/org/apache/lens/cube/parse/StorageTableResolver.java, > > line 526 > > <https://reviews.apache.org/r/52514/diff/3/?file=1529511#file1529511line526> > > > > This might skip expressions containing measures. We might have to > > iterate over queriedPhrases and look for measures in them. Will add the measures after iterating over the queriedPhrases > On Oct. 21, 2016, 9:06 a.m., Amareshwari Sriramadasu wrote: > > lens-cube/src/main/java/org/apache/lens/cube/parse/StorageTableResolver.java, > > line 118 > > <https://reviews.apache.org/r/52514/diff/3/?file=1529511#file1529511line118> > > > > Should we make default to 100% ? Also move the default value also as a > > constant to CubeQueryConfUtil . Agreed. Will move this to CubeQueryConfUtil. > On Oct. 21, 2016, 9:06 a.m., Amareshwari Sriramadasu wrote: > > lens-cube/src/main/java/org/apache/lens/cube/parse/CubeQueryConfUtil.java, > > line 49 > > <https://reviews.apache.org/r/52514/diff/3/?file=1529508#file1529508line49> > > > > Minor : > > > > Can we avoid the string concat with " + " ? Agreed. - Narayan --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/52514/#review153514 --- On Oct. 10, 2016, 7:45 a.m., Narayan Periwal wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/52514/ > --- > > (Updated Oct. 10, 2016, 7:45 a.m.) > > > Review request for lens. > > > Bugs: LENS-1333 > https://issues.apache.org/jira/browse/LENS-1333 > > > Repository: lens > > > Description > --- > > Though lens has partition registration being done whenever data is available, > there is no guarantee the partition registered is complete. There can be > different ways to know if the data is complete for partition. One option > could be to have a partition property saying whether it is complete or not. > Other could be to do a http call to another hosted service and more. > > Proposal here is to add an interface for DataCompletenessChecker and do the > check while resolving partitions. > > Here are some of the capabilities we would like to add in Lens : > # Lens will check partition existence first, if it exists, then check the > completeness percentage. If the completeness percentage is less than a > configured threshold (default should be 98, 99 or
[jira] [Updated] (LENS-1333) Add data completeness checker
[ https://issues.apache.org/jira/browse/LENS-1333?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Narayan Periwal updated LENS-1333: -- Status: Patch Available (was: In Progress) > Add data completeness checker > - > > Key: LENS-1333 > URL: https://issues.apache.org/jira/browse/LENS-1333 > Project: Apache Lens > Issue Type: New Feature > Components: cube >Reporter: Amareshwari Sriramadasu > Assignee: Narayan Periwal > Attachments: LENS-1333-v1.patch, LENS-1333-v2.patch > > > Though lens has partition registration being done whenever data is available, > there is no guarantee the partition registered is complete. There can be > different ways to know if the data is complete for partition. One option > could be to have a partition property saying whether it is complete or not. > Other could be to do a http call to another hosted service and more. > Proposal here is to add an interface for DataCompletenessChecker and do the > check while resolving partitions. > Here are some of the capabilities we would like to add in Lens : > # Lens will check partition existence first, if it exists, then check the > completeness percentage. If the completeness percentage is less than a > configured threshold (default should be 98, 99 or even 100), Lens will fail > the query. > # Lens's accept query on partial data will accept on incomplete data as well. > # Lens will also option to override the completeness percentage threshold > value at query level > # Lens will still have look ahead capability of daily being incomplete, then > it will union with hourly. > # If same measure is there in two different facts , Lens will we pick the one > with higher availability. > # In case of completeness percentage threshold missed, Lens will respond back > with available percentage. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (LENS-1333) Add data completeness checker
[ https://issues.apache.org/jira/browse/LENS-1333?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Narayan Periwal updated LENS-1333: -- Status: Patch Available (was: In Progress) > Add data completeness checker > - > > Key: LENS-1333 > URL: https://issues.apache.org/jira/browse/LENS-1333 > Project: Apache Lens > Issue Type: New Feature > Components: cube >Reporter: Amareshwari Sriramadasu > Assignee: Narayan Periwal > Attachments: LENS-1333-v1.patch, LENS-1333-v2.patch > > > Though lens has partition registration being done whenever data is available, > there is no guarantee the partition registered is complete. There can be > different ways to know if the data is complete for partition. One option > could be to have a partition property saying whether it is complete or not. > Other could be to do a http call to another hosted service and more. > Proposal here is to add an interface for DataCompletenessChecker and do the > check while resolving partitions. > Here are some of the capabilities we would like to add in Lens : > # Lens will check partition existence first, if it exists, then check the > completeness percentage. If the completeness percentage is less than a > configured threshold (default should be 98, 99 or even 100), Lens will fail > the query. > # Lens's accept query on partial data will accept on incomplete data as well. > # Lens will also option to override the completeness percentage threshold > value at query level > # Lens will still have look ahead capability of daily being incomplete, then > it will union with hourly. > # If same measure is there in two different facts , Lens will we pick the one > with higher availability. > # In case of completeness percentage threshold missed, Lens will respond back > with available percentage. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (LENS-1333) Add data completeness checker
[ https://issues.apache.org/jira/browse/LENS-1333?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Narayan Periwal updated LENS-1333: -- Status: In Progress (was: Patch Available) > Add data completeness checker > - > > Key: LENS-1333 > URL: https://issues.apache.org/jira/browse/LENS-1333 > Project: Apache Lens > Issue Type: New Feature > Components: cube >Reporter: Amareshwari Sriramadasu > Assignee: Narayan Periwal > Attachments: LENS-1333-v1.patch, LENS-1333-v2.patch > > > Though lens has partition registration being done whenever data is available, > there is no guarantee the partition registered is complete. There can be > different ways to know if the data is complete for partition. One option > could be to have a partition property saying whether it is complete or not. > Other could be to do a http call to another hosted service and more. > Proposal here is to add an interface for DataCompletenessChecker and do the > check while resolving partitions. > Here are some of the capabilities we would like to add in Lens : > # Lens will check partition existence first, if it exists, then check the > completeness percentage. If the completeness percentage is less than a > configured threshold (default should be 98, 99 or even 100), Lens will fail > the query. > # Lens's accept query on partial data will accept on incomplete data as well. > # Lens will also option to override the completeness percentage threshold > value at query level > # Lens will still have look ahead capability of daily being incomplete, then > it will union with hourly. > # If same measure is there in two different facts , Lens will we pick the one > with higher availability. > # In case of completeness percentage threshold missed, Lens will respond back > with available percentage. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (LENS-1333) Add data completeness checker
[ https://issues.apache.org/jira/browse/LENS-1333?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Narayan Periwal updated LENS-1333: -- Attachment: LENS-1333-v2.patch > Add data completeness checker > - > > Key: LENS-1333 > URL: https://issues.apache.org/jira/browse/LENS-1333 > Project: Apache Lens > Issue Type: New Feature > Components: cube >Reporter: Amareshwari Sriramadasu > Assignee: Narayan Periwal > Attachments: LENS-1333-v1.patch, LENS-1333-v2.patch > > > Though lens has partition registration being done whenever data is available, > there is no guarantee the partition registered is complete. There can be > different ways to know if the data is complete for partition. One option > could be to have a partition property saying whether it is complete or not. > Other could be to do a http call to another hosted service and more. > Proposal here is to add an interface for DataCompletenessChecker and do the > check while resolving partitions. > Here are some of the capabilities we would like to add in Lens : > # Lens will check partition existence first, if it exists, then check the > completeness percentage. If the completeness percentage is less than a > configured threshold (default should be 98, 99 or even 100), Lens will fail > the query. > # Lens's accept query on partial data will accept on incomplete data as well. > # Lens will also option to override the completeness percentage threshold > value at query level > # Lens will still have look ahead capability of daily being incomplete, then > it will union with hourly. > # If same measure is there in two different facts , Lens will we pick the one > with higher availability. > # In case of completeness percentage threshold missed, Lens will respond back > with available percentage. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (LENS-1340) Adding API for data completeness checker
[ https://issues.apache.org/jira/browse/LENS-1340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15561591#comment-15561591 ] Narayan Periwal commented on LENS-1340: --- Thanks [~prongs] > Adding API for data completeness checker > > > Key: LENS-1340 > URL: https://issues.apache.org/jira/browse/LENS-1340 > Project: Apache Lens > Issue Type: Sub-task > Components: cube > Reporter: Narayan Periwal > Assignee: Narayan Periwal > Fix For: 2.7 > > Attachments: LENS-1340-v2.patch, LENS-1340.patch > > > The data completeness API will take the factTag, startDate, endDate and > measure's list as input. > It will return the completeness percentage for each of the measures for the > given factTag and time range. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: Review Request 52514: Add data completeness checker
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/52514/ --- (Updated Oct. 10, 2016, 7:45 a.m.) Review request for lens. Bugs: LENS-1333 https://issues.apache.org/jira/browse/LENS-1333 Repository: lens Description --- Though lens has partition registration being done whenever data is available, there is no guarantee the partition registered is complete. There can be different ways to know if the data is complete for partition. One option could be to have a partition property saying whether it is complete or not. Other could be to do a http call to another hosted service and more. Proposal here is to add an interface for DataCompletenessChecker and do the check while resolving partitions. Here are some of the capabilities we would like to add in Lens : # Lens will check partition existence first, if it exists, then check the completeness percentage. If the completeness percentage is less than a configured threshold (default should be 98, 99 or even 100), Lens will fail the query. # Lens's accept query on partial data will accept on incomplete data as well. # Lens will also option to override the completeness percentage threshold value at query level # Lens will still have look ahead capability of daily being incomplete, then it will union with hourly. # If same measure is there in two different facts , Lens will we pick the one with higher availability. # In case of completeness percentage threshold missed, Lens will respond back with available percentage. Diffs (updated) - lens-cube/src/main/java/org/apache/lens/cube/metadata/CubeFactTable.java fb958c3 lens-cube/src/main/java/org/apache/lens/cube/metadata/MetastoreConstants.java 4585ef7 lens-cube/src/main/java/org/apache/lens/cube/parse/CandidateFact.java 01265a5 lens-cube/src/main/java/org/apache/lens/cube/parse/CandidateTablePruneCause.java 78fb21d lens-cube/src/main/java/org/apache/lens/cube/parse/CubeQueryConfUtil.java 49ed5ef lens-cube/src/main/java/org/apache/lens/cube/parse/DefaultChecker.java PRE-CREATION lens-cube/src/main/java/org/apache/lens/cube/parse/MaxCoveringFactResolver.java 13f1aa4 lens-cube/src/main/java/org/apache/lens/cube/parse/StorageTableResolver.java db26718 lens-cube/src/main/resources/olap-query-conf.xml 0c888ca lens-cube/src/test/java/org/apache/lens/cube/parse/CubeTestSetup.java f7f8af2 lens-cube/src/test/java/org/apache/lens/cube/parse/DefaultCompletenessChecker.java PRE-CREATION lens-cube/src/test/java/org/apache/lens/cube/parse/FieldsCannotBeQueriedTogetherTest.java 7afa32e lens-cube/src/test/java/org/apache/lens/cube/parse/TestBaseCubeQueries.java 6fb027a lens-cube/src/test/java/org/apache/lens/cube/parse/TestCubeRewriter.java b90d4d3 lens-server/src/main/resources/lenssession-default.xml b6bfa2e src/site/apt/admin/session-config.apt c4b3c04 src/site/apt/user/olap-query-conf.apt ce35ec4 Diff: https://reviews.apache.org/r/52514/diff/ Testing --- Thanks, Narayan Periwal
[jira] [Updated] (LENS-1333) Add data completeness checker
[ https://issues.apache.org/jira/browse/LENS-1333?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Narayan Periwal updated LENS-1333: -- Status: Open (was: Patch Available) > Add data completeness checker > - > > Key: LENS-1333 > URL: https://issues.apache.org/jira/browse/LENS-1333 > Project: Apache Lens > Issue Type: New Feature > Components: cube >Reporter: Amareshwari Sriramadasu > Assignee: Narayan Periwal > Attachments: LENS-1333-v1.patch > > > Though lens has partition registration being done whenever data is available, > there is no guarantee the partition registered is complete. There can be > different ways to know if the data is complete for partition. One option > could be to have a partition property saying whether it is complete or not. > Other could be to do a http call to another hosted service and more. > Proposal here is to add an interface for DataCompletenessChecker and do the > check while resolving partitions. > Here are some of the capabilities we would like to add in Lens : > # Lens will check partition existence first, if it exists, then check the > completeness percentage. If the completeness percentage is less than a > configured threshold (default should be 98, 99 or even 100), Lens will fail > the query. > # Lens's accept query on partial data will accept on incomplete data as well. > # Lens will also option to override the completeness percentage threshold > value at query level > # Lens will still have look ahead capability of daily being incomplete, then > it will union with hourly. > # If same measure is there in two different facts , Lens will we pick the one > with higher availability. > # In case of completeness percentage threshold missed, Lens will respond back > with available percentage. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Work started] (LENS-1333) Add data completeness checker
[ https://issues.apache.org/jira/browse/LENS-1333?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on LENS-1333 started by Narayan Periwal. - > Add data completeness checker > - > > Key: LENS-1333 > URL: https://issues.apache.org/jira/browse/LENS-1333 > Project: Apache Lens > Issue Type: New Feature > Components: cube >Reporter: Amareshwari Sriramadasu > Assignee: Narayan Periwal > Attachments: LENS-1333-v1.patch > > > Though lens has partition registration being done whenever data is available, > there is no guarantee the partition registered is complete. There can be > different ways to know if the data is complete for partition. One option > could be to have a partition property saying whether it is complete or not. > Other could be to do a http call to another hosted service and more. > Proposal here is to add an interface for DataCompletenessChecker and do the > check while resolving partitions. > Here are some of the capabilities we would like to add in Lens : > # Lens will check partition existence first, if it exists, then check the > completeness percentage. If the completeness percentage is less than a > configured threshold (default should be 98, 99 or even 100), Lens will fail > the query. > # Lens's accept query on partial data will accept on incomplete data as well. > # Lens will also option to override the completeness percentage threshold > value at query level > # Lens will still have look ahead capability of daily being incomplete, then > it will union with hourly. > # If same measure is there in two different facts , Lens will we pick the one > with higher availability. > # In case of completeness percentage threshold missed, Lens will respond back > with available percentage. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (LENS-1333) Add data completeness checker
[ https://issues.apache.org/jira/browse/LENS-1333?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Narayan Periwal updated LENS-1333: -- Attachment: LENS-1333-v1.patch > Add data completeness checker > - > > Key: LENS-1333 > URL: https://issues.apache.org/jira/browse/LENS-1333 > Project: Apache Lens > Issue Type: New Feature > Components: cube >Reporter: Amareshwari Sriramadasu > Assignee: Narayan Periwal > Attachments: LENS-1333-v1.patch > > > Though lens has partition registration being done whenever data is available, > there is no guarantee the partition registered is complete. There can be > different ways to know if the data is complete for partition. One option > could be to have a partition property saying whether it is complete or not. > Other could be to do a http call to another hosted service and more. > Proposal here is to add an interface for DataCompletenessChecker and do the > check while resolving partitions. > Here are some of the capabilities we would like to add in Lens : > # Lens will check partition existence first, if it exists, then check the > completeness percentage. If the completeness percentage is less than a > configured threshold (default should be 98, 99 or even 100), Lens will fail > the query. > # Lens's accept query on partial data will accept on incomplete data as well. > # Lens will also option to override the completeness percentage threshold > value at query level > # Lens will still have look ahead capability of daily being incomplete, then > it will union with hourly. > # If same measure is there in two different facts , Lens will we pick the one > with higher availability. > # In case of completeness percentage threshold missed, Lens will respond back > with available percentage. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (LENS-1333) Add data completeness checker
[ https://issues.apache.org/jira/browse/LENS-1333?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Narayan Periwal updated LENS-1333: -- Status: Patch Available (was: In Progress) > Add data completeness checker > - > > Key: LENS-1333 > URL: https://issues.apache.org/jira/browse/LENS-1333 > Project: Apache Lens > Issue Type: New Feature > Components: cube >Reporter: Amareshwari Sriramadasu > Assignee: Narayan Periwal > Attachments: LENS-1333-v1.patch > > > Though lens has partition registration being done whenever data is available, > there is no guarantee the partition registered is complete. There can be > different ways to know if the data is complete for partition. One option > could be to have a partition property saying whether it is complete or not. > Other could be to do a http call to another hosted service and more. > Proposal here is to add an interface for DataCompletenessChecker and do the > check while resolving partitions. > Here are some of the capabilities we would like to add in Lens : > # Lens will check partition existence first, if it exists, then check the > completeness percentage. If the completeness percentage is less than a > configured threshold (default should be 98, 99 or even 100), Lens will fail > the query. > # Lens's accept query on partial data will accept on incomplete data as well. > # Lens will also option to override the completeness percentage threshold > value at query level > # Lens will still have look ahead capability of daily being incomplete, then > it will union with hourly. > # If same measure is there in two different facts , Lens will we pick the one > with higher availability. > # In case of completeness percentage threshold missed, Lens will respond back > with available percentage. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: Review Request 52514: Add data completeness checker
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/52514/ --- (Updated Oct. 6, 2016, 10:45 a.m.) Review request for lens. Bugs: LENS-1333 https://issues.apache.org/jira/browse/LENS-1333 Repository: lens Description --- Though lens has partition registration being done whenever data is available, there is no guarantee the partition registered is complete. There can be different ways to know if the data is complete for partition. One option could be to have a partition property saying whether it is complete or not. Other could be to do a http call to another hosted service and more. Proposal here is to add an interface for DataCompletenessChecker and do the check while resolving partitions. Here are some of the capabilities we would like to add in Lens : # Lens will check partition existence first, if it exists, then check the completeness percentage. If the completeness percentage is less than a configured threshold (default should be 98, 99 or even 100), Lens will fail the query. # Lens's accept query on partial data will accept on incomplete data as well. # Lens will also option to override the completeness percentage threshold value at query level # Lens will still have look ahead capability of daily being incomplete, then it will union with hourly. # If same measure is there in two different facts , Lens will we pick the one with higher availability. # In case of completeness percentage threshold missed, Lens will respond back with available percentage. Diffs (updated) - lens-cube/src/main/java/org/apache/lens/cube/metadata/CubeFactTable.java fb958c3 lens-cube/src/main/java/org/apache/lens/cube/metadata/MetastoreConstants.java 4585ef7 lens-cube/src/main/java/org/apache/lens/cube/parse/CandidateFact.java 01265a5 lens-cube/src/main/java/org/apache/lens/cube/parse/CandidateTablePruneCause.java 78fb21d lens-cube/src/main/java/org/apache/lens/cube/parse/CubeQueryConfUtil.java 49ed5ef lens-cube/src/main/java/org/apache/lens/cube/parse/DefaultChecker.java PRE-CREATION lens-cube/src/main/java/org/apache/lens/cube/parse/MaxCoveringFactResolver.java 13f1aa4 lens-cube/src/main/java/org/apache/lens/cube/parse/StorageTableResolver.java db26718 lens-cube/src/main/resources/olap-query-conf.xml 0c888ca lens-cube/src/test/java/org/apache/lens/cube/parse/CubeTestSetup.java f7f8af2 lens-cube/src/test/java/org/apache/lens/cube/parse/DefaultCompletenessChecker.java PRE-CREATION lens-cube/src/test/java/org/apache/lens/cube/parse/FieldsCannotBeQueriedTogetherTest.java 7afa32e lens-cube/src/test/java/org/apache/lens/cube/parse/TestCubeRewriter.java b90d4d3 lens-server/src/main/resources/lenssession-default.xml b6bfa2e Diff: https://reviews.apache.org/r/52514/diff/ Testing --- Thanks, Narayan Periwal
Re: Review Request 52514: Add data completeness checker
> On Oct. 5, 2016, 6:49 a.m., Amareshwari Sriramadasu wrote: > > .gitignore, line 40 > > <https://reviews.apache.org/r/52514/diff/1/?file=1522015#file1522015line40> > > > > Can you avoid this change? I will revert this change. > On Oct. 5, 2016, 6:49 a.m., Amareshwari Sriramadasu wrote: > > lens-cube/src/main/java/org/apache/lens/cube/parse/CandidateTablePruneCause.java, > > line 120 > > <https://reviews.apache.org/r/52514/diff/1/?file=1522019#file1522019line120> > > > > Should be added after MISSING_PARTITIONS. > > > > Also lets change the text to "Data is incomplete. Details : %s" Done. On Oct. 5, 2016, 6:49 a.m., Narayan Periwal wrote: > > Can you add testcases for all capabilities listed in description? Adding test cases for all the capabilities listed. - Narayan --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/52514/#review151465 --- On Oct. 4, 2016, 2:10 p.m., Narayan Periwal wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/52514/ > --- > > (Updated Oct. 4, 2016, 2:10 p.m.) > > > Review request for lens. > > > Bugs: LENS-1333 > https://issues.apache.org/jira/browse/LENS-1333 > > > Repository: lens > > > Description > --- > > Though lens has partition registration being done whenever data is available, > there is no guarantee the partition registered is complete. There can be > different ways to know if the data is complete for partition. One option > could be to have a partition property saying whether it is complete or not. > Other could be to do a http call to another hosted service and more. > > Proposal here is to add an interface for DataCompletenessChecker and do the > check while resolving partitions. > > Here are some of the capabilities we would like to add in Lens : > # Lens will check partition existence first, if it exists, then check the > completeness percentage. If the completeness percentage is less than a > configured threshold (default should be 98, 99 or even 100), Lens will fail > the query. > # Lens's accept query on partial data will accept on incomplete data as well. > # Lens will also option to override the completeness percentage threshold > value at query level > # Lens will still have look ahead capability of daily being incomplete, then > it will union with hourly. > # If same measure is there in two different facts , Lens will we pick the one > with higher availability. > # In case of completeness percentage threshold missed, Lens will respond back > with available percentage. > > > Diffs > - > > .gitignore aca87dc > lens-cube/src/main/java/org/apache/lens/cube/metadata/CubeFactTable.java > fb958c3 > > lens-cube/src/main/java/org/apache/lens/cube/metadata/MetastoreConstants.java > 4585ef7 > lens-cube/src/main/java/org/apache/lens/cube/parse/CandidateFact.java > 01265a5 > > lens-cube/src/main/java/org/apache/lens/cube/parse/CandidateTablePruneCause.java > 78fb21d > lens-cube/src/main/java/org/apache/lens/cube/parse/CubeQueryConfUtil.java > 49ed5ef > lens-cube/src/main/java/org/apache/lens/cube/parse/DefaultChecker.java > PRE-CREATION > > lens-cube/src/main/java/org/apache/lens/cube/parse/MaxCoveringFactResolver.java > 13f1aa4 > > lens-cube/src/main/java/org/apache/lens/cube/parse/StorageTableResolver.java > db26718 > lens-cube/src/test/java/org/apache/lens/cube/parse/CubeTestSetup.java > f7f8af2 > > lens-cube/src/test/java/org/apache/lens/cube/parse/DefaultCompletenessChecker.java > PRE-CREATION > > lens-cube/src/test/java/org/apache/lens/cube/parse/FieldsCannotBeQueriedTogetherTest.java > 7afa32e > > Diff: https://reviews.apache.org/r/52514/diff/ > > > Testing > --- > > > Thanks, > > Narayan Periwal > >
[jira] [Updated] (LENS-1340) Adding API for data completeness checker
[ https://issues.apache.org/jira/browse/LENS-1340?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Narayan Periwal updated LENS-1340: -- Status: Patch Available (was: In Progress) > Adding API for data completeness checker > > > Key: LENS-1340 > URL: https://issues.apache.org/jira/browse/LENS-1340 > Project: Apache Lens > Issue Type: Sub-task > Components: cube > Reporter: Narayan Periwal > Assignee: Narayan Periwal > Fix For: 2.7 > > Attachments: LENS-1340-v2.patch, LENS-1340.patch > > > The data completeness API will take the factTag, startDate, endDate and > measure's list as input. > It will return the completeness percentage for each of the measures for the > given factTag and time range. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Review Request 52514: Add data completeness checker
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/52514/ --- Review request for lens. Bugs: LENS-1333 https://issues.apache.org/jira/browse/LENS-1333 Repository: lens Description --- Though lens has partition registration being done whenever data is available, there is no guarantee the partition registered is complete. There can be different ways to know if the data is complete for partition. One option could be to have a partition property saying whether it is complete or not. Other could be to do a http call to another hosted service and more. Proposal here is to add an interface for DataCompletenessChecker and do the check while resolving partitions. Here are some of the capabilities we would like to add in Lens : # Lens will check partition existence first, if it exists, then check the completeness percentage. If the completeness percentage is less than a configured threshold (default should be 98, 99 or even 100), Lens will fail the query. # Lens's accept query on partial data will accept on incomplete data as well. # Lens will also option to override the completeness percentage threshold value at query level # Lens will still have look ahead capability of daily being incomplete, then it will union with hourly. # If same measure is there in two different facts , Lens will we pick the one with higher availability. # In case of completeness percentage threshold missed, Lens will respond back with available percentage. Diffs - .gitignore aca87dc lens-cube/src/main/java/org/apache/lens/cube/metadata/CubeFactTable.java fb958c3 lens-cube/src/main/java/org/apache/lens/cube/metadata/MetastoreConstants.java 4585ef7 lens-cube/src/main/java/org/apache/lens/cube/parse/CandidateFact.java 01265a5 lens-cube/src/main/java/org/apache/lens/cube/parse/CandidateTablePruneCause.java 78fb21d lens-cube/src/main/java/org/apache/lens/cube/parse/CubeQueryConfUtil.java 49ed5ef lens-cube/src/main/java/org/apache/lens/cube/parse/DefaultChecker.java PRE-CREATION lens-cube/src/main/java/org/apache/lens/cube/parse/MaxCoveringFactResolver.java 13f1aa4 lens-cube/src/main/java/org/apache/lens/cube/parse/StorageTableResolver.java db26718 lens-cube/src/test/java/org/apache/lens/cube/parse/CubeTestSetup.java f7f8af2 lens-cube/src/test/java/org/apache/lens/cube/parse/DefaultCompletenessChecker.java PRE-CREATION lens-cube/src/test/java/org/apache/lens/cube/parse/FieldsCannotBeQueriedTogetherTest.java 7afa32e Diff: https://reviews.apache.org/r/52514/diff/ Testing --- Thanks, Narayan Periwal
[jira] [Work started] (LENS-1333) Add data completeness checker
[ https://issues.apache.org/jira/browse/LENS-1333?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on LENS-1333 started by Narayan Periwal. - > Add data completeness checker > - > > Key: LENS-1333 > URL: https://issues.apache.org/jira/browse/LENS-1333 > Project: Apache Lens > Issue Type: New Feature > Components: cube >Reporter: Amareshwari Sriramadasu > Assignee: Narayan Periwal > > Though lens has partition registration being done whenever data is available, > there is no guarantee the partition registered is complete. There can be > different ways to know if the data is complete for partition. One option > could be to have a partition property saying whether it is complete or not. > Other could be to do a http call to another hosted service and more. > Proposal here is to add an interface for DataCompletenessChecker and do the > check while resolving partitions. > Here are some of the capabilities we would like to add in Lens : > # Lens will check partition existence first, if it exists, then check the > completeness percentage. If the completeness percentage is less than a > configured threshold (default should be 98, 99 or even 100), Lens will fail > the query. > # Lens's accept query on partial data will accept on incomplete data as well. > # Lens will also option to override the completeness percentage threshold > value at query level > # Lens will still have look ahead capability of daily being incomplete, then > it will union with hourly. > # If same measure is there in two different facts , Lens will we pick the one > with higher availability. > # In case of completeness percentage threshold missed, Lens will respond back > with available percentage. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (LENS-1340) Adding API for data completeness checker
[ https://issues.apache.org/jira/browse/LENS-1340?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Narayan Periwal updated LENS-1340: -- Status: In Progress (was: Patch Available) > Adding API for data completeness checker > > > Key: LENS-1340 > URL: https://issues.apache.org/jira/browse/LENS-1340 > Project: Apache Lens > Issue Type: Sub-task > Components: cube > Reporter: Narayan Periwal > Assignee: Narayan Periwal > Fix For: 2.7 > > Attachments: LENS-1340-v2.patch, LENS-1340.patch > > > The data completeness API will take the factTag, startDate, endDate and > measure's list as input. > It will return the completeness percentage for each of the measures for the > given factTag and time range. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (LENS-1340) Adding API for data completeness checker
[ https://issues.apache.org/jira/browse/LENS-1340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15544865#comment-15544865 ] Narayan Periwal commented on LENS-1340: --- [~amareshwari], resolved the checkstyle issues and submitted the new patch. > Adding API for data completeness checker > > > Key: LENS-1340 > URL: https://issues.apache.org/jira/browse/LENS-1340 > Project: Apache Lens > Issue Type: Sub-task > Components: cube > Reporter: Narayan Periwal > Assignee: Narayan Periwal > Fix For: 2.7 > > Attachments: LENS-1340-v2.patch, LENS-1340.patch > > > The data completeness API will take the factTag, startDate, endDate and > measure's list as input. > It will return the completeness percentage for each of the measures for the > given factTag and time range. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (LENS-1340) Adding API for data completeness checker
[ https://issues.apache.org/jira/browse/LENS-1340?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Narayan Periwal updated LENS-1340: -- Attachment: LENS-1340-v2.patch > Adding API for data completeness checker > > > Key: LENS-1340 > URL: https://issues.apache.org/jira/browse/LENS-1340 > Project: Apache Lens > Issue Type: Sub-task > Components: cube > Reporter: Narayan Periwal > Assignee: Narayan Periwal > Fix For: 2.7 > > Attachments: LENS-1340-v2.patch, LENS-1340.patch > > > The data completeness API will take the factTag, startDate, endDate and > measure's list as input. > It will return the completeness percentage for each of the measures for the > given factTag and time range. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (LENS-1340) Adding API for data completeness checker
[ https://issues.apache.org/jira/browse/LENS-1340?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Narayan Periwal updated LENS-1340: -- Status: Patch Available (was: Reopened) > Adding API for data completeness checker > > > Key: LENS-1340 > URL: https://issues.apache.org/jira/browse/LENS-1340 > Project: Apache Lens > Issue Type: Sub-task > Components: cube > Reporter: Narayan Periwal > Assignee: Narayan Periwal > Fix For: 2.7 > > Attachments: LENS-1340-v2.patch, LENS-1340.patch > > > The data completeness API will take the factTag, startDate, endDate and > measure's list as input. > It will return the completeness percentage for each of the measures for the > given factTag and time range. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (LENS-1340) Adding API for data completeness checker
[ https://issues.apache.org/jira/browse/LENS-1340?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Narayan Periwal updated LENS-1340: -- Status: Patch Available (was: In Progress) > Adding API for data completeness checker > > > Key: LENS-1340 > URL: https://issues.apache.org/jira/browse/LENS-1340 > Project: Apache Lens > Issue Type: Sub-task > Components: cube > Reporter: Narayan Periwal > Assignee: Narayan Periwal > Attachments: LENS-1340.patch > > > The data completeness API will take the factTag, startDate, endDate and > measure's list as input. > It will return the completeness percentage for each of the measures for the > given factTag and time range. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (LENS-1340) Adding API for data completeness checker
[ https://issues.apache.org/jira/browse/LENS-1340?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Narayan Periwal updated LENS-1340: -- Attachment: LENS-1340.patch > Adding API for data completeness checker > > > Key: LENS-1340 > URL: https://issues.apache.org/jira/browse/LENS-1340 > Project: Apache Lens > Issue Type: Sub-task > Components: cube > Reporter: Narayan Periwal > Assignee: Narayan Periwal > Attachments: LENS-1340.patch > > > The data completeness API will take the factTag, startDate, endDate and > measure's list as input. > It will return the completeness percentage for each of the measures for the > given factTag and time range. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: Review Request 52259: Adding API for data completeness checker (LENS-1340)
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/52259/ --- (Updated Sept. 29, 2016, 10:27 a.m.) Review request for lens. Bugs: LENS-1340 https://issues.apache.org/jira/browse/LENS-1340 Repository: lens Description --- The data completeness API will take the factTag, startDate, endDate and measure's list as input. It will return the completeness percentage for each of the measures for the given factTag and time range. Diffs (updated) - lens-cube/src/main/java/org/apache/lens/cube/parse/DataCompletenessChecker.java PRE-CREATION Diff: https://reviews.apache.org/r/52259/diff/ Testing --- Thanks, Narayan Periwal
Re: Review Request 52259: Adding API for data completeness checker (LENS-1340)
> On Sept. 29, 2016, 10:02 a.m., Amareshwari Sriramadasu wrote: > > lens-cube/src/main/java/org/apache/lens/cube/parse/DataCompletenessChecker.java, > > line 30 > > <https://reviews.apache.org/r/52259/diff/4/?file=1514774#file1514774line30> > > > > Can you also describe significance of tagging here ? Added the description for significance of tags. - Narayan --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/52259/#review150832 --- On Sept. 29, 2016, 10:27 a.m., Narayan Periwal wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/52259/ > --- > > (Updated Sept. 29, 2016, 10:27 a.m.) > > > Review request for lens. > > > Bugs: LENS-1340 > https://issues.apache.org/jira/browse/LENS-1340 > > > Repository: lens > > > Description > --- > > The data completeness API will take the factTag, startDate, endDate and > measure's list as input. > It will return the completeness percentage for each of the measures for the > given factTag and time range. > > > Diffs > - > > > lens-cube/src/main/java/org/apache/lens/cube/parse/DataCompletenessChecker.java > PRE-CREATION > > Diff: https://reviews.apache.org/r/52259/diff/ > > > Testing > --- > > > Thanks, > > Narayan Periwal > >
Re: Review Request 52259: Adding API for data completeness checker (LENS-1340)
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/52259/ --- (Updated Sept. 29, 2016, 6:34 a.m.) Review request for lens. Bugs: LENS-1340 https://issues.apache.org/jira/browse/LENS-1340 Repository: lens Description --- The data completeness API will take the factTag, startDate, endDate and measure's list as input. It will return the completeness percentage for each of the measures for the given factTag and time range. Diffs (updated) - lens-cube/src/main/java/org/apache/lens/cube/parse/DataCompletenessChecker.java PRE-CREATION Diff: https://reviews.apache.org/r/52259/diff/ Testing --- Thanks, Narayan Periwal
Re: Review Request 52259: Adding API for data completeness checker (LENS-1340)
> On Sept. 29, 2016, 5:04 a.m., Amareshwari Sriramadasu wrote: > > lens-cube/src/main/java/org/apache/lens/cube/parse/DataCompletenessChecker.java, > > line 35 > > <https://reviews.apache.org/r/52259/diff/3/?file=1511766#file1511766line35> > > > > Will it be measure names here or measure tags? Will use measure tags for it. > On Sept. 29, 2016, 5:04 a.m., Amareshwari Sriramadasu wrote: > > lens-cube/src/main/java/org/apache/lens/cube/parse/DataCompletenessChecker.java, > > line 34 > > <https://reviews.apache.org/r/52259/diff/3/?file=1511766#file1511766line34> > > > > Mention end date is exclusive Will add this in the comment. > On Sept. 29, 2016, 5:04 a.m., Amareshwari Sriramadasu wrote: > > lens-cube/src/main/java/org/apache/lens/cube/parse/DataCompletenessChecker.java, > > line 32 > > <https://reviews.apache.org/r/52259/diff/3/?file=1511766#file1511766line32> > > > > Give more details about what does tag mean here. Added description for the fact tag. - Narayan --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/52259/#review150811 --- On Sept. 28, 2016, 9:25 a.m., Narayan Periwal wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/52259/ > --- > > (Updated Sept. 28, 2016, 9:25 a.m.) > > > Review request for lens. > > > Bugs: LENS-1340 > https://issues.apache.org/jira/browse/LENS-1340 > > > Repository: lens > > > Description > --- > > The data completeness API will take the factTag, startDate, endDate and > measure's list as input. > It will return the completeness percentage for each of the measures for the > given factTag and time range. > > > Diffs > - > > > lens-cube/src/main/java/org/apache/lens/cube/parse/DataCompletenessChecker.java > PRE-CREATION > > Diff: https://reviews.apache.org/r/52259/diff/ > > > Testing > --- > > > Thanks, > > Narayan Periwal > >
Re: Review Request 52259: Adding API for data completeness checker (LENS-1340)
> On Sept. 28, 2016, 8:59 a.m., Amareshwari Sriramadasu wrote: > > lens-cube/src/main/java/org/apache/lens/cube/parse/DataCompletenessChecker.java, > > line 34 > > <https://reviews.apache.org/r/52259/diff/2/?file=1510054#file1510054line34> > > > > Please add java for all method params, describing what they are. Added documentation for all method params. - Narayan --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/52259/#review150695 --- On Sept. 28, 2016, 9:25 a.m., Narayan Periwal wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/52259/ > --- > > (Updated Sept. 28, 2016, 9:25 a.m.) > > > Review request for lens. > > > Bugs: LENS-1340 > https://issues.apache.org/jira/browse/LENS-1340 > > > Repository: lens > > > Description > --- > > The data completeness API will take the factTag, startDate, endDate and > measure's list as input. > It will return the completeness percentage for each of the measures for the > given factTag and time range. > > > Diffs > - > > > lens-cube/src/main/java/org/apache/lens/cube/parse/DataCompletenessChecker.java > PRE-CREATION > > Diff: https://reviews.apache.org/r/52259/diff/ > > > Testing > --- > > > Thanks, > > Narayan Periwal > >
Re: Review Request 52259: Adding API for data completeness checker (LENS-1340)
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/52259/ --- (Updated Sept. 28, 2016, 9:25 a.m.) Review request for lens. Bugs: LENS-1340 https://issues.apache.org/jira/browse/LENS-1340 Repository: lens Description --- The data completeness API will take the factTag, startDate, endDate and measure's list as input. It will return the completeness percentage for each of the measures for the given factTag and time range. Diffs (updated) - lens-cube/src/main/java/org/apache/lens/cube/parse/DataCompletenessChecker.java PRE-CREATION Diff: https://reviews.apache.org/r/52259/diff/ Testing --- Thanks, Narayan Periwal
Re: Review Request 52259: Adding API for data completeness checker (LENS-1340)
> On Sept. 26, 2016, 1:19 p.m., Rajat Khandelwal wrote: > > lens-cube/src/main/java/org/apache/lens/cube/parse/DataCompletenessChecker.java, > > line 32 > > <https://reviews.apache.org/r/52259/diff/2/?file=1510054#file1510054line32> > > > > What interval are the keys in the map spaced over? > > > > Perhaps we need an UpdatePeriod argument? > > > > Also, do keep in mind that end date should be exclusive. > > > > And, if we are passing start, end and UpdatePeriod, it might be better > > to use `org.apache.lens.cube.metadata.TimePartitionRange` Keys are spaced over hourly intervals, this is fixed. Sure, will take care of this fact that end date is exclusive. - Narayan --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/52259/#review150396 ----------- On Sept. 26, 2016, 1:03 p.m., Narayan Periwal wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/52259/ > --- > > (Updated Sept. 26, 2016, 1:03 p.m.) > > > Review request for lens. > > > Bugs: LENS-1340 > https://issues.apache.org/jira/browse/LENS-1340 > > > Repository: lens > > > Description > --- > > The data completeness API will take the factTag, startDate, endDate and > measure's list as input. > It will return the completeness percentage for each of the measures for the > given factTag and time range. > > > Diffs > - > > > lens-cube/src/main/java/org/apache/lens/cube/parse/DataCompletenessChecker.java > PRE-CREATION > > Diff: https://reviews.apache.org/r/52259/diff/ > > > Testing > --- > > > Thanks, > > Narayan Periwal > >
Re: Review Request 52259: Adding API for data completeness checker (LENS-1340)
> On Sept. 26, 2016, 12:44 p.m., Rajat Khandelwal wrote: > > lens-cube/src/main/java/org/apache/lens/cube/parse/DataCompletenessChecker.java, > > line 29 > > <https://reviews.apache.org/r/52259/diff/1/?file=1510017#file1510017line29> > > > > Can you add documentation? e.g. What is the key `String` and what is > > supposed to be in the map. Added the Documentation. - Narayan --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/52259/#review150390 ------- On Sept. 26, 2016, 1:03 p.m., Narayan Periwal wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/52259/ > --- > > (Updated Sept. 26, 2016, 1:03 p.m.) > > > Review request for lens. > > > Bugs: LENS-1340 > https://issues.apache.org/jira/browse/LENS-1340 > > > Repository: lens > > > Description > --- > > The data completeness API will take the factTag, startDate, endDate and > measure's list as input. > It will return the completeness percentage for each of the measures for the > given factTag and time range. > > > Diffs > - > > > lens-cube/src/main/java/org/apache/lens/cube/parse/DataCompletenessChecker.java > PRE-CREATION > > Diff: https://reviews.apache.org/r/52259/diff/ > > > Testing > --- > > > Thanks, > > Narayan Periwal > >
Re: Review Request 52259: Adding API for data completeness checker (LENS-1340)
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/52259/ --- (Updated Sept. 26, 2016, 1:03 p.m.) Review request for lens. Bugs: LENS-1340 https://issues.apache.org/jira/browse/LENS-1340 Repository: lens Description --- The data completeness API will take the factTag, startDate, endDate and measure's list as input. It will return the completeness percentage for each of the measures for the given factTag and time range. Diffs (updated) - lens-cube/src/main/java/org/apache/lens/cube/parse/DataCompletenessChecker.java PRE-CREATION Diff: https://reviews.apache.org/r/52259/diff/ Testing --- Thanks, Narayan Periwal
[jira] [Commented] (LENS-1340) Adding API for data completeness checker
[ https://issues.apache.org/jira/browse/LENS-1340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15522813#comment-15522813 ] Narayan Periwal commented on LENS-1340: --- The review request is https://reviews.apache.org/r/52259/ > Adding API for data completeness checker > > > Key: LENS-1340 > URL: https://issues.apache.org/jira/browse/LENS-1340 > Project: Apache Lens > Issue Type: Sub-task > Components: cube > Reporter: Narayan Periwal > Assignee: Narayan Periwal > > The data completeness API will take the factTag, startDate, endDate and > measure's list as input. > It will return the completeness percentage for each of the measures for the > given factTag and time range. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Work started] (LENS-1340) Adding API for data completeness checker
[ https://issues.apache.org/jira/browse/LENS-1340?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on LENS-1340 started by Narayan Periwal. - > Adding API for data completeness checker > > > Key: LENS-1340 > URL: https://issues.apache.org/jira/browse/LENS-1340 > Project: Apache Lens > Issue Type: Sub-task > Components: cube > Reporter: Narayan Periwal > Assignee: Narayan Periwal > > The data completeness API will take the factTag, startDate, endDate and > measure's list as input. > It will return the completeness percentage for each of the measures for the > given factTag and time range. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Review Request 52259: Adding API for data completeness checker (LENS-1340)
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/52259/ --- Review request for lens. Bugs: LENS-1340 https://issues.apache.org/jira/browse/LENS-1340 Repository: lens Description --- The data completeness API will take the factTag, startDate, endDate and measure's list as input. It will return the completeness percentage for each of the measures for the given factTag and time range. Diffs - lens-cube/src/main/java/org/apache/lens/cube/parse/DataCompletenessChecker.java PRE-CREATION Diff: https://reviews.apache.org/r/52259/diff/ Testing --- Thanks, Narayan Periwal
[jira] [Created] (LENS-1340) Adding API for data completeness checker
Narayan Periwal created LENS-1340: - Summary: Adding API for data completeness checker Key: LENS-1340 URL: https://issues.apache.org/jira/browse/LENS-1340 Project: Apache Lens Issue Type: Sub-task Components: cube Reporter: Narayan Periwal Assignee: Narayan Periwal The data completeness API will take the factTag, startDate, endDate and measure's list as input. It will return the completeness percentage for each of the measures for the given factTag and time range. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
