[jira] [Commented] (YARN-7581) ATSv2 does not construct HBase filters correctly in HBase 2.0
[
https://issues.apache.org/jira/browse/YARN-7581?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16271415#comment-16271415
]
Haibo Chen commented on YARN-7581:
--
The final HBase filter is constructed in entity readers as
A FilterList based on fields to retrieve(metricstoretrieve, configstoretrieve,
etc)
AND
A FilterList based on atsv2 filters (info/conf/event/metric filters)
Attach a preliminary patch that extracts column families that are present only
in atsv2 filters,
and adds to the fields-to-retrieve-based FilterList a FamilyFilter for each of
them.
In the case of
TestTimelineReaderWebServicesHBaseStorage.testGetEntitiesConfigFilters(),
the patch will generate a new HBase filter
{code}
FilterList AND (2/2): [
FilterList AND (1/1): [
FilterList OR (2/2): [
SingleColumnValueFilter (c, config_param1, EQUAL, "value1"),
SingleColumnValueFilter (c, config_param1, EQUAL, "value3")
]
],
FilterList OR (2/2): [
FilterList AND (5/5): [
FamilyFilter (EQUAL, i),
QualifierFilter (NOT_EQUAL, e!),
QualifierFilter (NOT_EQUAL, i!),
QualifierFilter (NOT_EQUAL, s!),
QualifierFilter (NOT_EQUAL, r!)
],
FilterList AND (1/1): [
FamilyFilter(Equal, c)
]
]
]
{code}
> ATSv2 does not construct HBase filters correctly in HBase 2.0
> -
>
> Key: YARN-7581
> URL: https://issues.apache.org/jira/browse/YARN-7581
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: ATSv2
>Affects Versions: 3.0.0-beta1
>Reporter: Haibo Chen
>Assignee: Haibo Chen
> Attachments: YARN-7581.prelim.patch
>
>
> TestTimelineReaderWebServicesHBaseStorage.testGetEntitiesConfigFilters() and
> TestTimelineReaderWebServicesHBaseStorage.testGetEntitiesMetricFilters()
> start to fail after we upgrade HBase to 2.0-alpha4 (To reproduce locally,
> apply YARN-7581.prelim.patch that is attached in YARN-7346 and run the atsv2
> unit tests)
> *Error Message*
> [ERROR] Failures:
> [ERROR]
> TestTimelineReaderWebServicesHBaseStorage.testGetEntitiesConfigFilters:1266
> expected:<2> but was:<0>
> [ERROR]
> TestTimelineReaderWebServicesHBaseStorage.testGetEntitiesMetricFilters:1523
> expected:<1> but was:<0>
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7581) ATSv2 does not construct HBase filters correctly in HBase 2.0
[
https://issues.apache.org/jira/browse/YARN-7581?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16271360#comment-16271360
]
Haibo Chen commented on YARN-7581:
--
We have the relevant data (entity 5 that is supposed to be returned in the
result) in HBase as
| Row Key | Column Family| Column Qualifier | Cell Value |
|
||-|-|
| r1| c | config_param1|
value1 |
| r1| c | config_param2|
value2 |
| r1| c | cfg_param1 |
value3 |
| r1| i | info1|
cluster1|
When given the above filter, HBase regionserver happens to evaluate
the FamilyFilter first while iterating over the cells
Quoting analysis from [~appy]
> Looks like HBASE-13122 is the culprit.
> Since we do SVCF on cf "c" and FamilyFilter on "i", earlier scanner was still
> iterating on each cell of cf "c" and at some point found the ones we were
> filtering for.
> But after the change, we skipped the whole cf "c" after seeing the first
> cell, as a result SVCF didn't see other cells and failed to match the row.
> So despite HBASE-13122 being in 1.2.6 also, the difference came because of
> this filter:
{code}
FilterList OR (1/1):[
FilterList AND (5/5):[
FamilyFilter (EQUAL, i),
QualifierFilter (NOT_EQUAL, e!),
QualifierFilter (NOT_EQUAL, i!),
QualifierFilter (NOT_EQUAL, s!),
QualifierFilter (NOT_EQUAL, r!)]]]
{code}
> In 1.2.6, filter.filterKeyValue (filter=AND) will return SKIP_ROW, break out
> of loop and return "rc" which was initialized to SKIP. So we still keep going
> with current StoreScanner.
> But in 2.0.0, filter.internalFiterCell (filter=AND) will return SKIP_ROW, and
> rc will be set to SKIP_ROW before returning. As a result we skip the
> StoreScanner and as a result, all cells in this column family.
> ATSv2 does not construct HBase filters correctly in HBase 2.0
> -
>
> Key: YARN-7581
> URL: https://issues.apache.org/jira/browse/YARN-7581
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: ATSv2
>Affects Versions: 3.0.0-beta1
>Reporter: Haibo Chen
>Assignee: Haibo Chen
>
> TestTimelineReaderWebServicesHBaseStorage.testGetEntitiesConfigFilters() and
> TestTimelineReaderWebServicesHBaseStorage.testGetEntitiesMetricFilters()
> start to fail after we upgrade HBase to 2.0-alpha4 (To reproduce locally,
> apply YARN-7581.prelim.patch that is attached in YARN-7346 and run the atsv2
> unit tests)
> *Error Message*
> [ERROR] Failures:
> [ERROR]
> TestTimelineReaderWebServicesHBaseStorage.testGetEntitiesConfigFilters:1266
> expected:<2> but was:<0>
> [ERROR]
> TestTimelineReaderWebServicesHBaseStorage.testGetEntitiesMetricFilters:1523
> expected:<1> but was:<0>
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7581) ATSv2 does not construct HBase filters correctly in HBase 2.0
[
https://issues.apache.org/jira/browse/YARN-7581?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16271320#comment-16271320
]
Haibo Chen commented on YARN-7581:
--
Based on the discussion I had with our internal HBase folks, the issue has to
do how we construct HBase Filters in ATSv2.
Specifically, in the case of
TestTimelineReaderWebServicesHBaseStorage.testGetEntitiesConfigFilters(), the
query that
is failing the test,
{code}
URI uri = URI.create("http://localhost:; + getServerPort() + "/ws/v2/" +
"timeline/clusters/cluster1/apps/application_11_/" +
"entities/type1?conffilters=config_param1%20eq%20value1%20OR%20" +
"config_param1%20eq%20value3");
{code}
generates the following HBase filter
{code}
FilterList AND (2/2): [
FilterList AND (1/1): [
FilterList OR (2/2): [
SingleColumnValueFilter (c, config_param1, EQUAL, "value1"),
SingleColumnValueFilter (c, config_param1, EQUAL, "value3")
]
],
FilterList OR (1/1): [
FilterList AND (5/5): [
FamilyFilter (EQUAL, i),
QualifierFilter (NOT_EQUAL, e!),
QualifierFilter (NOT_EQUAL, i!),
QualifierFilter (NOT_EQUAL, s!),
QualifierFilter (NOT_EQUAL, r!)
]
]
]
{code}
and when the SingleColumnValueFilter are created (in
TimelineFilterUtils.createHBaseSingleColValueFilter()),
we call setFilterIfMissing(true) so that the whole row will be skipped if SCVF
does not see config_param1.
> ATSv2 does not construct HBase filters correctly in HBase 2.0
> -
>
> Key: YARN-7581
> URL: https://issues.apache.org/jira/browse/YARN-7581
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: ATSv2
>Affects Versions: 3.0.0-beta1
>Reporter: Haibo Chen
>Assignee: Haibo Chen
>
> TestTimelineReaderWebServicesHBaseStorage.testGetEntitiesConfigFilters() and
> TestTimelineReaderWebServicesHBaseStorage.testGetEntitiesMetricFilters()
> start to fail after we upgrade HBase to 2.0-alpha4 (To reproduce locally,
> apply YARN-7581.prelim.patch that is attached in YARN-7346 and run the atsv2
> unit tests)
> *Error Message*
> [ERROR] Failures:
> [ERROR]
> TestTimelineReaderWebServicesHBaseStorage.testGetEntitiesConfigFilters:1266
> expected:<2> but was:<0>
> [ERROR]
> TestTimelineReaderWebServicesHBaseStorage.testGetEntitiesMetricFilters:1523
> expected:<1> but was:<0>
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
