[GitHub] incubator-hawq pull request #837: HAWQ-779 support pxf filter pushdwon at th...

[jiadexin]

Github user jiadexin closed the pull request at:

https://github.com/apache/incubator-hawq/pull/837


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq pull request #837: HAWQ-779 support pxf filter pushdwon at th...

[jiadexin]

Github user jiadexin commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/837#discussion_r79295676
  
--- Diff: src/backend/optimizer/plan/createplan.c ---
@@ -1144,9 +1144,15 @@ static char** create_pxf_plan(char **segdb_file_map, 
RelOptInfo *rel, int total_


Relation relation = RelationIdGetRelation(planner_rt_fetch(scan_relid, 
ctx->root)->relid);
-   segdb_work_map = map_hddata_2gp_segments(uri_str, 
+   if (pxf_enable_filter_pushdown){
--- End diff --

you are right.
my C language unskilled, so...


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq pull request #837: HAWQ-779 support pxf filter pushdwon at th...

[jiadexin]

Github user jiadexin commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/837#discussion_r79295630
  
--- Diff: src/backend/optimizer/plan/createplan.c ---
@@ -1144,9 +1144,15 @@ static char** create_pxf_plan(char **segdb_file_map, 
RelOptInfo *rel, int total_


Relation relation = RelationIdGetRelation(planner_rt_fetch(scan_relid, 
ctx->root)->relid);
-   segdb_work_map = map_hddata_2gp_segments(uri_str, 
+   if (pxf_enable_filter_pushdown){
--- End diff --

you are right.
My C language unskilled, so...


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq pull request #837: HAWQ-779 support pxf filter pushdwon at th...

[hornn]

Github user hornn commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/837#discussion_r78508860
  
--- Diff: src/backend/optimizer/plan/createplan.c ---
@@ -1144,9 +1144,15 @@ static char** create_pxf_plan(char **segdb_file_map, 
RelOptInfo *rel, int total_


Relation relation = RelationIdGetRelation(planner_rt_fetch(scan_relid, 
ctx->root)->relid);
-   segdb_work_map = map_hddata_2gp_segments(uri_str, 
+   if (pxf_enable_filter_pushdown){
--- End diff --

can be refactored:
```
map_hddata_2gp_segments(uri_str,
   total_segs, 
segs_participating,
   relation, 
   pxf_enable_filter_pushdown ? 
ctx->root->parse->jointree->quals : NULL);
```


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq pull request #837: HAWQ-779 support pxf filter pushdwon at th...

[kavinderd]

Github user kavinderd commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/837#discussion_r77047594
  
--- Diff: 
pxf/pxf-hbase/src/main/java/org/apache/hawq/pxf/plugins/hbase/HBaseFilterBuilder.java
 ---
@@ -165,6 +165,14 @@ private Filter 
handleSimpleOperations(FilterParser.Operation opId,
 ByteArrayComparable comparator = 
getComparator(hbaseColumn.columnTypeCode(),
 constant.constant());
 
+if(operatorsMap.get(opId) == null){
+//HBase not support HDOP_LIKE, use 'NOT NULL' Comarator
--- End diff --

No, @hsyuan comment was that the comment should read "//HBase does not 
support..." HBase does not support the LIKE Filter as far as I am aware


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq pull request #837: HAWQ-779 support pxf filter pushdwon at th...

[jiadexin]

Github user jiadexin commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/837#discussion_r76547017
  
--- Diff: 
pxf/pxf-hbase/src/main/java/org/apache/hawq/pxf/plugins/hbase/HBaseFilterBuilder.java
 ---
@@ -165,6 +165,14 @@ private Filter 
handleSimpleOperations(FilterParser.Operation opId,
 ByteArrayComparable comparator = 
getComparator(hbaseColumn.columnTypeCode(),
 constant.constant());
 
+if(operatorsMap.get(opId) == null){
+//HBase not support HDOP_LIKE, use 'NOT NULL' Comarator
--- End diff --

should i  develop a HBase LIKE Filter ?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq pull request #837: HAWQ-779 support pxf filter pushdwon at th...

[jiadexin]

Github user jiadexin commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/837#discussion_r76546715
  
--- Diff: 
pxf/pxf-api/src/test/java/org/apache/hawq/pxf/api/FilterParserTest.java ---
@@ -215,6 +215,10 @@ public void parseColumnOnLeft() throws Exception {
 filter = "a1c2o7";
 op = Operation.HDOP_AND;
 runParseOneOperation("this filter was build from HDOP_AND", 
filter, op);
+
+filter = "a1c2o8";
+op = Operation.HDOP_LIKE;
+runParseOneOperation("this filter was build from HDOP_LIKE", 
filter, op);
--- End diff --

This reference to the previous code _`runParseOneOperation("this filter was 
build from HDOP_AND", filter, op)`_ ,  has been corrected.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq pull request #837: HAWQ-779 support pxf filter pushdwon at th...

[hsyuan]

Github user hsyuan commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/837#discussion_r76479540
  
--- Diff: 
pxf/pxf-hbase/src/main/java/org/apache/hawq/pxf/plugins/hbase/HBaseFilterBuilder.java
 ---
@@ -165,6 +165,14 @@ private Filter 
handleSimpleOperations(FilterParser.Operation opId,
 ByteArrayComparable comparator = 
getComparator(hbaseColumn.columnTypeCode(),
 constant.constant());
 
+if(operatorsMap.get(opId) == null){
+//HBase not support HDOP_LIKE, use 'NOT NULL' Comarator
--- End diff --

HBase does not support


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq pull request #837: HAWQ-779 support pxf filter pushdwon at th...

[GodenYao]

Github user GodenYao commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/837#discussion_r76469613
  
--- Diff: 
pxf/pxf-hbase/src/main/java/org/apache/hawq/pxf/plugins/hbase/HBaseFilterBuilder.java
 ---
@@ -165,6 +165,14 @@ private Filter 
handleSimpleOperations(FilterParser.Operation opId,
 ByteArrayComparable comparator = 
getComparator(hbaseColumn.columnTypeCode(),
 constant.constant());
 
+if(operatorsMap.get(opId) == null){
+//HBase not support HDOP_LIKE, use 'NOT NULL' Comarator
--- End diff --

comparator


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq pull request #837: HAWQ-779 support pxf filter pushdwon at th...

[hsyuan]

Github user hsyuan commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/837#discussion_r76370526
  
--- Diff: 
pxf/pxf-api/src/test/java/org/apache/hawq/pxf/api/FilterParserTest.java ---
@@ -215,6 +215,10 @@ public void parseColumnOnLeft() throws Exception {
 filter = "a1c2o7";
 op = Operation.HDOP_AND;
 runParseOneOperation("this filter was build from HDOP_AND", 
filter, op);
+
+filter = "a1c2o8";
+op = Operation.HDOP_LIKE;
+runParseOneOperation("this filter was build from HDOP_LIKE", 
filter, op);
--- End diff --

build -> built


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq pull request #837: HAWQ-779 support pxf filter pushdwon at th...

[jiadexin]

Github user jiadexin commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/837#discussion_r76368428
  
--- Diff: src/backend/optimizer/plan/createplan.c ---
@@ -1146,7 +1146,7 @@ static char** create_pxf_plan(char **segdb_file_map, 
RelOptInfo *rel, int total_
Relation relation = RelationIdGetRelation(planner_rt_fetch(scan_relid, 
ctx->root)->relid);
segdb_work_map = map_hddata_2gp_segments(uri_str, 

 total_segs, segs_participating,
-   
 relation, NULL);
+   
 relation, ctx->root->parse->jointree->quals);
--- End diff --

ok.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq pull request #837: HAWQ-779 support pxf filter pushdwon at th...

[kavinderd]

Github user kavinderd commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/837#discussion_r76098154
  
--- Diff: src/backend/optimizer/plan/createplan.c ---
@@ -1146,7 +1146,7 @@ static char** create_pxf_plan(char **segdb_file_map, 
RelOptInfo *rel, int total_
Relation relation = RelationIdGetRelation(planner_rt_fetch(scan_relid, 
ctx->root)->relid);
segdb_work_map = map_hddata_2gp_segments(uri_str, 

 total_segs, segs_participating,
-   
 relation, NULL);
+   
 relation, ctx->root->parse->jointree->quals);
--- End diff --

Can you ensure that the filters are only passed when the guc 
`pxf_enable_filter_pushdown` is enabled.  Such as is done here: 
https://github.com/apache/incubator-hawq/blob/316c5048f303513c40e8194b15024df12a3d76ad/src/backend/access/external/pxffilters.c#L554
 

Currently if this guc is disabled filters should not be passed to PXF


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq pull request #837: HAWQ-779 support pxf filter pushdwon at th...

[jiadexin]

Github user jiadexin commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/837#discussion_r74380940
  
--- Diff: src/backend/access/external/test/pxffilters_test.c ---
@@ -61,7 +62,7 @@ test__supported_filter_type(void **state)
 
/* go over pxf_supported_types array */
int nargs = sizeof(pxf_supported_types) / sizeof(Oid);
-   assert_int_equal(nargs, 12);
+   assert_int_equal(nargs, 13);
--- End diff --

This test is to check pxf_supported_types number, it's old value is 
hard-coded too(be 12), after an increase of DATEOID become 13.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq pull request #837: HAWQ-779 support pxf filter pushdwon at th...

[kavinderd]

Github user kavinderd commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/837#discussion_r74286229
  
--- Diff: src/backend/access/external/test/pxffilters_test.c ---
@@ -61,7 +62,7 @@ test__supported_filter_type(void **state)
 
/* go over pxf_supported_types array */
int nargs = sizeof(pxf_supported_types) / sizeof(Oid);
-   assert_int_equal(nargs, 12);
+   assert_int_equal(nargs, 13);
--- End diff --

Maybe have this value derived instead of hard-coded number


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq pull request #837: HAWQ-779 support pxf filter pushdwon at th...

[jiadexin]

GitHub user jiadexin opened a pull request:

https://github.com/apache/incubator-hawq/pull/837

HAWQ-779 support pxf filter pushdwon at the 'CREATE PLAN' stage ,and …

1.support  pxf filter pushdwon at the 'CREATE PLAN' stage -- 
src/backend/optimizer/plan/createplan.c
2.Due to '1' causes produce HAWQ-953 error, modify HiveDataFragmenter.java
3.add 'Date type'  filter and  'HDOP_LIKE' op -- 
pxffilters.h,pxffilters.c,FilterParser.java , and update the corresponding test 
-- FilterParserTest.java,pxffilters_test.c
4.Due to '2' cause,modified HBaseFilterBuilder.java to handle 'HDOP_LIKE' 
op.
5. 'Date filter' and  'HDOP_LIKE' used in 
pxf-solr/pxf-jdbc(https://github.com/inspur-insight/pxf-plugin)
6.By this amendment, I think: 'PXF Filter' architecture coupling is too 
high, I just want to add new types and op, but had to modify other components 
.i hope to improve the architecture .



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/inspur-insight/incubator-hawq HAWQ-779

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-hawq/pull/837.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #837


commit 3518b4af22af281909140bc011884420de540cc6
Author: Devin Jia 
Date:   2016-08-05T07:05:50Z

HAWQ-779 support pxf filter pushdwon at the 'CREATE PLAN' stage ,and more 
filter type & op




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---