[jira] [Assigned] (HAWQ-1098) build error when "configure --prefix" with different directory without running "make distclean" previously

2016-10-12 Thread Ming LI (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HAWQ-1098?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ming LI reassigned HAWQ-1098:
-

Assignee: Ming LI  (was: Lei Chang)

> build error when "configure --prefix" with different directory without 
> running "make distclean" previously
> --
>
> Key: HAWQ-1098
> URL: https://issues.apache.org/jira/browse/HAWQ-1098
> Project: Apache HAWQ
>  Issue Type: Bug
>  Components: Build
>Reporter: Ming LI
>Assignee: Ming LI
> Fix For: backlog
>
>
> Some customer report at:
> http://stackoverflow.com/questions/39217467/hawq-installation-on-redhat
> If "configure --prefix" with different directory without running "make 
> distclean" previously, it will report building error:
> {code}
> ld: warning: directory not found for option 
> '-L/Users/gpadmin/workspace/hawq2/apache-hawq/depends/libhdfs3/build/install/Users/gpadmin/workspace/hawq2/hawq-db-devel3/lib'
> ld: warning: directory not found for option 
> '-L/Users/gpadmin/workspace/hawq2/apache-hawq/depends/libyarn/build/install/Users/gpadmin/workspace/hawq2/hawq-db-devel3/lib'
> ld: library not found for -lhdfs3
> clang: error: linker command failed with exit code 1 (use -v to see 
> invocation)
> make[2]: *** [postgres] Error 1
> make[1]: *** [all] Error 2
> make: *** [all] Error 2
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-hawq pull request #960: HAWQ-1098. Fixed building error when "conf...

2016-10-12 Thread liming01 
GitHub user liming01 opened a pull request:

https://github.com/apache/incubator-hawq/pull/960

HAWQ-1098. Fixed building error when "configure --prefix=" different …

…directory without running "make distclean"

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/liming01/incubator-hawq mli/HAWQ-1098

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-hawq/pull/960.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #960


commit cb01cb5871dd66a6f1e0161f0b78ae785e0ed609
Author: Ming LI 
Date:   2016-10-13T05:37:46Z

HAWQ-1098. Fixed building error when "configure --prefix=" different 
directory without running "make distclean"




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (HAWQ-1098) build error when "configure --prefix" with different directory without running "make distclean" previously

2016-10-12 Thread Ming LI (JIRA) 
Ming LI created HAWQ-1098:
-

 Summary: build error when "configure --prefix" with different 
directory without running "make distclean" previously
 Key: HAWQ-1098
 URL: https://issues.apache.org/jira/browse/HAWQ-1098
 Project: Apache HAWQ
  Issue Type: Bug
  Components: Build
Reporter: Ming LI
Assignee: Lei Chang
 Fix For: backlog


Some customer report at:
http://stackoverflow.com/questions/39217467/hawq-installation-on-redhat

If "configure --prefix" with different directory without running "make 
distclean" previously, it will report building error:
{code}
ld: warning: directory not found for option 
'-L/Users/gpadmin/workspace/hawq2/apache-hawq/depends/libhdfs3/build/install/Users/gpadmin/workspace/hawq2/hawq-db-devel3/lib'
ld: warning: directory not found for option 
'-L/Users/gpadmin/workspace/hawq2/apache-hawq/depends/libyarn/build/install/Users/gpadmin/workspace/hawq2/hawq-db-devel3/lib'
ld: library not found for -lhdfs3
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make[2]: *** [postgres] Error 1
make[1]: *** [all] Error 2
make: *** [all] Error 2
{code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-hawq pull request #959: HAWQ-1084. Fixed memory allocation for tab...

2016-10-12 Thread sansanichfb 
Github user sansanichfb commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/959#discussion_r83113822
  
--- Diff: src/bin/psql/describe.c ---
@@ -4316,10 +4316,10 @@ describePxfTable(const char *profile, const char 
*pattern, bool verbose)
if (verbose)
{
sourcefieldtype = PQgetvalue(res, i, 4);
-   total_fields = PQgetvalue(res, i, 5);
+   total_fields = atoi(PQgetvalue(res, i, 5));
--- End diff --

No, atoi() doesn't guarantee a positive value, it just takes char, assumes 
it's a int value and converts it to int value. The SQL `SELECT t.path, 
t.itemname, t.fieldname, t.fieldtype, sourcefieldtype, COUNT() OVER(PARTITION 
BY path, itemname) as total_fields FROM pxf_get_item_fields('Hive', '*') t` 
guarantees that total_fields will be >=0. I would say we could fix function 
pg_local_calloc so it should take only positive value.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq pull request #958: HAWQ-1048. Support OR, NOT logical operato...

2016-10-12 Thread kavinderd 
Github user kavinderd commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/958#discussion_r83111789
  
--- Diff: src/backend/access/external/pxffilters.c ---
@@ -31,14 +31,14 @@
 #include "utils/guc.h"
 #include "utils/lsyscache.h"
 
-static List* pxf_make_filter_list(List* quals);
+static List* pxf_make_expression_items_list(List *quals, Node *parent, 
bool *logicalOpsNum);
 static void pxf_free_filter(PxfFilterDesc* filter);
-static void pxf_free_filter_list(List *filters);
 static char* pxf_serialize_filter_list(List *filters);
 static bool opexpr_to_pxffilter(OpExpr *expr, PxfFilterDesc *filter);
 static bool supported_filter_type(Oid type);
 static void const_to_str(Const *constval, StringInfo buf);
 static List* append_attr_from_var(Var* var, List* attrs);
+static void enrichTrivialExpression(List *expressionItems);
--- End diff --

change function signature to match style with underscores 
`enrich_trivial_expression`


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Assigned] (HAWQ-1084) Sometimes psql crashes with "out of memory" on \d hcatalog.* command

2016-10-12 Thread Oleksandr Diachenko (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HAWQ-1084?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Oleksandr Diachenko reassigned HAWQ-1084:
-

Assignee: Oleksandr Diachenko  (was: Vineet Goel)

> Sometimes psql crashes with "out of memory" on \d hcatalog.* command
> 
>
> Key: HAWQ-1084
> URL: https://issues.apache.org/jira/browse/HAWQ-1084
> Project: Apache HAWQ
>  Issue Type: Bug
>  Components: PXF
>Reporter: Oleksandr Diachenko
>Assignee: Oleksandr Diachenko
> Fix For: backlog
>
>
> Normally it returns definition of all Hive tables:
> {code}
> PXF Hive Table "default.hive_abc"
>  Column |  Type  
> +
>  t0 | text
>  num1   | int4
>  d1 | float8
>  t1 | text
> PXF Hive Table "default.hive_abc222"
>  Column |  Type  
> +
>  d1 | float8
>  t0 | text
>  num1   | int4
>  t1 | text
> PXF Hive Table "default.hive_avro_table"
>  Column |  Type  
> +
>  t0 | text
>  d1 | float8
>  num1   | int4
>  t1 | text
> PXF Hive Table "default.hive_binary"
>  Column | Type  
> +---
>  b1 | bytea
> PXF Hive Table "default.hive_collections_table"
>  Column |  Type  
> +
>  ut1| text
>  sr1| text
>  m1 | text
>  a1 | text
>  s1 | text
>  f1 | float4
> PXF Hive Table "default.hive_many_partitioned_table"
>  Column |   Type
> +---
>  s1 | text
>  s2 | text
>  n1 | int4
>  d1 | float8
>  dc1| numeric
>  tm | timestamp
>  f  | float4
>  bg | int8
>  b  | bool
>  tn | int2
>  sml| int2
>  dt | date
>  vc1| varchar
>  c1 | bpchar
>  bin| bytea
> PXF Hive Table "default.hive_orc_all_types"
>  Column |   Type
> +---
>  bin| bytea
>  c1 | bpchar
>  vc1| varchar
>  dt | date
>  sml| int2
>  tn | int2
>  b  | bool
>  bg | int8
>  f  | float4
>  tm | timestamp
>  dc1| numeric
>  d1 | float8
>  n1 | int4
>  s2 | text
>  s1 | text
> PXF Hive Table "default.hive_orc_snappy"
>  Column |  Type  
> +
>  t0 | text
>  t1 | text
>  num1   | int4
>  d1 | float8
> PXF Hive Table "default.hive_orc_table"
>  Column |  Type  
> +
>  d1 | float8
>  num1   | int4
>  t1 | text
>  t0 | text
> PXF Hive Table "default.hive_orc_zlib"
>  Column |  Type  
> +
>  t0 | text
>  t1 | text
>  num1   | int4
>  d1 | float8
> PXF Hive Table "default.hive_parquet_table"
>  Column |  Type  
> +
>  d1 | float8
>  num1   | int4
>  t1 | text
>  t0 | text
> PXF Hive Table "default.hive_partitioned_clustered_sorted_table"
>  Column |  Type  
> +
>  fmt| text
>  d1 | float8
>  num1   | int4
>  t1 | text
>  t0 | text
> PXF Hive Table "default.hive_partitioned_clustered_table"
>  Column |  Type  
> +
>  fmt| text
>  d1 | float8
>  num1   | int4
>  t1 | text
>  t0 | text
> PXF Hive Table "default.hive_partitioned_skewed_stored_table"
>  Column |  Type  
> +
>  fmt| text
>  d1 | float8
>  num1   | int4
>  t1 | text
>  t0 | text
> PXF Hive Table "default.hive_partitioned_skewed_table"
>  Column |  Type  
> +
>  fmt| text
>  t0 | text
>  d1 | float8
>  num1   | int4
>  t1 | text
> PXF Hive Table "default.hive_partitioned_table"
>  Column |  Type  
> +
>  t0 | text
>  t1 | text
>  num1   | int4
>  d1 | float8
>  fmt| text
> PXF Hive Table "default.hive_rc_table"
>  Column |  Type  
> +
>  t0 | text
>  d1 | float8
>  num1   | int4
>  t1 | text
> PXF Hive Table "default.hive_rc_table_no_serde"
>  Column |  Type  
> +
>  d1 | float8
>  num1   | int4
>  t1 | text
>  t0 | text
> PXF Hive Table "default.hive_sequence_table"
>  Column |  Type  
> +
>  t0 | text
>  t1 | text
>  num1   | int4
>  d1 | float8
> PXF Hive Table "default.hive_small_data"
>  Column |  Type  
> +
>  d1 | float8
>  n1 | int4
>  s2 | text
>  s1 | text
> PXF Hive Table "default.hive_small_data_no_data_file"
>  Column |  Type  
> +
>  d1 | float8
>  n1 | int4
>  s2 | text
>  s1 | text
> PXF Hive Table "default.hive_table"
>  Column |  Type  
> +
>  s1 | text
>  n1 | int4
>  d1 | float8
>  bg | int8
>  b  | bool
> PXF Hive Table "default.hive_types"
>  Column |   Type
> +---
>  bin| bytea
>  c1 | bpchar
>  vc1| varchar
>  dt | date
>  sml| int2
>  tn | int2
>  b  | bool
>  bg | int8
>  f  |

[GitHub] incubator-hawq pull request #957: HAWQ-963. PXF support for IS_NULL and IS_N...

2016-10-12 Thread sansanichfb 
Github user sansanichfb commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/957#discussion_r83111298
  
--- Diff: 
pxf/pxf-api/src/main/java/org/apache/hawq/pxf/api/FilterParser.java ---
@@ -63,7 +63,9 @@
 HDOP_EQ,
 HDOP_NE,
 HDOP_AND,
-HDOP_LIKE
+HDOP_LIKE,
--- End diff --

Does it makes sense to separate UnaryOperation to new enum?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq pull request #957: HAWQ-963. PXF support for IS_NULL and IS_N...

2016-10-12 Thread sansanichfb 
Github user sansanichfb commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/957#discussion_r83110772
  
--- Diff: 
pxf/pxf-api/src/main/java/org/apache/hawq/pxf/api/FilterParser.java ---
@@ -63,7 +63,9 @@
 HDOP_EQ,
 HDOP_NE,
 HDOP_AND,
-HDOP_LIKE
+HDOP_LIKE,
+HDOP_ISNULL,
--- End diff --

Make them HDOP_IS_NULL, HDOP_IS_NOT_NULL?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq issue #959: HAWQ-1084. Fixed memory allocation for table prin...

2016-10-12 Thread kavinderd 
Github user kavinderd commented on the issue:

https://github.com/apache/incubator-hawq/pull/959
  
+1


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq pull request #959: HAWQ-1084. Fixed memory allocation for tab...

2016-10-12 Thread kavinderd 
Github user kavinderd commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/959#discussion_r83110560
  
--- Diff: src/bin/psql/describe.c ---
@@ -4316,10 +4316,10 @@ describePxfTable(const char *profile, const char 
*pattern, bool verbose)
if (verbose)
{
sourcefieldtype = PQgetvalue(res, i, 4);
-   total_fields = PQgetvalue(res, i, 5);
+   total_fields = atoi(PQgetvalue(res, i, 5));
--- End diff --

Ah, good catch


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq pull request #959: HAWQ-1084. Fixed memory allocation for tab...

2016-10-12 Thread sansanichfb 
Github user sansanichfb commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/959#discussion_r83110356
  
--- Diff: src/bin/psql/describe.c ---
@@ -4316,10 +4316,10 @@ describePxfTable(const char *profile, const char 
*pattern, bool verbose)
if (verbose)
{
sourcefieldtype = PQgetvalue(res, i, 4);
-   total_fields = PQgetvalue(res, i, 5);
+   total_fields = atoi(PQgetvalue(res, i, 5));
--- End diff --

PQgetvalue returns char* and if you assign it directly to int variable, it 
will be just some random int value. When this value happened to be positive, it 
would just allocate enormous chunk of memory but in case of negative 
value(mostly when you just started psql, so heap is small) it would fail with 
oom.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq pull request #959: HAWQ-1084. Fixed memory allocation for tab...

2016-10-12 Thread kavinderd 
Github user kavinderd commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/959#discussion_r83109712
  
--- Diff: src/bin/psql/describe.c ---
@@ -4316,10 +4316,10 @@ describePxfTable(const char *profile, const char 
*pattern, bool verbose)
if (verbose)
{
sourcefieldtype = PQgetvalue(res, i, 4);
-   total_fields = PQgetvalue(res, i, 5);
+   total_fields = atoi(PQgetvalue(res, i, 5));
--- End diff --

Why was an OOM being thrown without `atoi`?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq pull request #958: HAWQ-1048. Support OR, NOT logical operato...

2016-10-12 Thread sansanichfb 
Github user sansanichfb commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/958#discussion_r83109402
  
--- Diff: src/backend/access/external/pxffilters.c ---
@@ -626,16 +683,53 @@ char *serializePxfFilterQuals(List *quals)
 
if (pxf_enable_filter_pushdown)
{
-   List *filters = pxf_make_filter_list(quals);
 
-   result  = pxf_serialize_filter_list(filters);
-   pxf_free_filter_list(filters);
+   int logicalOpsNum = 0;
+   List *expressionItems = pxf_make_expression_items_list(quals, 
NULL, );
+
+   //Trivial expression means list of OpExpr implicitly ANDed
+   bool isTrivialExpression = logicalOpsNum == 0 && 
expressionItems && expressionItems->length > 1;
--- End diff --

In case of ORCA it does, but for standard optimizer it doesn't.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq pull request #959: HAWQ-1084. Fixed memory allocation for tab...

2016-10-12 Thread sansanichfb 
GitHub user sansanichfb opened a pull request:

https://github.com/apache/incubator-hawq/pull/959

HAWQ-1084. Fixed memory allocation for table printing.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/sansanichfb/incubator-hawq HAWQ-1084

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-hawq/pull/959.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #959






---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq pull request #958: HAWQ-1048. Support OR, NOT logical operato...

2016-10-12 Thread kavinderd 
Github user kavinderd commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/958#discussion_r83108994
  
--- Diff: src/backend/access/external/pxffilters.c ---
@@ -135,7 +135,40 @@ dbop_pxfop_map pxf_supported_opr[] =
{1097 /* date_gt */, PXFOP_GT},
{1096 /* date_le */, PXFOP_LE},
{1098 /* date_ge */, PXFOP_GE},
-   {1094 /* date_ne */, PXFOP_NE}
+   {1094 /* date_ne */, PXFOP_NE},
+
+   /* float8 */
+   {Float8EqualOperator  /* float8eq */, PXFOP_EQ},
+   {672 /* float8lt */, PXFOP_LT},
+   {674 /* float8gt */, PXFOP_GT},
+   {673 /* float8le */, PXFOP_LE},
+   {675 /* float8ge */, PXFOP_GE},
+   {671 /* float8ne */, PXFOP_NE},
+
+   /* float48 */
+   {1120 /* float48eq */, PXFOP_EQ},
+   {1122 /* float48lt */, PXFOP_LT},
+   {1123 /* float48gt */, PXFOP_GT},
+   {1124 /* float48le */, PXFOP_LE},
+   {1125 /* float48ge */, PXFOP_GE},
+   {1121 /* float48ne */, PXFOP_NE},
+
+   /* bpchar */
+   {BPCharEqualOperator  /* bpchareq */, PXFOP_EQ},
+   {1058  /* bpcharlt */, PXFOP_LT},
+   {1060 /* bpchargt */, PXFOP_GT},
+   {1059 /* bpcharle */, PXFOP_LE},
+   {1061 /* bpcharge */, PXFOP_GE},
+   {1057 /* bpcharne */, PXFOP_NE}
+
+   /* bytea */
+   // TODO: uncomment ocne HAWQ-1085 is done
--- End diff --

typo 'once'


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq pull request #958: HAWQ-1048. Support OR, NOT logical operato...

2016-10-12 Thread sansanichfb 
GitHub user sansanichfb opened a pull request:

https://github.com/apache/incubator-hawq/pull/958

HAWQ-1048. Support OR, NOT logical operators in the HAWQ/PXF Bridge.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/sansanichfb/incubator-hawq HAWQ-1048

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-hawq/pull/958.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #958


commit cd186f6fb3ab6bc74833ef5184da03f113ee7995
Author: Oleksandr Diachenko 
Date:   2016-09-17T02:00:50Z

HAWQ-1048. Draft implementation.

commit 7ef7e0fbfe332d28755175045b1835f5c1117e25
Author: Oleksandr Diachenko 
Date:   2016-09-19T21:05:22Z

HAWQ-1048. Added free logic.

commit 7a9cc88df6b53bbe6cb9b6e5735f08da1a651a42
Author: Oleksandr Diachenko 
Date:   2016-09-20T00:01:25Z

HAWQ-1048. Updated comments.

commit 0ff70373b3b700028892c72fd9e1fb516bbcaa25
Author: Oleksandr Diachenko 
Date:   2016-09-22T01:19:18Z

HAWQ-1048. Return NULL instead of empty string.

commit 694ad0383cce829057636365a87b92056bb52e03
Author: Oleksandr Diachenko 
Date:   2016-09-22T18:43:19Z

 HAWQ-1048. Discard whole filter if at least one operator is not supported.

commit 98bc759fe34363121049fd2f08ed6d72679082b9
Author: Oleksandr Diachenko 
Date:   2016-09-22T20:34:20Z

 HAWQ-1048. Added filter-push down support for float8.

commit d1873951dbca3928fef948044a948d5477ae7fe9
Author: Oleksandr Diachenko 
Date:   2016-09-22T23:45:57Z

 HAWQ-1048. Added filter-push down support for bpchar, bytea.

commit 9225016bdff957afb6f4d3d21c65d2daaa454b05
Author: Oleksandr Diachenko 
Date:   2016-09-22T23:46:46Z

Merge branch 'master' into HAWQ-1048

commit d563ab5b184b07b86406659ebe0abec7e4915d81
Author: Oleksandr Diachenko 
Date:   2016-09-24T01:35:41Z

HAWQ-1048. Changed logic to support more than one logical operator in list.

commit 38cb4b0040122992837c9655d52241b6b9757529
Author: Oleksandr Diachenko 
Date:   2016-09-26T19:23:32Z

HAWQ-1048. Updated log level to DEBUG1.

commit dffc760de7c1ed9ec1a40e5e94d16be202e75b97
Author: Oleksandr Diachenko 
Date:   2016-10-03T20:17:10Z

HAWQ-1048. Fixed unit-tests.

commit bbfde70d90ceece472a4c5df17761b0369b4936d
Author: Oleksandr Diachenko 
Date:   2016-10-06T21:53:18Z

HAWQ-1048. Do not send filter string on fragmenter call.

commit 2ebc98a191f4086a10f79b1dc00b4bb6934a7e5f
Author: Oleksandr Diachenko 
Date:   2016-10-06T21:55:13Z

HAWQ-1048. Do not send filter string on fragmenter call.

commit 071c300808f4f35c254f9b66e8e4ddde8e8483d4
Author: Oleksandr Diachenko 
Date:   2016-10-08T00:41:17Z

HAWQ-1048. Fixed filter builder for ORCA/default optimizer.

commit 8f03f3aba638490a49714e0afda3f003376fa768
Author: Oleksandr Diachenko 
Date:   2016-10-10T19:34:27Z

HAWQ-1048. Added support for float48.

commit d4f071af998355d680a2cf81fb2d28c3063b3734
Author: Oleksandr Diachenko 
Date:   2016-10-11T22:27:46Z

HAWQ-1048. Updated unit-tests.

commit ce6606b66d2b192dbca9e734eaab6c3fe848df37
Author: Oleksandr Diachenko 
Date:   2016-10-11T23:35:58Z

HAWQ-1048. Disabled freeing logic.

commit f44e0b2f5a9bdfcf7eaf491034d1ae850197dafe
Author: Oleksandr Diachenko 
Date:   2016-10-11T23:49:47Z

HAWQ-1048. Fixed free logic for BoolExpr nodes.

commit 0e2d341449d8217184150310bd3bf21083ef803c
Author: Oleksandr Diachenko 
Date:   2016-10-11T23:51:38Z

HAWQ-1048. Fixed free logic for BoolExpr nodes.

commit 83560c5bc73449d0dff382313d26704eb2e68742
Author: Oleksandr Diachenko 
Date:   2016-10-11T23:57:30Z

HAWQ-1048. Renamed free method.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq pull request #913: HAWQ-1048. Support OR, NOT logical operato...

2016-10-12 Thread sansanichfb 
Github user sansanichfb closed the pull request at:

https://github.com/apache/incubator-hawq/pull/913


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq pull request #906: HAWQ-964. Support for OR and NOT Logical O...

2016-10-12 Thread kavinderd 
Github user kavinderd commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/906#discussion_r83103736
  
--- Diff: 
pxf/pxf-hive/src/main/java/org/apache/hawq/pxf/plugins/hive/HiveDataFragmenter.java
 ---
@@ -404,24 +411,41 @@ private String buildFilterStringForHive() throws 
Exception {
 HiveFilterBuilder eval = new HiveFilterBuilder(inputData);
 Object filter = eval.getFilterObject(filterInput);
 
-String prefix = "";
-
-if (filter instanceof List) {
-
-for (Object f : (List) filter) {
-if (buildSingleFilter(f, filtersString, prefix)) {
-// Set 'and' operator between each matched partition 
filter.
-prefix = " and ";
-}
-}
-
+if (filter instanceof LogicalFilter) {
+buildCompoundFilter((LogicalFilter) filter, filtersString);
 } else {
-buildSingleFilter(filter, filtersString, prefix);
+buildSingleFilter(filter, filtersString, "");
 }
 
 return filtersString.toString();
 }
 
+private void buildCompoundFilter(LogicalFilter filter, StringBuilder 
filterString) throws Exception{
+String prefix;
+switch(filter.getOperator()) {
--- End diff --

Since LogicalOperation is generic, do you think we should put a very Hive 
specific string in that enum?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq pull request #906: HAWQ-964. Support for OR and NOT Logical O...

2016-10-12 Thread denalex 
Github user denalex commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/906#discussion_r83099497
  
--- Diff: 
pxf/pxf-api/src/main/java/org/apache/hawq/pxf/api/FilterParser.java ---
@@ -396,4 +387,12 @@ private Operation reverseOp(Operation operation) {
 operatorTranslationMap.put(8, Operation.HDOP_LIKE);
 return operatorTranslationMap;
 }
+
+static private Map 
initLogicalOperatorTransMap() {
+Map integerLogicalOperationMap = new 
HashMap<>();
--- End diff --

please review 
http://stackoverflow.com/questions/5292790/convert-integer-value-to-matching-java-enum
 for discussion on dangers of relying on ordinal values. I think for us, we 
might want to maintain explicit number as the private member of enum, then 
build and cache the array once for all future access.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq pull request #906: HAWQ-964. Support for OR and NOT Logical O...

2016-10-12 Thread kavinderd 
Github user kavinderd commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/906#discussion_r83100399
  
--- Diff: 
pxf/pxf-hbase/src/main/java/org/apache/hawq/pxf/plugins/hbase/HBaseFilterBuilder.java
 ---
@@ -52,12 +52,15 @@
  */
 public class HBaseFilterBuilder implements FilterParser.FilterBuilder {
 private Map 
operatorsMap;
+private Map 
logicalOperatorsMap;
 private byte[] startKey;
 private byte[] endKey;
 private HBaseTupleDescription tupleDescription;
+private static final String NOT_OP = "l2";
--- End diff --

Because HBase doesn't support NOT pushdown so we ignore any filters with 
NOT only in HBase


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq pull request #906: HAWQ-964. Support for OR and NOT Logical O...

2016-10-12 Thread kavinderd 
Github user kavinderd commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/906#discussion_r83098043
  
--- Diff: 
pxf/pxf-api/src/main/java/org/apache/hawq/pxf/api/FilterParser.java ---
@@ -396,4 +387,12 @@ private Operation reverseOp(Operation operation) {
 operatorTranslationMap.put(8, Operation.HDOP_LIKE);
 return operatorTranslationMap;
 }
+
+static private Map 
initLogicalOperatorTransMap() {
+Map integerLogicalOperationMap = new 
HashMap<>();
--- End diff --

@denalex Thanks. I had stumbled on the same thing


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq pull request #906: HAWQ-964. Support for OR and NOT Logical O...

2016-10-12 Thread denalex 
Github user denalex commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/906#discussion_r83097860
  
--- Diff: 
pxf/pxf-api/src/main/java/org/apache/hawq/pxf/api/FilterParser.java ---
@@ -396,4 +387,12 @@ private Operation reverseOp(Operation operation) {
 operatorTranslationMap.put(8, Operation.HDOP_LIKE);
 return operatorTranslationMap;
 }
+
+static private Map 
initLogicalOperatorTransMap() {
+Map integerLogicalOperationMap = new 
HashMap<>();
--- End diff --

check here: 
http://stackoverflow.com/questions/11047756/getting-enum-associated-with-int-value


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq pull request #906: HAWQ-964. Support for OR and NOT Logical O...

2016-10-12 Thread kavinderd 
Github user kavinderd commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/906#discussion_r83097937
  
--- Diff: 
pxf/pxf-api/src/main/java/org/apache/hawq/pxf/api/FilterParser.java ---
@@ -396,4 +387,12 @@ private Operation reverseOp(Operation operation) {
 operatorTranslationMap.put(8, Operation.HDOP_LIKE);
 return operatorTranslationMap;
 }
+
+static private Map 
initLogicalOperatorTransMap() {
+Map integerLogicalOperationMap = new 
HashMap<>();
--- End diff --

Nvm. Enum.values() returns an array of values


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq pull request #906: HAWQ-964. Support for OR and NOT Logical O...

2016-10-12 Thread kavinderd 
Github user kavinderd commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/906#discussion_r83097391
  
--- Diff: 
pxf/pxf-api/src/main/java/org/apache/hawq/pxf/api/FilterParser.java ---
@@ -396,4 +387,12 @@ private Operation reverseOp(Operation operation) {
 operatorTranslationMap.put(8, Operation.HDOP_LIKE);
 return operatorTranslationMap;
 }
+
+static private Map 
initLogicalOperatorTransMap() {
+Map integerLogicalOperationMap = new 
HashMap<>();
--- End diff --

I think we'll need to go with an array. I haven't found a way to do a 
reverse lookup of a Enum based on `int` values. Enum.valueOf() takes a String.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq pull request #957: HAWQ-963. PXF support for IS_NULL and IS_N...

2016-10-12 Thread denalex 
GitHub user denalex opened a pull request:

https://github.com/apache/incubator-hawq/pull/957

HAWQ-963. PXF support for IS_NULL and IS_NOT_NULL conditions



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/denalex/incubator-hawq alex-HAWQ-963

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-hawq/pull/957.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #957


commit 3f7894df71d0c8f4bf086d747b33073e00daa50d
Author: Alexander Denissov 
Date:   2016-10-12T20:46:33Z

HAWQ-963. PXF support for IS_NULL and IS_NOT_NULL conditions




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq pull request #906: HAWQ-964. Support for OR and NOT Logical O...

2016-10-12 Thread sansanichfb 
Github user sansanichfb commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/906#discussion_r83094033
  
--- Diff: 
pxf/pxf-hive/src/main/java/org/apache/hawq/pxf/plugins/hive/HiveDataFragmenter.java
 ---
@@ -404,24 +411,41 @@ private String buildFilterStringForHive() throws 
Exception {
 HiveFilterBuilder eval = new HiveFilterBuilder(inputData);
 Object filter = eval.getFilterObject(filterInput);
 
-String prefix = "";
-
-if (filter instanceof List) {
-
-for (Object f : (List) filter) {
-if (buildSingleFilter(f, filtersString, prefix)) {
-// Set 'and' operator between each matched partition 
filter.
-prefix = " and ";
-}
-}
-
+if (filter instanceof LogicalFilter) {
+buildCompoundFilter((LogicalFilter) filter, filtersString);
 } else {
-buildSingleFilter(filter, filtersString, prefix);
+buildSingleFilter(filter, filtersString, "");
 }
 
 return filtersString.toString();
 }
 
+private void buildCompoundFilter(LogicalFilter filter, StringBuilder 
filterString) throws Exception{
+String prefix;
+switch(filter.getOperator()) {
--- End diff --

I would add prefix as a member value to LogicalOperation enum and just 
invoke getter here.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq pull request #906: HAWQ-964. Support for OR and NOT Logical O...

2016-10-12 Thread sansanichfb 
Github user sansanichfb commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/906#discussion_r83092740
  
--- Diff: 
pxf/pxf-hive/src/main/java/org/apache/hawq/pxf/plugins/hive/HiveAccessor.java 
---
@@ -205,55 +217,138 @@ private boolean isFiltered(List 
partitionFields,
 return testOneFilter(partitionFields, filter, inputData);
 }
 
-/*
- * We are testing one filter against all the partition fields. The 
filter
- * has the form "fieldA = valueA". The partitions have the form
- * 
partitionOne=valueOne/partitionTwo=ValueTwo/partitionThree=valueThree 1.
- * For a filter to match one of the partitions, lets say partitionA for
- * example, we need: fieldA = partittionOne and valueA = valueOne. If 
this
- * condition occurs, we return true. 2. If fieldA does not match any 
one of
- * the partition fields we also return true, it means we ignore this 
filter
- * because it is not on a partition field. 3. If fieldA = 
partittionOne and
- * valueA != valueOne, then we return false.
- */
-private boolean testOneFilter(List partitionFields,
-  Object filter, InputData input) {
-// Let's look first at the filter
-FilterParser.BasicFilter bFilter = (FilterParser.BasicFilter) 
filter;
+private boolean testForUnsupportedOperators(List filterList) {
+boolean nonAndOp = true;
+for (Object filter : filterList) {
+if (filter instanceof LogicalFilter) {
+if (((LogicalFilter) filter).getOperator() != 
FilterParser.LogicalOperation.HDOP_AND)
+return false;
+if (((LogicalFilter) filter).getFilterList() != null)
+nonAndOp = 
testForUnsupportedOperators(((LogicalFilter) filter).getFilterList());
+}
+}
+return nonAndOp;
+}
 
-boolean isFilterOperationEqual = (bFilter.getOperation() == 
FilterParser.Operation.HDOP_EQ);
-if (!isFilterOperationEqual) /*
+private boolean testForPartitionEquality(List 
partitionFields, List filterList, InputData input) {
+boolean partitionAllowed = true;
+for (Object filter : filterList) {
+if (filter instanceof BasicFilter) {
+BasicFilter bFilter = (BasicFilter) filter;
+boolean isFilterOperationEqual = (bFilter.getOperation() 
== FilterParser.Operation.HDOP_EQ);
+if (!isFilterOperationEqual) /*
   * in case this is not an "equality 
filter"
   * we ignore it here - in partition
   * filtering
   */{
-return true;
-}
+return true;
+}
+
+int filterColumnIndex = bFilter.getColumn().index();
+String filterValue = 
bFilter.getConstant().constant().toString();
+ColumnDescriptor filterColumn = 
input.getColumn(filterColumnIndex);
+String filterColumnName = filterColumn.columnName();
 
-int filterColumnIndex = bFilter.getColumn().index();
-String filterValue = bFilter.getConstant().constant().toString();
-ColumnDescriptor filterColumn = input.getColumn(filterColumnIndex);
-String filterColumnName = filterColumn.columnName();
+for (HivePartition partition : partitionFields) {
+if (filterColumnName.equals(partition.name)) {
 
-for (HivePartition partition : partitionFields) {
-if (filterColumnName.equals(partition.name)) {
 /*
  * the filter field matches a partition field, but the 
values do
  * not match
  */
-return filterValue.equals(partition.val);
-}
-}
+boolean keepPartition = 
filterValue.equals(partition.val);
+
+/*
+ * If the string comparison fails then we should 
check the comparison of
+ * the two operands as typed values
+ */
+if (!keepPartition && 
!partition.val.equals("__HIVE_DEFAULT_PARTITION__")){
--- End diff --

Make "__HIVE_DEFAULT_PARTITION__" a constant?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at

[GitHub] incubator-hawq pull request #906: HAWQ-964. Support for OR and NOT Logical O...

2016-10-12 Thread sansanichfb 
Github user sansanichfb commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/906#discussion_r83090586
  
--- Diff: 
pxf/pxf-api/src/main/java/org/apache/hawq/pxf/api/BasicFilter.java ---
@@ -0,0 +1,37 @@
+package org.apache.hawq.pxf.api;
--- End diff --

Missing Apache header?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq pull request #906: HAWQ-964. Support for OR and NOT Logical O...

2016-10-12 Thread sansanichfb 
Github user sansanichfb commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/906#discussion_r83092607
  
--- Diff: 
pxf/pxf-hive/src/main/java/org/apache/hawq/pxf/plugins/hive/HiveAccessor.java 
---
@@ -205,55 +217,138 @@ private boolean isFiltered(List 
partitionFields,
 return testOneFilter(partitionFields, filter, inputData);
 }
 
-/*
- * We are testing one filter against all the partition fields. The 
filter
- * has the form "fieldA = valueA". The partitions have the form
- * 
partitionOne=valueOne/partitionTwo=ValueTwo/partitionThree=valueThree 1.
- * For a filter to match one of the partitions, lets say partitionA for
- * example, we need: fieldA = partittionOne and valueA = valueOne. If 
this
- * condition occurs, we return true. 2. If fieldA does not match any 
one of
- * the partition fields we also return true, it means we ignore this 
filter
- * because it is not on a partition field. 3. If fieldA = 
partittionOne and
- * valueA != valueOne, then we return false.
- */
-private boolean testOneFilter(List partitionFields,
-  Object filter, InputData input) {
-// Let's look first at the filter
-FilterParser.BasicFilter bFilter = (FilterParser.BasicFilter) 
filter;
+private boolean testForUnsupportedOperators(List filterList) {
--- End diff --

Method name is a bit confusing. Could we rename it to something like 
hasUnsupportedOperator?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq pull request #906: HAWQ-964. Support for OR and NOT Logical O...

2016-10-12 Thread denalex 
Github user denalex commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/906#discussion_r83044785
  
--- Diff: 
pxf/pxf-hive/src/main/java/org/apache/hawq/pxf/plugins/hive/HiveDataFragmenter.java
 ---
@@ -405,24 +411,41 @@ private String buildFilterStringForHive() throws 
Exception {
 HiveFilterBuilder eval = new HiveFilterBuilder(inputData);
 Object filter = eval.getFilterObject(filterInput);
 
-String prefix = "";
-
-if (filter instanceof List) {
-
-for (Object f : (List) filter) {
-if (buildSingleFilter(f, filtersString, prefix)) {
-// Set 'and' operator between each matched partition 
filter.
-prefix = " and ";
-}
-}
-
+if (filter instanceof LogicalFilter) {
+buildCompoundFilter((LogicalFilter) filter, filtersString);
 } else {
-buildSingleFilter(filter, filtersString, prefix);
+buildSingleFilter(filter, filtersString, "");
 }
 
 return filtersString.toString();
 }
 
+private void buildCompoundFilter(LogicalFilter filter, StringBuilder 
filterString) throws Exception{
+String prefix;
+switch(filter.getOperator()) {
+case HDOP_AND:
+prefix = " and ";
+break;
+case HDOP_OR:
+prefix = " or ";
+break;
+case HDOP_NOT:
+prefix = " not ";
+break;
+default:
+prefix = "";
+}
+
+for (int i = 0; i < filter.getFilterList().size(); i++) {
--- End diff --

for (Object f : filter.getFilterList() )

is more compact notation that collapses 2 lines into one


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---