[jira] [Closed] (HAWQ-1508) Fix travis broken caused by libssl path

[Hongxu Ma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HAWQ-1508?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hongxu Ma closed HAWQ-1508.
---
Resolution: Fixed

fixed

> Fix travis broken caused by libssl path
> ---
>
> Key: HAWQ-1508
> URL: https://issues.apache.org/jira/browse/HAWQ-1508
> Project: Apache HAWQ
>  Issue Type: Sub-task
>  Components: libhdfs
>Reporter: Hongxu Ma
>Assignee: Hongxu Ma
> Fix For: 2.3.0.0-incubating
>
>
> https://travis-ci.org/apache/incubator-hawq/builds/257989127?utm_source=github_status_medium=notification
> {code}
> CMake Error at 
> /usr/local/Cellar/cmake/3.6.2/share/cmake/Modules/FindPackageHandleStandardArgs.cmake:148
>  (message):
>   Could NOT find SSL (missing: SSL_INCLUDE_DIR)
> Call Stack (most recent call first):
>   
> /usr/local/Cellar/cmake/3.6.2/share/cmake/Modules/FindPackageHandleStandardArgs.cmake:388
>  (_FPHSA_FAILURE_MESSAGE)
>   CMake/FindSSL.cmake:24 (FIND_PACKAGE_HANDLE_STANDARD_ARGS)
>   CMakeLists.txt:24 (FIND_PACKAGE)
> -- Configuring incomplete, errors occurred!
> See also 
> "/Users/travis/build/apache/incubator-hawq/depends/libhdfs3/build/CMakeFiles/CMakeOutput.log".
> failed to configure the project
> make[1]: *** [pre-config] Error 1
> make: *** [all] Error 2
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (HAWQ-1509) Support TDE read function

[Hongxu Ma (JIRA)]

Hongxu Ma created HAWQ-1509:
---

 Summary: Support TDE read function
 Key: HAWQ-1509
 URL: https://issues.apache.org/jira/browse/HAWQ-1509
 Project: Apache HAWQ
  Issue Type: Sub-task
  Components: libhdfs
Reporter: Hongxu Ma
Assignee: Radar Lei
 Fix For: 2.3.0.0-incubating


Currently, we have already supported TDE write.
Then will support TDE read in this JIRA.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] incubator-hawq issue #1269: HAWQ-1506. Support multi-append a file within en...

[wengyanqing]

Github user wengyanqing commented on the issue:

https://github.com/apache/incubator-hawq/pull/1269
  
LGTM. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (HAWQ-1198) Fragmenter should return only relevant fragments for partitioned tables when X-GP-FILTER passed

[Shubham Sharma (JIRA)]

[ 
https://issues.apache.org/jira/browse/HAWQ-1198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16103334#comment-16103334
 ] 

Shubham Sharma commented on HAWQ-1198:
--

In HiveDataFragmenter while building a BasicFilter all operators apart from "=" 
are [ignored( code reference) | 
https://github.com/apache/incubator-hawq/blob/master/pxf/pxf-hive/src/main/java/org/apache/hawq/pxf/plugins/hive/HiveDataFragmenter.java#L405-#L411
 ]. This causes hive to fetch irrelevant fragments for every query that does 
not have an "equal to" operator.


[Pull request 1272 | https://github.com/apache/incubator-hawq/pull/1272] 
addresses this issue. After the changes tested few queries and filtering out 
irrelevant fragments give a 10X performance increase while querying hive 
through pxf.


Test queries

{code}
between_operator_test.sql
set client_min_messages to debug;
\timing
select * from hawq_hive where datelocal between '2016-05-27' and '2016-05-29';

in_operator_test.sql
set client_min_messages to debug;
\timing
select * from hawq_hive a where a.datelocal in 
('2016-05-29','2016-05-28','2016-05-27');

logical_operator_test.sql
set client_min_messages to debug;
\timing
select * from hawq_hive where datelocal <= '2016-05-29' and datelocal 
>='2016-05-27';

union_operator_test.sql
set client_min_messages to debug;
\timing
select count(*) from (select * from hawq_hive a where a.datelocal = 
'2016-05-29'  union all select * from hawq_hive b where 
b.datelocal='2016-05-28' union all select * from hawq_hive b where 
b.datelocal='2016-05-27' ) fs;

{code} 

Before code changes

{code}

psql -f between_operator_test.sql &> before/between_operator_test.out
psql -f union_operator_test.sql &> before/union_operator_test.out
psql -f logical_operator_test.sql &> before/logical_operator_test.out
psql -f in_operator_test.sql &> before/in_operator_test.out


[gpadmin@localhost hive_fragmenter]$ grep -i "Fragment list" before/*

Number of fragments filtered 136, should be three

before/between_operator_test.out:psql:between_operator_test.sql:3: DEBUG2:  
Fragment list: (136 elements, pxf_isilon = false)
before/between_operator_test.out:psql:between_operator_test.sql:3: DEBUG2:  
Fragment list: (136 elements, pxf_isilon = false)

before/in_operator_test.out:psql:in_operator_test.sql:3: DEBUG2:  Fragment 
list: (136 elements, pxf_isilon = false)
before/in_operator_test.out:psql:in_operator_test.sql:3: DEBUG2:  Fragment 
list: (136 elements, pxf_isilon = false)

before/logical_operator_test.out:psql:logical_operator_test.sql:3: DEBUG2:  
Fragment list: (136 elements, pxf_isilon = false)
before/logical_operator_test.out:psql:logical_operator_test.sql:3: DEBUG2:  
Fragment list: (136 elements, pxf_isilon = false)

before/union_operator_test.out:psql:union_operator_test.sql:3: DEBUG2:  
Fragment list: (1 elements, pxf_isilon = false)
before/union_operator_test.out:psql:union_operator_test.sql:3: DEBUG2:  
Fragment list: (1 elements, pxf_isilon = false)
before/union_operator_test.out:psql:union_operator_test.sql:3: DEBUG2:  
Fragment list: (1 elements, pxf_isilon = false)
before/union_operator_test.out:psql:union_operator_test.sql:3: DEBUG2:  
Fragment list: (1 elements, pxf_isilon = false)
before/union_operator_test.out:psql:union_operator_test.sql:3: DEBUG2:  
Fragment list: (1 elements, pxf_isilon = false)
before/union_operator_test.out:psql:union_operator_test.sql:3: DEBUG2:  
Fragment list: (1 elements, pxf_isilon = false)

[gpadmin@localhost hive_fragmenter]$ grep Time before/*

before/between_operator_test.out:Time: 4485.182 ms
before/in_operator_test.out:Time: 2285.578 ms
before/logical_operator_test.out:Time: 2508.315 ms
before/union_operator_test.out:Time: 609.298 ms

{code}


After code changes

{code}

[gpadmin@localhost hive_fragmenter]$ psql -f between_operator_test.sql &> 
after/between_operator_test.out
[gpadmin@localhost hive_fragmenter]$ psql -f union_operator_test.sql &> 
after/union_operator_test.out
[gpadmin@localhost hive_fragmenter]$ psql -f logical_operator_test.sql &> 
after/logical_operator_test.out
[gpadmin@localhost hive_fragmenter]$ psql -f in_operator_test.sql &> 
after/in_operator_test.out

Number of fragments filtered reduced to three except for IN operator for which 
filtering is not implemented yet.

[gpadmin@localhost hive_fragmenter]$ grep -i "Fragment list" after/*
after/between_operator_test.out:psql:between_operator_test.sql:3: DEBUG2:  
Fragment list: (3 elements, pxf_isilon = false)
after/between_operator_test.out:psql:between_operator_test.sql:3: DEBUG2:  
Fragment list: (3 elements, pxf_isilon = false)

after/in_operator_test.out:psql:in_operator_test.sql:3: DEBUG2:  Fragment list: 
(136 elements, pxf_isilon = false)
after/in_operator_test.out:psql:in_operator_test.sql:3: DEBUG2:  Fragment list: 
(136 elements, pxf_isilon = false)

after/logical_operator_test.out:psql:logical_operator_test.sql:3: DEBUG2:  
Fragment list: (3 elements, 

[GitHub] incubator-hawq pull request #1272: HAWQ-1198 - Fix to filter out irrelevant ...

[outofmem0ry]

GitHub user outofmem0ry opened a pull request:

https://github.com/apache/incubator-hawq/pull/1272

HAWQ-1198 - Fix to filter out irrelevant fragments while accessing Hive



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/outofmem0ry/incubator-hawq feature/HAWQ-1198

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-hawq/pull/1272.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1272


commit 4289b1c2a178e713b152fed705614c16341db01b
Author: Shubham Sharma 
Date:   2017-07-27T15:17:55Z

HAWQ-1198 - Fix to filter out irrelevant fragments while accesing Hive




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (HAWQ-786) Framework to support pluggable formats and file systems

[Lei Chang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HAWQ-786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16103037#comment-16103037
 ] 

Lei Chang commented on HAWQ-786:


[~rlei] This feature has been implemented in Oushu commercial hawq version. 
currently we are refactoring the code and making it work on oss hawq. After it 
is ready, we will update this JIRA. 

> Framework to support pluggable formats and file systems
> ---
>
> Key: HAWQ-786
> URL: https://issues.apache.org/jira/browse/HAWQ-786
> Project: Apache HAWQ
>  Issue Type: Wish
>  Components: Storage
>Reporter: Lei Chang
>Assignee: Lei Chang
> Fix For: backlog
>
> Attachments: ORCdesign-v0.1-2016-06-17.pdf
>
>
> In current HAWQ, two native formats are supported: AO and parquet. Now we 
> want to support ORC. A framework to support naive c/c++ pluggable format is 
> needed to support ORC more easily. And it can also be potentially used for 
> fast external data access.
> And there are a lot of requests for supporting S3, Ceph and other file 
> systems, and this is closely related to pluggable formats, so this JIRA is 
> proposing a framework to support both.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] incubator-hawq issue #1271: HAWQ-1508. Fix travis broken caused by libssl pa...

[radarwave]

Github user radarwave commented on the issue:

https://github.com/apache/incubator-hawq/pull/1271
  
Travis have been broken for days, thanks @interma to bring it back.

+1


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (HAWQ-1508) Fix travis broken caused by libssl path

[Hongxu Ma (JIRA)]

Hongxu Ma created HAWQ-1508:
---

 Summary: Fix travis broken caused by libssl path
 Key: HAWQ-1508
 URL: https://issues.apache.org/jira/browse/HAWQ-1508
 Project: Apache HAWQ
  Issue Type: Sub-task
  Components: libhdfs
Reporter: Hongxu Ma
Assignee: Radar Lei
 Fix For: 2.3.0.0-incubating


https://travis-ci.org/apache/incubator-hawq/builds/257989127?utm_source=github_status_medium=notification
{code}
CMake Error at 
/usr/local/Cellar/cmake/3.6.2/share/cmake/Modules/FindPackageHandleStandardArgs.cmake:148
 (message):

  Could NOT find SSL (missing: SSL_INCLUDE_DIR)

Call Stack (most recent call first):

  
/usr/local/Cellar/cmake/3.6.2/share/cmake/Modules/FindPackageHandleStandardArgs.cmake:388
 (_FPHSA_FAILURE_MESSAGE)

  CMake/FindSSL.cmake:24 (FIND_PACKAGE_HANDLE_STANDARD_ARGS)

  CMakeLists.txt:24 (FIND_PACKAGE)

-- Configuring incomplete, errors occurred!

See also 
"/Users/travis/build/apache/incubator-hawq/depends/libhdfs3/build/CMakeFiles/CMakeOutput.log".

failed to configure the project

make[1]: *** [pre-config] Error 1

make: *** [all] Error 2
{code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (HAWQ-1508) Fix travis broken caused by libssl path

[Hongxu Ma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HAWQ-1508?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hongxu Ma reassigned HAWQ-1508:
---

Assignee: Hongxu Ma  (was: Radar Lei)

> Fix travis broken caused by libssl path
> ---
>
> Key: HAWQ-1508
> URL: https://issues.apache.org/jira/browse/HAWQ-1508
> Project: Apache HAWQ
>  Issue Type: Sub-task
>  Components: libhdfs
>Reporter: Hongxu Ma
>Assignee: Hongxu Ma
> Fix For: 2.3.0.0-incubating
>
>
> https://travis-ci.org/apache/incubator-hawq/builds/257989127?utm_source=github_status_medium=notification
> {code}
> CMake Error at 
> /usr/local/Cellar/cmake/3.6.2/share/cmake/Modules/FindPackageHandleStandardArgs.cmake:148
>  (message):
>   Could NOT find SSL (missing: SSL_INCLUDE_DIR)
> Call Stack (most recent call first):
>   
> /usr/local/Cellar/cmake/3.6.2/share/cmake/Modules/FindPackageHandleStandardArgs.cmake:388
>  (_FPHSA_FAILURE_MESSAGE)
>   CMake/FindSSL.cmake:24 (FIND_PACKAGE_HANDLE_STANDARD_ARGS)
>   CMakeLists.txt:24 (FIND_PACKAGE)
> -- Configuring incomplete, errors occurred!
> See also 
> "/Users/travis/build/apache/incubator-hawq/depends/libhdfs3/build/CMakeFiles/CMakeOutput.log".
> failed to configure the project
> make[1]: *** [pre-config] Error 1
> make: *** [all] Error 2
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (HAWQ-1507) HAWQ unittest compile fail on SuSE

[Ivan Weng (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HAWQ-1507?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ivan Weng resolved HAWQ-1507.
-
   Resolution: Fixed
Fix Version/s: 2.3.0.0-incubating

> HAWQ unittest compile fail on SuSE
> --
>
> Key: HAWQ-1507
> URL: https://issues.apache.org/jira/browse/HAWQ-1507
> Project: Apache HAWQ
>  Issue Type: Bug
>  Components: Build
>Reporter: Ivan Weng
>Assignee: Ivan Weng
> Fix For: 2.3.0.0-incubating
>
>
> HAWQ make unittest-check fail on SuSE. Because missing -lpthread in 
> Makefile.mock



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HAWQ-786) Framework to support pluggable formats and file systems

[Radar Lei (JIRA)]

[ 
https://issues.apache.org/jira/browse/HAWQ-786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16102879#comment-16102879
 ] 

Radar Lei commented on HAWQ-786:


Hi [~lei_chang], glad to see this feature is scheduled in next release. We are 
starting to do some spike on Parquet 2.0 support, which is related with the 
pluggable storage feature your are working on. So we want to know more details 
about the interfaces between pluggable service and the parquet file format, 
this can help a lot to our spike.

As the design doc in this jira is for ORC only and might be out of date, would 
you update the design doc to include pluggable storage? Thanks a lot.

> Framework to support pluggable formats and file systems
> ---
>
> Key: HAWQ-786
> URL: https://issues.apache.org/jira/browse/HAWQ-786
> Project: Apache HAWQ
>  Issue Type: Wish
>  Components: Storage
>Reporter: Lei Chang
>Assignee: Lei Chang
> Fix For: backlog
>
> Attachments: ORCdesign-v0.1-2016-06-17.pdf
>
>
> In current HAWQ, two native formats are supported: AO and parquet. Now we 
> want to support ORC. A framework to support naive c/c++ pluggable format is 
> needed to support ORC more easily. And it can also be potentially used for 
> fast external data access.
> And there are a lot of requests for supporting S3, Ceph and other file 
> systems, and this is closely related to pluggable formats, so this JIRA is 
> proposing a framework to support both.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] incubator-hawq issue #1270: HAWQ-1507. Fix unittest-check fail on SuSE

[interma]

Github user interma commented on the issue:

https://github.com/apache/incubator-hawq/pull/1270
  
+1


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq issue #1270: HAWQ-1507. Fix unittest-check fail on SuSE

[radarwave]

Github user radarwave commented on the issue:

https://github.com/apache/incubator-hawq/pull/1270
  
LGTM +1


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq pull request #1270: HAWQ-1507. Fix unittest-check fail on SuS...

[wengyanqing]

GitHub user wengyanqing opened a pull request:

https://github.com/apache/incubator-hawq/pull/1270

HAWQ-1507. Fix unittest-check fail on SuSE



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/wengyanqing/incubator-hawq HAWQ-1507

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-hawq/pull/1270.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1270


commit 3ee258ed17291cf2ce33694e954df956194528c1
Author: Ivan 
Date:   2017-07-27T07:28:49Z

HAWQ-1507. Fix unittest-check fail on SuSE




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Assigned] (HAWQ-1507) HAWQ unittest compile fail on SuSE

[Ivan Weng (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HAWQ-1507?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ivan Weng reassigned HAWQ-1507:
---

Assignee: Ivan Weng  (was: Radar Lei)

> HAWQ unittest compile fail on SuSE
> --
>
> Key: HAWQ-1507
> URL: https://issues.apache.org/jira/browse/HAWQ-1507
> Project: Apache HAWQ
>  Issue Type: Bug
>  Components: Build
>Reporter: Ivan Weng
>Assignee: Ivan Weng
>
> HAWQ make unittest-check fail on SuSE. Because missing -lpthread in 
> Makefile.mock



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (HAWQ-1507) HAWQ unittest compile fail on SuSE

[Ivan Weng (JIRA)]

Ivan Weng created HAWQ-1507:
---

 Summary: HAWQ unittest compile fail on SuSE
 Key: HAWQ-1507
 URL: https://issues.apache.org/jira/browse/HAWQ-1507
 Project: Apache HAWQ
  Issue Type: Bug
  Components: Build
Reporter: Ivan Weng
Assignee: Radar Lei


HAWQ make unittest-check fail on SuSE. Because missing -lpthread in 
Makefile.mock



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)