[GitHub] incubator-hawq pull request #1230: HAWQ-1453. Fixed relation_close() error a...
Github user hsyuan commented on a diff in the pull request: https://github.com/apache/incubator-hawq/pull/1230#discussion_r114932353 --- Diff: src/backend/commands/analyze.c --- @@ -308,7 +308,7 @@ void analyzeStmt(VacuumStmt *stmt, List *relids, int preferred_seg_num) ListCell*le1 = NULL; int successCount = 0, failCount = 0; StringInfoData failNames; - ResourceOwner owner, oldOwner; + ResourceOwner asOwner, oldOwner1; --- End diff -- if `as` of `asOwner` means `analyzeStmt`, we'd better add comments or explanation on the both vars. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq pull request #1187: HAWQ-1408. Fixed crash when alloc not eno...
Github user hsyuan commented on a diff in the pull request: https://github.com/apache/incubator-hawq/pull/1187#discussion_r108066084 --- Diff: src/backend/access/appendonly/appendonlywriter.c --- @@ -923,30 +923,34 @@ usedByConcurrentTransaction(AOSegfileStatus *segfilestat) */ int addCandidateSegno(AOSegfileStatus **maxSegno4Seg, int segment_num, AOSegfileStatus *candidate_status, bool isKeepHash) --- End diff -- Can you give me an example? I don't get it. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq pull request #1187: HAWQ-1408. Fixed crash when alloc not eno...
Github user hsyuan commented on a diff in the pull request:
https://github.com/apache/incubator-hawq/pull/1187#discussion_r108030497
--- Diff: src/backend/access/appendonly/appendonlywriter.c ---
@@ -1136,26 +1140,22 @@ List *SetSegnoForWrite(List *existing_segnos, Oid
relid, int segment_num,
/* If the found segfile status are still no enough,
* we need to get new ones from the status pool.
*/
-if (remaining_num > 0)
-{
-//generate new segment_num to make sure that in keepHash
mode, all segment node has at least one segfile is writable
-int newAllocatedNum = remaining_num;
-for(int i= 1; i<= newAllocatedNum; i++)
-{
-int new_status = AORelGetSegfileStatus(aoentry);
-if (new_status == NEXT_END_OF_LIST)
-{
-LWLockRelease(AOSegFileLock);
-ereport(ERROR, (errmsg("cannot open more than %d
append-only table segment files cocurrently",
-MaxAORelSegFileStatus)));
-}
-AOSegfileStatusPool[new_status].segno =
++aoentry->max_seg_no;
-AOSegfileStatusPool[new_status].next =
aoentry->head_rel_segfile.next;
-aoentry->head_rel_segfile.next = new_status;
-remaining_num = addCandidateSegno(maxSegno4Segment,
segment_num, [new_status], keepHash);
-}
-Assert(remaining_num==0);//make sure all segno got a
candidate
-}
+ while(remaining_num>0)
--- End diff --
and here
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq pull request #1187: HAWQ-1408. Fixed crash when alloc not eno...
Github user hsyuan commented on a diff in the pull request:
https://github.com/apache/incubator-hawq/pull/1187#discussion_r108030444
--- Diff: src/backend/access/appendonly/appendonlywriter.c ---
@@ -923,30 +923,34 @@ usedByConcurrentTransaction(AOSegfileStatus
*segfilestat)
*/
int
addCandidateSegno(AOSegfileStatus **maxSegno4Seg, int segment_num,
AOSegfileStatus *candidate_status, bool isKeepHash)
-{
+{
+ int remaining_num = segment_num;
if(isKeepHash){
- int remaining_num = segment_num;
int idx = (candidate_status->segno+segment_num-1) %
segment_num;// from 1 to segment_num-1, then 0(segment_num)
if (NULL==maxSegno4Seg[idx] || maxSegno4Seg[idx]->segno >
candidate_status->segno) //using the min seg no firstly.
maxSegno4Seg[idx] = candidate_status;
for(int i=0; i<segment_num; i++)
{
if(NULL!=maxSegno4Seg[i] && maxSegno4Seg[i]->segno > 0){
-remaining_num--;
+ remaining_num--;
}
}
- return remaining_num;
}else{
+ bool assigned = false; // candidate_status assigned?
+ remaining_num = 0;
for(int i=0; i<segment_num; i++){
--- End diff --
```c
for(int i=0; i<segment_num; i++)
```
To
```c
for (int i = 0; i < segment_num; i++)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq pull request #1187: HAWQ-1408. Fixed crash when alloc not eno...
Github user hsyuan commented on a diff in the pull request: https://github.com/apache/incubator-hawq/pull/1187#discussion_r108030401 --- Diff: src/backend/access/appendonly/appendonlywriter.c --- @@ -923,30 +923,34 @@ usedByConcurrentTransaction(AOSegfileStatus *segfilestat) */ int addCandidateSegno(AOSegfileStatus **maxSegno4Seg, int segment_num, AOSegfileStatus *candidate_status, bool isKeepHash) --- End diff -- The indentation of this function is horrible. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq pull request #1189: HAWQ-1409. Send AGG-TYPE header to PXF
Github user hsyuan commented on a diff in the pull request:
https://github.com/apache/incubator-hawq/pull/1189#discussion_r108025770
--- Diff: src/backend/executor/nodeAgg.c ---
@@ -1950,6 +1950,21 @@ ExecInitAgg(Agg *node, EState *estate, int eflags)
* initialize child nodes
*/
outerPlan = outerPlan(node);
+ if (outerPlan->type == T_ExternalScan) {
+ /*
+* Hack to indicate to PXF when there is an external scan
+*/
+ if (aggstate->aggs && list_length(aggstate->aggs) == 1) {
+ foreach(l, aggstate->aggs) {
+ AggrefExprState *aggrefstate = (AggrefExprState
*) lfirst(l);
--- End diff --
just by changing `lfirst(l)` to `linitial(aggstate->aggs)`, you can remove
`foreach` loop.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq pull request #1187: HAWQ-1408. Fixed crash when alloc not eno...
Github user hsyuan commented on a diff in the pull request:
https://github.com/apache/incubator-hawq/pull/1187#discussion_r108025472
--- Diff: src/backend/access/appendonly/appendonlywriter.c ---
@@ -1136,26 +1140,22 @@ List *SetSegnoForWrite(List *existing_segnos, Oid
relid, int segment_num,
/* If the found segfile status are still no enough,
* we need to get new ones from the status pool.
*/
-if (remaining_num > 0)
-{
-//generate new segment_num to make sure that in keepHash
mode, all segment node has at least one segfile is writable
-int newAllocatedNum = remaining_num;
-for(int i= 1; i<= newAllocatedNum; i++)
-{
-int new_status = AORelGetSegfileStatus(aoentry);
-if (new_status == NEXT_END_OF_LIST)
-{
-LWLockRelease(AOSegFileLock);
-ereport(ERROR, (errmsg("cannot open more than %d
append-only table segment files cocurrently",
-MaxAORelSegFileStatus)));
-}
-AOSegfileStatusPool[new_status].segno =
++aoentry->max_seg_no;
-AOSegfileStatusPool[new_status].next =
aoentry->head_rel_segfile.next;
-aoentry->head_rel_segfile.next = new_status;
-remaining_num = addCandidateSegno(maxSegno4Segment,
segment_num, [new_status], keepHash);
-}
-Assert(remaining_num==0);//make sure all segno got a
candidate
-}
+ while(remaining_num>0)
+ {
+ //generate new segment_num to make sure that in
keepHash mode, all segment node has at least one segfile is writable
+ int new_status = AORelGetSegfileStatus(aoentry);
+ if (new_status == NEXT_END_OF_LIST)
+ {
+ LWLockRelease(AOSegFileLock);
+ ereport(ERROR, (errmsg("cannot open
more than %d append-only table segment files cocurrently",
--- End diff --
`cocurrently` typo
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq pull request #1187: HAWQ-1408. Fixed crash when alloc not eno...
Github user hsyuan commented on a diff in the pull request:
https://github.com/apache/incubator-hawq/pull/1187#discussion_r108025409
--- Diff: src/backend/access/appendonly/appendonlywriter.c ---
@@ -923,30 +923,34 @@ usedByConcurrentTransaction(AOSegfileStatus
*segfilestat)
*/
int
addCandidateSegno(AOSegfileStatus **maxSegno4Seg, int segment_num,
AOSegfileStatus *candidate_status, bool isKeepHash)
-{
+{
+ int remaining_num = segment_num;
--- End diff --
The space of this chunk of code is messy.
`{}` placement is not consistent, even in a single function.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq pull request #1187: HAWQ-1408. Fixed crash when alloc not eno...
Github user hsyuan commented on a diff in the pull request:
https://github.com/apache/incubator-hawq/pull/1187#discussion_r108025364
--- Diff: src/backend/access/appendonly/appendonlywriter.c ---
@@ -923,30 +923,34 @@ usedByConcurrentTransaction(AOSegfileStatus
*segfilestat)
*/
int
addCandidateSegno(AOSegfileStatus **maxSegno4Seg, int segment_num,
AOSegfileStatus *candidate_status, bool isKeepHash)
-{
+{
+ int remaining_num = segment_num;
if(isKeepHash){
- int remaining_num = segment_num;
int idx = (candidate_status->segno+segment_num-1) %
segment_num;// from 1 to segment_num-1, then 0(segment_num)
if (NULL==maxSegno4Seg[idx] || maxSegno4Seg[idx]->segno >
candidate_status->segno) //using the min seg no firstly.
maxSegno4Seg[idx] = candidate_status;
for(int i=0; i<segment_num; i++)
{
if(NULL!=maxSegno4Seg[i] && maxSegno4Seg[i]->segno > 0){
-remaining_num--;
+ remaining_num--;
}
}
- return remaining_num;
}else{
+ int assigned = false; // candidate_status assigned?
--- End diff --
Why not bool?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq pull request #1189: HAWQ-1409. Send AGG-TYPE header to PXF
Github user hsyuan commented on a diff in the pull request:
https://github.com/apache/incubator-hawq/pull/1189#discussion_r108025216
--- Diff: src/backend/executor/nodeAgg.c ---
@@ -1950,6 +1950,21 @@ ExecInitAgg(Agg *node, EState *estate, int eflags)
* initialize child nodes
*/
outerPlan = outerPlan(node);
+ if (outerPlan->type == T_ExternalScan) {
+ /*
+* Hack to indicate to PXF when there is an external scan
+*/
+ if (aggstate->aggs && list_length(aggstate->aggs) == 1) {
+ foreach(l, aggstate->aggs) {
--- End diff --
if the size of `aggstate->aggs` is already 1, do we need use `foreach` to
iterate?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq pull request #1189: HAWQ-1409. Send AGG-TYPE header to PXF
Github user hsyuan commented on a diff in the pull request:
https://github.com/apache/incubator-hawq/pull/1189#discussion_r108025159
--- Diff: src/backend/executor/nodeAgg.c ---
@@ -1950,6 +1950,21 @@ ExecInitAgg(Agg *node, EState *estate, int eflags)
* initialize child nodes
*/
outerPlan = outerPlan(node);
+ if (outerPlan->type == T_ExternalScan) {
+ /*
+* Hack to indicate to PXF when there is an external scan
+*/
+ if (aggstate->aggs && list_length(aggstate->aggs) == 1) {
--- End diff --
Do we need check `aggstate->aggs` is null or not? if it is null,
`list_length` will return 0.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq pull request #1189: HAWQ-1409. Send AGG-TYPE header to PXF
Github user hsyuan commented on a diff in the pull request:
https://github.com/apache/incubator-hawq/pull/1189#discussion_r108025134
--- Diff: src/backend/executor/nodeAgg.c ---
@@ -1950,6 +1950,21 @@ ExecInitAgg(Agg *node, EState *estate, int eflags)
* initialize child nodes
*/
outerPlan = outerPlan(node);
+ if (outerPlan->type == T_ExternalScan) {
--- End diff --
Better use `IsA(outerPlan, ExternalScan)`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq pull request #1189: HAWQ-1409. Send AGG-TYPE header to PXF
Github user hsyuan commented on a diff in the pull request:
https://github.com/apache/incubator-hawq/pull/1189#discussion_r108025072
--- Diff: src/backend/access/external/fileam.c ---
@@ -459,12 +459,20 @@ external_stopscan(FileScanDesc scan)
*
*/
ExternalSelectDesc
-external_getnext_init(PlanState *state) {
+external_getnext_init(PlanState *state, ExternalScanState *es_state) {
ExternalSelectDesc desc = (ExternalSelectDesc)
palloc0(sizeof(ExternalSelectDescData));
+ Plan *rootPlan;
if (state != NULL)
{
desc->projInfo = state->ps_ProjInfo;
+ /*
+* If we have an agg type then our parent is an Agg node
+*/
+ rootPlan = state->state->es_plannedstmt->planTree;
+ if (rootPlan->type == T_Agg && es_state->parent_agg_type) {
--- End diff --
Recommend to use `IsA(rootPlan, Agg)`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq pull request #1189: HAWQ-1409. Send AGG-TYPE header to PXF
Github user hsyuan commented on a diff in the pull request:
https://github.com/apache/incubator-hawq/pull/1189#discussion_r108025017
--- Diff: src/backend/executor/nodeAgg.c ---
@@ -1950,6 +1950,21 @@ ExecInitAgg(Agg *node, EState *estate, int eflags)
* initialize child nodes
*/
outerPlan = outerPlan(node);
+ if (outerPlan->type == T_ExternalScan) {
+ /*
+* Hack to indicate to PXF when there is an external scan
+*/
+ if (aggstate->aggs && list_length(aggstate->aggs) == 1) {
+ foreach(l, aggstate->aggs) {
+ AggrefExprState *aggrefstate = (AggrefExprState
*) lfirst(l);
+ Aggref *aggref = (Aggref *)
aggrefstate->xprstate.expr;
+ //Only dealing with one agg
+ if (aggref->aggfnoid == 2147 ||
aggref->aggfnoid == 2803) {
--- End diff --
Would be better use `COUNT_ANY_OID` and `COUNT_STAR_OID`...
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq pull request #1189: HAWQ-1409. Send AGG-TYPE header to PXF
Github user hsyuan commented on a diff in the pull request:
https://github.com/apache/incubator-hawq/pull/1189#discussion_r108024814
--- Diff: src/backend/access/external/pxfheaders.c ---
@@ -110,6 +110,20 @@ void build_http_header(PxfInputData *input)
else
churl_headers_append(headers, "X-GP-HAS-FILTER", "0");
+ /* Aggregate information */
+ if (input->agg_type) {
+ char agg_groups_str[sizeof(int32)];
--- End diff --
Where is this var used?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq pull request #1001: HAWQ-1136. Disable .psqlrc in minirepro
Github user hsyuan closed the pull request at: https://github.com/apache/incubator-hawq/pull/1001 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq issue #1001: HAWQ-1136. Disable .psqlrc in minirepro
Github user hsyuan commented on the issue: https://github.com/apache/incubator-hawq/pull/1001 @paul-guo- @linwen @liming01 Please take a look. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq pull request #1001: HAWQ-1136. Disable .psqlrc in minirepro
GitHub user hsyuan opened a pull request:
https://github.com/apache/incubator-hawq/pull/1001
HAWQ-1136. Disable .psqlrc in minirepro
.psqlrc can create unexpected output and changes in formatting that don't
play nice with parse_oids().
```
psql database --pset footer -Atq -h localhost -p 5432 -U gpadmin -f
/tmp/20161012232709/toolkit.sql
{"relids": "573615536", "funcids": ""}
Time: 2.973 ms
```
Generates an Exception:
```
Traceback (most recent call last):
File "/usr/local/greenplum-db/./bin/minirepro", line 386, in
main()
File "/usr/local/greenplum-db/./bin/minirepro", line 320, in main
mr_query = parse_oids(cursor, json_str)
File "/usr/local/greenplum-db/./bin/minirepro", line 151, in parse_oids
result.relids = json.loads(json_oids)['relids']
File "/usr/local/greenplum-db/ext/python/lib/python2.6/json/__init__.py",
line 307, in loads
return _default_decoder.decode(s)
File "/usr/local/greenplum-db/ext/python/lib/python2.6/json/decoder.py",
line 322, in decode
raise ValueError(errmsg("Extra data", s, end, len(s)))
ValueError: Extra data: line 2 column 1 - line 3 column 1 (char 39 - 54)
```
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/hsyuan/incubator-hawq HAWQ-1136
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-hawq/pull/1001.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1001
commit 4a9eb3f513fef7b1d483de46fb7462b2013864d7
Author: Haisheng Yuan <hy...@pivotal.io>
Date: 2016-11-03T11:05:14Z
HAWQ-1136. Disable .psqlrc in minirepro
.psqlrc can create unexpected output and changes in formatting that don't
play nice with parse_oids().
```
psql database --pset footer -Atq -h localhost -p 5432 -U gpadmin -f
/tmp/20161012232709/toolkit.sql
{"relids": "573615536", "funcids": ""}
Time: 2.973 ms
```
Generates an Exception:
```
Traceback (most recent call last):
File "/usr/local/greenplum-db/./bin/minirepro", line 386, in
main()
File "/usr/local/greenplum-db/./bin/minirepro", line 320, in main
mr_query = parse_oids(cursor, json_str)
File "/usr/local/greenplum-db/./bin/minirepro", line 151, in parse_oids
result.relids = json.loads(json_oids)['relids']
File "/usr/local/greenplum-db/ext/python/lib/python2.6/json/__init__.py",
line 307, in loads
return _default_decoder.decode(s)
File "/usr/local/greenplum-db/ext/python/lib/python2.6/json/decoder.py",
line 322, in decode
raise ValueError(errmsg("Extra data", s, end, len(s)))
ValueError: Extra data: line 2 column 1 - line 3 column 1 (char 39 - 54)
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq pull request #991: HAWQ-1131. Fix error message difference be...
Github user hsyuan closed the pull request at: https://github.com/apache/incubator-hawq/pull/991 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq issue #991: HAWQ-1131. Fix error message difference between O...
Github user hsyuan commented on the issue: https://github.com/apache/incubator-hawq/pull/991 Pushed to master: https://github.com/apache/incubator-hawq/commit/56b44f8595b4ba1ac9c49b775d5110e9be1275b2 https://github.com/apache/incubator-hawq/commit/31ea7d08ede7d5bc3108454731ba11bdcf1092d1 https://github.com/apache/incubator-hawq/commit/fed12f1a57ac64a8c7cab867f31b2930dddc2744 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq pull request #991: HAWQ-1131. Fix error message difference be...
GitHub user hsyuan opened a pull request: https://github.com/apache/incubator-hawq/pull/991 HAWQ-1131. Fix error message difference between Orca and Planner - In ORCA, due to the way exception handled previously we do warning first and then later print error referring that message. In this commit, we enhanced the exception handling so we just print a single error message. - Removed 'PQO unable to generate a plan' or 'Aborting PQO plan generation' message and make the error message as close as the planner. - Updated error message with filename and line number from where the exception is raised. - Updated regression test expected output files Patches are ported the following commits from GPDB: https://github.com/greenplum-db/gpdb/commit/4eb5db7a6bc8ac3faeca761193dc37e2c70e1ea3 https://github.com/greenplum-db/gpdb/commit/d41ebde141104dd3db6fa29ac46b4096ee7d4825 Author: Karthikeyan Jambu Rajaraman <karthi@gmail.com> You can merge this pull request into a Git repository by running: $ git pull https://github.com/hsyuan/incubator-hawq master Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-hawq/pull/991.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #991 commit 339c6102750f21c3d7762505b5da134b51244207 Author: Haisheng Yuan <hy...@pivotal.io> Date: 2016-11-01T07:27:24Z HAWQ-1131. Fix error message difference between Orca and Planner - In ORCA, due to the way exception handled previously we do warning first and then later print error referring that message. In this commit, we enhanced the exception handling so we just print a single error message. - Removed 'PQO unable to generate a plan' or 'Aborting PQO plan generation' message and make the error message as close as the planner. - Updated error message with filename and line number from where the exception is raised. - Updated regression test expected output files Patches are ported the following commits from GPDB: https://github.com/greenplum-db/gpdb/commit/4eb5db7a6bc8ac3faeca761193dc37e2c70e1ea3 https://github.com/greenplum-db/gpdb/commit/d41ebde141104dd3db6fa29ac46b4096ee7d4825 Author: Karthikeyan Jambu Rajaraman <karthi@gmail.com> --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq issue #896: [#126526845] Port recent changes from gpdb for ha...
Github user hsyuan commented on the issue: https://github.com/apache/incubator-hawq/pull/896 It is not a simple move. It is kind of rewrite sql into cpp test classes. Since this PR is just porting, we can do it in later changes. @liming01 Can you merge this in? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq pull request #837: HAWQ-779 support pxf filter pushdwon at th...
Github user hsyuan commented on a diff in the pull request:
https://github.com/apache/incubator-hawq/pull/837#discussion_r76479540
--- Diff:
pxf/pxf-hbase/src/main/java/org/apache/hawq/pxf/plugins/hbase/HBaseFilterBuilder.java
---
@@ -165,6 +165,14 @@ private Filter
handleSimpleOperations(FilterParser.Operation opId,
ByteArrayComparable comparator =
getComparator(hbaseColumn.columnTypeCode(),
constant.constant());
+if(operatorsMap.get(opId) == null){
+//HBase not support HDOP_LIKE, use 'NOT NULL' Comarator
--- End diff --
HBase does not support
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq issue #839: HAWQ-983. Fix bug that minirepro generates wrong ...
Github user hsyuan commented on the issue: https://github.com/apache/incubator-hawq/pull/839 Pushed to master: https://github.com/apache/incubator-hawq/commit/19124b7da032b61d9215520483b67a109246e363 https://github.com/apache/incubator-hawq/commit/08a1c6c2af705df850b1b9bbd6188d0cd2036b42 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq pull request #839: HAWQ-983. Fix bug that minirepro generates...
Github user hsyuan closed the pull request at: https://github.com/apache/incubator-hawq/pull/839 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq issue #838: HAWQ-934. Populate canSetTag of PlannedStmt from ...
Github user hsyuan commented on the issue: https://github.com/apache/incubator-hawq/pull/838 pushed to master: https://github.com/apache/incubator-hawq/commit/a17647be029a42c85e659cfc7fe253708a036bd7 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq pull request #838: HAWQ-934. Populate canSetTag of PlannedStm...
Github user hsyuan closed the pull request at: https://github.com/apache/incubator-hawq/pull/838 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq pull request #837: HAWQ-779 support pxf filter pushdwon at th...
Github user hsyuan commented on a diff in the pull request:
https://github.com/apache/incubator-hawq/pull/837#discussion_r76370526
--- Diff:
pxf/pxf-api/src/test/java/org/apache/hawq/pxf/api/FilterParserTest.java ---
@@ -215,6 +215,10 @@ public void parseColumnOnLeft() throws Exception {
filter = "a1c2o7";
op = Operation.HDOP_AND;
runParseOneOperation("this filter was build from HDOP_AND",
filter, op);
+
+filter = "a1c2o8";
+op = Operation.HDOP_LIKE;
+runParseOneOperation("this filter was build from HDOP_LIKE",
filter, op);
--- End diff --
build -> built
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq pull request #845: HAWQ-999. Treat hash table as random when ...
Github user hsyuan commented on a diff in the pull request:
https://github.com/apache/incubator-hawq/pull/845#discussion_r76369938
--- Diff: src/backend/cdb/cdbdatalocality.c ---
@@ -3791,12 +3791,27 @@ run_allocation_algorithm(SplitAllocResult *result,
List *virtual_segments, Query
targetPolicy = GpPolicyFetch(CurrentMemoryContext, myrelid);
bool isRelationHash = is_relation_hash(targetPolicy);
+ int fileCountInRelation = list_length(rel_data->files);
+ bool FileCountBucketNumMismatch = false;
+ if (targetPolicy->bucketnum > 0) {
+ FileCountBucketNumMismatch = fileCountInRelation %
--- End diff --
why not `fileCountBucketNumMismatch = fileCountInRelation %
targetPolicy->bucketnum != 0`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq pull request #845: HAWQ-999. Treat hash table as random when ...
Github user hsyuan commented on a diff in the pull request:
https://github.com/apache/incubator-hawq/pull/845#discussion_r76369662
--- Diff: src/backend/utils/misc/guc.c ---
@@ -3176,6 +3177,17 @@ static struct config_bool ConfigureNamesBool[] =
},
{
+ {"allow_file_count_bucket_num_mismatch", PGC_POSTMASTER,
CLIENT_CONN_LOCALE,
+ gettext_noop("allow hash table to be treated as random when
file count and"
--- End diff --
allow file count and bucket number mismatch by viewing hash distributed
table as randomly distributed table
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq pull request #845: HAWQ-999. Treat hash table as random when ...
Github user hsyuan commented on a diff in the pull request:
https://github.com/apache/incubator-hawq/pull/845#discussion_r76369259
--- Diff: src/backend/cdb/cdbdatalocality.c ---
@@ -3791,12 +3791,27 @@ run_allocation_algorithm(SplitAllocResult *result,
List *virtual_segments, Query
targetPolicy = GpPolicyFetch(CurrentMemoryContext, myrelid);
bool isRelationHash = is_relation_hash(targetPolicy);
+ int fileCountInRelation = list_length(rel_data->files);
+ bool FileCountBucketNumMismatch = false;
+ if (targetPolicy->bucketnum > 0) {
+ FileCountBucketNumMismatch = fileCountInRelation %
+ targetPolicy->bucketnum == 0 ? false : true;
+ }
+ if (FileCountBucketNumMismatch &&
!allow_file_count_bucket_num_mismatch) {
+ elog(ERROR, "file count %d in catalog is not in proportion to
the bucket "
+ "number %d of hash table with oid=%u, some data may be
lost, if you "
--- End diff --
number %d of distribution policy for table with oid = %u, some data may be
missing or invalid.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq pull request #845: HAWQ-999. Treat hash table as random when ...
Github user hsyuan commented on a diff in the pull request: https://github.com/apache/incubator-hawq/pull/845#discussion_r76369524 --- Diff: src/backend/utils/misc/guc.c --- @@ -450,6 +450,7 @@ char *Debug_dtm_action_protocol_str; /* Enable check for compatibility of encoding and locale in createdb */ bool gp_encoding_check_locale_compatibility; +bool allow_file_count_bucket_num_mismatch; --- End diff -- indent --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq pull request #845: HAWQ-999. Treat hash table as random when ...
Github user hsyuan commented on a diff in the pull request:
https://github.com/apache/incubator-hawq/pull/845#discussion_r76369429
--- Diff: src/backend/cdb/cdbdatalocality.c ---
@@ -3791,12 +3791,27 @@ run_allocation_algorithm(SplitAllocResult *result,
List *virtual_segments, Query
targetPolicy = GpPolicyFetch(CurrentMemoryContext, myrelid);
bool isRelationHash = is_relation_hash(targetPolicy);
+ int fileCountInRelation = list_length(rel_data->files);
+ bool FileCountBucketNumMismatch = false;
+ if (targetPolicy->bucketnum > 0) {
+ FileCountBucketNumMismatch = fileCountInRelation %
+ targetPolicy->bucketnum == 0 ? false : true;
+ }
+ if (FileCountBucketNumMismatch &&
!allow_file_count_bucket_num_mismatch) {
+ elog(ERROR, "file count %d in catalog is not in proportion to
the bucket "
+ "number %d of hash table with oid=%u, some data may be
lost, if you "
+ "still want to continue the query by considering the
table as random, set GUC "
--- End diff --
You can force running the query by viewing the table as randomly
distributed table. set GUC allow_file_count_bucket_num_mismatch on to enable
this feature.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq pull request #845: HAWQ-999. Treat hash table as random when ...
Github user hsyuan commented on a diff in the pull request: https://github.com/apache/incubator-hawq/pull/845#discussion_r76368733 --- Diff: src/backend/cdb/cdbdatalocality.c --- @@ -3791,12 +3791,27 @@ run_allocation_algorithm(SplitAllocResult *result, List *virtual_segments, Query targetPolicy = GpPolicyFetch(CurrentMemoryContext, myrelid); bool isRelationHash = is_relation_hash(targetPolicy); + int fileCountInRelation = list_length(rel_data->files); + bool FileCountBucketNumMismatch = false; --- End diff -- why is this var capital leading? `FileCountBucketNumMismatch` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq pull request #845: HAWQ-999. Treat hash table as random when ...
Github user hsyuan commented on a diff in the pull request:
https://github.com/apache/incubator-hawq/pull/845#discussion_r76368378
--- Diff: src/backend/cdb/cdbdatalocality.c ---
@@ -3822,8 +3837,13 @@ run_allocation_algorithm(SplitAllocResult *result,
List *virtual_segments, Query
uint64_t before_run_allocate_hash_or_random =
gettime_microsec();
/*allocate hash relation*/
if (isRelationHash) {
- if (context->keep_hash &&
assignment_context.virtual_segment_num
- == targetPolicy->bucketnum) {
+ /*
+ * if file count of the table is not equal to or multiple of
+ * bucket number, we should process it as random table.
+ */
+ if (context->keep_hash
+ && assignment_context.virtual_segment_num==
targetPolicy->bucketnum
--- End diff --
space before ==
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq pull request #861: HAWQ-1001. Implement HAWQ user ACL check t...
Github user hsyuan commented on a diff in the pull request:
https://github.com/apache/incubator-hawq/pull/861#discussion_r76368047
--- Diff: src/backend/utils/misc/guc.c ---
@@ -4323,6 +4324,16 @@ static struct config_bool ConfigureNamesBool[] =
},
{
+{"enable_ranger", PGC_POSTMASTER, CONN_AUTH_SETTINGS,
+ gettext_noop("support to using ranger to manage hawq privilege."),
--- End diff --
Enable Apache Ranger for HAWQ privilege management.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq pull request #807: HAWQ-897. Add feature test for create tabl...
Github user hsyuan commented on a diff in the pull request:
https://github.com/apache/incubator-hawq/pull/807#discussion_r76184425
--- Diff: src/test/feature/catalog/test_create_table.cpp ---
@@ -22,26 +22,26 @@ class TestCreateTable : public ::testing::Test {
TEST_F(TestCreateTable, TestCreateTable1) {
hawq::test::SQLUtility util;
// prepare
- util.execute("DROP TABLE IF EXISTS aggtest");
--- End diff --
The easier way to do this is creating a new schema `X`, set the default
search_path to `X`. At the end of the test, drop the schema. In this way, you
don't need to take care of dropping tables when you add new test cases. This is
how GPDB is doing.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq issue #839: HAWQ-983. Fix bug that minirepro generates wrong ...
Github user hsyuan commented on the issue: https://github.com/apache/incubator-hawq/pull/839 @d @oarap please take a look. This is ported from gpdb. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq issue #838: HAWQ-934. Populate canSetTag of PlannedStmt from ...
Github user hsyuan commented on the issue: https://github.com/apache/incubator-hawq/pull/838 @vraghavan78 @d @oarap need you guys kind review. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq issue #839: HAWQ-983. Fix bug that minirepro generates wrong ...
Github user hsyuan commented on the issue: https://github.com/apache/incubator-hawq/pull/839 ping, will anyone work on merging this pull request? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq issue #838: HAWQ-934. Populate canSetTag of PlannedStmt from ...
Github user hsyuan commented on the issue: https://github.com/apache/incubator-hawq/pull/838 ping, any updates? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq issue #839: HAWQ-983. Fix bug that minirepro generates wrong ...
Github user hsyuan commented on the issue: https://github.com/apache/incubator-hawq/pull/839 @changleicn @liming01 Please take a look. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq issue #806: HAWQ-938. Remove ivy.xml in gpopt and read orca v...
Github user hsyuan commented on the issue: https://github.com/apache/incubator-hawq/pull/806 pushed to master: https://github.com/apache/incubator-hawq/commit/78c2766578102501e67c133df942fc6355d56c8e --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq pull request #800: HAWQ-936. Add GUC for array expansion in O...
Github user hsyuan closed the pull request at: https://github.com/apache/incubator-hawq/pull/800 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq pull request #806: HAWQ-938. Remove ivy.xml in gpopt and read...
Github user hsyuan closed the pull request at: https://github.com/apache/incubator-hawq/pull/806 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq issue #800: HAWQ-936. Add GUC for array expansion in ORCA opt...
Github user hsyuan commented on the issue: https://github.com/apache/incubator-hawq/pull/800 pushed to master: https://github.com/apache/incubator-hawq/commit/950065bb0db627992be6b96afb4671b3c5ab1c90 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq pull request #799: HAWQ-934. Populate canSetTag of PlannedStm...
Github user hsyuan closed the pull request at: https://github.com/apache/incubator-hawq/pull/799 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq issue #799: HAWQ-934. Populate canSetTag of PlannedStmt from ...
Github user hsyuan commented on the issue: https://github.com/apache/incubator-hawq/pull/799 Pushed to master: https://github.com/apache/incubator-hawq/commit/eae0585f6109534c7ead553604c44b1dd483d8a3 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq issue #795: HAWQ-860. Fix ORCA wrong plan when correlated sub...
Github user hsyuan commented on the issue: https://github.com/apache/incubator-hawq/pull/795 Pushed to master: https://github.com/apache/incubator-hawq/commit/e10f9a4e17f3012bb5afb98c53172dedefc8854e https://github.com/apache/incubator-hawq/commit/4e08f10fc87301682721d6bffcb74a188ac7b595 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq pull request #795: HAWQ-860. Fix ORCA wrong plan when correla...
Github user hsyuan closed the pull request at: https://github.com/apache/incubator-hawq/pull/795 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq issue #799: HAWQ-934. Populate canSetTag of PlannedStmt from ...
Github user hsyuan commented on the issue: https://github.com/apache/incubator-hawq/pull/799 @changleicn @wengyanqing @paul-guo- Please take a look. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq issue #806: HAWQ-938. Remove ivy.xml in gpopt and read orca v...
Github user hsyuan commented on the issue: https://github.com/apache/incubator-hawq/pull/806 Thanks, will take care of it. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq issue #800: HAWQ-936. Add GUC for array expansion in ORCA opt...
Github user hsyuan commented on the issue: https://github.com/apache/incubator-hawq/pull/800 This PR will give https://github.com/apache/incubator-hawq/pull/795 a free ride. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq issue #806: HAWQ-938. Remove ivy.xml in gpopt and read orca v...
Github user hsyuan commented on the issue: https://github.com/apache/incubator-hawq/pull/806 @changleicn @yaoj2 @wengyanqing Please take a look. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq pull request #806: HAWQ-938. Remove ivy.xml in gpopt and read...
GitHub user hsyuan opened a pull request: https://github.com/apache/incubator-hawq/pull/806 HAWQ-938. Remove ivy.xml in gpopt and read orca version from header file The old mechanism extracted the version numbers from the Ivy config file, which doesn't do the right thing if you build without Ivy. Using the version headers is simpler, anyway. Also removed `ivy.xml` and `ivy-build.xml` under `gpopt` folder. You can merge this pull request into a Git repository by running: $ git pull https://github.com/hsyuan/incubator-hawq HAWQ-938 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-hawq/pull/806.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #806 commit 2a2a89cc6b950d4067a8e4d8a5e79b2f7b1cf839 Author: Haisheng Yuan <hy...@pivotal.io> Date: 2016-07-20T20:14:25Z HAWQ-938. Remove ivy.xml in gpopt and read orca version from header file The old mechanism extracted the version numbers from the Ivy config file, which doesn't do the right thing if you build without Ivy. Using the version headers is simpler, anyway. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq pull request #800: HAWQ-936. Add GUC for array expansion in O...
GitHub user hsyuan opened a pull request: https://github.com/apache/incubator-hawq/pull/800 HAWQ-936. Add GUC for array expansion in ORCA optimizer Consider the query with the following pattern 'select * from foo where foo.a IN (1,2,3,...)'. Currently, when the number of constants in the IN subquery is large, the query optimization time is unacceptable. This is stopping customers from turning Orca on by default since many of the queries are generated queries with such a pattern. The root cause is due to the expansion of the IN subquery into an expression in disjunctive normal form. The objective of this patch is to add a guc parameter 'optimizer_array_expansion_threshold' to control the maximum number of array elements in IN array list. By default, the GUC value is set to 25, which means if the array size is larger than 25, the array expansion is disabled. You can merge this pull request into a Git repository by running: $ git pull https://github.com/hsyuan/incubator-hawq HAWQ-936 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-hawq/pull/800.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #800 commit 6acc01abbd22ad17ad041d122f01aa072b48a9f6 Author: Haisheng Yuan and Lubomir Petrov <hy...@pivotal.io> Date: 2016-07-19T00:05:14Z HAWQ-936. Add GUC for array expansion in ORCA optimizer Consider the query with the following pattern 'select * from foo where foo.a IN (1,2,3,...)'. Currently, when the number of constants in the IN subquery is large, the query optimization time is unacceptable. This is stopping customers from turning Orca on by default since many of the queries are generated queries with such a pattern. The root cause is due to the expansion of the IN subquery into an expression in disjunctive normal form. The objective of this patch is to add a guc parameter 'optimizer_array_expansion_threshold' to control the maximum number of array elements in IN array list. By default, the GUC value is set to 25, which means if the array size is larger than 25, the array expansion is disabled. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq pull request #:
Github user hsyuan commented on the pull request: https://github.com/apache/incubator-hawq/commit/5506a22819d58c18681b7a37de58fa4299026b4e#commitcomment-18290655 How can I run the test? `boolean`, `type_sanity` etc are still in file `src/test/regress/known_good_schedule`. This test schedule is going to fail. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq issue #794: HAWQ-926. Remove pycrypto from HAWQ source code
Github user hsyuan commented on the issue: https://github.com/apache/incubator-hawq/pull/794 This PR should be closed, right? It is already in: https://github.com/apache/incubator-hawq/commit/ac031357b3b29978772f1e7478d97d21da653df4 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq pull request #799: HAWQ-934. Populate canSetTag of PlannedStm...
GitHub user hsyuan opened a pull request: https://github.com/apache/incubator-hawq/pull/799 HAWQ-934. Populate canSetTag of PlannedStmt from Query object HAWQ generated an error if a single query resulted in multiple query plans because of rule transformation and the plans were produced by PQO. This is because of an incorrect directive in the plan to lock the same resource more than once. This issue has been fixed. You can merge this pull request into a Git repository by running: $ git pull https://github.com/hsyuan/incubator-hawq HAWQ-934 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-hawq/pull/799.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #799 commit 8ccca05f62a0a8e315fa21799d5401a792470477 Author: Haisheng Yuan and Lubomir Petrov <hy...@pivotal.io> Date: 2016-07-18T21:51:39Z HAWQ-934. Populate canSetTag of PlannedStmt from Query object --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq issue #797: HAWQ-622. Update libhdfs3 readme, clean old strin...
Github user hsyuan commented on the issue: https://github.com/apache/incubator-hawq/pull/797 +1 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq issue #795: HAWQ-860. Fix ORCA wrong plan when correlated sub...
Github user hsyuan commented on the issue: https://github.com/apache/incubator-hawq/pull/795 @yaoj2 @wengyanqing @zhangh43 Please take a look. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq pull request #795: HAWQ-860. Fix ORCA wrong plan when correla...
GitHub user hsyuan opened a pull request: https://github.com/apache/incubator-hawq/pull/795 HAWQ-860. Fix ORCA wrong plan when correlated subquery contains set-returning functions ORCA 1.633 returns wrong result for the following query: ```sql select 0 is distinct from (select count(1) from (select unnest(array[1, 2, 3])) as foo); ?column? -- t t t (3 rows) ``` Correct result should be: ```sql ?column? -- t (1 row) ``` This bug was fixed by bumping ORCA version to 1.634. For detail information, see ORCA pull request: https://github.com/greenplum-db/gporca/pull/49 You can merge this pull request into a Git repository by running: $ git pull https://github.com/hsyuan/incubator-hawq HAWQ-860 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-hawq/pull/795.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #795 commit eef9fe3c330119bdeaecc8f5b18382d1b9568e46 Author: Haisheng Yuan and Omer Arap <hy...@pivotal.io> Date: 2016-07-14T19:16:40Z Fix ORCA wrong plan when correlated subquery contains set-returning functions --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq issue #720: HAWQ-823 - Initial Amazon S3 External Table Suppo...
Github user hsyuan commented on the issue: https://github.com/apache/incubator-hawq/pull/720 Glad to see this porting from GPDB S3 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
