date:20161101

[jira] [Created] (HAWQ-1136) Disable .psqlrc in minirepro

2016-11-01 Thread Haisheng Yuan (JIRA)

Haisheng Yuan created HAWQ-1136:
---

 Summary: Disable .psqlrc in minirepro
 Key: HAWQ-1136
 URL: https://issues.apache.org/jira/browse/HAWQ-1136
 Project: Apache HAWQ
  Issue Type: Bug
  Components: Command Line Tools
Reporter: Haisheng Yuan
Assignee: Lei Chang


.psqlrc can create unexpected output and changes in formatting that don't play 
nice with parse_oids().

```
psql database --pset footer -Atq -h localhost -p 5432 -U gpadmin -f 
/tmp/20161012232709/toolkit.sql

{"relids": "573615536", "funcids": ""}
Time: 2.973 ms
```

Generates an Exception:
```
Traceback (most recent call last):
  File "/usr/local/greenplum-db/./bin/minirepro", line 386, in 
main()
  File "/usr/local/greenplum-db/./bin/minirepro", line 320, in main
mr_query = parse_oids(cursor, json_str)
  File "/usr/local/greenplum-db/./bin/minirepro", line 151, in parse_oids
result.relids = json.loads(json_oids)['relids']
  File "/usr/local/greenplum-db/ext/python/lib/python2.6/json/__init__.py", 
line 307, in loads
return _default_decoder.decode(s)
  File "/usr/local/greenplum-db/ext/python/lib/python2.6/json/decoder.py", line 
322, in decode
raise ValueError(errmsg("Extra data", s, end, len(s)))
ValueError: Extra data: line 2 column 1 - line 3 column 1 (char 39 - 54)
```



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (HAWQ-1136) Disable .psqlrc in minirepro

2016-11-01 Thread Haisheng Yuan (JIRA)


 [ 
https://issues.apache.org/jira/browse/HAWQ-1136?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Haisheng Yuan reassigned HAWQ-1136:
---

Assignee: Haisheng Yuan  (was: Lei Chang)

> Disable .psqlrc in minirepro
> 
>
> Key: HAWQ-1136
> URL: https://issues.apache.org/jira/browse/HAWQ-1136
> Project: Apache HAWQ
>  Issue Type: Bug
>  Components: Command Line Tools
>Reporter: Haisheng Yuan
>Assignee: Haisheng Yuan
>
> .psqlrc can create unexpected output and changes in formatting that don't 
> play nice with parse_oids().
> ```
> psql database --pset footer -Atq -h localhost -p 5432 -U gpadmin -f 
> /tmp/20161012232709/toolkit.sql
> {"relids": "573615536", "funcids": ""}
> Time: 2.973 ms
> ```
> Generates an Exception:
> ```
> Traceback (most recent call last):
>   File "/usr/local/greenplum-db/./bin/minirepro", line 386, in 
> main()
>   File "/usr/local/greenplum-db/./bin/minirepro", line 320, in main
> mr_query = parse_oids(cursor, json_str)
>   File "/usr/local/greenplum-db/./bin/minirepro", line 151, in parse_oids
> result.relids = json.loads(json_oids)['relids']
>   File "/usr/local/greenplum-db/ext/python/lib/python2.6/json/__init__.py", 
> line 307, in loads
> return _default_decoder.decode(s)
>   File "/usr/local/greenplum-db/ext/python/lib/python2.6/json/decoder.py", 
> line 322, in decode
> raise ValueError(errmsg("Extra data", s, end, len(s)))
> ValueError: Extra data: line 2 column 1 - line 3 column 1 (char 39 - 54)
> ```



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-hawq issue #992: HAWQ-1133. Replace print with logger output in ha...

2016-11-01 Thread xunzhang

Github user xunzhang commented on the issue:

https://github.com/apache/incubator-hawq/pull/992
  
also cc @liming01 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq pull request #990: HAWQ-1128. Support HAWQ register tables wi...

2016-11-01 Thread xunzhang

Github user xunzhang closed the pull request at:

https://github.com/apache/incubator-hawq/pull/990


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq issue #990: HAWQ-1128. Support HAWQ register tables with same...

2016-11-01 Thread xunzhang

Github user xunzhang commented on the issue:

https://github.com/apache/incubator-hawq/pull/990
  
Merged, thanks.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Resolved] (HAWQ-1128) Support HAWQ register tables with same file name in different schema

2016-11-01 Thread hongwu (JIRA)


 [ 
https://issues.apache.org/jira/browse/HAWQ-1128?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

hongwu resolved HAWQ-1128.
--
Resolution: Implemented

> Support HAWQ register tables with same file name in different schema
> 
>
> Key: HAWQ-1128
> URL: https://issues.apache.org/jira/browse/HAWQ-1128
> Project: Apache HAWQ
>  Issue Type: Sub-task
>  Components: Command Line Tools
>Affects Versions: 2.0.1.0-incubating
>Reporter: Chunling Wang
>Assignee: hongwu
> Fix For: 2.0.1.0-incubating
>
>
> Now, in HAWQ Register, it can not distinguish tables with same file name but 
> in different schema, which are regarded as same table. We should save and use 
> schema information for HAWQ register.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-hawq issue #994: HAWQ-1135. MADlib: Raising exception leads to dat...

2016-11-01 Thread liming01

Github user liming01 commented on the issue:

https://github.com/apache/incubator-hawq/pull/994
  
+1


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq issue #994: HAWQ-1135. MADlib: Raising exception leads to dat...

2016-11-01 Thread huor

Github user huor commented on the issue:

https://github.com/apache/incubator-hawq/pull/994
  
Nice fix, Hubert! Is it possible to simplify the reproduction case and make 
it MADlib independent so that this scenario can be added to hawq regression 
test?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq issue #994: HAWQ-1135. MADlib: Raising exception leads to dat...

2016-11-01 Thread xunzhang

Github user xunzhang commented on the issue:

https://github.com/apache/incubator-hawq/pull/994
  
ð» 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq pull request #994: HAWQ-1135. MADlib: Raising exception leads...

2016-11-01 Thread zhangh43

GitHub user zhangh43 opened a pull request:

https://github.com/apache/incubator-hawq/pull/994

HAWQ-1135. MADlib: Raising exception leads to database connection terâ¦

â¦mination.

To solve this bug, we remove AtSubAbort_ActiveQueryResource() when 
subtransaction abort.

In sub transaction, ActiveQueryResource should not be set NULL.
Sub transaction inherits resource from parent.
Parent transaction may still use ActiveQueryResource to reference the 
original resource.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/zhangh43/incubator-hawq hawq1135

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-hawq/pull/994.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #994


commit f01c1181f84acd115490ca1799cd320e526d8709
Author: hzhang2 
Date:   2016-11-02T02:24:04Z

HAWQ-1135. MADlib: Raising exception leads to database connection 
termination.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq issue #990: HAWQ-1128. Support HAWQ register tables with same...

2016-11-01 Thread linwen

Github user linwen commented on the issue:

https://github.com/apache/incubator-hawq/pull/990
  
+1 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (HAWQ-1135) MADlib: Raising exception leads to database connection termination.

2016-11-01 Thread Hubert Zhang (JIRA)


 [ 
https://issues.apache.org/jira/browse/HAWQ-1135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hubert Zhang updated HAWQ-1135:
---
Summary: MADlib: Raising exception leads to database connection 
termination.  (was: MADlib: Raising exception leads to database connection 
termination)

> MADlib: Raising exception leads to database connection termination.
> ---
>
> Key: HAWQ-1135
> URL: https://issues.apache.org/jira/browse/HAWQ-1135
> Project: Apache HAWQ
>  Issue Type: Bug
>  Components: Core
>Reporter: Ming LI
>Assignee: Hubert Zhang
>
> MADlib tests on HAWQ 2.0 Nightly builds fails due to server terminating it's 
> connection. The failed tests are testing for bad input by returning an 
> exception on specific user inputs. These exceptions are raised cleanly in 
> other platforms including HAWQ 2.0 and all Greenplum DBs.
> Reproduction Steps
> Install MADlib using the RPM and HAWQ install script.
> Run attached script (called hawq_2.0.1_test.sql)
> Current error message is
> {{
> server closed the connection unexpectedly
> This probably means the server terminated abnormally
> before or while processing the request.
> }}
> Expected error is
> {{
> ERROR: spiexceptions.InvalidParameterValue: Function 
> "madlib.lmf_igd_transition(double precision[],integer,integer,double 
> precision,double precision[],integer,integer,integer,double precision,double 
> precision)": Invalid type conversion. Null where not expected.
> }}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (HAWQ-1135) MADlib: Raising exception leads to database connection termination

2016-11-01 Thread Hubert Zhang (JIRA)


 [ 
https://issues.apache.org/jira/browse/HAWQ-1135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hubert Zhang reassigned HAWQ-1135:
--

Assignee: Hubert Zhang  (was: Lei Chang)

> MADlib: Raising exception leads to database connection termination
> --
>
> Key: HAWQ-1135
> URL: https://issues.apache.org/jira/browse/HAWQ-1135
> Project: Apache HAWQ
>  Issue Type: Bug
>  Components: Core
>Reporter: Ming LI
>Assignee: Hubert Zhang
>
> MADlib tests on HAWQ 2.0 Nightly builds fails due to server terminating it's 
> connection. The failed tests are testing for bad input by returning an 
> exception on specific user inputs. These exceptions are raised cleanly in 
> other platforms including HAWQ 2.0 and all Greenplum DBs.
> Reproduction Steps
> Install MADlib using the RPM and HAWQ install script.
> Run attached script (called hawq_2.0.1_test.sql)
> Current error message is
> {{
> server closed the connection unexpectedly
> This probably means the server terminated abnormally
> before or while processing the request.
> }}
> Expected error is
> {{
> ERROR: spiexceptions.InvalidParameterValue: Function 
> "madlib.lmf_igd_transition(double precision[],integer,integer,double 
> precision,double precision[],integer,integer,integer,double precision,double 
> precision)": Invalid type conversion. Null where not expected.
> }}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HAWQ-1135) MADlib: Raising exception leads to database connection termination

2016-11-01 Thread Ming LI (JIRA)


[ 
https://issues.apache.org/jira/browse/HAWQ-1135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15627383#comment-15627383
 ] 

Ming LI commented on HAWQ-1135:
---

1) Execute SQL:
select madlib.lmf_igd_run(
  'LMF_output_table'::varchar,
  'madlibtestdata.mlens100k'::varchar,
  'user_id'::varchar,
  'movie_id'::varchar,
  'rating'::varchar,
  943::integer,
  1682::integer,
  10::integer,
  0.01::double precision,
  NULL::double precision,
  10::integer,
  1e-3::double precision
  );
  

2) gdb executor process:
b planner
c
c 17
b siglongjmp


(gdb) f 1
(gdb) p *edata
$2 = {elevel = 20, output_to_server = 1 '\001', output_to_client = 1 '\001', 
show_funcname = 0 '\000',
  omit_location = 1 '\001', fatal_return = 0 '\000', hide_stmt = 0 '\000', 
send_alert = 0 '\000',
  filename = 0x7f68bf30fd08 
"/data/home/gpdbchina/madlib-1.9.1-build/incubator-madlib/src/ports/hawq/../greenplum/dbconnector/../../postgres/dbconnector/UDF_impl.hpp",
 lineno = 210,
  funcname = 0x7f68bf3104d4 "call", domain = 0xc6b0bb "postgres-8.2", 
sqlerrcode = 50856066,
  message = 0x1c42950 "Function \"madlib.lmf_igd_transition(double 
precision[],integer,integer,double precision,double 
precision[],integer,integer,integer,double precision,double precision)\": 
Invalid type conversion. Null wh"..., detail = 0x0, detail_log = 0x0, hint = 
0x0,
  context = 0x1c42eb0 "SQL statement \"\n", ' ' , "SELECT\n", 
' ' , "1 AS _iteration,\n", ' ' , "(\n", ' 
' , "SELECT\n", ' ' , 
"madlib.lmf_igd_step(\n", ' ' , "(_src.user_id)"..., 
cursorpos = 0,
  internalpos = 0, internalquery = 0x0, saved_errno = 11, stacktracearray = 
{0x95d52a, 0x7f68bf2382e9,
0x6fb00b, 0x6faeab, 0x6fb394, 0x6fc17d, 0x6fce08, 0x6fc08a, 0x6df92f, 
0x7183ff, 0xaab907, 0x6d608a,
0x72f44d, 0x72ef58, 0x72c1d9, 0x7f68c44a8011, 0x7f68c44a7a54, 
0x7f68c41ca9d4, 0x7f68c41cc647,
0x7f68c41caa94, 0x7f68c41cc647, 0x7f68c415fd9d, 0x7f68c4138c63, 
0x7f68c41c9460, 0x7f68c41cbb7f,
0x7f68c41cc647, 0x7f68c41cc722, 0x7f68c44a1e97, 0x7f68c44a155c, 
0x7f68c449faf3},
  stacktracesize = 30, printstack = 0 '\000'}
(gdb) bt


#0  0x003dc100e150 in siglongjmp () from /lib64/libpthread.so.0
#1  0x0095d6f7 in errfinish (dummy=0) at elog.c:578
#2  0x7f68bf238321 in long 
madlib::dbconnector::postgres::UDF::call(FunctionCallInfoData*)
 ()
   from /usr/local/madlib/Versions/1.9.1/ports/hawq/2.0/lib/libmadlib.so
#3  0x006fb00b in invoke_agg_trans_func (transfn=0x2f7cff8, numargs=9, 
transValue=37836256,
noTransvalue=0x2f7a431 "", transValueIsNull=0x2f7a430 "", transtypeByVal=0 
'\000', transtypeLen=-1,
fcinfo=0x7fff61be3500, funcctx=0x2fb5b28, tuplecontext=0x244d0b0, 
mem_manager=0x2fb5e00)
at nodeAgg.c:471
#4  0x006faeab in advance_transition_function (aggstate=0x2fb5b28, 
peraggstate=0x2f7cfc0,
pergroupstate=0x2f7a428, fcinfo=0x7fff61be3500, mem_manager=0x2fb5e00) at 
nodeAgg.c:392
#5  0x006fb394 in advance_aggregates (aggstate=0x2fb5b28, 
pergroup=0x2f7a428,
mem_manager=0x2fb5e00) at nodeAgg.c:618
#6  0x006fc17d in agg_retrieve_scalar (aggstate=0x2fb5b28) at 
nodeAgg.c:1173
#7  0x006fce08 in agg_retrieve_direct (aggstate=0x2fb5b28) at 
nodeAgg.c:1693
#8  0x006fc08a in ExecAgg (node=0x2fb5b28) at nodeAgg.c:1138
#9  0x006df92f in ExecProcNode (node=0x2fb5b28) at execProcnode.c:979
#10 0x007183ff in ExecSetParamPlan (node=0x2fb5808, econtext=0x2f74a10, 
gbl_queryDesc=0x2fa8550)
at nodeSubplan.c:1161
#11 0x00aab907 in preprocess_initplans (queryDesc=0x2fa8550) at 
cdbsubplan.c:171
#12 0x006d608a in ExecutorStart (queryDesc=0x2fa8550, eflags=0) at 
execMain.c:929
#13 0x0072f44d in _SPI_pquery (queryDesc=0x2fa8550, fire_triggers=1 
'\001', tcount=0)
at spi.c:2214
#14 0x0072ef58 in _SPI_execute_plan (plan=0x242be00, Values=0x244c108,
Nulls=0x21e5a78 "notice", snapshot=0x0, crosscheck_snapshot=0x0, 
read_only=0 '\000',
fire_triggers=1 '\001', tcount=0) at spi.c:1972
#15 0x0072c1d9 in SPI_execute_plan (plan=0x242be00, Values=0x244c108, 
Nulls=0x21e5a78 "notice",
read_only=0 '\000', tcount=0) at spi.c:520
#16 0x7f68c44a8011 in PLy_spi_execute_plan (ob=0x228c8b8, list=0x222e3b0, 
limit=0)
at plpython.c:3737
#17 0x7f68c44a7a54 in PLy_spi_execute (self=0x0, args=0x2221098) at 
plpython.c:3635
#18 0x7f68c41ca9d4 in PyEval_EvalFrameEx () from 
/usr/lib64/libpython2.6.so.1.0
#19 0x7f68c41cc647 in PyEval_EvalCodeEx () from 
/usr/lib64/libpython2.6.so.1.0
#20 0x7f68c41caa94 in PyEval_EvalFrameEx () from 
/usr/lib64/libpython2.6.so.1.0
#21 0x7f68c41cc647 in PyEval_EvalCodeEx () from 
/usr/lib64/libpython2.6.so.1.0
#22 0x7f68c415fd9d in ?? () from /usr/lib64/libpython2.6.so.1.0
#23 0x7f68c4138c63 in PyObject_Call () from

[jira] [Created] (HAWQ-1135) MADlib: Raising exception leads to database connection termination

2016-11-01 Thread Ming LI (JIRA)

Ming LI created HAWQ-1135:
-

 Summary: MADlib: Raising exception leads to database connection 
termination
 Key: HAWQ-1135
 URL: https://issues.apache.org/jira/browse/HAWQ-1135
 Project: Apache HAWQ
  Issue Type: Bug
  Components: Core
Reporter: Ming LI
Assignee: Lei Chang


MADlib tests on HAWQ 2.0 Nightly builds fails due to server terminating it's 
connection. The failed tests are testing for bad input by returning an 
exception on specific user inputs. These exceptions are raised cleanly in other 
platforms including HAWQ 2.0 and all Greenplum DBs.
Reproduction Steps
Install MADlib using the RPM and HAWQ install script.
Run attached script (called hawq_2.0.1_test.sql)
Current error message is
{{
server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
}}
Expected error is
{{
ERROR: spiexceptions.InvalidParameterValue: Function 
"madlib.lmf_igd_transition(double precision[],integer,integer,double 
precision,double precision[],integer,integer,integer,double precision,double 
precision)": Invalid type conversion. Null where not expected.
}}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HAWQ-1119) create new documentation topic for PXF writable profiles

2016-11-01 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/HAWQ-1119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15627161#comment-15627161
 ] 

ASF GitHub Bot commented on HAWQ-1119:
--

Github user dyozie commented on a diff in the pull request:

https://github.com/apache/incubator-hawq-docs/pull/46#discussion_r86052740
  
--- Diff: pxf/HDFSWritablePXF.html.md.erb ---
@@ -0,0 +1,416 @@
+---
+title: Writing Data to HDFS
+---
+
+The PXF HDFS plug-in supports writable external tables using the 
`HdfsTextSimple` and `SequenceWritable` profiles.  You might create a writable 
table to export data from a HAWQ internal table to binary or text HDFS files.
+
+Use the `HdfsTextSimple` profile when writing text data. Use the 
`SequenceWritable` profile when dealing with binary data.
+
+This section describes how to use these PXF profiles to create writable 
external tables.
+
+**Note**: Tables that you create with writable profiles can only be used 
for INSERT operations.  If you want to query inserted data, you must define a 
separate external readable table that references the new HDFS file using the 
equivalent readable profile.  ??You can also create a Hive table to access the 
HDFS file.??
+
+## Prerequisites
+
+Before working with HDFS file data using HAWQ and PXF, ensure that:
+
+-   The HDFS plug-in is installed on all cluster nodes. See [Installing 
PXF Plug-ins](InstallPXFPlugins.html) for PXF plug-in installation information.
+-   All HDFS users have read permissions to HDFS services.
+-   HDFS write permissions are provided to a restricted set of users.
+
+## Writing to PXF External Tables
+The PXF HDFS plug-in supports two writable profiles: `HdfsTextSimple` and 
`SequenceWritable`.
+
+Use the following syntax to create a HAWQ external writable table 
representing HDFS data: 
+
+``` sql
+CREATE WRITABLE EXTERNAL TABLE  
+(   [, ...] | LIKE  )
+LOCATION ('pxf://[:]/
+
?PROFILE=HdfsTextSimple|SequenceWritable[&=[...]]')
+FORMAT '[TEXT|CSV|CUSTOM]' ();
+```
+
+HDFS-plug-in-specific keywords and values used in the [CREATE EXTERNAL 
TABLE](../reference/sql/CREATE-EXTERNAL-TABLE.html) call are described in the 
table below.
+
+| Keyword  | Value |
+|---|-|
+| \[:\]| The HDFS NameNode and port. |
+| \| The path to the file in the HDFS data store. |
+| PROFILE| The `PROFILE` keyword must specify one of the values 
`HdfsTextSimple` or `SequenceWritable`. |
+| \  | \ is profile-specific. These 
options are discussed in the next topic.|
+| FORMAT 'TEXT' | Use '`TEXT`' `FORMAT` with the `HdfsTextSimple` profile 
to create a plain-text-delimited file at the location specified by 
\. The `HdfsTextSimple` '`TEXT`' `FORMAT` supports only the 
built-in `(delimiter=)` \. |
+| FORMAT 'CSV' | Use '`CSV`' `FORMAT` with the `HdfsTextSimple` profile to 
create a comma-separated-value file at the location specified by 
\.  |
+| FORMAT 'CUSTOM' | Use the `'CUSTOM'` `FORMAT` with the 
`SequenceWritable` profile. The `SequenceWritable` '`CUSTOM`' `FORMAT` supports 
only the built-in `(formatter='pxfwritable_export)` (write) and 
`(formatter='pxfwritable_import)` (read) \.
+
+**Note**: When creating PXF external tables, you cannot use the `HEADER` 
option in your `FORMAT` specification.
+
+## Custom Options
+
+The `HdfsTextSimple` and `SequenceWritable` profiles support the following 
custom options:
+
+| Option  | Value Description | Profile |
+|---|-||
+| COMPRESSION_CODEC| The compression codec Java class name. If this 
option is not provided, no data compression is performed. Supported compression 
codecs include: `org.apache.hadoop.io.compress.DefaultCodec` and 
`org.apache.hadoop.io.compress.BZip2Codec` | HdfsTextSimple, SequenceWritable |
+||  `org.apache.hadoop.io.compress.GzipCodec` | HdfsTextSimple |
+| COMPRESSION_TYPE| The compression type to employ; supported values 
are `RECORD` (the default) or `BLOCK`. | HdfsTextSimple, SequenceWritable |
+| DATA-SCHEMA| The name of the writer serialization/deserialization 
class. The jar file in which this class resides must be in the PXF class path. 
This option is required for the `SequenceWritable` profile and has no default 
value. | SequenceWritable|
+| THREAD-SAFE | Boolean value determining if a table query can run in 
multi-threaded mode. The default value is `TRUE`. Set this option to `FALSE` to 
handle all requests in a single thread for operations that are not thread-safe 
(for example, compression). |

[jira] [Commented] (HAWQ-1119) create new documentation topic for PXF writable profiles

2016-11-01 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/HAWQ-1119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15627162#comment-15627162
 ] 

ASF GitHub Bot commented on HAWQ-1119:
--

Github user dyozie commented on a diff in the pull request:

https://github.com/apache/incubator-hawq-docs/pull/46#discussion_r86050016
  
--- Diff: pxf/HDFSWritablePXF.html.md.erb ---
@@ -0,0 +1,416 @@
+---
+title: Writing Data to HDFS
+---
+
+The PXF HDFS plug-in supports writable external tables using the 
`HdfsTextSimple` and `SequenceWritable` profiles.  You might create a writable 
table to export data from a HAWQ internal table to binary or text HDFS files.
+
+Use the `HdfsTextSimple` profile when writing text data. Use the 
`SequenceWritable` profile when dealing with binary data.
+
+This section describes how to use these PXF profiles to create writable 
external tables.
+
+**Note**: Tables that you create with writable profiles can only be used 
for INSERT operations.  If you want to query inserted data, you must define a 
separate external readable table that references the new HDFS file using the 
equivalent readable profile.  ??You can also create a Hive table to access the 
HDFS file.??
+
+## Prerequisites
+
+Before working with HDFS file data using HAWQ and PXF, ensure that:
+
+-   The HDFS plug-in is installed on all cluster nodes. See [Installing 
PXF Plug-ins](InstallPXFPlugins.html) for PXF plug-in installation information.
+-   All HDFS users have read permissions to HDFS services.
+-   HDFS write permissions are provided to a restricted set of users.
+
+## Writing to PXF External Tables
+The PXF HDFS plug-in supports two writable profiles: `HdfsTextSimple` and 
`SequenceWritable`.
+
+Use the following syntax to create a HAWQ external writable table 
representing HDFS data: 
+
+``` sql
+CREATE WRITABLE EXTERNAL TABLE  
+(   [, ...] | LIKE  )
+LOCATION ('pxf://[:]/
+
?PROFILE=HdfsTextSimple|SequenceWritable[&=[...]]')
+FORMAT '[TEXT|CSV|CUSTOM]' ();
+```
+
+HDFS-plug-in-specific keywords and values used in the [CREATE EXTERNAL 
TABLE](../reference/sql/CREATE-EXTERNAL-TABLE.html) call are described in the 
table below.
+
+| Keyword  | Value |
+|---|-|
+| \[:\]| The HDFS NameNode and port. |
+| \| The path to the file in the HDFS data store. |
+| PROFILE| The `PROFILE` keyword must specify one of the values 
`HdfsTextSimple` or `SequenceWritable`. |
+| \  | \ is profile-specific. These 
options are discussed in the next topic.|
+| FORMAT 'TEXT' | Use '`TEXT`' `FORMAT` with the `HdfsTextSimple` profile 
to create a plain-text-delimited file at the location specified by 
\. The `HdfsTextSimple` '`TEXT`' `FORMAT` supports only the 
built-in `(delimiter=)` \. |
+| FORMAT 'CSV' | Use '`CSV`' `FORMAT` with the `HdfsTextSimple` profile to 
create a comma-separated-value file at the location specified by 
\.  |
+| FORMAT 'CUSTOM' | Use the `'CUSTOM'` `FORMAT` with the 
`SequenceWritable` profile. The `SequenceWritable` '`CUSTOM`' `FORMAT` supports 
only the built-in `(formatter='pxfwritable_export)` (write) and 
`(formatter='pxfwritable_import)` (read) \.
+
+**Note**: When creating PXF external tables, you cannot use the `HEADER` 
option in your `FORMAT` specification.
+
+## Custom Options
+
+The `HdfsTextSimple` and `SequenceWritable` profiles support the following 
custom options:
+
+| Option  | Value Description | Profile |
+|---|-||
+| COMPRESSION_CODEC| The compression codec Java class name. If this 
option is not provided, no data compression is performed. Supported compression 
codecs include: `org.apache.hadoop.io.compress.DefaultCodec` and 
`org.apache.hadoop.io.compress.BZip2Codec` | HdfsTextSimple, SequenceWritable |
+||  `org.apache.hadoop.io.compress.GzipCodec` | HdfsTextSimple |
+| COMPRESSION_TYPE| The compression type to employ; supported values 
are `RECORD` (the default) or `BLOCK`. | HdfsTextSimple, SequenceWritable |
+| DATA-SCHEMA| The name of the writer serialization/deserialization 
class. The jar file in which this class resides must be in the PXF class path. 
This option is required for the `SequenceWritable` profile and has no default 
value. | SequenceWritable|
+| THREAD-SAFE | Boolean value determining if a table query can run in 
multi-threaded mode. The default value is `TRUE`. Set this option to `FALSE` to 
handle all requests in a single thread for operations that are not thread-safe 
(for example, compression). |

[jira] [Commented] (HAWQ-1119) create new documentation topic for PXF writable profiles

2016-11-01 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/HAWQ-1119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15627158#comment-15627158
 ] 

ASF GitHub Bot commented on HAWQ-1119:
--

Github user dyozie commented on a diff in the pull request:

https://github.com/apache/incubator-hawq-docs/pull/46#discussion_r86049010
  
--- Diff: pxf/HDFSWritablePXF.html.md.erb ---
@@ -0,0 +1,416 @@
+---
+title: Writing Data to HDFS
+---
+
+The PXF HDFS plug-in supports writable external tables using the 
`HdfsTextSimple` and `SequenceWritable` profiles.  You might create a writable 
table to export data from a HAWQ internal table to binary or text HDFS files.
+
+Use the `HdfsTextSimple` profile when writing text data. Use the 
`SequenceWritable` profile when dealing with binary data.
+
+This section describes how to use these PXF profiles to create writable 
external tables.
+
+**Note**: Tables that you create with writable profiles can only be used 
for INSERT operations.  If you want to query inserted data, you must define a 
separate external readable table that references the new HDFS file using the 
equivalent readable profile.  ??You can also create a Hive table to access the 
HDFS file.??
+
+## Prerequisites
+
+Before working with HDFS file data using HAWQ and PXF, ensure that:
+
+-   The HDFS plug-in is installed on all cluster nodes. See [Installing 
PXF Plug-ins](InstallPXFPlugins.html) for PXF plug-in installation information.
+-   All HDFS users have read permissions to HDFS services.
+-   HDFS write permissions are provided to a restricted set of users.
+
+## Writing to PXF External Tables
+The PXF HDFS plug-in supports two writable profiles: `HdfsTextSimple` and 
`SequenceWritable`.
+
+Use the following syntax to create a HAWQ external writable table 
representing HDFS data: 
+
+``` sql
+CREATE WRITABLE EXTERNAL TABLE  
+(   [, ...] | LIKE  )
+LOCATION ('pxf://[:]/
+
?PROFILE=HdfsTextSimple|SequenceWritable[&=[...]]')
+FORMAT '[TEXT|CSV|CUSTOM]' ();
+```
+
+HDFS-plug-in-specific keywords and values used in the [CREATE EXTERNAL 
TABLE](../reference/sql/CREATE-EXTERNAL-TABLE.html) call are described in the 
table below.
+
+| Keyword  | Value |
+|---|-|
+| \[:\]| The HDFS NameNode and port. |
+| \| The path to the file in the HDFS data store. |
+| PROFILE| The `PROFILE` keyword must specify one of the values 
`HdfsTextSimple` or `SequenceWritable`. |
+| \  | \ is profile-specific. These 
options are discussed in the next topic.|
+| FORMAT 'TEXT' | Use '`TEXT`' `FORMAT` with the `HdfsTextSimple` profile 
to create a plain-text-delimited file at the location specified by 
\. The `HdfsTextSimple` '`TEXT`' `FORMAT` supports only the 
built-in `(delimiter=)` \. |
+| FORMAT 'CSV' | Use '`CSV`' `FORMAT` with the `HdfsTextSimple` profile to 
create a comma-separated-value file at the location specified by 
\.  |
+| FORMAT 'CUSTOM' | Use the `'CUSTOM'` `FORMAT` with the 
`SequenceWritable` profile. The `SequenceWritable` '`CUSTOM`' `FORMAT` supports 
only the built-in `(formatter='pxfwritable_export)` (write) and 
`(formatter='pxfwritable_import)` (read) \.
+
+**Note**: When creating PXF external tables, you cannot use the `HEADER` 
option in your `FORMAT` specification.
+
+## Custom Options
+
+The `HdfsTextSimple` and `SequenceWritable` profiles support the following 
custom options:
+
+| Option  | Value Description | Profile |
+|---|-||
+| COMPRESSION_CODEC| The compression codec Java class name. If this 
option is not provided, no data compression is performed. Supported compression 
codecs include: `org.apache.hadoop.io.compress.DefaultCodec` and 
`org.apache.hadoop.io.compress.BZip2Codec` | HdfsTextSimple, SequenceWritable |
+||  `org.apache.hadoop.io.compress.GzipCodec` | HdfsTextSimple |
+| COMPRESSION_TYPE| The compression type to employ; supported values 
are `RECORD` (the default) or `BLOCK`. | HdfsTextSimple, SequenceWritable |
+| DATA-SCHEMA| The name of the writer serialization/deserialization 
class. The jar file in which this class resides must be in the PXF class path. 
This option is required for the `SequenceWritable` profile and has no default 
value. | SequenceWritable|
+| THREAD-SAFE | Boolean value determining if a table query can run in 
multi-threaded mode. The default value is `TRUE`. Set this option to `FALSE` to 
handle all requests in a single thread for operations that are not thread-safe 
(for example, compression). |

[jira] [Commented] (HAWQ-1119) create new documentation topic for PXF writable profiles

2016-11-01 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/HAWQ-1119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15627164#comment-15627164
 ] 

ASF GitHub Bot commented on HAWQ-1119:
--

Github user dyozie commented on a diff in the pull request:

https://github.com/apache/incubator-hawq-docs/pull/46#discussion_r86052363
  
--- Diff: pxf/HDFSWritablePXF.html.md.erb ---
@@ -0,0 +1,416 @@
+---
+title: Writing Data to HDFS
+---
+
+The PXF HDFS plug-in supports writable external tables using the 
`HdfsTextSimple` and `SequenceWritable` profiles.  You might create a writable 
table to export data from a HAWQ internal table to binary or text HDFS files.
+
+Use the `HdfsTextSimple` profile when writing text data. Use the 
`SequenceWritable` profile when dealing with binary data.
+
+This section describes how to use these PXF profiles to create writable 
external tables.
+
+**Note**: Tables that you create with writable profiles can only be used 
for INSERT operations.  If you want to query inserted data, you must define a 
separate external readable table that references the new HDFS file using the 
equivalent readable profile.  ??You can also create a Hive table to access the 
HDFS file.??
+
+## Prerequisites
+
+Before working with HDFS file data using HAWQ and PXF, ensure that:
+
+-   The HDFS plug-in is installed on all cluster nodes. See [Installing 
PXF Plug-ins](InstallPXFPlugins.html) for PXF plug-in installation information.
+-   All HDFS users have read permissions to HDFS services.
+-   HDFS write permissions are provided to a restricted set of users.
+
+## Writing to PXF External Tables
+The PXF HDFS plug-in supports two writable profiles: `HdfsTextSimple` and 
`SequenceWritable`.
+
+Use the following syntax to create a HAWQ external writable table 
representing HDFS data: 
+
+``` sql
+CREATE WRITABLE EXTERNAL TABLE  
+(   [, ...] | LIKE  )
+LOCATION ('pxf://[:]/
+
?PROFILE=HdfsTextSimple|SequenceWritable[&=[...]]')
+FORMAT '[TEXT|CSV|CUSTOM]' ();
+```
+
+HDFS-plug-in-specific keywords and values used in the [CREATE EXTERNAL 
TABLE](../reference/sql/CREATE-EXTERNAL-TABLE.html) call are described in the 
table below.
+
+| Keyword  | Value |
+|---|-|
+| \[:\]| The HDFS NameNode and port. |
+| \| The path to the file in the HDFS data store. |
+| PROFILE| The `PROFILE` keyword must specify one of the values 
`HdfsTextSimple` or `SequenceWritable`. |
+| \  | \ is profile-specific. These 
options are discussed in the next topic.|
+| FORMAT 'TEXT' | Use '`TEXT`' `FORMAT` with the `HdfsTextSimple` profile 
to create a plain-text-delimited file at the location specified by 
\. The `HdfsTextSimple` '`TEXT`' `FORMAT` supports only the 
built-in `(delimiter=)` \. |
+| FORMAT 'CSV' | Use '`CSV`' `FORMAT` with the `HdfsTextSimple` profile to 
create a comma-separated-value file at the location specified by 
\.  |
+| FORMAT 'CUSTOM' | Use the `'CUSTOM'` `FORMAT` with the 
`SequenceWritable` profile. The `SequenceWritable` '`CUSTOM`' `FORMAT` supports 
only the built-in `(formatter='pxfwritable_export)` (write) and 
`(formatter='pxfwritable_import)` (read) \.
+
+**Note**: When creating PXF external tables, you cannot use the `HEADER` 
option in your `FORMAT` specification.
+
+## Custom Options
+
+The `HdfsTextSimple` and `SequenceWritable` profiles support the following 
custom options:
+
+| Option  | Value Description | Profile |
+|---|-||
+| COMPRESSION_CODEC| The compression codec Java class name. If this 
option is not provided, no data compression is performed. Supported compression 
codecs include: `org.apache.hadoop.io.compress.DefaultCodec` and 
`org.apache.hadoop.io.compress.BZip2Codec` | HdfsTextSimple, SequenceWritable |
+||  `org.apache.hadoop.io.compress.GzipCodec` | HdfsTextSimple |
+| COMPRESSION_TYPE| The compression type to employ; supported values 
are `RECORD` (the default) or `BLOCK`. | HdfsTextSimple, SequenceWritable |
+| DATA-SCHEMA| The name of the writer serialization/deserialization 
class. The jar file in which this class resides must be in the PXF class path. 
This option is required for the `SequenceWritable` profile and has no default 
value. | SequenceWritable|
+| THREAD-SAFE | Boolean value determining if a table query can run in 
multi-threaded mode. The default value is `TRUE`. Set this option to `FALSE` to 
handle all requests in a single thread for operations that are not thread-safe 
(for example, compression). |

[jira] [Commented] (HAWQ-1119) create new documentation topic for PXF writable profiles

2016-11-01 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/HAWQ-1119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15627159#comment-15627159
 ] 

ASF GitHub Bot commented on HAWQ-1119:
--

Github user dyozie commented on a diff in the pull request:

https://github.com/apache/incubator-hawq-docs/pull/46#discussion_r86050201
  
--- Diff: pxf/HDFSWritablePXF.html.md.erb ---
@@ -0,0 +1,416 @@
+---
+title: Writing Data to HDFS
+---
+
+The PXF HDFS plug-in supports writable external tables using the 
`HdfsTextSimple` and `SequenceWritable` profiles.  You might create a writable 
table to export data from a HAWQ internal table to binary or text HDFS files.
+
+Use the `HdfsTextSimple` profile when writing text data. Use the 
`SequenceWritable` profile when dealing with binary data.
+
+This section describes how to use these PXF profiles to create writable 
external tables.
+
+**Note**: Tables that you create with writable profiles can only be used 
for INSERT operations.  If you want to query inserted data, you must define a 
separate external readable table that references the new HDFS file using the 
equivalent readable profile.  ??You can also create a Hive table to access the 
HDFS file.??
+
+## Prerequisites
+
+Before working with HDFS file data using HAWQ and PXF, ensure that:
+
+-   The HDFS plug-in is installed on all cluster nodes. See [Installing 
PXF Plug-ins](InstallPXFPlugins.html) for PXF plug-in installation information.
+-   All HDFS users have read permissions to HDFS services.
+-   HDFS write permissions are provided to a restricted set of users.
+
+## Writing to PXF External Tables
+The PXF HDFS plug-in supports two writable profiles: `HdfsTextSimple` and 
`SequenceWritable`.
+
+Use the following syntax to create a HAWQ external writable table 
representing HDFS data: 
+
+``` sql
+CREATE WRITABLE EXTERNAL TABLE  
+(   [, ...] | LIKE  )
+LOCATION ('pxf://[:]/
+
?PROFILE=HdfsTextSimple|SequenceWritable[&=[...]]')
+FORMAT '[TEXT|CSV|CUSTOM]' ();
+```
+
+HDFS-plug-in-specific keywords and values used in the [CREATE EXTERNAL 
TABLE](../reference/sql/CREATE-EXTERNAL-TABLE.html) call are described in the 
table below.
+
+| Keyword  | Value |
+|---|-|
+| \[:\]| The HDFS NameNode and port. |
+| \| The path to the file in the HDFS data store. |
+| PROFILE| The `PROFILE` keyword must specify one of the values 
`HdfsTextSimple` or `SequenceWritable`. |
+| \  | \ is profile-specific. These 
options are discussed in the next topic.|
+| FORMAT 'TEXT' | Use '`TEXT`' `FORMAT` with the `HdfsTextSimple` profile 
to create a plain-text-delimited file at the location specified by 
\. The `HdfsTextSimple` '`TEXT`' `FORMAT` supports only the 
built-in `(delimiter=)` \. |
+| FORMAT 'CSV' | Use '`CSV`' `FORMAT` with the `HdfsTextSimple` profile to 
create a comma-separated-value file at the location specified by 
\.  |
+| FORMAT 'CUSTOM' | Use the `'CUSTOM'` `FORMAT` with the 
`SequenceWritable` profile. The `SequenceWritable` '`CUSTOM`' `FORMAT` supports 
only the built-in `(formatter='pxfwritable_export)` (write) and 
`(formatter='pxfwritable_import)` (read) \.
+
+**Note**: When creating PXF external tables, you cannot use the `HEADER` 
option in your `FORMAT` specification.
+
+## Custom Options
+
+The `HdfsTextSimple` and `SequenceWritable` profiles support the following 
custom options:
+
+| Option  | Value Description | Profile |
+|---|-||
+| COMPRESSION_CODEC| The compression codec Java class name. If this 
option is not provided, no data compression is performed. Supported compression 
codecs include: `org.apache.hadoop.io.compress.DefaultCodec` and 
`org.apache.hadoop.io.compress.BZip2Codec` | HdfsTextSimple, SequenceWritable |
+||  `org.apache.hadoop.io.compress.GzipCodec` | HdfsTextSimple |
+| COMPRESSION_TYPE| The compression type to employ; supported values 
are `RECORD` (the default) or `BLOCK`. | HdfsTextSimple, SequenceWritable |
+| DATA-SCHEMA| The name of the writer serialization/deserialization 
class. The jar file in which this class resides must be in the PXF class path. 
This option is required for the `SequenceWritable` profile and has no default 
value. | SequenceWritable|
+| THREAD-SAFE | Boolean value determining if a table query can run in 
multi-threaded mode. The default value is `TRUE`. Set this option to `FALSE` to 
handle all requests in a single thread for operations that are not thread-safe 
(for example, compression). |

[jira] [Commented] (HAWQ-1119) create new documentation topic for PXF writable profiles

2016-11-01 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/HAWQ-1119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15627160#comment-15627160
 ] 

ASF GitHub Bot commented on HAWQ-1119:
--

Github user dyozie commented on a diff in the pull request:

https://github.com/apache/incubator-hawq-docs/pull/46#discussion_r86049842
  
--- Diff: pxf/HDFSWritablePXF.html.md.erb ---
@@ -0,0 +1,416 @@
+---
+title: Writing Data to HDFS
+---
+
+The PXF HDFS plug-in supports writable external tables using the 
`HdfsTextSimple` and `SequenceWritable` profiles.  You might create a writable 
table to export data from a HAWQ internal table to binary or text HDFS files.
+
+Use the `HdfsTextSimple` profile when writing text data. Use the 
`SequenceWritable` profile when dealing with binary data.
+
+This section describes how to use these PXF profiles to create writable 
external tables.
+
+**Note**: Tables that you create with writable profiles can only be used 
for INSERT operations.  If you want to query inserted data, you must define a 
separate external readable table that references the new HDFS file using the 
equivalent readable profile.  ??You can also create a Hive table to access the 
HDFS file.??
+
+## Prerequisites
+
+Before working with HDFS file data using HAWQ and PXF, ensure that:
+
+-   The HDFS plug-in is installed on all cluster nodes. See [Installing 
PXF Plug-ins](InstallPXFPlugins.html) for PXF plug-in installation information.
+-   All HDFS users have read permissions to HDFS services.
+-   HDFS write permissions are provided to a restricted set of users.
+
+## Writing to PXF External Tables
+The PXF HDFS plug-in supports two writable profiles: `HdfsTextSimple` and 
`SequenceWritable`.
+
+Use the following syntax to create a HAWQ external writable table 
representing HDFS data: 
+
+``` sql
+CREATE WRITABLE EXTERNAL TABLE  
+(   [, ...] | LIKE  )
+LOCATION ('pxf://[:]/
+
?PROFILE=HdfsTextSimple|SequenceWritable[&=[...]]')
+FORMAT '[TEXT|CSV|CUSTOM]' ();
+```
+
+HDFS-plug-in-specific keywords and values used in the [CREATE EXTERNAL 
TABLE](../reference/sql/CREATE-EXTERNAL-TABLE.html) call are described in the 
table below.
+
+| Keyword  | Value |
+|---|-|
+| \[:\]| The HDFS NameNode and port. |
+| \| The path to the file in the HDFS data store. |
+| PROFILE| The `PROFILE` keyword must specify one of the values 
`HdfsTextSimple` or `SequenceWritable`. |
+| \  | \ is profile-specific. These 
options are discussed in the next topic.|
+| FORMAT 'TEXT' | Use '`TEXT`' `FORMAT` with the `HdfsTextSimple` profile 
to create a plain-text-delimited file at the location specified by 
\. The `HdfsTextSimple` '`TEXT`' `FORMAT` supports only the 
built-in `(delimiter=)` \. |
+| FORMAT 'CSV' | Use '`CSV`' `FORMAT` with the `HdfsTextSimple` profile to 
create a comma-separated-value file at the location specified by 
\.  |
+| FORMAT 'CUSTOM' | Use the `'CUSTOM'` `FORMAT` with the 
`SequenceWritable` profile. The `SequenceWritable` '`CUSTOM`' `FORMAT` supports 
only the built-in `(formatter='pxfwritable_export)` (write) and 
`(formatter='pxfwritable_import)` (read) \.
+
+**Note**: When creating PXF external tables, you cannot use the `HEADER` 
option in your `FORMAT` specification.
+
+## Custom Options
+
+The `HdfsTextSimple` and `SequenceWritable` profiles support the following 
custom options:
+
+| Option  | Value Description | Profile |
+|---|-||
+| COMPRESSION_CODEC| The compression codec Java class name. If this 
option is not provided, no data compression is performed. Supported compression 
codecs include: `org.apache.hadoop.io.compress.DefaultCodec` and 
`org.apache.hadoop.io.compress.BZip2Codec` | HdfsTextSimple, SequenceWritable |
+||  `org.apache.hadoop.io.compress.GzipCodec` | HdfsTextSimple |
+| COMPRESSION_TYPE| The compression type to employ; supported values 
are `RECORD` (the default) or `BLOCK`. | HdfsTextSimple, SequenceWritable |
+| DATA-SCHEMA| The name of the writer serialization/deserialization 
class. The jar file in which this class resides must be in the PXF class path. 
This option is required for the `SequenceWritable` profile and has no default 
value. | SequenceWritable|
+| THREAD-SAFE | Boolean value determining if a table query can run in 
multi-threaded mode. The default value is `TRUE`. Set this option to `FALSE` to 
handle all requests in a single thread for operations that are not thread-safe 
(for example, compression). |

[jira] [Commented] (HAWQ-1119) create new documentation topic for PXF writable profiles

2016-11-01 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/HAWQ-1119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15627163#comment-15627163
 ] 

ASF GitHub Bot commented on HAWQ-1119:
--

Github user dyozie commented on a diff in the pull request:

https://github.com/apache/incubator-hawq-docs/pull/46#discussion_r86051029
  
--- Diff: pxf/HDFSWritablePXF.html.md.erb ---
@@ -0,0 +1,416 @@
+---
+title: Writing Data to HDFS
+---
+
+The PXF HDFS plug-in supports writable external tables using the 
`HdfsTextSimple` and `SequenceWritable` profiles.  You might create a writable 
table to export data from a HAWQ internal table to binary or text HDFS files.
+
+Use the `HdfsTextSimple` profile when writing text data. Use the 
`SequenceWritable` profile when dealing with binary data.
+
+This section describes how to use these PXF profiles to create writable 
external tables.
+
+**Note**: Tables that you create with writable profiles can only be used 
for INSERT operations.  If you want to query inserted data, you must define a 
separate external readable table that references the new HDFS file using the 
equivalent readable profile.  ??You can also create a Hive table to access the 
HDFS file.??
+
+## Prerequisites
+
+Before working with HDFS file data using HAWQ and PXF, ensure that:
+
+-   The HDFS plug-in is installed on all cluster nodes. See [Installing 
PXF Plug-ins](InstallPXFPlugins.html) for PXF plug-in installation information.
+-   All HDFS users have read permissions to HDFS services.
+-   HDFS write permissions are provided to a restricted set of users.
+
+## Writing to PXF External Tables
+The PXF HDFS plug-in supports two writable profiles: `HdfsTextSimple` and 
`SequenceWritable`.
+
+Use the following syntax to create a HAWQ external writable table 
representing HDFS data: 
+
+``` sql
+CREATE WRITABLE EXTERNAL TABLE  
+(   [, ...] | LIKE  )
+LOCATION ('pxf://[:]/
+
?PROFILE=HdfsTextSimple|SequenceWritable[&=[...]]')
+FORMAT '[TEXT|CSV|CUSTOM]' ();
+```
+
+HDFS-plug-in-specific keywords and values used in the [CREATE EXTERNAL 
TABLE](../reference/sql/CREATE-EXTERNAL-TABLE.html) call are described in the 
table below.
+
+| Keyword  | Value |
+|---|-|
+| \[:\]| The HDFS NameNode and port. |
+| \| The path to the file in the HDFS data store. |
+| PROFILE| The `PROFILE` keyword must specify one of the values 
`HdfsTextSimple` or `SequenceWritable`. |
+| \  | \ is profile-specific. These 
options are discussed in the next topic.|
+| FORMAT 'TEXT' | Use '`TEXT`' `FORMAT` with the `HdfsTextSimple` profile 
to create a plain-text-delimited file at the location specified by 
\. The `HdfsTextSimple` '`TEXT`' `FORMAT` supports only the 
built-in `(delimiter=)` \. |
+| FORMAT 'CSV' | Use '`CSV`' `FORMAT` with the `HdfsTextSimple` profile to 
create a comma-separated-value file at the location specified by 
\.  |
+| FORMAT 'CUSTOM' | Use the `'CUSTOM'` `FORMAT` with the 
`SequenceWritable` profile. The `SequenceWritable` '`CUSTOM`' `FORMAT` supports 
only the built-in `(formatter='pxfwritable_export)` (write) and 
`(formatter='pxfwritable_import)` (read) \.
+
+**Note**: When creating PXF external tables, you cannot use the `HEADER` 
option in your `FORMAT` specification.
+
+## Custom Options
+
+The `HdfsTextSimple` and `SequenceWritable` profiles support the following 
custom options:
+
+| Option  | Value Description | Profile |
+|---|-||
+| COMPRESSION_CODEC| The compression codec Java class name. If this 
option is not provided, no data compression is performed. Supported compression 
codecs include: `org.apache.hadoop.io.compress.DefaultCodec` and 
`org.apache.hadoop.io.compress.BZip2Codec` | HdfsTextSimple, SequenceWritable |
+||  `org.apache.hadoop.io.compress.GzipCodec` | HdfsTextSimple |
+| COMPRESSION_TYPE| The compression type to employ; supported values 
are `RECORD` (the default) or `BLOCK`. | HdfsTextSimple, SequenceWritable |
+| DATA-SCHEMA| The name of the writer serialization/deserialization 
class. The jar file in which this class resides must be in the PXF class path. 
This option is required for the `SequenceWritable` profile and has no default 
value. | SequenceWritable|
+| THREAD-SAFE | Boolean value determining if a table query can run in 
multi-threaded mode. The default value is `TRUE`. Set this option to `FALSE` to 
handle all requests in a single thread for operations that are not thread-safe 
(for example, compression). |

[jira] [Commented] (HAWQ-1134) Add Bigtop layout specific pxf-private classpath

2016-11-01 Thread Goden Yao (JIRA)


[ 
https://issues.apache.org/jira/browse/HAWQ-1134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15627138#comment-15627138
 ] 

Goden Yao commented on HAWQ-1134:
-

I think the plan is to remove PHD/HDP specific classpath files as we only have 
1 hadoop distribution to work with at the moment. the bigtop one you added 
could be the standard one for all. 

> Add Bigtop layout specific pxf-private classpath
> 
>
> Key: HAWQ-1134
> URL: https://issues.apache.org/jira/browse/HAWQ-1134
> Project: Apache HAWQ
>  Issue Type: Improvement
>  Components: PXF
>Reporter: Roman Shaposhnik
>Assignee: Roman Shaposhnik
>
> Currently PXF ships with HDP and PHD specific classpath files. It would be 
> great to have Bigtop specific one.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-hawq pull request #993: HAWQ-1134. Add Bigtop layout specific pxf-...

2016-11-01 Thread rvs

GitHub user rvs opened a pull request:

https://github.com/apache/incubator-hawq/pull/993

HAWQ-1134. Add Bigtop layout specific pxf-private classpath



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/rvs/incubator-hawq HAWQ-1134

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-hawq/pull/993.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #993


commit ad8f4e52f7993ca6091457ea51b6f91278364545
Author: Roman Shaposhnik 
Date:   2016-11-01T23:03:49Z

HAWQ-1134. Add Bigtop layout specific pxf-private classpath




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Assigned] (HAWQ-1134) Add Bigtop layout specific pxf-private classpath

2016-11-01 Thread Roman Shaposhnik (JIRA)


 [ 
https://issues.apache.org/jira/browse/HAWQ-1134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Roman Shaposhnik reassigned HAWQ-1134:
--

Assignee: Roman Shaposhnik  (was: Lei Chang)

> Add Bigtop layout specific pxf-private classpath
> 
>
> Key: HAWQ-1134
> URL: https://issues.apache.org/jira/browse/HAWQ-1134
> Project: Apache HAWQ
>  Issue Type: Improvement
>  Components: PXF
>Reporter: Roman Shaposhnik
>Assignee: Roman Shaposhnik
>
> Currently PXF ships with HDP and PHD specific classpath files. It would be 
> great to have Bigtop specific one.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HAWQ-1134) Add Bigtop layout specific pxf-private classpath

2016-11-01 Thread Roman Shaposhnik (JIRA)

Roman Shaposhnik created HAWQ-1134:
--

 Summary: Add Bigtop layout specific pxf-private classpath
 Key: HAWQ-1134
 URL: https://issues.apache.org/jira/browse/HAWQ-1134
 Project: Apache HAWQ
  Issue Type: Improvement
  Components: PXF
Reporter: Roman Shaposhnik
Assignee: Lei Chang


Currently PXF ships with HDP and PHD specific classpath files. It would be 
great to have Bigtop specific one.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HAWQ-1119) create new documentation topic for PXF writable profiles

2016-11-01 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/HAWQ-1119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15626949#comment-15626949
 ] 

ASF GitHub Bot commented on HAWQ-1119:
--

Github user dyozie commented on a diff in the pull request:

https://github.com/apache/incubator-hawq-docs/pull/46#discussion_r86042974
  
--- Diff: pxf/HDFSWritablePXF.html.md.erb ---
@@ -0,0 +1,410 @@
+---
+title: Writing Data to HDFS
+---
+
+The PXF HDFS plug-in supports writable external tables using the 
`HdfsTextSimple` and `SequenceWritable` profiles.  You might create a writable 
table to export data from a HAWQ internal table to HDFS.
+
+This section describes how to use these PXF profiles to create writable 
external tables.
+
+**Note**: You cannot directly query data in a HAWQ writable table.  After 
creating the external writable table, you must create a HAWQ readable external 
table accessing the HDFS file, then query that table. ??You can also create a 
Hive table to access the HDFS file.??
+
+## Prerequisites
+
+Before working with HDFS file data using HAWQ and PXF, ensure that:
+
+-   The HDFS plug-in is installed on all cluster nodes. See [Installing 
PXF Plug-ins](InstallPXFPlugins.html) for PXF plug-in installation information.
+-   All HDFS users have read permissions to HDFS services and that write 
permissions have been restricted to specific users.
+
+## Writing to PXF External Tables
+The PXF HDFS plug-in supports writable two profiles: `HdfsTextSimple` and 
`SequenceWritable`.
+
+Use the following syntax to create a HAWQ external writable table 
representing HDFS data: 
+
+``` sql
+CREATE EXTERNAL WRITABLE TABLE  
+(   [, ...] | LIKE  )
+LOCATION ('pxf://[:]/
+
?PROFILE=HdfsTextSimple|SequenceWritable[&=[...]]')
+FORMAT '[TEXT|CSV|CUSTOM]' ();
+```
+
+HDFS-plug-in-specific keywords and values used in the [CREATE EXTERNAL 
TABLE](../reference/sql/CREATE-EXTERNAL-TABLE.html) call are described in the 
table below.
+
+| Keyword  | Value |
+|---|-|
+| \[:\]| The HDFS NameNode and port. |
+| \| The path to the file in the HDFS data store. |
+| PROFILE| The `PROFILE` keyword must specify one of the values 
`HdfsTextSimple` or `SequenceWritable`. |
+| \  | \ is profile-specific. These 
options are discussed in the next topic.|
--- End diff --

Thanks - that sounds good to me.


> create new documentation topic for PXF writable profiles
> 
>
> Key: HAWQ-1119
> URL: https://issues.apache.org/jira/browse/HAWQ-1119
> Project: Apache HAWQ
>  Issue Type: Improvement
>  Components: Documentation
>Reporter: Lisa Owen
>Assignee: David Yozie
> Fix For: 2.0.1.0-incubating
>
>
> certain profiles supported by the existing PXF plug-ins support writable 
> tables.  create some documentation content for these profiles.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HAWQ-1119) create new documentation topic for PXF writable profiles

2016-11-01 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/HAWQ-1119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15626933#comment-15626933
 ] 

ASF GitHub Bot commented on HAWQ-1119:
--

Github user lisakowen commented on a diff in the pull request:

https://github.com/apache/incubator-hawq-docs/pull/46#discussion_r86042160
  
--- Diff: pxf/HDFSWritablePXF.html.md.erb ---
@@ -0,0 +1,410 @@
+---
+title: Writing Data to HDFS
+---
+
+The PXF HDFS plug-in supports writable external tables using the 
`HdfsTextSimple` and `SequenceWritable` profiles.  You might create a writable 
table to export data from a HAWQ internal table to HDFS.
+
+This section describes how to use these PXF profiles to create writable 
external tables.
+
+**Note**: You cannot directly query data in a HAWQ writable table.  After 
creating the external writable table, you must create a HAWQ readable external 
table accessing the HDFS file, then query that table. ??You can also create a 
Hive table to access the HDFS file.??
+
+## Prerequisites
+
+Before working with HDFS file data using HAWQ and PXF, ensure that:
+
+-   The HDFS plug-in is installed on all cluster nodes. See [Installing 
PXF Plug-ins](InstallPXFPlugins.html) for PXF plug-in installation information.
+-   All HDFS users have read permissions to HDFS services and that write 
permissions have been restricted to specific users.
+
+## Writing to PXF External Tables
+The PXF HDFS plug-in supports writable two profiles: `HdfsTextSimple` and 
`SequenceWritable`.
+
+Use the following syntax to create a HAWQ external writable table 
representing HDFS data: 
+
+``` sql
+CREATE EXTERNAL WRITABLE TABLE  
+(   [, ...] | LIKE  )
+LOCATION ('pxf://[:]/
+
?PROFILE=HdfsTextSimple|SequenceWritable[&=[...]]')
+FORMAT '[TEXT|CSV|CUSTOM]' ();
+```
+
+HDFS-plug-in-specific keywords and values used in the [CREATE EXTERNAL 
TABLE](../reference/sql/CREATE-EXTERNAL-TABLE.html) call are described in the 
table below.
+
+| Keyword  | Value |
+|---|-|
+| \[:\]| The HDFS NameNode and port. |
+| \| The path to the file in the HDFS data store. |
+| PROFILE| The `PROFILE` keyword must specify one of the values 
`HdfsTextSimple` or `SequenceWritable`. |
+| \  | \ is profile-specific. These 
options are discussed in the next topic.|
--- End diff --

profile-options is a better name, i agree.  all of the plug-in pages are 
currently using custom-options.  i will make the change to profile-options for 
all plug-ins in a separate PR after this is accepted.  sound ok?


> create new documentation topic for PXF writable profiles
> 
>
> Key: HAWQ-1119
> URL: https://issues.apache.org/jira/browse/HAWQ-1119
> Project: Apache HAWQ
>  Issue Type: Improvement
>  Components: Documentation
>Reporter: Lisa Owen
>Assignee: David Yozie
> Fix For: 2.0.1.0-incubating
>
>
> certain profiles supported by the existing PXF plug-ins support writable 
> tables.  create some documentation content for these profiles.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-hawq issue #981: HAWQ-1114. Implement filter-push down for IN on H...

2016-11-01 Thread shivzone

Github user shivzone commented on the issue:

https://github.com/apache/incubator-hawq/pull/981
  
Why aren't we adding PXFOP_IN to PxfOperatorCode enum ?
For the other operators, PXF service expects to see the opcode value to 
correspond to the PxfOperatorCode enum


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq pull request #981: HAWQ-1114. Implement filter-push down for ...

2016-11-01 Thread shivzone

Github user shivzone commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/981#discussion_r85998628
  
--- Diff: src/backend/access/external/pxffilters.c ---
@@ -357,6 +393,46 @@ pxf_serialize_filter_list(List *expressionItems)
 
switch (tag)
{
+   case T_Var:
+   {
+   elog(DEBUG1, "pxf_serialize_filter_list: node 
tag %d (T_Var)", tag);
+   PxfFilterDesc *filter = (PxfFilterDesc *) 
palloc0(sizeof(PxfFilterDesc));
+   Var *var = (Var *) node;
+   if (var_to_pxffilter(var, filter))
+   {
+   PxfOperand l = filter->l;
+   PxfOperand r = filter->r;
+   PxfOperatorCode o = filter->op;
+   if (pxfoperand_is_attr(l) && 
pxfoperand_is_scalar_const(r))
--- End diff --

Can we have a scenario which is left operand is scalar const and right 
operand is attirbute ?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq pull request #992: HAWQ-1133. Replace print with logger outpu...

2016-11-01 Thread xunzhang

GitHub user xunzhang opened a pull request:

https://github.com/apache/incubator-hawq/pull/992

HAWQ-1133. Replace print with logger output in hawqregister.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/xunzhang/incubator-hawq HAWQ-1133

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-hawq/pull/992.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #992


commit 3336cae66ec67789c6e23177e44f14486db127eb
Author: xunzhang 
Date:   2016-11-01T11:39:29Z

HAWQ-1133. Replace print with logger output in hawqregister.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq issue #992: HAWQ-1133. Replace print with logger output in ha...

2016-11-01 Thread xunzhang

Github user xunzhang commented on the issue:

https://github.com/apache/incubator-hawq/pull/992
  
cc @ictmalili @zhangh43 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (HAWQ-1133) Should print out date/time information in hawq register output.

2016-11-01 Thread hongwu (JIRA)

hongwu created HAWQ-1133:


 Summary: Should print out date/time information in hawq register 
output.
 Key: HAWQ-1133
 URL: https://issues.apache.org/jira/browse/HAWQ-1133
 Project: Apache HAWQ
  Issue Type: Sub-task
  Components: Command Line Tools
Reporter: hongwu
Assignee: Lei Chang
 Fix For: 2.0.1.0-incubating






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (HAWQ-1133) Should print out date/time information in hawq register output.

2016-11-01 Thread hongwu (JIRA)


 [ 
https://issues.apache.org/jira/browse/HAWQ-1133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

hongwu reassigned HAWQ-1133:


Assignee: hongwu  (was: Lei Chang)

> Should print out date/time information in hawq register output.
> ---
>
> Key: HAWQ-1133
> URL: https://issues.apache.org/jira/browse/HAWQ-1133
> Project: Apache HAWQ
>  Issue Type: Sub-task
>  Components: Command Line Tools
>Reporter: hongwu
>Assignee: hongwu
> Fix For: 2.0.1.0-incubating
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HAWQ-1128) Support HAWQ register tables with same file name in different schema

2016-11-01 Thread hongwu (JIRA)


 [ 
https://issues.apache.org/jira/browse/HAWQ-1128?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

hongwu updated HAWQ-1128:
-
Fix Version/s: (was: backlog)
   2.0.1.0-incubating

> Support HAWQ register tables with same file name in different schema
> 
>
> Key: HAWQ-1128
> URL: https://issues.apache.org/jira/browse/HAWQ-1128
> Project: Apache HAWQ
>  Issue Type: Sub-task
>  Components: Command Line Tools
>Affects Versions: 2.0.1.0-incubating
>Reporter: Chunling Wang
>Assignee: hongwu
> Fix For: 2.0.1.0-incubating
>
>
> Now, in HAWQ Register, it can not distinguish tables with same file name but 
> in different schema, which are regarded as same table. We should save and use 
> schema information for HAWQ register.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-hawq issue #990: HAWQ-1128. Support HAWQ register tables with same...

2016-11-01 Thread xunzhang

Github user xunzhang commented on the issue:

https://github.com/apache/incubator-hawq/pull/990
  
also cc @linwen 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (HAWQ-1128) Support HAWQ register tables with same file name in different schema

2016-11-01 Thread hongwu (JIRA)


 [ 
https://issues.apache.org/jira/browse/HAWQ-1128?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

hongwu updated HAWQ-1128:
-
Affects Version/s: 2.0.1.0-incubating

> Support HAWQ register tables with same file name in different schema
> 
>
> Key: HAWQ-1128
> URL: https://issues.apache.org/jira/browse/HAWQ-1128
> Project: Apache HAWQ
>  Issue Type: Sub-task
>  Components: Command Line Tools
>Affects Versions: 2.0.1.0-incubating
>Reporter: Chunling Wang
>Assignee: hongwu
> Fix For: 2.0.1.0-incubating
>
>
> Now, in HAWQ Register, it can not distinguish tables with same file name but 
> in different schema, which are regarded as same table. We should save and use 
> schema information for HAWQ register.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HAWQ-1132) HAWQ should throw error when we insert data in a hash table and the virtual segment number is 1

2016-11-01 Thread Chunling Wang (JIRA)

Chunling Wang created HAWQ-1132:
---

 Summary: HAWQ should throw error when we insert data in a hash 
table and the virtual segment number is 1
 Key: HAWQ-1132
 URL: https://issues.apache.org/jira/browse/HAWQ-1132
 Project: Apache HAWQ
  Issue Type: Bug
  Components: Core, Planner, Query Execution
Reporter: Chunling Wang
Assignee: Lei Chang


If we set  virtual segment number is 1, and create a hash table (default hash 
number is 6), we will just get a warning message in a no partition table when 
we insert a tuple. And we can even not get any message in a partition table. 
When we select from this table, HAWQ throws error. 

No partition table:
{code}
postgres=# set enforce_virtual_segment_number = 1;
SET
postgres=# create table t(id int) DISTRIBUTED BY (id);
CREATE TABLE
postgres=# insert into t values(1);
WARNING:  skipping "t" --- error returned: file count 1 in catalog is not in 
proportion to the bucket number 6 of hash table with oid=2966724, some data may 
be lost, if you still want to continue the query by considering the table as 
random, set GUC allow_file_count_bucket_num_mismatch to on and try again.
INFO:  ANALYZE completed. Success: 0, Failure: 1 (t)
INSERT 0 1
postgres=# select * from t;
ERROR:  file count 1 in catalog is not in proportion to the bucket number 6 of 
hash table with oid=2966724, some data may be lost, if you still want to 
continue the query by considering the table as random, set GUC 
allow_file_count_bucket_num_mismatch to on and try again. 
(cdbdatalocality.c:3801)
postgres=#
{code}

Partition table:
{code}
postgres=# set enforce_virtual_segment_number = 1;
SET
postgres=# CREATE TABLE t (id int, rank int, year int, gender char(1), count 
int ) DISTRIBUTED BY (id) PARTITION BY LIST (gender) ( PARTITION girls 
VALUES ('F'), PARTITION boys VALUES ('M'), DEFAULT PARTITION other );
NOTICE:  CREATE TABLE will create partition "t_1_prt_girls" for table "t"
NOTICE:  CREATE TABLE will create partition "t_1_prt_boys" for table "t"
NOTICE:  CREATE TABLE will create partition "t_1_prt_other" for table "t"
CREATE TABLE
postgres=# insert into t values(51, 1, 1, 'F', 1);
INSERT 0 1
postgres=# select * from t;
ERROR:  file count 1 in catalog is not in proportion to the bucket number 6 of 
hash table with oid=2966703, some data may be lost, if you still want to 
continue the query by considering the table as random, set GUC 
allow_file_count_bucket_num_mismatch to on and try again. 
(cdbdatalocality.c:3801)
postgres=#
{code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HAWQ-1117) RM crash when init db after configure with param '--enable-cassert'

2016-11-01 Thread Xiang Sheng (JIRA)


 [ 
https://issues.apache.org/jira/browse/HAWQ-1117?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xiang Sheng updated HAWQ-1117:
--
Summary: RM crash when init db after configure with param 
'--enable-cassert'  (was: can't start hawq  cluster)

> RM crash when init db after configure with param '--enable-cassert'
> ---
>
> Key: HAWQ-1117
> URL: https://issues.apache.org/jira/browse/HAWQ-1117
> Project: Apache HAWQ
>  Issue Type: Bug
>  Components: Core
>Reporter: Devin Jia
>Assignee: Lei Chang
>
> after i upgrade hawq to 2.0.1 and build, the hawq cluster can't start.
> 1.configure and build:
> {quote}
> ./configure --prefix=/opt/hawq-build --enable-depend --enable-cassert 
> --enable-debug
> make && make install
> {quote}
> 2. start error:
> {quote}
> [gpadmin@hmaster pg_log]$ more 
> /home/gpadmin/hawq-data-directory/masterdd/pg_log/hawq-2016-10-20_133056.csv 
> 2016-10-20 13:30:56.549712 
> CST,"gpadmin","template1",p3279,th-266811104,"[local]",,2016-10-20 13:30:56 
> CST,0,,,seg-1,"FATAL","57P03","the database system is in recovery 
> mode",,,
> 0,,"postmaster.c",2656,
> 2016-10-20 13:30:56.556630 
> CST,,,p3280,th-2668111040,,,seg-1,"LOG","0","database system 
> was interrupted at 2016-10-20 13:22:51 CST",,,0,,"xlog.c",6229,
> 2016-10-20 13:30:56.558414 
> CST,,,p3280,th-2668111040,,,seg-1,"LOG","0","checkpoint 
> record is at 0/857ED8",,,0,,"xlog.c",6306,
> 2016-10-20 13:30:56.558464 
> CST,,,p3280,th-2668111040,,,seg-1,"LOG","0","redo record is 
> at 0/857ED8; undo record is at 0/0; shutdown TRUE",,,0,,"xlog.c",6340,
> 2016-10-20 13:30:56.558495 
> CST,,,p3280,th-2668111040,,,seg-1,"LOG","0","next transaction 
> ID: 0/963; next OID: 10896",,,0,,"xlog.c",6344,
> 2016-10-20 13:30:56.558522 
> CST,,,p3280,th-2668111040,,,seg-1,"LOG","0","next 
> MultiXactId: 1; next MultiXactOffset: 0",,,0,,"xlog.c",6347,
> 2016-10-20 13:30:56.558559 
> CST,,,p3280,th-2668111040,,,seg-1,"LOG","0","database system 
> was not properly shut down; automatic recovery in 
> progress",,,0,,"xlog.c",6436,
> 2016-10-20 13:30:56.563303 
> CST,,,p3280,th-2668111040,,,seg-1,"LOG","0","record with zero 
> length at 0/857F28",,,0,,"xlog.c",4110,
> 2016-10-20 13:30:56.563348 
> CST,,,p3280,th-2668111040,,,seg-1,"LOG","0","no record for 
> redo after checkpoint, skip redo and proceed for recovery 
> pass",,,0,,"xlog.c",6500,
> 2016-10-20 13:30:56.563411 
> CST,,,p3280,th-2668111040,,,seg-1,"LOG","0","end of 
> transaction log location is 0/857F28",,,0,,"xlog.c",6584,
> 2016-10-20 13:30:56.568795 
> CST,,,p3280,th-2668111040,,,seg-1,"LOG","0","Finished startup 
> pass 1.  Proceeding to startup crash recovery passes 2 and 
> 3.",,,0,,"xlog.c",681
> 8,
> 2016-10-20 13:30:56.580641 
> CST,,,p3281,th-2668111040,,,seg-1,"LOG","0","Finished startup 
> crash recovery pass 2",,,0,,"xlog.c",6989,
> 2016-10-20 13:30:56.595325 
> CST,,,p3282,th-2668111040,,,seg-1,"LOG","0","recovery restart 
> point at 0/857ED8","xlog redo checkpoint: redo 0/857ED8; undo 0/0; tli 1; 
> xid 0/
> 963; oid 10896; multi 1; offset 0; shutdown
> REDO PASS 3 @ 0/857ED8; LSN 0/857F28: prev 0/857E88; xid 0: XLOG - 
> checkpoint: redo 0/857ED8; undo 0/0; tli 1; xid 0/963; oid 10896; multi 1; 
> offset 0; shutdown",,0,,"xlog.c",8331,
> 2016-10-20 13:30:56.595390 
> CST,,,p3282,th-2668111040,,,seg-1,"LOG","0","record with zero 
> length at 0/857F28",,,0,,"xlog.c",4110,
> 2016-10-20 13:30:56.595477 
> CST,,,p3282,th-2668111040,,,seg-1,"LOG","0","Oldest active 
> transaction from prepared transactions 963",,,0,,"xlog.c",5998,
> 2016-10-20 13:30:56.603266 
> CST,,,p3282,th-2668111040,,,seg-1,"LOG","0","database system 
> is ready",,,0,,"xlog.c",6024,
> 2016-10-20 13:30:56.603314 
> CST,,,p3282,th-2668111040,,,seg-1,"LOG","0","PostgreSQL 
> 8.2.15 (Greenplum Database 4.2.0 build 1) (HAWQ 2.0.1.0 build dev) on 
> x86_64-unknown-linux
> -gnu, compiled by GCC gcc (GCC) 4.8.2 20140120 (Red Hat 4.8.2-15) compiled on 
> Oct 20 2016 12:27:04 (with assert checking)",,,0,,"xlog.c",6034,
> 2016-10-20 13:30:56.607520 
> CST,,,p3282,th-2668111040,,,seg-1,"LOG","0","Finished startup 
> crash recovery pass 3",,,0,,"xlog.c",7133,
> 2016-10-20 13:30:56.632316 
> CST,,,p3283,th-2668111040,,,seg-1,"LOG","0","Finished startup 
> integrity checking",,,0,,"xlog.c",7161,
> 2016-10-20 13:30:56.645485 
> CST,,,p3290,th-2668111040,con4,,seg-1,"LOG","0","Resource 
> manager starts accepting resource request. Listening

[jira] [Commented] (HAWQ-1129) plr-hawq default install directory should be the same as HAWQ install directory

2016-11-01 Thread Radar Lei (JIRA)


[ 
https://issues.apache.org/jira/browse/HAWQ-1129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15624717#comment-15624717
 ] 

Radar Lei commented on HAWQ-1129:
-

Closed by:

https://github.com/apache/incubator-hawq/pull/989

> plr-hawq default install directory should be the same as HAWQ install 
> directory
> ---
>
> Key: HAWQ-1129
> URL: https://issues.apache.org/jira/browse/HAWQ-1129
> Project: Apache HAWQ
>  Issue Type: Bug
>  Components: Command Line Tools
>Reporter: Radar Lei
>Assignee: Radar Lei
> Fix For: 2.0.1.0-incubating
>
>
> Currently plr-hawq default install path is '/usr/local/hawq_2_0_1_0-2.0.1.0', 
> this is different from the hawq default install path 
> '/usr/local/hawq_2_0_1_0'.
> We should keep them the same.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (HAWQ-1129) plr-hawq default install directory should be the same as HAWQ install directory

2016-11-01 Thread Radar Lei (JIRA)


 [ 
https://issues.apache.org/jira/browse/HAWQ-1129?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Radar Lei resolved HAWQ-1129.
-
Resolution: Fixed

> plr-hawq default install directory should be the same as HAWQ install 
> directory
> ---
>
> Key: HAWQ-1129
> URL: https://issues.apache.org/jira/browse/HAWQ-1129
> Project: Apache HAWQ
>  Issue Type: Bug
>  Components: Command Line Tools
>Reporter: Radar Lei
>Assignee: Radar Lei
> Fix For: 2.0.1.0-incubating
>
>
> Currently plr-hawq default install path is '/usr/local/hawq_2_0_1_0-2.0.1.0', 
> this is different from the hawq default install path 
> '/usr/local/hawq_2_0_1_0'.
> We should keep them the same.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (HAWQ-1131) Fix error message difference between Orca and Planner

2016-11-01 Thread Haisheng Yuan (JIRA)


 [ 
https://issues.apache.org/jira/browse/HAWQ-1131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Haisheng Yuan reassigned HAWQ-1131:
---

Assignee: Haisheng Yuan  (was: Lei Chang)

> Fix error message difference between Orca and Planner
> -
>
> Key: HAWQ-1131
> URL: https://issues.apache.org/jira/browse/HAWQ-1131
> Project: Apache HAWQ
>  Issue Type: Bug
>  Components: Optimizer
>Reporter: Haisheng Yuan
>Assignee: Haisheng Yuan
>
> In ORCA, due to the way exception handled previously we do warning first
> and then later print error referring that message. Only a single message 
> should be printed. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HAWQ-1131) Fix error message difference between Orca and Planner

2016-11-01 Thread Haisheng Yuan (JIRA)


 [ 
https://issues.apache.org/jira/browse/HAWQ-1131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Haisheng Yuan updated HAWQ-1131:

Summary: Fix error message difference between Orca and Planner  (was: Fix 
error message between Orca and Planner)

> Fix error message difference between Orca and Planner
> -
>
> Key: HAWQ-1131
> URL: https://issues.apache.org/jira/browse/HAWQ-1131
> Project: Apache HAWQ
>  Issue Type: Bug
>  Components: Optimizer
>Reporter: Haisheng Yuan
>Assignee: Lei Chang
>
> In ORCA, due to the way exception handled previously we do warning first
> and then later print error referring that message. Only a single message 
> should be printed. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-hawq pull request #991: HAWQ-1131. Fix error message difference be...

2016-11-01 Thread hsyuan

GitHub user hsyuan opened a pull request:

https://github.com/apache/incubator-hawq/pull/991

HAWQ-1131. Fix error message difference between Orca and Planner

- In ORCA, due to the way exception handled previously we do warning first
and then later print error referring that message. In this commit, we
enhanced the exception handling so we just print a single error message.
- Removed 'PQO unable to generate a plan' or 'Aborting PQO plan
generation' message and make the error message as close as the planner.
- Updated error message with filename and line number from where the 
exception
is raised.
- Updated regression test expected output files

Patches are ported the following commits from GPDB:

https://github.com/greenplum-db/gpdb/commit/4eb5db7a6bc8ac3faeca761193dc37e2c70e1ea3

https://github.com/greenplum-db/gpdb/commit/d41ebde141104dd3db6fa29ac46b4096ee7d4825

Author: Karthikeyan Jambu Rajaraman 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/hsyuan/incubator-hawq master

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-hawq/pull/991.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #991


commit 339c6102750f21c3d7762505b5da134b51244207
Author: Haisheng Yuan 
Date:   2016-11-01T07:27:24Z

HAWQ-1131. Fix error message difference between Orca and Planner

- In ORCA, due to the way exception handled previously we do warning first
and then later print error referring that message. In this commit, we
enhanced the exception handling so we just print a single error message.
- Removed 'PQO unable to generate a plan' or 'Aborting PQO plan
generation' message and make the error message as close as the planner.
- Updated error message with filename and line number from where the 
exception
is raised.
- Updated regression test expected output files

Patches are ported the following commits from GPDB:

https://github.com/greenplum-db/gpdb/commit/4eb5db7a6bc8ac3faeca761193dc37e2c70e1ea3

https://github.com/greenplum-db/gpdb/commit/d41ebde141104dd3db6fa29ac46b4096ee7d4825

Author: Karthikeyan Jambu Rajaraman 




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (HAWQ-1131) Fix error message between Orca and Planner

2016-11-01 Thread Haisheng Yuan (JIRA)

Haisheng Yuan created HAWQ-1131:
---

 Summary: Fix error message between Orca and Planner
 Key: HAWQ-1131
 URL: https://issues.apache.org/jira/browse/HAWQ-1131
 Project: Apache HAWQ
  Issue Type: Bug
  Components: Optimizer
Reporter: Haisheng Yuan
Assignee: Lei Chang


In ORCA, due to the way exception handled previously we do warning first
and then later print error referring that message. Only a single message should 
be printed. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

45 matches

Mail list logo