[GitHub] incubator-hawq pull request #1267: HAWQ-1504 - Fixed namenode hang during do...

2017-09-04 Thread outofmem0ry 
Github user outofmem0ry closed the pull request at:

https://github.com/apache/incubator-hawq/pull/1267


---

[jira] [Commented] (HAWQ-786) Framework to support pluggable formats and file systems

2017-09-04 Thread Yi Jin (JIRA) 

[ 
https://issues.apache.org/jira/browse/HAWQ-786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16152365#comment-16152365
 ] 

Yi Jin commented on HAWQ-786:
-

Now let's assign Ruilong to complete this feature. Thanks.

> Framework to support pluggable formats and file systems
> ---
>
> Key: HAWQ-786
> URL: https://issues.apache.org/jira/browse/HAWQ-786
> Project: Apache HAWQ
>  Issue Type: Wish
>  Components: Storage
>Reporter: Lei Chang
>Assignee: Ruilong Huo
> Fix For: 2.3.0.0-incubating
>
> Attachments: ORCdesign-v0.1-2016-06-17.pdf
>
>
> In current HAWQ, two native formats are supported: AO and parquet. Now we 
> want to support ORC. A framework to support naive c/c++ pluggable format is 
> needed to support ORC more easily. And it can also be potentially used for 
> fast external data access.
> And there are a lot of requests for supporting S3, Ceph and other file 
> systems, and this is closely related to pluggable formats, so this JIRA is 
> proposing a framework to support both.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (HAWQ-786) Framework to support pluggable formats and file systems

2017-09-04 Thread Yi Jin (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HAWQ-786?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yi Jin reassigned HAWQ-786:
---

Assignee: Ruilong Huo  (was: Lei Chang)

> Framework to support pluggable formats and file systems
> ---
>
> Key: HAWQ-786
> URL: https://issues.apache.org/jira/browse/HAWQ-786
> Project: Apache HAWQ
>  Issue Type: Wish
>  Components: Storage
>Reporter: Lei Chang
>Assignee: Ruilong Huo
> Fix For: 2.3.0.0-incubating
>
> Attachments: ORCdesign-v0.1-2016-06-17.pdf
>
>
> In current HAWQ, two native formats are supported: AO and parquet. Now we 
> want to support ORC. A framework to support naive c/c++ pluggable format is 
> needed to support ORC more easily. And it can also be potentially used for 
> fast external data access.
> And there are a lot of requests for supporting S3, Ceph and other file 
> systems, and this is closely related to pluggable formats, so this JIRA is 
> proposing a framework to support both.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] incubator-hawq issue #1263: HAWQ-1495 Corrected answer file to match insert ...

2017-09-04 Thread radarwave 
Github user radarwave commented on the issue:

https://github.com/apache/incubator-hawq/pull/1263
  
@outofmem0ry Please squash your commits and rewrite the commit messages as 
now the answer file is not changed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq issue #1267: HAWQ-1504 - Fixed namenode hang during docker co...

2017-09-04 Thread radarwave 
Github user radarwave commented on the issue:

https://github.com/apache/incubator-hawq/pull/1267
  
@outofmem0ry Code merged, please close this PR. Thanks.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq issue #1285: HAWQ-1520. Create filespace should also skip hdf...

2017-09-04 Thread linwen 
Github user linwen commented on the issue:

https://github.com/apache/incubator-hawq/pull/1285
  
+1 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq issue #1285: HAWQ-1520. Create filespace should also skip hdf...

2017-09-04 Thread radarwave 
Github user radarwave commented on the issue:

https://github.com/apache/incubator-hawq/pull/1285
  
LGTM


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Assigned] (HAWQ-1265) Support Complex Hive Schema's

2017-09-04 Thread Radar Lei (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HAWQ-1265?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Radar Lei reassigned HAWQ-1265:
---

Assignee: Oleksandr Diachenko  (was: Radar Lei)

> Support Complex Hive Schema's
> -
>
> Key: HAWQ-1265
> URL: https://issues.apache.org/jira/browse/HAWQ-1265
> Project: Apache HAWQ
>  Issue Type: Improvement
>  Components: Hcatalog, PXF
>Reporter: Michael Andre Pearce
>Assignee: Oleksandr Diachenko
>
> Currently if in hive you have Avro or other formats, where the schema is 
> complex, you cannot currently query the fields in the complex object via 
> hcatalog/pxf integration.
> In terms of Avro Schema - records, enums and complex arrays, do not work. 
> Hive fully supports this complex/object notation, as does many of the other 
> SQL tools in the Hadoop eco-system (Spark, Impala, Drill).
> These all seem to support the same styled solution of using dots for complex 
> object/path negotiation:
> SELECT schema.table.fieldA.nestedRecordFieldS FROM myavrotable;



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (HAWQ-1445) PXF JDBC Security, Extra Props, and Max Queries

2017-09-04 Thread Radar Lei (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HAWQ-1445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Radar Lei reassigned HAWQ-1445:
---

Assignee: Oleksandr Diachenko  (was: Radar Lei)

> PXF JDBC Security, Extra Props, and Max Queries
> ---
>
> Key: HAWQ-1445
> URL: https://issues.apache.org/jira/browse/HAWQ-1445
> Project: Apache HAWQ
>  Issue Type: Improvement
>  Components: PXF
>Reporter: Jon Roberts
>Assignee: Oleksandr Diachenko
>
> 1) Security
> The example has the username and password in the connection string.  
> LOCATION ('pxf://localhost:51200/demodb.myclass'
>   '?PROFILE=JDBC'
>   '_DRIVER=com.mysql.jdbc.Driver'
>   
> '_URL=jdbc:mysql://192.168.200.6:3306/demodb=root=root'
>   )
> This creates  security issue because anyone that can connect to the database 
> will be able to see the username and password of the JDBC connection.
> I suggest changing the URL to a connection profile that points to a file 
> outside of the database.  For Greenplum database and S3, the LOCATION syntax 
> includes "config=/path/to/config_file".  The config_file contains the S3 
> credentials.  This seems like a good pattern to follow here too.
> 2) Extra Properties
> Some JDBC drivers will need many additional properties beyond the URL and 
> this requires setting it with a put to a Properties variable.  An example of 
> this is Oracle's defaultRowPrefetch property that needs to be updated from 
> the default of 10 which is designed for OLTP to something larger like 2000 
> which is more ideal for data extracts.  
> Additionally, you will need the ability to set the isolation level which is 
> done with setTransactionIsolation on the Connection.  I don't believe you can 
> set this on the connection URL either.  Many SQL Server and DB2 database 
> still don't use snapshot isolation and use dirty reads instead to prevent 
> blocking locks.  The configuration file I suggested above will need an "extra 
> properties" variable that is a delimited list of key/value pairs so you can 
> add multiple extra properties.
> 3) Max Queries
> The external table definition doesn't limit how many concurrent queries can 
> be executed on the remote server.  It would be pretty simple to create a 
> single external table using PXF JDBC that would issue thousands of concurrent 
> queries to a single source database when doing a single SELECT in HAWQ.
> Initially, we should add a max_queries variable to the configuration file 
> that I'm suggesting, that will reject queries from proceeding when a greater 
> number of PXF instances are being requested than the max_queries variable.  
> Longer term, we should implement a queueing system so we can support external 
> tables that partitions data from the source at a very small grain but without 
> killing the source database.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (HAWQ-1301) Decomission existing regression tests related to PXF and move them to feature test framwork

2017-09-04 Thread Radar Lei (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HAWQ-1301?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Radar Lei reassigned HAWQ-1301:
---

Assignee: Oleksandr Diachenko  (was: Radar Lei)

> Decomission existing regression tests related to PXF and move them to feature 
> test framwork
> ---
>
> Key: HAWQ-1301
> URL: https://issues.apache.org/jira/browse/HAWQ-1301
> Project: Apache HAWQ
>  Issue Type: Improvement
>  Components: PXF
>Reporter: Oleksandr Diachenko
>Assignee: Oleksandr Diachenko
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Closed] (HAWQ-1300) hawq cannot compile with Bison 3.x.

2017-09-04 Thread Radar Lei (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HAWQ-1300?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Radar Lei closed HAWQ-1300.
---
Resolution: Workaround

> hawq cannot compile with Bison 3.x.
> ---
>
> Key: HAWQ-1300
> URL: https://issues.apache.org/jira/browse/HAWQ-1300
> Project: Apache HAWQ
>  Issue Type: Bug
>  Components: Build
>Reporter: Lei Chang
>Assignee: Radar Lei
> Fix For: backlog
>
>
> Yes, I met similar issue, Bison 3.x does not work for HAWQ now.
> On Mon, Jan 30, 2017 at 12:37 PM, Dmitry Bouzolin <
> dbouzo...@yahoo.com.invalid> wrote:
> > Hi Lei,
> > I use Bison 3.0.2. And looks actually like a bug in gram.c source for this
> > Bison version.The function refers yyscanner which is not defined. I will
> > reach out Bison bug list.Thanks for reply!
> >
> > On Sunday, January 29, 2017 8:09 PM, Lei Chang 
> > wrote:
> >
> >
> >  Hi Dmitry,
> >
> > Which bison version do you use? Looks this is a known issue when compiling
> > hawq on latest bison (3.x) version.  Bison 2.x version should work.
> >
> > Thanks
> > Lei
> >
> >
> >
> >
> > On Mon, Jan 30, 2017 at 3:41 AM, Dmitry Bouzolin <
> > dbouzo...@yahoo.com.invalid> wrote:
> >
> > > Hi All,
> > > Yes, I know arch linux is not supported, however I appreciate any clues
> > on
> > > why the build would fail like so:
> > >
> > > make -C caql allmake[4]: Entering directory
> > '/data/src/incubator-hawq/src/
> > > backend/catalog/caql'
> > > gcc -O3 -std=gnu99  -Wall -Wmissing-prototypes -Wpointer-arith
> > > -Wendif-labels -Wformat-security -fno-strict-aliasing -fwrapv
> > > -fno-aggressive-loop-optimizations  -I/usr/include/libxml2
> > > -I../../../../src/include -D_GNU_SOURCE  -I/data/src/incubator-hawq/
> > > depends/libhdfs3/build/install/opt/hawq/include
> > > -I/data/src/incubator-hawq/depends/libyarn/build/install/
> > opt/hawq/include
> > > -c -o gram.o gram.c
> > > gram.c: In function ‘caql_yyparse’:
> > > gram.c:1368:41: error: ‘yyscanner’ undeclared (first use in this
> > function)
> > >yychar = yylex (, , yyscanner);
> > >  ^
> > > gram.c:1368:41: note: each undeclared identifier is reported only once
> > for
> > > each function it appears in
> > > : recipe for target 'gram.o' failed
> > >
> > > If I build on CentOS, I get different make like for this target and build
> > > succeeds:
> > > make -C caql all
> > > make[4]: Entering directory `/data/src/incubator-hawq/src/
> > > backend/catalog/caql'
> > > gcc -O3 -std=gnu99  -Wall -Wmissing-prototypes -Wpointer-arith
> > > -Wendif-labels -Wformat-security -fno-strict-aliasing -fwrapv
> > > -fno-aggressive-loop-optimizations  -I/usr/include/libxml2
> > > -I../../../../src/include -D_GNU_SOURCE  -I/data/src/incubator-hawq/
> > > depends/libhdfs3/build/install/opt/hawq/include
> > > -I/data/src/incubator-hawq/depends/libyarn/build/install/
> > opt/hawq/include
> > > -c -o caqlanalyze.o caqlanalyze.c
> > >
> > > The difference is in input and output file. The same line in Arch
> > > completes successfully. All dependencies are in place.
> > >
> > > Thanks, Dmitry.
> > >



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] incubator-hawq issue #1285: HAWQ-1520. Create filespace should also skip hdf...

2017-09-04 Thread interma 
Github user interma commented on the issue:

https://github.com/apache/incubator-hawq/pull/1285
  
Have already tested, init passed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq issue #1285: HAWQ-1520. Create filespace should also skip hdf...

2017-09-04 Thread interma 
Github user interma commented on the issue:

https://github.com/apache/incubator-hawq/pull/1285
  
@linwen @radarwave @wengyanqing help to review, thanks!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (HAWQ-127) Create CI projects for HAWQ releases

2017-09-04 Thread Radar Lei (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HAWQ-127?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Radar Lei updated HAWQ-127:
---
Fix Version/s: (was: 2.3.0.0-incubating)
   backlog

> Create CI projects for HAWQ releases
> 
>
> Key: HAWQ-127
> URL: https://issues.apache.org/jira/browse/HAWQ-127
> Project: Apache HAWQ
>  Issue Type: Bug
>  Components: Build
>Affects Versions: 2.1.0.0-incubating
>Reporter: Lei Chang
>Assignee: Radar Lei
> Fix For: backlog
>
>
> Create Jenkins projects that build HAWQ binary, source tarballs and docker 
> images, and run sanity tests including at least installcheck-good tests for 
> each commit.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (HAWQ-1488) PXF HiveVectorizedORC profile should support Timestamp

2017-09-04 Thread Radar Lei (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HAWQ-1488?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Radar Lei reassigned HAWQ-1488:
---

Assignee: Oleksandr Diachenko  (was: Radar Lei)

> PXF HiveVectorizedORC profile should support Timestamp
> --
>
> Key: HAWQ-1488
> URL: https://issues.apache.org/jira/browse/HAWQ-1488
> Project: Apache HAWQ
>  Issue Type: Improvement
>Reporter: Oleksandr Diachenko
>Assignee: Oleksandr Diachenko
> Fix For: backlog
>
>
> As for now Timestamp datatype is not supported in HiveVectorizedORC.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (HAWQ-1270) Plugged storage back-ends for HAWQ

2017-09-04 Thread Radar Lei (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HAWQ-1270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Radar Lei reassigned HAWQ-1270:
---

Assignee: Yi Jin  (was: Radar Lei)

> Plugged storage back-ends for HAWQ
> --
>
> Key: HAWQ-1270
> URL: https://issues.apache.org/jira/browse/HAWQ-1270
> Project: Apache HAWQ
>  Issue Type: Improvement
>Reporter: Dmitry Buzolin
>Assignee: Yi Jin
>
> Since HAWQ only depends on Hadoop and Parquet for columnar format support, I 
> would like to propose pluggable storage backend design for Hawq. Hadoop is 
> already supported but there is Ceph -  a distributed, storage system which 
> offers standard Posix compliant file system, object and a block storage. Ceph 
> is also data location aware, written in C++. and is more sophisticated 
> storage backend compare to Hadoop at this time. It provides replicated and 
> erasure encoded storage pools, Other great features of Ceph are: snapshots 
> and an algorithmic approach to map data to the nodes rather than having 
> centrally managed namenodes. I don't think HDFS offers any of these features. 
> In terms of performance, Ceph should be faster than HFDS since it is written 
> on C++ and because it doesn't have scalability limitations when mapping data 
> to storage pools, compare to Hadoop, where name node is such point of 
> contention.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (HAWQ-1303) Load each partition as separate table for heterogenous tables in HCatalog

2017-09-04 Thread Radar Lei (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HAWQ-1303?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Radar Lei reassigned HAWQ-1303:
---

Assignee: Oleksandr Diachenko  (was: Radar Lei)

> Load each partition as separate table for heterogenous tables in HCatalog
> -
>
> Key: HAWQ-1303
> URL: https://issues.apache.org/jira/browse/HAWQ-1303
> Project: Apache HAWQ
>  Issue Type: Improvement
>  Components: Hcatalog, PXF
>Reporter: Oleksandr Diachenko
>Assignee: Oleksandr Diachenko
>
> Changes introduced in HAWQ-1228 made HAWQ use optimal profile/format for Hive 
> tables. But there is a limitation when HAWQ loads Hive tables into memory, it 
> loads them as one table even if a table has multiple partitions with 
> different output formats(GPDBWritable, TEXT). Thus currently it uses 
> GBDBWritable format for that case. The idea is to load each partition set of 
> one output format as a separate table, so not optimal profile, but optimal 
> output format could be used.
> Example: 
> We have Hive table with four partitions of following formats - Text, RC, ORC, 
> Sequence file.
> Currently, HAWQ will load it to memory with GPDBWritable format.
> GPDBWritable format is optimal for HiveORC, Hive profiles but not optimal for 
> HIveText and HiveRC profiles.
> With proposed changes, HAWQ should load two tables with TEXT and GPDBWritable 
> formats and use following pairs to read partitions - HiveText/TEXT, 
> HiveRC/TEXT, HiveORC/GPDBWritable, Hive/GPDBWritable.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (HAWQ-1288) Create a standalone PXF command line client

2017-09-04 Thread Radar Lei (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HAWQ-1288?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Radar Lei reassigned HAWQ-1288:
---

Assignee: Oleksandr Diachenko  (was: Radar Lei)

> Create a standalone PXF command line client
> ---
>
> Key: HAWQ-1288
> URL: https://issues.apache.org/jira/browse/HAWQ-1288
> Project: Apache HAWQ
>  Issue Type: Sub-task
>  Components: PXF
>Reporter: Roman Shaposhnik
>Assignee: Oleksandr Diachenko
>
> In order for PXF to start feeling like a standalone component it would be 
> great if we could create a command line client along the lines of hadoop fs 
> ... CLI.
> A related benefit here is a much crisper articulation of PXF APIs in code 
> (something that is currently pretty difficult to untangle from the only PXF 
> client that exists -- HAWQ's own C stub)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Closed] (HAWQ-1245) can HAWQ support alternate python module deployment directory?

2017-09-04 Thread Radar Lei (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HAWQ-1245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Radar Lei closed HAWQ-1245.
---
   Resolution: Won't Fix
Fix Version/s: backlog

> can HAWQ support alternate python module deployment directory?
> --
>
> Key: HAWQ-1245
> URL: https://issues.apache.org/jira/browse/HAWQ-1245
> Project: Apache HAWQ
>  Issue Type: Improvement
>  Components: Command Line Tools
>Reporter: Lisa Owen
>Assignee: Radar Lei
>Priority: Minor
> Fix For: backlog
>
>
> HAWQ no longer embeds python and is now using the system python installation. 
>  with this change, installing a new python module now requires root/sudo 
> access to the system python directories.  is there any reason why HAWQ would 
> not be able to support deploying python modules to an alternate directory 
> that is owned by gpadmin?  or using a python virtual environment?



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Closed] (HAWQ-1516) add hawq start "userinput.ask_yesno"

2017-09-04 Thread Radar Lei (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HAWQ-1516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Radar Lei closed HAWQ-1516.
---
   Resolution: Not A Bug
Fix Version/s: backlog

This works as design.

> add hawq start "userinput.ask_yesno" 
> -
>
> Key: HAWQ-1516
> URL: https://issues.apache.org/jira/browse/HAWQ-1516
> Project: Apache HAWQ
>  Issue Type: Improvement
>  Components: Command Line Tools
>Reporter: wangbincmss
>Assignee: Radar Lei
>Priority: Minor
> Fix For: backlog
>
>
> both  hawq  init and   hawq  stop  have "ask_yesno" ,   hawq  start  not  
> have "ask_yesno"
> "ask_yesno" will make users think twice for whether there is an alive cluster 
> and whether to proceed, I think   hawq  start should have "ask_yesno"  like 
> hawq  init and   hawq  stop



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (HAWQ-1518) Add a UDF for showing whether the data directory is an encryption zone

2017-09-04 Thread Radar Lei (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HAWQ-1518?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Radar Lei reassigned HAWQ-1518:
---

Assignee: Amy  (was: Radar Lei)

> Add a UDF for showing whether the data directory is an encryption zone
> --
>
> Key: HAWQ-1518
> URL: https://issues.apache.org/jira/browse/HAWQ-1518
> Project: Apache HAWQ
>  Issue Type: Sub-task
>  Components: Catalog
>Reporter: Hongxu Ma
>Assignee: Amy
> Fix For: 2.3.0.0-incubating
>
>
> A UDF(read only) for showing whether the data directory is an encryption zone.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (HAWQ-1504) Namenode hangs during restart of docker environment configured using incubator-hawq/contrib/hawq-docker/

2017-09-04 Thread Radar Lei (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HAWQ-1504?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Radar Lei resolved HAWQ-1504.
-
   Resolution: Fixed
Fix Version/s: 2.3.0.0-incubating

Fixed by Shubham Sharma.

> Namenode hangs during restart of docker environment configured using 
> incubator-hawq/contrib/hawq-docker/
> 
>
> Key: HAWQ-1504
> URL: https://issues.apache.org/jira/browse/HAWQ-1504
> Project: Apache HAWQ
>  Issue Type: Bug
>  Components: Command Line Tools
>Reporter: Shubham Sharma
>Assignee: Radar Lei
>Priority: Minor
> Fix For: 2.3.0.0-incubating
>
>
> After setting up an environment using instructions provided under 
> incubator-hawq/contrib/hawq-docker/, while trying to restart docker 
> containers namenode hangs and tries a namenode -format during every start.
> Steps to reproduce this issue - 
> - Navigate to incubator-hawq/contrib/hawq-docker
> - make stop
> - make start
> - docker exec -it centos7-namenode bash
> - ps -ef | grep java
> You can see namenode -format running.
> {code}
> [gpadmin@centos7-namenode data]$ ps -ef | grep java
> hdfs1110  1 00:56 ?00:00:06 
> /etc/alternatives/java_sdk/bin/java -Dproc_namenode -Xmx1000m 
> -Dhdfs.namenode=centos7-namenode -Dhadoop.log.dir=/var/log/hadoop/hdfs 
> -Dhadoop.log.file=hadoop.log -Dhadoop.home.dir=/usr/hdp/2.5.0.0-1245/hadoop 
> -Dhadoop.id.str= -Dhadoop.root.logger=INFO,console 
> -Djava.library.path=:/usr/hdp/2.5.0.0-1245/hadoop/lib/native/Linux-amd64-64:/usr/hdp/2.5.0.0-1245/hadoop/lib/native
>  -Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true 
> -Dhadoop.security.logger=INFO,NullAppender 
> org.apache.hadoop.hdfs.server.namenode.NameNode -format
> {code}
> Since namenode -format runs in interactive mode and at this stage it is 
> waiting for a (Yes/No) response, the namenode will remain stuck forever. This 
> makes hdfs unavailable.
> Root cause of the problem - 
> In the dockerfiles present under 
> incubator-hawq/contrib/hawq-docker/centos6-docker/hawq-test and 
> incubator-hawq/contrib/hawq-docker/centos7-docker/hawq-test, the docker 
> directive ENTRYPOINT executes entrypoin.sh during startup.
> The entrypoint.sh in turn executes start-hdfs.sh. start-dfs.sh checks for the 
> following - 
> {code}
> if [ ! -d /tmp/hdfs/name/current ]; then
> su -l hdfs -c "hdfs namenode -format"
>   fi
> {code}
> My assumption is it looks for fsimage and edit logs. If they are not present 
> the script assumes that this a first time initialization and namenode format 
> should be done. However, path /tmp/hdfs/name/current does not exist on 
> namenode. 
> From namenode logs it is clear that fsimage and edit logs are written under 
> /tmp/hadoop-hdfs/dfs/name/current.
> {code}
> 2017-07-18 00:55:20,892 INFO org.apache.hadoop.hdfs.server.namenode.FSImage: 
> No edit log streams selected.
> 2017-07-18 00:55:20,893 INFO org.apache.hadoop.hdfs.server.namenode.FSImage: 
> Planning to load image: 
> FSImageFile(file=/tmp/hadoop-hdfs/dfs/name/current/fsimage_000,
>  cpktTxId=000)
> 2017-07-18 00:55:20,995 INFO 
> org.apache.hadoop.hdfs.server.namenode.FSImageFormatPBINode: Loading 1 INodes.
> 2017-07-18 00:55:21,064 INFO 
> org.apache.hadoop.hdfs.server.namenode.FSImageFormatProtobuf: Loaded FSImage 
> in 0 seconds.
> 2017-07-18 00:55:21,065 INFO org.apache.hadoop.hdfs.server.namenode.FSImage: 
> Loaded image for txid 0 from 
> /tmp/hadoop-hdfs/dfs/name/current/fsimage_000
> 2017-07-18 00:55:21,084 INFO 
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Need to save fs image? 
> false (staleImage=false, haEnabled=false, isRollingUpgrade=false)
> 2017-07-18 00:55:21,084 INFO 
> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at 1
> {code}
> Thus wrong path in 
> incubator-hawq/contrib/hawq-docker/centos*-docker/hawq-test/start-hdfs.sh 
> causes namenode to hang during each restart of the containers making hdfs 
> unavailable.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (HAWQ-1521) Idle QE Processes Can't Quit After An Interval

2017-09-04 Thread Radar Lei (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HAWQ-1521?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Radar Lei reassigned HAWQ-1521:
---

Assignee: Lin Wen  (was: Radar Lei)

> Idle QE Processes Can't Quit After An Interval
> --
>
> Key: HAWQ-1521
> URL: https://issues.apache.org/jira/browse/HAWQ-1521
> Project: Apache HAWQ
>  Issue Type: Bug
>Reporter: Lin Wen
>Assignee: Lin Wen
>
> After a query is finished, there are some idle QE processes on segments. 
> These QE processes are expected to quit after a time interval, this interval 
> is controlled by a GUC gp_vmem_idle_resource_timeout, the default value is 18 
> seconds.
> However, this does't act as expected. Idle QE processes on segments always 
> exist there, unless the QD process quit. 
> The reason is in postgres.c, the codes to enable this timer can't get 
> executed. function gangsExist() always return false, since gang related 
> structures are all NULL.
>   if (IdleSessionGangTimeout > 0 && gangsExist())
>   if (!enable_sig_alarm( IdleSessionGangTimeout /* ms */, false))
>   elog(FATAL, "could not set timer for client wait 
> timeout");



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (HAWQ-1524) Travis CI build failure after upgrading protobuf to 3.4

2017-09-04 Thread Radar Lei (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HAWQ-1524?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Radar Lei resolved HAWQ-1524.
-
   Resolution: Fixed
Fix Version/s: 2.3.0.0-incubating

Fixed by Shubham Sharma.

> Travis CI build failure after upgrading protobuf to 3.4
> ---
>
> Key: HAWQ-1524
> URL: https://issues.apache.org/jira/browse/HAWQ-1524
> Project: Apache HAWQ
>  Issue Type: Bug
>  Components: Build
>Reporter: Shubham Sharma
>Assignee: Radar Lei
> Fix For: 2.3.0.0-incubating
>
>
> After upgrading the protobuf version to 3.4 , CI pipeline fails with below 
> errors. From the error message it looks like it is a problem with namespace 
> resolution while declaring stringstream and ostringstream
> {code}
> Error message -
> /Users/travis/build/apache/incubator-hawq/depends/libyarn/src/libyarnclient/LibYarnClient.cpp:248:9:
> error: unknown type name 'stringstream'; did you mean
> 'std::stringstream'?
> stringstream ss;
> ^~~~
> std::stringstream
> /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../include/c++/v1/iosfwd:153:38:
> note: 'std::stringstream' declared here
> typedef basic_stringstream stringstream;
> /Users/travis/build/apache/incubator-hawq/depends/libyarn/src/libyarnclient/LibYarnClient.cpp:299:13:
> error: unknown type name 'ostringstream'; did you mean
> 'std::ostringstream'?
> ostringstream key;
> ^
> std::ostringstream
> /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../include/c++/v1/iosfwd:152:38:
> note: 'std::ostringstream' declared here
> typedef basic_ostringstreamostringstream;
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)