date:20170213

[GitHub] incubator-hawq issue #1116: HAWQ-1323. Add license header added to each sour...

2017-02-13 Thread paul-guo-

Github user paul-guo- commented on the issue:

https://github.com/apache/incubator-hawq/pull/1116
  
+1


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq pull request #1116: HAWQ-1323. Add license header added to ea...

2017-02-13 Thread paul-guo-

Github user paul-guo- commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/1116#discussion_r100971424
  
--- Diff: pom.xml ---
@@ -129,7 +129,42 @@
   src/pl/**/*
   src/port/*
   src/template/*
-  src/test/**/*
+  src/test/**/*.in
+  src/test/**/*.mdp
+  
src/test/feature/UDF/ans/function_basics.ans.orca
+  
src/test/feature/UDF/ans/function_basics.ranger.ans.orca
+  
src/test/feature/UDF/ans/function_basics.ans.planner
+  
src/test/feature/UDF/ans/function_basics.ranger.ans.planner
+  src/test/**/*.paq
+  src/test/**/*paq
+  src/test/**/usage2case2/*
+  src/test/**/*.tbl
+  src/test/**/*.xml
+  src/test/**/*.yml
+  src/test/**/arch_config
+  src/test/regress/bkuprestore_schedule
+  
src/test/regress/current_good_schedule.EXCLUDE
+  src/test/regress/goh_schedule
+  src/test/regress/makeschedule
+  src/test/performance/sqls/crtsimple
+  src/test/performance/sqls/crtsimpleidx
+  src/test/performance/sqls/drpsimple
+  src/test/performance/sqls/orbsimple
+  src/test/performance/sqls/slcsimple
+  src/test/**/Makefile
+  src/test/performance/results/PgSQL.970926
+  src/test/performance/sqls/connection
+  src/test/feature/doxygen_template
+  src/test/regress/expected_statuses
+  src/test/feature/lib/global_init_file
+  src/test/feature/ao/sql/init_file
--- End diff --

replace all init_file lines with one line?
src/test/feature/**/init_file ?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq issue #1116: HAWQ-1323. Add license header added to each sour...

2017-02-13 Thread huor

Github user huor commented on the issue:

https://github.com/apache/incubator-hawq/pull/1116
  
+1


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq issue #1119: HAWQ-1328. Add deny and exclude policy template ...

2017-02-13 Thread zhangh43

Github user zhangh43 commented on the issue:

https://github.com/apache/incubator-hawq/pull/1119
  
@denalex @linwen Please review it.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq pull request #1119: HAWQ-1328. Add deny and exclude policy te...

2017-02-13 Thread zhangh43

GitHub user zhangh43 opened a pull request:

https://github.com/apache/incubator-hawq/pull/1119

HAWQ-1328. Add deny and exclude policy template for hawq service in ranger

add option "enableDenyAndExceptionsInPolicies": "true" in 
ranger-servicedef-hawq.json to open deny and exclude policy template by default.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/zhangh43/incubator-hawq hawq1328

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-hawq/pull/1119.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1119


commit c9a8770fa85e0529fe0e89e6e1aef651c56687cf
Author: hubertzhang 
Date:   2017-02-14T06:20:13Z

HAWQ-1328. Add deny and exclude policy template for hawq service in ranger.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Assigned] (HAWQ-1328) Add deny and exclude policy template for hawq service in ranger.

2017-02-13 Thread Hubert Zhang (JIRA)


 [ 
https://issues.apache.org/jira/browse/HAWQ-1328?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hubert Zhang reassigned HAWQ-1328:
--

Assignee: Hubert Zhang  (was: Ed Espino)

> Add deny and exclude policy template for hawq service in ranger.
> 
>
> Key: HAWQ-1328
> URL: https://issues.apache.org/jira/browse/HAWQ-1328
> Project: Apache HAWQ
>  Issue Type: Sub-task
>  Components: Security
>Reporter: Hubert Zhang
>Assignee: Hubert Zhang
> Fix For: backlog
>
>
> Currently, there is no template of deny and exclude policy for HAWQ service 
> in Ranger, we need to open this option in ranger-servicedef-hawq.json by 
> default.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Assigned] (HAWQ-256) Integrate Security with Apache Ranger

2017-02-13 Thread Hubert Zhang (JIRA)


 [ 
https://issues.apache.org/jira/browse/HAWQ-256?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hubert Zhang reassigned HAWQ-256:
-

Assignee: Lili Ma  (was: Hubert Zhang)

> Integrate Security with Apache Ranger
> -
>
> Key: HAWQ-256
> URL: https://issues.apache.org/jira/browse/HAWQ-256
> Project: Apache HAWQ
>  Issue Type: New Feature
>  Components: Security
>Reporter: Michael Andre Pearce (IG)
>Assignee: Lili Ma
> Fix For: backlog
>
> Attachments: HAWQRangerSupportDesign.pdf, 
> HAWQRangerSupportDesign_v0.2.pdf, HAWQRangerSupportDesign_v0.3.pdf
>
>
> Integrate security with Apache Ranger for a unified Hadoop security solution. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Assigned] (HAWQ-256) Integrate Security with Apache Ranger

2017-02-13 Thread Hubert Zhang (JIRA)


 [ 
https://issues.apache.org/jira/browse/HAWQ-256?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hubert Zhang reassigned HAWQ-256:
-

Assignee: Hubert Zhang  (was: Lili Ma)

> Integrate Security with Apache Ranger
> -
>
> Key: HAWQ-256
> URL: https://issues.apache.org/jira/browse/HAWQ-256
> Project: Apache HAWQ
>  Issue Type: New Feature
>  Components: Security
>Reporter: Michael Andre Pearce (IG)
>Assignee: Hubert Zhang
> Fix For: backlog
>
> Attachments: HAWQRangerSupportDesign.pdf, 
> HAWQRangerSupportDesign_v0.2.pdf, HAWQRangerSupportDesign_v0.3.pdf
>
>
> Integrate security with Apache Ranger for a unified Hadoop security solution. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (HAWQ-1328) Add deny and exclude policy template for hawq service in ranger.

2017-02-13 Thread Hubert Zhang (JIRA)

Hubert Zhang created HAWQ-1328:
--

 Summary: Add deny and exclude policy template for hawq service in 
ranger.
 Key: HAWQ-1328
 URL: https://issues.apache.org/jira/browse/HAWQ-1328
 Project: Apache HAWQ
  Issue Type: Sub-task
Reporter: Hubert Zhang
Assignee: Ed Espino


Currently, there is no template of deny and exclude policy for HAWQ service in 
Ranger, we need to open this option in ranger-servicedef-hawq.json by default.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] incubator-hawq issue #1118: HAWQ-1326. Cancel the query earlier if one of th...

2017-02-13 Thread paul-guo-

Github user paul-guo- commented on the issue:

https://github.com/apache/incubator-hawq/pull/1118
  
@jiny2 @huor @ictmalili 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq pull request #1118: HAWQ-1326. Cancel the query earlier if on...

2017-02-13 Thread paul-guo-

GitHub user paul-guo- opened a pull request:

https://github.com/apache/incubator-hawq/pull/1118

HAWQ-1326. Cancel the query earlier if one of the segments for the quâ¦

â¦ery crashes

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/paul-guo-/incubator-hawq Merge2

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-hawq/pull/1118.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1118


commit 47a42b423b60f72df24381da6df6c6e2c8acbc32
Author: Paul Guo 
Date:   2017-02-06T11:54:57Z

HAWQ-1326. Cancel the query earlier if one of the segments for the query 
crashes




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (HAWQ-1326) Cancel the query earlier if one of the segments for the query crashes

2017-02-13 Thread Paul Guo (JIRA)


 [ 
https://issues.apache.org/jira/browse/HAWQ-1326?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Paul Guo updated HAWQ-1326:
---
Summary: Cancel the query earlier if one of the segments for the query 
crashes  (was: Cancel the query if one of the segments for the query crashes)

> Cancel the query earlier if one of the segments for the query crashes
> -
>
> Key: HAWQ-1326
> URL: https://issues.apache.org/jira/browse/HAWQ-1326
> Project: Apache HAWQ
>  Issue Type: Bug
>Reporter: Paul Guo
>Assignee: Paul Guo
> Fix For: 2.2.0.0-incubating
>
>
> QD thread could hang in the loop of poll() since: 1) The alive segments could 
> wait at the interconnect for the dead segment until interconnect timeout (by 
> default 1 hour). 2) In the QD thread poll() will not sense the system-down 
> until kernel tcp keepalive messaging is triggered, however the keepalive 
> timeout is a bit long (2 hours by default on rhel6.x) and it could be 
> configured via procfs only.
> A proper solution would be using the RM heartbeat mechanism:
> RM maintains a global ID lists (stable cross node adding or removing) for all 
> nodes and keeps updating the health state via userspace heartbeat mechanism, 
> thus we could maintain a bitmap in shared memory which keeps the latest node 
> healthy info updated then we could use it in QD code, i.e. Cancel the query 
> if finding the segment node, which handles part of the query, is down.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Assigned] (HAWQ-1326) Cancel the query if one of the segments for the query crashes

2017-02-13 Thread Paul Guo (JIRA)


 [ 
https://issues.apache.org/jira/browse/HAWQ-1326?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Paul Guo reassigned HAWQ-1326:
--

Assignee: Paul Guo  (was: Ed Espino)

> Cancel the query if one of the segments for the query crashes
> -
>
> Key: HAWQ-1326
> URL: https://issues.apache.org/jira/browse/HAWQ-1326
> Project: Apache HAWQ
>  Issue Type: Bug
>Reporter: Paul Guo
>Assignee: Paul Guo
> Fix For: 2.2.0.0-incubating
>
>
> QD thread could hang in the loop of poll() since: 1) The alive segments could 
> wait at the interconnect for the dead segment until interconnect timeout (by 
> default 1 hour). 2) In the QD thread poll() will not sense the system-down 
> until kernel tcp keepalive messaging is triggered, however the keepalive 
> timeout is a bit long (2 hours by default on rhel6.x) and it could be 
> configured via procfs only.
> A proper solution would be using the RM heartbeat mechanism:
> RM maintains a global ID lists (stable cross node adding or removing) for all 
> nodes and keeps updating the health state via userspace heartbeat mechanism, 
> thus we could maintain a bitmap in shared memory which keeps the latest node 
> healthy info updated then we could use it in QD code, i.e. Cancel the query 
> if finding the segment node, which handles part of the query, is down.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Closed] (HAWQ-1327) Move ID from struct SegStatData to struct SegInfoData so that ID could be used in QD.

2017-02-13 Thread Paul Guo (JIRA)


 [ 
https://issues.apache.org/jira/browse/HAWQ-1327?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Paul Guo closed HAWQ-1327.
--
Resolution: Fixed

>  Move ID from struct SegStatData to struct SegInfoData so that ID could be 
> used in QD.
> --
>
> Key: HAWQ-1327
> URL: https://issues.apache.org/jira/browse/HAWQ-1327
> Project: Apache HAWQ
>  Issue Type: Sub-task
>Reporter: Paul Guo
>Assignee: Paul Guo
> Fix For: 2.2.0.0-incubating
>
>
> This is the sub-JIRA for
> HAWQ-1326. Cancel the query if one of the segments for the query crashes
> The summary is quite clear.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] incubator-hawq issue #1117: HAWQ-1327. Move ID from struct SegStatData to st...

2017-02-13 Thread paul-guo-

Github user paul-guo- commented on the issue:

https://github.com/apache/incubator-hawq/pull/1117
  
Merged.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq pull request #1117: HAWQ-1327. Move ID from struct SegStatDat...

2017-02-13 Thread paul-guo-

Github user paul-guo- closed the pull request at:

https://github.com/apache/incubator-hawq/pull/1117


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Assigned] (HAWQ-1327) Move ID from struct SegStatData to struct SegInfoData so that ID could be used in QD.

2017-02-13 Thread Paul Guo (JIRA)


 [ 
https://issues.apache.org/jira/browse/HAWQ-1327?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Paul Guo reassigned HAWQ-1327:
--

Assignee: Paul Guo  (was: Ed Espino)

>  Move ID from struct SegStatData to struct SegInfoData so that ID could be 
> used in QD.
> --
>
> Key: HAWQ-1327
> URL: https://issues.apache.org/jira/browse/HAWQ-1327
> Project: Apache HAWQ
>  Issue Type: Sub-task
>Reporter: Paul Guo
>Assignee: Paul Guo
> Fix For: 2.2.0.0-incubating
>
>
> This is the sub-JIRA for
> HAWQ-1326. Cancel the query if one of the segments for the query crashes
> The summary is quite clear.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] incubator-hawq issue #1117: HAWQ-1327. Move ID from struct SegStatData to st...

2017-02-13 Thread paul-guo-

Github user paul-guo- commented on the issue:

https://github.com/apache/incubator-hawq/pull/1117
  
@huor 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq pull request #1117: HAWQ-1327. Move ID from struct SegStatDat...

2017-02-13 Thread paul-guo-

GitHub user paul-guo- opened a pull request:

https://github.com/apache/incubator-hawq/pull/1117

HAWQ-1327. Move ID from struct SegStatData to struct SegInfoData so tâ¦

â¦hat ID could be used in QD.

This also fixes some warnings and some minor bugs which are found during 
warning fixing.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/paul-guo-/incubator-hawq RM

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-hawq/pull/1117.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1117


commit 0e279f358f40ce2736239853b79d707ff9fc7bfd
Author: Paul Guo 
Date:   2017-02-04T10:09:14Z

HAWQ-1327. Move ID from struct SegStatData to struct SegInfoData so that ID 
could be used in QD.

This also fixes some warnings and some minor bugs which are found during 
warning fixing.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Resolved] (HAWQ-1324) Query cancel cause segment to go into Crash recovery

2017-02-13 Thread Ming LI (JIRA)


 [ 
https://issues.apache.org/jira/browse/HAWQ-1324?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ming LI resolved HAWQ-1324.
---
   Resolution: Fixed
Fix Version/s: backlog

> Query cancel cause segment to go into Crash recovery
> 
>
> Key: HAWQ-1324
> URL: https://issues.apache.org/jira/browse/HAWQ-1324
> Project: Apache HAWQ
>  Issue Type: Bug
>Reporter: Ming LI
>Assignee: Ming LI
> Fix For: backlog
>
>
> A query was cancelled due to this connection issue to HDFS on Isilon. Seg26 
> then went into crash recovery due to a INSERT query being cancelled. What 
> should be the expected behaviour when HDFS becomes unavailable and a Query 
> fails due to HDFS unavailability.
> Below is the HDFS error
> {code}
> 2017-01-04 03:04:08.382615 
> JST,"carund","dwhrun",p574246,th1862944896,"192.168.10.12","47554",2017-01-04 
> 03:03:08 JST,0,con198952,,seg29,"FATAL","08006","connection to client 
> lost",,,0,,"postgres.c",3518,
> 2017-01-04 03:04:08.420099 
> JST,,,p755778,th18629448960,,,seg-1,"LOG","0","3rd party 
> error log:
> 2017-01-04 03:04:08.419969, p574222, th140507423066240, ERROR Handle 
> Exception: NamenodeImpl.cpp: 670: Unexpected error: status: 
> STATUS_FILE_NOT_AVAILABLE = 0xC467 Path: 
> hawq_default/16385/16563/802748/26 with path=
> ""/hawq_default/16385/16563/802748/26"", 
> clientname=libhdfs3_client_random_866998528_count_1_pid_574222_tid_140507423066240
> @ Hdfs::Internal::UnWrapper Hdfs::HdfsIOException, Hdfs::Internal::Nothing, Hdfs::Internal::Nothing, 
> Hdfs::Internal::Nothing, Hdfs::Internal::Nothing, Hdfs::Internal::Nothing , 
> Hdfs::Internal::Nothing, Hdfs::Internal::Nothing, Hdfs::Internal::Nothing, 
> Hdfs::Internal::Nothing>::unwrap(char const, int)
> @ Hdfs::Internal::UnWrapper Hdfs::UnresolvedLinkException, Hdfs::HdfsIOException, 
> Hdfs::Internal::Nothing, Hdfs::Internal::Nothing, Hdfs::Internal::Nothing, 
> Hdfs::Internal::Not hing, Hdfs::Internal::Nothing, Hdfs::Internal::Nothing, 
> Hdfs::Internal::Nothing, Hdfs::Internal::Nothing>::unwrap(char const, int)
> @ Hdfs::Internal::NamenodeImpl::fsync(std::string const&, std::string const&)
> @ Hdfs::Internal::NamenodeProxy::fsync(std::string const&, std::string const&)
> @ Hdfs::Internal::OutputStreamImpl::closePipeline()
> @ Hdfs::Internal::OutputStreamImpl::close()
> @ hdfsCloseFile
> @ gpfs_hdfs_closefile
> @ HdfsCloseFile
> @ HdfsFileClose
> @ CleanupTempFiles
> @ AbortTransaction
> @ AbortCurrentTransaction
> @ PostgresMain
> @ BackendStartup
> @ ServerLoop
> @ PostmasterMain
> @ main
> @ Unknown
> @ Unknown""SysLoggerMain","syslogger.c",518,
> 2017-01-04 03:04:08.420272 
> JST,"carund","dwhrun",p574222,th1862944896,"192.168.10.12","47550",2017-01-04 
> 03:03:08 
> JST,40678725,con198952,cmd4,seg25,,,x40678725,sx1,"WARNING","58030","could 
> not close file 7 : (hdfs://ffd
> lakehd.ffwin.fujifilm.co.jp:8020/hawq_default/16385/16563/802748/26) errno 
> 5","Unexpected error: status: STATUS_FILE_NOT_AVAILABLE = 0xC467 Path: 
> hawq_default/16385/16563/802748/26 with path=""/hawq_default/16385/16
> 563/802748/26"", 
> clientname=libhdfs3_client_random_866998528_count_1_pid_574222_tid_140507423066240",,0,,"fd.c",2762,
> {code}
> Segment 26 going into Crash recovery - from seg26 log file
> {code}
> 2017-01-04 03:04:08.420314 
> JST,"carund","dwhrun",p574222,th1862944896,"192.168.10.12","47550",2017-01-04 
> 03:03:08 
> JST,40678725,con198952,cmd4,seg25,,,x40678725,sx1,"LOG","08006","could not 
> send data to client: 接続が相
> 手からリセットされました",,,0,,"pqcomm.c",1292,
> 2017-01-04 03:04:08.420358 
> JST,"carund","dwhrun",p574222,th1862944896,"192.168.10.12","47550",2017-01-04 
> 03:03:08 JST,0,con198952,,seg25,"LOG","08006","could not send data to 
> client: パイプが切断されました",,,0,
> ,"pqcomm.c",1292,
> 2017-01-04 03:04:08.420375 
> JST,"carund","dwhrun",p574222,th1862944896,"192.168.10.12","47550",2017-01-04 
> 03:03:08 JST,0,con198952,,seg25,"FATAL","08006","connection to client 
> lost",,,0,,"postgres.c",3518,
> 2017-01-04 03:04:08.950354 
> JST,,,p755773,th18629448960,,,seg-1,"LOG","0","server process 
> (PID 574240) was terminated by signal 11: Segmentation 
> fault",,,0,,"postmaster.c",4748,
> 2017-01-04 03:04:08.950403 
> JST,,,p755773,th18629448960,,,seg-1,"LOG","0","terminating 
> any other active server processes",,,0,,"postmaster.c",4486,
> 2017-01-04 03:04:08.954044 
> JST,,,p41605,th18629448960,,,seg-1,"LOG","0","Segment RM 
> exits.",,,0,,"resourcemanager.c",340,
> 2017-01-04 03:04:08.954078 
> JST,,,p41605,th18629448960,,,seg-1,"LOG","0","Clean up 
> handler in message server is called.",,,0,,"rmcomm_MessageServer.c",105,
>

[jira] [Created] (HAWQ-1327) Move ID from struct SegStatData to struct SegInfoData so that ID could be used in QD.

2017-02-13 Thread Paul Guo (JIRA)

Paul Guo created HAWQ-1327:
--

 Summary:  Move ID from struct SegStatData to struct SegInfoData so 
that ID could be used in QD.
 Key: HAWQ-1327
 URL: https://issues.apache.org/jira/browse/HAWQ-1327
 Project: Apache HAWQ
  Issue Type: Sub-task
Reporter: Paul Guo
Assignee: Ed Espino


This is the sub-JIRA for
HAWQ-1326. Cancel the query if one of the segments for the query crashes

The summary is quite clear.




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (HAWQ-1326) Cancel the query if one of the segments for the query crashes

2017-02-13 Thread Paul Guo (JIRA)

Paul Guo created HAWQ-1326:
--

 Summary: Cancel the query if one of the segments for the query 
crashes
 Key: HAWQ-1326
 URL: https://issues.apache.org/jira/browse/HAWQ-1326
 Project: Apache HAWQ
  Issue Type: Bug
Reporter: Paul Guo
Assignee: Ed Espino
 Fix For: 2.2.0.0-incubating


QD thread could hang in the loop of poll() since: 1) The alive segments could 
wait at the interconnect for the dead segment until interconnect timeout (by 
default 1 hour). 2) In the QD thread poll() will not sense the system-down 
until kernel tcp keepalive messaging is triggered, however the keepalive 
timeout is a bit long (2 hours by default on rhel6.x) and it could be 
configured via procfs only.

A proper solution would be using the RM heartbeat mechanism:

RM maintains a global ID lists (stable cross node adding or removing) for all 
nodes and keeps updating the health state via userspace heartbeat mechanism, 
thus we could maintain a bitmap in shared memory which keeps the latest node 
healthy info updated then we could use it in QD code, i.e. Cancel the query if 
finding the segment node, which handles part of the query, is down.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] incubator-hawq issue #1116: HAWQ-1323. Add license header added to each sour...

2017-02-13 Thread amyrazz44

Github user amyrazz44 commented on the issue:

https://github.com/apache/incubator-hawq/pull/1116
  
@huor @paul-guo- @linwen please help review this pull request. thank you


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq pull request #1116: HAWQ-1323. Add license header added to ea...

2017-02-13 Thread amyrazz44

GitHub user amyrazz44 opened a pull request:

https://github.com/apache/incubator-hawq/pull/1116

HAWQ-1323. Add license header added to each source file in test directory



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/amyrazz44/incubator-hawq License

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-hawq/pull/1116.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1116


commit 418726453b705a2762ca6b8c384660e5a4a7d8be
Author: amyrazz44 
Date:   2017-02-14T02:08:03Z

HAWQ-1323. Add license header added to each source file in test directory

commit df7c51e306bebc44bc550a159e7b63c13a0529c4
Author: amyrazz44 
Date:   2017-02-14T02:46:29Z

HAWQ-1323. Add license header added to each source file in test directory

commit fd663500671b8629b17d65063787a0ebd02cb813
Author: amyrazz44 
Date:   2017-02-14T02:54:13Z

HAWQ-1323. Add license header added to each source file in test directory




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Resolved] (HAWQ-762) Hive aggregation queries through PXF sometimes hang

2017-02-13 Thread Kavinder Dhaliwal (JIRA)


 [ 
https://issues.apache.org/jira/browse/HAWQ-762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kavinder Dhaliwal resolved HAWQ-762.

   Resolution: Fixed
Fix Version/s: (was: backlog)
   2.1.0.0-incubating

> Hive aggregation queries through PXF sometimes hang
> ---
>
> Key: HAWQ-762
> URL: https://issues.apache.org/jira/browse/HAWQ-762
> Project: Apache HAWQ
>  Issue Type: Bug
>  Components: Hcatalog, PXF
>Affects Versions: 2.0.0.0-incubating
>Reporter: Oleksandr Diachenko
>Assignee: Kavinder Dhaliwal
>  Labels: performance
> Fix For: 2.1.0.0-incubating
>
>
> Reproduce Steps:
> {code}
> select count(*) from hcatalog.default.hivetable;
> {code}
> sometimes, this query will hang and we see from pxf logs that Hive thrift 
> server cannot be connected from PXF agent. 
> While users can still visit hive metastore (through HUE) and execute the same 
> query.
> After a restart of PXF agent, this query goes through without issues.
> *Troubleshooting Guide*
> - check catalina.out (tomcat) and pxf-service.log to see if the query request 
> gets to tomcat/pxf webapp, any exceptions happened during the time window
> - enable {code}log_min_messages=DEBUG2{code} to see at which step the query 
> is stuck
> - try:
> {code}
> curl http:///pxf/ProtocolVersion
> {code}
> where URI is the hostname or IP of the machine you installed PXF, port is 
> usually 51200 if you didn’t change it.
> The response you’ll get if PXF service is running OK:
> {code}
> {version: v14}
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Assigned] (HAWQ-762) Hive aggregation queries through PXF sometimes hang

2017-02-13 Thread Kavinder Dhaliwal (JIRA)


 [ 
https://issues.apache.org/jira/browse/HAWQ-762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kavinder Dhaliwal reassigned HAWQ-762:
--

Assignee: Kavinder Dhaliwal  (was: Goden Yao)

> Hive aggregation queries through PXF sometimes hang
> ---
>
> Key: HAWQ-762
> URL: https://issues.apache.org/jira/browse/HAWQ-762
> Project: Apache HAWQ
>  Issue Type: Bug
>  Components: Hcatalog, PXF
>Affects Versions: 2.0.0.0-incubating
>Reporter: Oleksandr Diachenko
>Assignee: Kavinder Dhaliwal
>  Labels: performance
> Fix For: 2.1.0.0-incubating
>
>
> Reproduce Steps:
> {code}
> select count(*) from hcatalog.default.hivetable;
> {code}
> sometimes, this query will hang and we see from pxf logs that Hive thrift 
> server cannot be connected from PXF agent. 
> While users can still visit hive metastore (through HUE) and execute the same 
> query.
> After a restart of PXF agent, this query goes through without issues.
> *Troubleshooting Guide*
> - check catalina.out (tomcat) and pxf-service.log to see if the query request 
> gets to tomcat/pxf webapp, any exceptions happened during the time window
> - enable {code}log_min_messages=DEBUG2{code} to see at which step the query 
> is stuck
> - try:
> {code}
> curl http:///pxf/ProtocolVersion
> {code}
> where URI is the hostname or IP of the machine you installed PXF, port is 
> usually 51200 if you didn’t change it.
> The response you’ll get if PXF service is running OK:
> {code}
> {version: v14}
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (HAWQ-1324) Query cancel cause segment to go into Crash recovery

2017-02-13 Thread Ming LI (JIRA)

Ming LI created HAWQ-1324:
-

 Summary: Query cancel cause segment to go into Crash recovery
 Key: HAWQ-1324
 URL: https://issues.apache.org/jira/browse/HAWQ-1324
 Project: Apache HAWQ
  Issue Type: Bug
Reporter: Ming LI
Assignee: Ed Espino


A query was cancelled due to this connection issue to HDFS on Isilon. Seg26 
then went into crash recovery due to a INSERT query being cancelled. What 
should be the expected behaviour when HDFS becomes unavailable and a Query 
fails due to HDFS unavailability.
Below is the HDFS error
{code}
2017-01-04 03:04:08.382615 
JST,"carund","dwhrun",p574246,th1862944896,"192.168.10.12","47554",2017-01-04 
03:03:08 JST,0,con198952,,seg29,"FATAL","08006","connection to client 
lost",,,0,,"postgres.c",3518,
2017-01-04 03:04:08.420099 
JST,,,p755778,th18629448960,,,seg-1,"LOG","0","3rd party error 
log:
2017-01-04 03:04:08.419969, p574222, th140507423066240, ERROR Handle Exception: 
NamenodeImpl.cpp: 670: Unexpected error: status: STATUS_FILE_NOT_AVAILABLE = 
0xC467 Path: hawq_default/16385/16563/802748/26 with path=
""/hawq_default/16385/16563/802748/26"", 
clientname=libhdfs3_client_random_866998528_count_1_pid_574222_tid_140507423066240
@ Hdfs::Internal::UnWrapper::unwrap(char const, int)
@ Hdfs::Internal::UnWrapper::unwrap(char const, int)
@ Hdfs::Internal::NamenodeImpl::fsync(std::string const&, std::string const&)
@ Hdfs::Internal::NamenodeProxy::fsync(std::string const&, std::string const&)
@ Hdfs::Internal::OutputStreamImpl::closePipeline()
@ Hdfs::Internal::OutputStreamImpl::close()
@ hdfsCloseFile
@ gpfs_hdfs_closefile
@ HdfsCloseFile
@ HdfsFileClose
@ CleanupTempFiles
@ AbortTransaction
@ AbortCurrentTransaction
@ PostgresMain
@ BackendStartup
@ ServerLoop
@ PostmasterMain
@ main
@ Unknown
@ Unknown""SysLoggerMain","syslogger.c",518,
2017-01-04 03:04:08.420272 
JST,"carund","dwhrun",p574222,th1862944896,"192.168.10.12","47550",2017-01-04 
03:03:08 
JST,40678725,con198952,cmd4,seg25,,,x40678725,sx1,"WARNING","58030","could not 
close file 7 : (hdfs://ffd
lakehd.ffwin.fujifilm.co.jp:8020/hawq_default/16385/16563/802748/26) errno 
5","Unexpected error: status: STATUS_FILE_NOT_AVAILABLE = 0xC467 Path: 
hawq_default/16385/16563/802748/26 with path=""/hawq_default/16385/16
563/802748/26"", 
clientname=libhdfs3_client_random_866998528_count_1_pid_574222_tid_140507423066240",,0,,"fd.c",2762,
{code}
Segment 26 going into Crash recovery - from seg26 log file
{code}
2017-01-04 03:04:08.420314 
JST,"carund","dwhrun",p574222,th1862944896,"192.168.10.12","47550",2017-01-04 
03:03:08 JST,40678725,con198952,cmd4,seg25,,,x40678725,sx1,"LOG","08006","could 
not send data to client: 接続が相
手からリセットされました",,,0,,"pqcomm.c",1292,
2017-01-04 03:04:08.420358 
JST,"carund","dwhrun",p574222,th1862944896,"192.168.10.12","47550",2017-01-04 
03:03:08 JST,0,con198952,,seg25,"LOG","08006","could not send data to 
client: パイプが切断されました",,,0,
,"pqcomm.c",1292,
2017-01-04 03:04:08.420375 
JST,"carund","dwhrun",p574222,th1862944896,"192.168.10.12","47550",2017-01-04 
03:03:08 JST,0,con198952,,seg25,"FATAL","08006","connection to client 
lost",,,0,,"postgres.c",3518,
2017-01-04 03:04:08.950354 
JST,,,p755773,th18629448960,,,seg-1,"LOG","0","server process 
(PID 574240) was terminated by signal 11: Segmentation 
fault",,,0,,"postmaster.c",4748,
2017-01-04 03:04:08.950403 
JST,,,p755773,th18629448960,,,seg-1,"LOG","0","terminating any 
other active server processes",,,0,,"postmaster.c",4486,
2017-01-04 03:04:08.954044 
JST,,,p41605,th18629448960,,,seg-1,"LOG","0","Segment RM 
exits.",,,0,,"resourcemanager.c",340,
2017-01-04 03:04:08.954078 
JST,,,p41605,th18629448960,,,seg-1,"LOG","0","Clean up handler 
in message server is called.",,,0,,"rmcomm_MessageServer.c",105,
2017-01-04 03:04:08.972706 
JST,,,p574711,th1862944896,"192.168.10.12","48121",2017-01-04 03:04:08 
JST,0,,,seg-1,"LOG","0","PID 574308 in cancel request did not match 
any process",,,0,,"postmaster.c",3166
,
2017-01-04 03:04:08.976211 
JST,,,p574712,th1862944896,"192.168.10.12","48127",2017-01-04 03:04:08 
JST,0,,,seg-1,"LOG","0","PID 574320 in cancel request did not match 
any

[jira] [Updated] (HAWQ-1323) Add license header added to each source file in test directory

2017-02-13 Thread Ruilong Huo (JIRA)


 [ 
https://issues.apache.org/jira/browse/HAWQ-1323?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ruilong Huo updated HAWQ-1323:
--
Description: Need to add license header each source file in src/test 
directory including *.c, *.cpp, *.h, *.py, *.pl, *.java, *.sh, make file. The 
pom.xml need to be updated accordingly to reflect it.  (was: Need to add 
license header each source file in src/test directory including *.c, *.cpp, 
*.h, *.py, *.pl, *.java, *.sh, make file.)

> Add license header added to each source file in test directory
> --
>
> Key: HAWQ-1323
> URL: https://issues.apache.org/jira/browse/HAWQ-1323
> Project: Apache HAWQ
>  Issue Type: Bug
>  Components: Tests
>Affects Versions: 2.1.0.0-incubating
>Reporter: Ruilong Huo
>Assignee: Jiali Yao
> Fix For: 2.2.0.0-incubating
>
>
> Need to add license header each source file in src/test directory including 
> *.c, *.cpp, *.h, *.py, *.pl, *.java, *.sh, make file. The pom.xml need to be 
> updated accordingly to reflect it.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HAWQ-1324) Query cancel cause segment to go into Crash recovery

2017-02-13 Thread Ming LI (JIRA)


[ 
https://issues.apache.org/jira/browse/HAWQ-1324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15863267#comment-15863267
 ] 

Ming LI commented on HAWQ-1324:
---

Hi all,

The root cause is backtrace() is not a safe functions to call from a signal 
handler. 

It is similar to below problem:
http://stackoverflow.com/questions/6371028/what-makes-backtrace-crashsigsegv-on-linux-64-bit
{code}
The documentation for signal handling 
(http://pubs.opengroup.org/onlinepubs/009695399/functions/xsh_chap02_04.html) 
defines the list of safe functions to call from a signal handler, you must not 
use any other functions, including backtrace. (search for async-signal-safe in 
that document)
{code}

So the fix should be similar with 
https://issues.apache.org/jira/browse/HAWQ-978. 

Need more time to verify the fix.  Thanks.

> Query cancel cause segment to go into Crash recovery
> 
>
> Key: HAWQ-1324
> URL: https://issues.apache.org/jira/browse/HAWQ-1324
> Project: Apache HAWQ
>  Issue Type: Bug
>Reporter: Ming LI
>Assignee: Ed Espino
>
> A query was cancelled due to this connection issue to HDFS on Isilon. Seg26 
> then went into crash recovery due to a INSERT query being cancelled. What 
> should be the expected behaviour when HDFS becomes unavailable and a Query 
> fails due to HDFS unavailability.
> Below is the HDFS error
> {code}
> 2017-01-04 03:04:08.382615 
> JST,"carund","dwhrun",p574246,th1862944896,"192.168.10.12","47554",2017-01-04 
> 03:03:08 JST,0,con198952,,seg29,"FATAL","08006","connection to client 
> lost",,,0,,"postgres.c",3518,
> 2017-01-04 03:04:08.420099 
> JST,,,p755778,th18629448960,,,seg-1,"LOG","0","3rd party 
> error log:
> 2017-01-04 03:04:08.419969, p574222, th140507423066240, ERROR Handle 
> Exception: NamenodeImpl.cpp: 670: Unexpected error: status: 
> STATUS_FILE_NOT_AVAILABLE = 0xC467 Path: 
> hawq_default/16385/16563/802748/26 with path=
> ""/hawq_default/16385/16563/802748/26"", 
> clientname=libhdfs3_client_random_866998528_count_1_pid_574222_tid_140507423066240
> @ Hdfs::Internal::UnWrapper Hdfs::HdfsIOException, Hdfs::Internal::Nothing, Hdfs::Internal::Nothing, 
> Hdfs::Internal::Nothing, Hdfs::Internal::Nothing, Hdfs::Internal::Nothing , 
> Hdfs::Internal::Nothing, Hdfs::Internal::Nothing, Hdfs::Internal::Nothing, 
> Hdfs::Internal::Nothing>::unwrap(char const, int)
> @ Hdfs::Internal::UnWrapper Hdfs::UnresolvedLinkException, Hdfs::HdfsIOException, 
> Hdfs::Internal::Nothing, Hdfs::Internal::Nothing, Hdfs::Internal::Nothing, 
> Hdfs::Internal::Not hing, Hdfs::Internal::Nothing, Hdfs::Internal::Nothing, 
> Hdfs::Internal::Nothing, Hdfs::Internal::Nothing>::unwrap(char const, int)
> @ Hdfs::Internal::NamenodeImpl::fsync(std::string const&, std::string const&)
> @ Hdfs::Internal::NamenodeProxy::fsync(std::string const&, std::string const&)
> @ Hdfs::Internal::OutputStreamImpl::closePipeline()
> @ Hdfs::Internal::OutputStreamImpl::close()
> @ hdfsCloseFile
> @ gpfs_hdfs_closefile
> @ HdfsCloseFile
> @ HdfsFileClose
> @ CleanupTempFiles
> @ AbortTransaction
> @ AbortCurrentTransaction
> @ PostgresMain
> @ BackendStartup
> @ ServerLoop
> @ PostmasterMain
> @ main
> @ Unknown
> @ Unknown""SysLoggerMain","syslogger.c",518,
> 2017-01-04 03:04:08.420272 
> JST,"carund","dwhrun",p574222,th1862944896,"192.168.10.12","47550",2017-01-04 
> 03:03:08 
> JST,40678725,con198952,cmd4,seg25,,,x40678725,sx1,"WARNING","58030","could 
> not close file 7 : (hdfs://ffd
> lakehd.ffwin.fujifilm.co.jp:8020/hawq_default/16385/16563/802748/26) errno 
> 5","Unexpected error: status: STATUS_FILE_NOT_AVAILABLE = 0xC467 Path: 
> hawq_default/16385/16563/802748/26 with path=""/hawq_default/16385/16
> 563/802748/26"", 
> clientname=libhdfs3_client_random_866998528_count_1_pid_574222_tid_140507423066240",,0,,"fd.c",2762,
> {code}
> Segment 26 going into Crash recovery - from seg26 log file
> {code}
> 2017-01-04 03:04:08.420314 
> JST,"carund","dwhrun",p574222,th1862944896,"192.168.10.12","47550",2017-01-04 
> 03:03:08 
> JST,40678725,con198952,cmd4,seg25,,,x40678725,sx1,"LOG","08006","could not 
> send data to client: 接続が相
> 手からリセットされました",,,0,,"pqcomm.c",1292,
> 2017-01-04 03:04:08.420358 
> JST,"carund","dwhrun",p574222,th1862944896,"192.168.10.12","47550",2017-01-04 
> 03:03:08 JST,0,con198952,,seg25,"LOG","08006","could not send data to 
> client: パイプが切断されました",,,0,
> ,"pqcomm.c",1292,
> 2017-01-04 03:04:08.420375 
> JST,"carund","dwhrun",p574222,th1862944896,"192.168.10.12","47550",2017-01-04 
> 03:03:08 JST,0,con198952,,seg25,"FATAL","08006","connection to client 
> lost",,,0,,"postgres.c",3518,
> 2017-01-04 03:04:08.950354 
> JST,,,p755773,th18629448960,,,seg-1,"LOG","0","server process

[jira] [Assigned] (HAWQ-1324) Query cancel cause segment to go into Crash recovery

2017-02-13 Thread Ming LI (JIRA)


 [ 
https://issues.apache.org/jira/browse/HAWQ-1324?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ming LI reassigned HAWQ-1324:
-

Assignee: Ming LI  (was: Ed Espino)

> Query cancel cause segment to go into Crash recovery
> 
>
> Key: HAWQ-1324
> URL: https://issues.apache.org/jira/browse/HAWQ-1324
> Project: Apache HAWQ
>  Issue Type: Bug
>Reporter: Ming LI
>Assignee: Ming LI
>
> A query was cancelled due to this connection issue to HDFS on Isilon. Seg26 
> then went into crash recovery due to a INSERT query being cancelled. What 
> should be the expected behaviour when HDFS becomes unavailable and a Query 
> fails due to HDFS unavailability.
> Below is the HDFS error
> {code}
> 2017-01-04 03:04:08.382615 
> JST,"carund","dwhrun",p574246,th1862944896,"192.168.10.12","47554",2017-01-04 
> 03:03:08 JST,0,con198952,,seg29,"FATAL","08006","connection to client 
> lost",,,0,,"postgres.c",3518,
> 2017-01-04 03:04:08.420099 
> JST,,,p755778,th18629448960,,,seg-1,"LOG","0","3rd party 
> error log:
> 2017-01-04 03:04:08.419969, p574222, th140507423066240, ERROR Handle 
> Exception: NamenodeImpl.cpp: 670: Unexpected error: status: 
> STATUS_FILE_NOT_AVAILABLE = 0xC467 Path: 
> hawq_default/16385/16563/802748/26 with path=
> ""/hawq_default/16385/16563/802748/26"", 
> clientname=libhdfs3_client_random_866998528_count_1_pid_574222_tid_140507423066240
> @ Hdfs::Internal::UnWrapper Hdfs::HdfsIOException, Hdfs::Internal::Nothing, Hdfs::Internal::Nothing, 
> Hdfs::Internal::Nothing, Hdfs::Internal::Nothing, Hdfs::Internal::Nothing , 
> Hdfs::Internal::Nothing, Hdfs::Internal::Nothing, Hdfs::Internal::Nothing, 
> Hdfs::Internal::Nothing>::unwrap(char const, int)
> @ Hdfs::Internal::UnWrapper Hdfs::UnresolvedLinkException, Hdfs::HdfsIOException, 
> Hdfs::Internal::Nothing, Hdfs::Internal::Nothing, Hdfs::Internal::Nothing, 
> Hdfs::Internal::Not hing, Hdfs::Internal::Nothing, Hdfs::Internal::Nothing, 
> Hdfs::Internal::Nothing, Hdfs::Internal::Nothing>::unwrap(char const, int)
> @ Hdfs::Internal::NamenodeImpl::fsync(std::string const&, std::string const&)
> @ Hdfs::Internal::NamenodeProxy::fsync(std::string const&, std::string const&)
> @ Hdfs::Internal::OutputStreamImpl::closePipeline()
> @ Hdfs::Internal::OutputStreamImpl::close()
> @ hdfsCloseFile
> @ gpfs_hdfs_closefile
> @ HdfsCloseFile
> @ HdfsFileClose
> @ CleanupTempFiles
> @ AbortTransaction
> @ AbortCurrentTransaction
> @ PostgresMain
> @ BackendStartup
> @ ServerLoop
> @ PostmasterMain
> @ main
> @ Unknown
> @ Unknown""SysLoggerMain","syslogger.c",518,
> 2017-01-04 03:04:08.420272 
> JST,"carund","dwhrun",p574222,th1862944896,"192.168.10.12","47550",2017-01-04 
> 03:03:08 
> JST,40678725,con198952,cmd4,seg25,,,x40678725,sx1,"WARNING","58030","could 
> not close file 7 : (hdfs://ffd
> lakehd.ffwin.fujifilm.co.jp:8020/hawq_default/16385/16563/802748/26) errno 
> 5","Unexpected error: status: STATUS_FILE_NOT_AVAILABLE = 0xC467 Path: 
> hawq_default/16385/16563/802748/26 with path=""/hawq_default/16385/16
> 563/802748/26"", 
> clientname=libhdfs3_client_random_866998528_count_1_pid_574222_tid_140507423066240",,0,,"fd.c",2762,
> {code}
> Segment 26 going into Crash recovery - from seg26 log file
> {code}
> 2017-01-04 03:04:08.420314 
> JST,"carund","dwhrun",p574222,th1862944896,"192.168.10.12","47550",2017-01-04 
> 03:03:08 
> JST,40678725,con198952,cmd4,seg25,,,x40678725,sx1,"LOG","08006","could not 
> send data to client: 接続が相
> 手からリセットされました",,,0,,"pqcomm.c",1292,
> 2017-01-04 03:04:08.420358 
> JST,"carund","dwhrun",p574222,th1862944896,"192.168.10.12","47550",2017-01-04 
> 03:03:08 JST,0,con198952,,seg25,"LOG","08006","could not send data to 
> client: パイプが切断されました",,,0,
> ,"pqcomm.c",1292,
> 2017-01-04 03:04:08.420375 
> JST,"carund","dwhrun",p574222,th1862944896,"192.168.10.12","47550",2017-01-04 
> 03:03:08 JST,0,con198952,,seg25,"FATAL","08006","connection to client 
> lost",,,0,,"postgres.c",3518,
> 2017-01-04 03:04:08.950354 
> JST,,,p755773,th18629448960,,,seg-1,"LOG","0","server process 
> (PID 574240) was terminated by signal 11: Segmentation 
> fault",,,0,,"postmaster.c",4748,
> 2017-01-04 03:04:08.950403 
> JST,,,p755773,th18629448960,,,seg-1,"LOG","0","terminating 
> any other active server processes",,,0,,"postmaster.c",4486,
> 2017-01-04 03:04:08.954044 
> JST,,,p41605,th18629448960,,,seg-1,"LOG","0","Segment RM 
> exits.",,,0,,"resourcemanager.c",340,
> 2017-01-04 03:04:08.954078 
> JST,,,p41605,th18629448960,,,seg-1,"LOG","0","Clean up 
> handler in message server is called.",,,0,,"rmcomm_MessageServer.c",105,
> 2017-01-04 03:04:08.972706 
>

[jira] [Comment Edited] (HAWQ-1324) Query cancel cause segment to go into Crash recovery

2017-02-13 Thread Ming LI (JIRA)


[ 
https://issues.apache.org/jira/browse/HAWQ-1324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15863274#comment-15863274
 ] 

Ming LI edited comment on HAWQ-1324 at 2/13/17 7:14 AM:


The complete fix for this defect should be similar with PostgreSQL 9.6:
1) Function StatementCancelHandler() can not directly call ProcessInterrupts() 
which cascade call some unsafe functions. Only simple logic ( e.g. set variable 
flags) can be included in signal handler function,  and call 
ProcessInterrupts() wherever the code pointer the signal can be triggered and 
executed.
2) Forward all related fixes from postgresql to hawq should be very complex 
task. We currently just offer the fix which can not completely fix this kinds 
of crash, but it reduce the possibility of this kind of crash. 

Thanks.


was (Author: mli):
The compete fix for this defect should be similar with PostgreSQL 9.6:
1) Function StatementCancelHandler() can not directly call ProcessInterrupts() 
which cascade call some unsafe functions. Only simple logic ( e.g. set variable 
flags) can be included in signal handler function,  and call 
ProcessInterrupts() wherever the code pointer the signal can be triggered and 
executed.
2) Forward all related fixes from postgresql to hawq should be very complex 
task. We currently just offer the fix which can not completely fix this kinds 
of crash, but it reduce the possibility of this kind of crash. 

Thanks.

> Query cancel cause segment to go into Crash recovery
> 
>
> Key: HAWQ-1324
> URL: https://issues.apache.org/jira/browse/HAWQ-1324
> Project: Apache HAWQ
>  Issue Type: Bug
>Reporter: Ming LI
>Assignee: Ming LI
>
> A query was cancelled due to this connection issue to HDFS on Isilon. Seg26 
> then went into crash recovery due to a INSERT query being cancelled. What 
> should be the expected behaviour when HDFS becomes unavailable and a Query 
> fails due to HDFS unavailability.
> Below is the HDFS error
> {code}
> 2017-01-04 03:04:08.382615 
> JST,"carund","dwhrun",p574246,th1862944896,"192.168.10.12","47554",2017-01-04 
> 03:03:08 JST,0,con198952,,seg29,"FATAL","08006","connection to client 
> lost",,,0,,"postgres.c",3518,
> 2017-01-04 03:04:08.420099 
> JST,,,p755778,th18629448960,,,seg-1,"LOG","0","3rd party 
> error log:
> 2017-01-04 03:04:08.419969, p574222, th140507423066240, ERROR Handle 
> Exception: NamenodeImpl.cpp: 670: Unexpected error: status: 
> STATUS_FILE_NOT_AVAILABLE = 0xC467 Path: 
> hawq_default/16385/16563/802748/26 with path=
> ""/hawq_default/16385/16563/802748/26"", 
> clientname=libhdfs3_client_random_866998528_count_1_pid_574222_tid_140507423066240
> @ Hdfs::Internal::UnWrapper Hdfs::HdfsIOException, Hdfs::Internal::Nothing, Hdfs::Internal::Nothing, 
> Hdfs::Internal::Nothing, Hdfs::Internal::Nothing, Hdfs::Internal::Nothing , 
> Hdfs::Internal::Nothing, Hdfs::Internal::Nothing, Hdfs::Internal::Nothing, 
> Hdfs::Internal::Nothing>::unwrap(char const, int)
> @ Hdfs::Internal::UnWrapper Hdfs::UnresolvedLinkException, Hdfs::HdfsIOException, 
> Hdfs::Internal::Nothing, Hdfs::Internal::Nothing, Hdfs::Internal::Nothing, 
> Hdfs::Internal::Not hing, Hdfs::Internal::Nothing, Hdfs::Internal::Nothing, 
> Hdfs::Internal::Nothing, Hdfs::Internal::Nothing>::unwrap(char const, int)
> @ Hdfs::Internal::NamenodeImpl::fsync(std::string const&, std::string const&)
> @ Hdfs::Internal::NamenodeProxy::fsync(std::string const&, std::string const&)
> @ Hdfs::Internal::OutputStreamImpl::closePipeline()
> @ Hdfs::Internal::OutputStreamImpl::close()
> @ hdfsCloseFile
> @ gpfs_hdfs_closefile
> @ HdfsCloseFile
> @ HdfsFileClose
> @ CleanupTempFiles
> @ AbortTransaction
> @ AbortCurrentTransaction
> @ PostgresMain
> @ BackendStartup
> @ ServerLoop
> @ PostmasterMain
> @ main
> @ Unknown
> @ Unknown""SysLoggerMain","syslogger.c",518,
> 2017-01-04 03:04:08.420272 
> JST,"carund","dwhrun",p574222,th1862944896,"192.168.10.12","47550",2017-01-04 
> 03:03:08 
> JST,40678725,con198952,cmd4,seg25,,,x40678725,sx1,"WARNING","58030","could 
> not close file 7 : (hdfs://ffd
> lakehd.ffwin.fujifilm.co.jp:8020/hawq_default/16385/16563/802748/26) errno 
> 5","Unexpected error: status: STATUS_FILE_NOT_AVAILABLE = 0xC467 Path: 
> hawq_default/16385/16563/802748/26 with path=""/hawq_default/16385/16
> 563/802748/26"", 
> clientname=libhdfs3_client_random_866998528_count_1_pid_574222_tid_140507423066240",,0,,"fd.c",2762,
> {code}
> Segment 26 going into Crash recovery - from seg26 log file
> {code}
> 2017-01-04 03:04:08.420314 
> JST,"carund","dwhrun",p574222,th1862944896,"192.168.10.12","47550",2017-01-04 
> 03:03:08 
> JST,40678725,con198952,cmd4,seg25,,,x40678725,sx1,"LOG","08006","could not 
> send

[jira] [Created] (HAWQ-1325) Allow queries related to pg_temp if ranger is enable

2017-02-13 Thread Lin Wen (JIRA)

Lin Wen created HAWQ-1325:
-

 Summary: Allow queries related to pg_temp if ranger is enable
 Key: HAWQ-1325
 URL: https://issues.apache.org/jira/browse/HAWQ-1325
 Project: Apache HAWQ
  Issue Type: Sub-task
Reporter: Lin Wen
Assignee: Ed Espino
 Fix For: 2.2.0.0-incubating


Queries related to temp will send request to RPS, asking the privilege of 
schema "pg_temp_XXX", like this:

./hawq-2017-02-13_142852.csv:2017-02-13 14:29:29.718445 
CST,"linw","postgres",p71787,th-1324481600,"[local]",,2017-02-13 14:29:01 CST,  
  8477,con13,cmd3,seg-1,,,x8477,sx1,"DEBUG3","0","send json request to 
ranger : { ""requestId"": ""3"", ""user"": ""linw"", ""clientIp"": 
""127.0.0.1"", ""context"": ""select * from temp1;"", ""access"": [ { 
""resource"": { ""database"": ""postgres"", ""schema"": ""pg_temp_13"", 
""table"": ""temp1"" }, ""privileges"": [ ""select"" ] } ] }",,"select * 
from temp1;",0,,"rangerrest.c",454,

In order to better control, for pg_temp_XX schema and objects in that schema, 
we should fall back these checks to catalog without sending requests to RPS. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HAWQ-1323) Add license header added to each source file in test directory

2017-02-13 Thread Ruilong Huo (JIRA)


 [ 
https://issues.apache.org/jira/browse/HAWQ-1323?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ruilong Huo updated HAWQ-1323:
--
Affects Version/s: 2.1.0.0-incubating

> Add license header added to each source file in test directory
> --
>
> Key: HAWQ-1323
> URL: https://issues.apache.org/jira/browse/HAWQ-1323
> Project: Apache HAWQ
>  Issue Type: Bug
>  Components: Tests
>Affects Versions: 2.1.0.0-incubating
>Reporter: Ruilong Huo
>Assignee: Jiali Yao
> Fix For: 2.2.0.0-incubating
>
>
> Need to add license header each source file in src/test directory including 
> *.c, *.cpp, *.h, *.py, *.pl, *.java, *.sh, make file. The pom.xml need to be 
> updated accordingly to reflect it.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Resolved] (HAWQ-1319) Add ASF header to java files and remove jar files introduced with HAWQ-922

2017-02-13 Thread Ruilong Huo (JIRA)


 [ 
https://issues.apache.org/jira/browse/HAWQ-1319?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ruilong Huo resolved HAWQ-1319.
---
Resolution: Fixed

> Add ASF header to java files and remove jar files introduced with HAWQ-922
> --
>
> Key: HAWQ-1319
> URL: https://issues.apache.org/jira/browse/HAWQ-1319
> Project: Apache HAWQ
>  Issue Type: Task
>  Components: Build
>Affects Versions: 2.1.0.0-incubating
>Reporter: Ed Espino
>Assignee: Ruilong Huo
> Fix For: 2.1.0.0-incubating
>
>
> The following two jar files were introduce with HAWQ-922:
> * src/test/feature/UDF/sql/PLJavaAdd.jar
> * src/test/feature/UDF/sql/PLJavauAdd.jar
> Alan Gates identified these files during his IPMC review for Apache HAWQ 
> 2.1.0.0 RC3.
> These files were missed by our RAT check due to the fact the "src/test" 
> directory is explicitly excluded (src/test/**/*).  To 
> avoid this in the future, this should be updated to exclude only the files 
> necessary (example: binary test files).
> The source is readily available in the same directory and these jar files 
> should be removed.
> The following java files should contain the ASF license header.  Are there 
> any other files added that should contain the ASF license header?



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Closed] (HAWQ-1319) Add ASF header to java files and remove jar files introduced with HAWQ-922

2017-02-13 Thread Ruilong Huo (JIRA)


 [ 
https://issues.apache.org/jira/browse/HAWQ-1319?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ruilong Huo closed HAWQ-1319.
-

> Add ASF header to java files and remove jar files introduced with HAWQ-922
> --
>
> Key: HAWQ-1319
> URL: https://issues.apache.org/jira/browse/HAWQ-1319
> Project: Apache HAWQ
>  Issue Type: Task
>  Components: Build
>Affects Versions: 2.1.0.0-incubating
>Reporter: Ed Espino
>Assignee: Ruilong Huo
> Fix For: 2.1.0.0-incubating
>
>
> The following two jar files were introduce with HAWQ-922:
> * src/test/feature/UDF/sql/PLJavaAdd.jar
> * src/test/feature/UDF/sql/PLJavauAdd.jar
> Alan Gates identified these files during his IPMC review for Apache HAWQ 
> 2.1.0.0 RC3.
> These files were missed by our RAT check due to the fact the "src/test" 
> directory is explicitly excluded (src/test/**/*).  To 
> avoid this in the future, this should be updated to exclude only the files 
> necessary (example: binary test files).
> The source is readily available in the same directory and these jar files 
> should be removed.
> The following java files should contain the ASF license header.  Are there 
> any other files added that should contain the ASF license header?



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (HAWQ-1323) Add license header added to each source file in test directory

2017-02-13 Thread Ruilong Huo (JIRA)

Ruilong Huo created HAWQ-1323:
-

 Summary: Add license header added to each source file in test 
directory
 Key: HAWQ-1323
 URL: https://issues.apache.org/jira/browse/HAWQ-1323
 Project: Apache HAWQ
  Issue Type: Bug
  Components: Tests
Reporter: Ruilong Huo
Assignee: Jiali Yao
 Fix For: 2.2.0.0-incubating


Need to add license header each source file in src/test directory including 
*.c, *.cpp, *.h, *.py, *.pl, *.java, *.sh, make file.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HAWQ-1324) Query cancel cause segment to go into Crash recovery

2017-02-13 Thread Ming LI (JIRA)


[ 
https://issues.apache.org/jira/browse/HAWQ-1324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15863274#comment-15863274
 ] 

Ming LI commented on HAWQ-1324:
---

The compete fix for this defect should be similar with PostgreSQL 9.6:
1) Function StatementCancelHandler() can not directly call ProcessInterrupts() 
which cascade call some unsafe functions. Only simple logic ( e.g. set variable 
flags) can be included in signal handler function,  and call 
ProcessInterrupts() wherever the code pointer the signal can be triggered and 
executed.
2) Forward all related fixes from postgresql to hawq should be very complex 
task. We currently just offer the fix which can not completely fix this kinds 
of crash, but it reduce the possibility of this kind of crash. 

Thanks.

> Query cancel cause segment to go into Crash recovery
> 
>
> Key: HAWQ-1324
> URL: https://issues.apache.org/jira/browse/HAWQ-1324
> Project: Apache HAWQ
>  Issue Type: Bug
>Reporter: Ming LI
>Assignee: Ed Espino
>
> A query was cancelled due to this connection issue to HDFS on Isilon. Seg26 
> then went into crash recovery due to a INSERT query being cancelled. What 
> should be the expected behaviour when HDFS becomes unavailable and a Query 
> fails due to HDFS unavailability.
> Below is the HDFS error
> {code}
> 2017-01-04 03:04:08.382615 
> JST,"carund","dwhrun",p574246,th1862944896,"192.168.10.12","47554",2017-01-04 
> 03:03:08 JST,0,con198952,,seg29,"FATAL","08006","connection to client 
> lost",,,0,,"postgres.c",3518,
> 2017-01-04 03:04:08.420099 
> JST,,,p755778,th18629448960,,,seg-1,"LOG","0","3rd party 
> error log:
> 2017-01-04 03:04:08.419969, p574222, th140507423066240, ERROR Handle 
> Exception: NamenodeImpl.cpp: 670: Unexpected error: status: 
> STATUS_FILE_NOT_AVAILABLE = 0xC467 Path: 
> hawq_default/16385/16563/802748/26 with path=
> ""/hawq_default/16385/16563/802748/26"", 
> clientname=libhdfs3_client_random_866998528_count_1_pid_574222_tid_140507423066240
> @ Hdfs::Internal::UnWrapper Hdfs::HdfsIOException, Hdfs::Internal::Nothing, Hdfs::Internal::Nothing, 
> Hdfs::Internal::Nothing, Hdfs::Internal::Nothing, Hdfs::Internal::Nothing , 
> Hdfs::Internal::Nothing, Hdfs::Internal::Nothing, Hdfs::Internal::Nothing, 
> Hdfs::Internal::Nothing>::unwrap(char const, int)
> @ Hdfs::Internal::UnWrapper Hdfs::UnresolvedLinkException, Hdfs::HdfsIOException, 
> Hdfs::Internal::Nothing, Hdfs::Internal::Nothing, Hdfs::Internal::Nothing, 
> Hdfs::Internal::Not hing, Hdfs::Internal::Nothing, Hdfs::Internal::Nothing, 
> Hdfs::Internal::Nothing, Hdfs::Internal::Nothing>::unwrap(char const, int)
> @ Hdfs::Internal::NamenodeImpl::fsync(std::string const&, std::string const&)
> @ Hdfs::Internal::NamenodeProxy::fsync(std::string const&, std::string const&)
> @ Hdfs::Internal::OutputStreamImpl::closePipeline()
> @ Hdfs::Internal::OutputStreamImpl::close()
> @ hdfsCloseFile
> @ gpfs_hdfs_closefile
> @ HdfsCloseFile
> @ HdfsFileClose
> @ CleanupTempFiles
> @ AbortTransaction
> @ AbortCurrentTransaction
> @ PostgresMain
> @ BackendStartup
> @ ServerLoop
> @ PostmasterMain
> @ main
> @ Unknown
> @ Unknown""SysLoggerMain","syslogger.c",518,
> 2017-01-04 03:04:08.420272 
> JST,"carund","dwhrun",p574222,th1862944896,"192.168.10.12","47550",2017-01-04 
> 03:03:08 
> JST,40678725,con198952,cmd4,seg25,,,x40678725,sx1,"WARNING","58030","could 
> not close file 7 : (hdfs://ffd
> lakehd.ffwin.fujifilm.co.jp:8020/hawq_default/16385/16563/802748/26) errno 
> 5","Unexpected error: status: STATUS_FILE_NOT_AVAILABLE = 0xC467 Path: 
> hawq_default/16385/16563/802748/26 with path=""/hawq_default/16385/16
> 563/802748/26"", 
> clientname=libhdfs3_client_random_866998528_count_1_pid_574222_tid_140507423066240",,0,,"fd.c",2762,
> {code}
> Segment 26 going into Crash recovery - from seg26 log file
> {code}
> 2017-01-04 03:04:08.420314 
> JST,"carund","dwhrun",p574222,th1862944896,"192.168.10.12","47550",2017-01-04 
> 03:03:08 
> JST,40678725,con198952,cmd4,seg25,,,x40678725,sx1,"LOG","08006","could not 
> send data to client: 接続が相
> 手からリセットされました",,,0,,"pqcomm.c",1292,
> 2017-01-04 03:04:08.420358 
> JST,"carund","dwhrun",p574222,th1862944896,"192.168.10.12","47550",2017-01-04 
> 03:03:08 JST,0,con198952,,seg25,"LOG","08006","could not send data to 
> client: パイプが切断されました",,,0,
> ,"pqcomm.c",1292,
> 2017-01-04 03:04:08.420375 
> JST,"carund","dwhrun",p574222,th1862944896,"192.168.10.12","47550",2017-01-04 
> 03:03:08 JST,0,con198952,,seg25,"FATAL","08006","connection to client 
> lost",,,0,,"postgres.c",3518,
> 2017-01-04 03:04:08.950354 
> JST,,,p755773,th18629448960,,,seg-1,"LOG","0","server process 
> (PID 574240) was terminated by signal 11:

[GitHub] incubator-hawq issue #1115: HAWQ-1324. Fixed crash at query cancel, signal h...

2017-02-13 Thread wengyanqing

Github user wengyanqing commented on the issue:

https://github.com/apache/incubator-hawq/pull/1115
  
LGTM


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

38 matches

Mail list logo