[GitHub] incubator-hawq pull request: HAWQ-199. Add license header for PXF ...

[linwen]

GitHub user linwen opened a pull request:

https://github.com/apache/incubator-hawq/pull/141

HAWQ-199. Add license header for PXF source files

Add license header for PXF source files.
Please review. Thank you!

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/linwen/incubator-hawq hawq-199

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-hawq/pull/141.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #141


commit 7b5c7fedfd0714b8d07e636c25b523a69151e171
Author: Wen Lin 
Date:   2015-11-27T12:51:17Z

HAWQ-199. Add license header for PXF source files




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: HAWQ on YARN - continuous container allocation

[Alexey Grishchenko]

You can also find HAWQ log in attachment. As expected, HAWQ tries to
releaseResources, but somehow it returns success

On Fri, Nov 27, 2015 at 2:10 PM, Alexey Grishchenko 
wrote:

> Hi, guys
>
> I've got an issue with running HAWQ 2.0 on YARN
> On starting HAWQ successfully registers YARN application and starts
> allocating containers. But it never stops allocating them. Regardless the
> amount of vcores and memory you give YARN to manage, HAWQ would allocate
> containers until it eat all the available resources. After this, all the
> queries start to hang.
>
> What I can see in the RM logs (full log is attached):
> 2015-11-27 05:34:59,214 WARN  resourcemanager.RMAuditLogger
> (RMAuditLogger.java:logFailure(215)) - USER=gpadmin OPERATION=AM Released
> Container TARGET=SchedulerRESULT=FAILURE  DESCRIPTION=Trying to
> release container not owned by app or with invalid id.
>  PERMISSIONS=Unauthorized access or invalid container
>  APPID=application_1448630699339_0002
>  CONTAINERID=container_1448630699339_0002_01_08
>
> Do you know the possible reason for this?
> Using HAWQ 2.0.0.0_beta build 18453 on a single node with PHD 3.3.2.0
> (YARN 2.7.1)
>
> --
> Alexey Grishchenko, http://0x0fff.com
>



-- 
Alexey Grishchenko, http://0x0fff.com


gpdb-2015-11-27_061656.csv.gz
Description: GNU Zip compressed data

Re: HAWQ on YARN - continuous container allocation

[Alexey Grishchenko]

Ok, now I see that the container name is formed in a wrong way.

2015-11-27 06:18:02,991 INFO  rmcontainer.RMContainerImpl
(RMContainerImpl.java:handle(417)) -
*container_e08_1448630699339_0003_01_03* Container Transitioned from
NEW to ALLOCATED
2015-11-27 06:18:02,991 INFO  resourcemanager.RMAuditLogger
(RMAuditLogger.java:logSuccess(106)) - USER=gpadmin OPERATION=AM Allocated
Container TARGET=SchedulerApp RESULT=SUCCESS
APPID=application_1448630699339_0003
CONTAINERID=container_e08_1448630699339_0003_01_03
2015-11-27 06:18:02,991 INFO  scheduler.SchedulerNode
(SchedulerNode.java:allocateContainer(154)) - Assigned container
container_e08_1448630699339_0003_01_03 of capacity  on host hawq20.pivotal.io:45454, which has 3 containers,
 used and  available after
allocation
2015-11-27 06:18:02,991 INFO  capacity.LeafQueue
(LeafQueue.java:assignContainer(1616)) - assignedContainer application
attempt=appattempt_1448630699339_0003_01 container=Container:
[ContainerId: container_e08_1448630699339_0003_01_03, NodeId:
hawq20.pivotal.io:45454, NodeHttpAddress: hawq20.pivotal.io:8042, Resource:
, Priority: 1, Token: null, ] queue=default:
capacity=1.0, absoluteCapacity=1.0, usedResources=,
usedCapacity=0.17519248, absoluteUsedCapacity=0.17519248, numApps=1,
numContainers=2 clusterResource=
2015-11-27 06:18:03,850 INFO  rmcontainer.RMContainerImpl
(RMContainerImpl.java:handle(417)) -
container_e08_1448630699339_0003_01_03 Container Transitioned from
ALLOCATED to ACQUIRED
2015-11-27 06:18:03,877 WARN  resourcemanager.RMAuditLogger
(RMAuditLogger.java:logFailure(215)) - USER=gpadmin IP=192.168.220.128
OPERATION=AM
Released Container TARGET=Scheduler RESULT=FAILURE DESCRIPTION=Trying to
release container not owned by app or with invalid id. PERMISSIONS=Unauthorized
access or invalid container APPID=application_1448630699339_0003
CONTAINERID=*container_1448630699339_0003_01_03*

Container is allocated with the name
container_e08_1448630699339_0003_01_03, but when HAWQ tries to shut it
down it uses the name container_1448630699339_0003_01_03, which is
wrong because there is no container with this name


On Fri, Nov 27, 2015 at 2:29 PM, Alexey Grishchenko 
wrote:

> You can also find HAWQ log in attachment. As expected, HAWQ tries to
> releaseResources, but somehow it returns success
>
> On Fri, Nov 27, 2015 at 2:10 PM, Alexey Grishchenko <
> programme...@gmail.com> wrote:
>
>> Hi, guys
>>
>> I've got an issue with running HAWQ 2.0 on YARN
>> On starting HAWQ successfully registers YARN application and starts
>> allocating containers. But it never stops allocating them. Regardless the
>> amount of vcores and memory you give YARN to manage, HAWQ would allocate
>> containers until it eat all the available resources. After this, all the
>> queries start to hang.
>>
>> What I can see in the RM logs (full log is attached):
>> 2015-11-27 05:34:59,214 WARN  resourcemanager.RMAuditLogger
>> (RMAuditLogger.java:logFailure(215)) - USER=gpadmin OPERATION=AM Released
>> Container TARGET=SchedulerRESULT=FAILURE  DESCRIPTION=Trying to
>> release container not owned by app or with invalid id.
>>  PERMISSIONS=Unauthorized access or invalid container
>>  APPID=application_1448630699339_0002
>>  CONTAINERID=container_1448630699339_0002_01_08
>>
>> Do you know the possible reason for this?
>> Using HAWQ 2.0.0.0_beta build 18453 on a single node with PHD 3.3.2.0
>> (YARN 2.7.1)
>>
>> --
>> Alexey Grishchenko, http://0x0fff.com
>>
>
>
>
> --
> Alexey Grishchenko, http://0x0fff.com
>



-- 
Alexey Grishchenko, http://0x0fff.com

Re: Performance issue about HAWQ 2.0 beta

[Jiali Yao]

Hi Leon

Thanks for providing it. The result is not as we expected. In our
performance test, we found the performance is comparable with 1.3.
Could you please some more information:
1. Get segment configuration information from 1.3 and 2.0
select * from gp_segment_configuration ;
2. Could you please run "explain analyze" to get more statistic information?
3. Want to confirm with you: The result run in yarn mode ,right? Also I see
your previous email to indicate there is some error in yarn, these query is
also from that test round, right?

Thanks

Jiali

On Fri, Nov 27, 2015 at 3:43 PM, Leon Zhang  wrote:

> Hi, HAWQ Developers:
>
>As my previous email hint, I run TPC-DS test on our development.
> Comparing with previous version 1.3.x, we can see the performance
> improvement on most of queries.
>
>But the problem is performance reduction for *some* queries. For
> example, the query64, the running time increase from 10754.688 ms
> to 68884.731 ms . I am not sure if any changes were made that increase the
> running time?
>
>In order to discuss the detail about this issue, I would like use the
> query10. The running time increase from 1795.746 ms to 744919.251 ms. I
> also attache the sql about this query, and the query plan for this query.
>
>Thanks
>
>

[GitHub] incubator-hawq pull request: HAWQ-195. Correct the spelling of par...

[linwen]

Github user linwen closed the pull request at:

https://github.com/apache/incubator-hawq/pull/137


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: Problem about Hawq on YARN

[Wen Lin]

Hi, Leon,

The log you attached maybe known bug in hawq dev team or an unknown one.
It's hard to conclude based on it.
Would you please send a full log? Then we can know what happened before the
error message was printed.

Thank you!

On Fri, Nov 27, 2015 at 3:23 PM, Leon Zhang  wrote:

> Hi, HAWQ Developers:
>
>  Thanks for the friendly hawq community, they help me to setup the
> hawq-2.0 on YARN. After I execute the TPC-DS queries, I encounter some yarn
> error for part of queries.
>
>  For instance, the query1:
> -- start query 12 in stream 0 using template query1.tpl and seed 1711160884
> with customer_total_return as
> (select sr_customer_sk as ctr_customer_sk
> ,sr_store_sk as ctr_store_sk
> ,sum(SR_REVERSED_CHARGE) as ctr_total_return
> from store_returns
> ,date_dim
> where sr_returned_date_sk = d_date_sk
> and d_year =1998
> group by sr_customer_sk
> ,sr_store_sk)
>  select  c_customer_id
> from customer_total_return ctr1
> ,store
> ,customer
> where ctr1.ctr_total_return > (select avg(ctr_total_return)*1.2
> from customer_total_return ctr2
> where ctr1.ctr_store_sk = ctr2.ctr_store_sk)
> and s_store_sk = ctr1.ctr_store_sk
> and s_state = 'TN'
> and ctr1.ctr_customer_sk = c_customer_sk
> order by c_customer_id
> limit 100;
> psql:/mnt/xiaolin/query/scale_1/query_0.sql:502: WARNING:  Sync RPC
> framework (inet) finds exception raised.
> psql:/mnt/xiaolin/query/scale_1/query_0.sql:502: ERROR:  failed to return
> resource to HAWQ resource manager because of failing to receive content.
> (pquery.c:983)
>
>  And many other errors like this:
> $ grep psql:/mnt/xiaolin/query/scale_1/query_0.sql
> hawq_yarn/log_1/log_1.log
> psql:/mnt/xiaolin/query/scale_1/query_0.sql:502: WARNING:  Sync RPC
> framework (inet) finds exception raised.
> psql:/mnt/xiaolin/query/scale_1/query_0.sql:502: ERROR:  failed to return
> resource to HAWQ resource manager because of failing to receive content.
> (pquery.c:983)
> psql:/mnt/xiaolin/query/scale_1/query_0.sql:1103: WARNING:  Sync RPC
> framework (inet) finds exception raised.
> psql:/mnt/xiaolin/query/scale_1/query_0.sql:1103: ERROR:  failed to return
> resource to HAWQ resource manager because of failing to receive content.
> (pquery.c:983)
> psql:/mnt/xiaolin/query/scale_1/query_0.sql:1132: ERROR:  failed to acquire
> resource because of too many unavailable segments. (pquery.c:798)
> psql:/mnt/xiaolin/query/scale_1/query_0.sql:1997: WARNING:  Sync RPC
> framework (inet) finds exception raised.
> psql:/mnt/xiaolin/query/scale_1/query_0.sql:1997: ERROR:  failed to return
> resource to HAWQ resource manager because of failing to receive content.
> (pquery.c:983)
> psql:/mnt/xiaolin/query/scale_1/query_0.sql:2150: WARNING:  Sync RPC
> framework (inet) finds exception raised.
> psql:/mnt/xiaolin/query/scale_1/query_0.sql:2150: ERROR:  failed to return
> resource to HAWQ resource manager because of failing to receive content.
> (pquery.c:983)
> psql:/mnt/xiaolin/query/scale_1/query_0.sql:2455: ERROR:  insufficient
> memory reserved for statement (execHHashagg.c:732)  (seg12 dserver2:40404
> pid=331172) (dispatcher.c:1701)
> psql:/mnt/xiaolin/query/scale_1/query_0.sql:2543: WARNING:  Sync RPC
> framework (inet) finds exception raised.
> psql:/mnt/xiaolin/query/scale_1/query_0.sql:2543: ERROR:  failed to return
> resource to HAWQ resource manager because of failing to receive content.
> (pquery.c:983)
> psql:/mnt/xiaolin/query/scale_1/query_0.sql:2639: WARNING:  Sync RPC
> framework (inet) finds exception raised.
> psql:/mnt/xiaolin/query/scale_1/query_0.sql:2639: ERROR:  failed to return
> resource to HAWQ resource manager because of failing to receive content.
> (pquery.c:983)
> psql:/mnt/xiaolin/query/scale_1/query_0.sql:2661: ERROR:  failed to acquire
> resource because of too many unavailable segments. (pquery.c:798)
> psql:/mnt/xiaolin/query/scale_1/query_0.sql:2720: ERROR:  failed to acquire
> resource because of too many unavailable segments. (pquery.c:798)
> psql:/mnt/xiaolin/query/scale_1/query_0.sql:2742: ERROR:  failed to acquire
> resource because of too many unavailable segments. (pquery.c:798)
> psql:/mnt/xiaolin/query/scale_1/query_0.sql:2794: ERROR:  failed to acquire
> resource because of too many unavailable segments. (pquery.c:798)
> psql:/mnt/xiaolin/query/scale_1/query_0.sql:2902: ERROR:  failed to acquire
> resource because of too many unavailable segments. (pquery.c:798)
> psql:/mnt/xiaolin/query/scale_1/query_0.sql:2935: ERROR:  failed to acquire
> resource because of too many unavailable segments. (pquery.c:798)
> psql:/mnt/xiaolin/query/scale_1/query_0.sql:3182: WARNING:  Sync RPC
> framework (inet) finds exception raised.
> psql:/mnt/xiaolin/query/scale_1/query_0.sql:3182: ERROR:  failed to return
> resource to HAWQ resource manager because of failing to receive content.
> (pquery.c:983)
> psql:/mnt/xiaolin/query/scale_1/query_0.sql:3286: ERROR:  failed to acquire
> resource because of too many unavailable 

[GitHub] incubator-hawq pull request: Hawq-184. Add license header

[linwen]

Github user linwen closed the pull request at:

https://github.com/apache/incubator-hawq/pull/139


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---