[
https://issues.apache.org/jira/browse/HAWQ-91?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15071002#comment-15071002
]
Alexey Grishchenko commented on HAWQ-91:
----------------------------------------
1. What is your HAWQ configuration (select * from gp_segment_configuration)
2. Do you see this error only when loading the data with gpload or also when
querying the system?
3. What is the configuration of gpload in terms of error limit and error tables?
4. Try to load small part of data. Get first 1000-100000 lines of the file with
"head" command and load them using gpfdist, not gpload, and check the result
I suspect that the issue is related to the error limit which is set too high
and HAWQ uses all the RAM to store your erroneous entries
> “Out of memory” error when use gpload to load data
> --------------------------------------------------
>
> Key: HAWQ-91
> URL: https://issues.apache.org/jira/browse/HAWQ-91
> Project: Apache HAWQ
> Issue Type: Bug
> Components: Command Line Tools
> Reporter: dingyuanpu
> Assignee: Lei Chang
>
> I have some problems with HAWQ : My HAWQ version is 1.3 on HDP2.2.6 ,which is
> on 4 servers with x86 system(256G memory and 1T hard disk for each)
> The detail information is follow:
> I used the gpload tools to upload the store_sales.dat(the data is 188G) of
> TPC-DS, the errors are:
> 2015-10-27 01:24:51|INFO|gpload session started 2015-10-27 01:24:51
> 2015-10-27 01:24:51|INFO|setting schema 'public' for table 'store_sales'
> 2015-10-27 01:24:52|INFO|started gpfdist -p 8081 -P 8082 -f
> "tpc500g-data/store_sales_aa_aa_aa" -t 30
> 2015-10-27 01:30:25|ERROR|ERROR: Out of memory (seg0 node1.fd.h3c.com:40000
> pid=74456)
> DETAIL:
> VM Protect failed to allocate 8388608 bytes, 7 MB available
> External table ext_gpload20151027_012451_543181, line N/A of
> gpfdist://node2:8081/tpc500g-data/store_sales_aa_aa_aa: ""
> encountered while running INSERT INTO public."store_sales"
> ("ss_sold_date_sk","ss_sold_time_sk","ss_item_sk","ss_customer_sk","ss_cdemo_sk","ss_hdemo_sk","ss_addr_sk","ss_store_sk","ss_promo_sk","ss_ticket_number","ss_quantity","ss_wholesale_cost","ss_list_price","ss_sales_price","ss_ext_discount_amt","ss_ext_sales_price","ss_ext_wholesale_cost","ss_ext_list_price","ss_ext_tax","ss_coupon_amt","ss_net_paid","ss_net_paid_inc_tax","ss_net_profit")
> SELECT
> "ss_sold_date_sk","ss_sold_time_sk","ss_item_sk","ss_customer_sk","ss_cdemo_sk","ss_hdemo_sk","ss_addr_sk","ss_store_sk","ss_promo_sk","ss_ticket_number","ss_quantity","ss_wholesale_cost","ss_list_price","ss_sales_price","ss_ext_discount_amt","ss_ext_sales_price","ss_ext_wholesale_cost","ss_ext_list_price","ss_ext_tax","ss_coupon_amt","ss_net_paid","ss_net_paid_inc_tax","ss_net_profit"
> FROM ext_gpload20151027_012451_543181
> 2015-10-27 01:30:25|INFO|rows Inserted = 0
> 2015-10-27 01:30:25|INFO|rows Updated = 0
> 2015-10-27 01:30:25|INFO|data formatting errors = 0
> 2015-10-27 01:30:25|INFO|gpload failed
> I have used the following command to modify the parameters,the errors still
> exist:
> gpconfig -c gp_vmem_protect_limit -v 8192MB (I have also tried
> 4096、8192、16384、32768、81920、245760、262144)
> gpstop –r
> please help me to solve the problem ,thanks.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
