test On Mon, May 16, 2016 at 10:59 PM, < [email protected]> wrote:
> > user Digest 17 May 2016 02:59:27 -0000 Issue 67 > > Topics (messages 212 through 215) > > Re: no partition for partitioning key make gpload failed. > 212 by: Lei Chang > 213 by: Gagan Brahmi > 214 by: yin.zhb.163.com > > Re: HAWQ YARN RPC Errors > 215 by: Wen Lin > > Administrivia: > > --------------------------------------------------------------------- > To post to the list, e-mail: [email protected] > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: > [email protected] > > ---------------------------------------------------------------------- > > > > ---------- Forwarded message ---------- > From: Lei Chang <[email protected]> > To: user <[email protected]> > Cc: dev <[email protected]> > Date: Sun, 15 May 2016 09:16:11 +0800 > Subject: Re: no partition for partitioning key make gpload failed. > > Hi, i think it is because your data inserted has a partition key not in > the range. for example, i can reproduce this on my laptop by the follow > 'insert'. You can change you partition table definition accordingly. > > postgres=# insert into stocks > values(1,1,'1','1',1,1,'1','20170101'::date); > > > ERROR: no partition for partitioning key (seg1 localhost:40000 pid=35447) > Thanks > Lei > > > On Sat, May 14, 2016 at 11:34 PM, [email protected] <[email protected]> wrote: > >> Hi,all: >> I am using hawq 1.1.0,When I using gpload there is a problem: >> >> [gpadmin@stars71 ~]$ psql -d postgres >> postgres=# create table stocks >> postgres-# ( >> postgres(# storeno bigint, >> postgres(# itemno bigint, >> postgres(# itemname text, >> postgres(# spec text, >> postgres(# itemseq bigint, >> postgres(# stocks bigint, >> postgres(# unit text, >> postgres(# ymd date >> postgres(# ) >> postgres-# distributed by (itemno) >> postgres-# partition by range(ymd) >> postgres-# ( >> >> postgres(# partition p2015 start ('20150101'::date) end ('20160101'::date), >> postgres(# partition p2016 start ('20160101'::date) end ('20170101'::date) >> postgres(# ); >> >> NOTICE: CREATE TABLE will create partition "stocks_1_prt_p2015" for table >> "stocks" >> >> NOTICE: CREATE TABLE will create partition "stocks_1_prt_p2016" for table >> "stocks" >> CREATE TABLE >> postgres=# \q >> >> [gpadmin@stars71 >> ctl]$ gpfdist -d /data/data/ -p 8081 -l /home/gpadmin/log & >> [gpadmin@stars71 ctl]$ cat stocks.ctl >> --- >> VERSION: 1.0.0.1 >> DATABASE: postgres >> USER: gpadmin >> HOST: stars71 >> PORT: 5432 >> GPLOAD: >> INPUT: >> - SOURCE: >> LOCAL_HOSTNAME: >> - stars71 >> PORT: 8081 >> FILE: >> - /data/data/* >> - COLUMNS: >> - storeno: bigint >> - itemno: bigint >> - itemname: text >> - spec: text >> - itemseq: bigint >> - stocks: bigint >> - unit: text >> - ymd: date >> - FORMAT: text >> - DELIMITER: E'\t' >> - ERROR_LIMIT: 25 >> - ERROR_TABLE: stocks_err >> OUTPUT: >> - TABLE: stocks >> - MODE: INSERT >> [gpadmin@stars71 ctl]$ gpload -f stocks.ctl >> 2016-05-14 23:24:19|INFO|gpload session started 2016-05-14 23:24:19 >> 2016-05-14 23:24:19|INFO|setting schema 'public' for table 'stocks' >> >> 2016-05-14 23:24:20|INFO|started gpfdist -p 8081 -P 8082 -f "/data/data/*" >> -t 30 >> >> 2016-05-14 23:24:20|WARN|A gpload control file processing warning occurred. >> A delimiter must have a length of one. Special characters must be quoted. >> gpload will assume this is a sql escape character sequence. >> 2016-05-14 23:24:20|ERROR|ERROR: *no partition for partitioning key* >> (seg5 localhost:40000 pid=14591) >> >> encountered while running INSERT INTO public."stocks" >> ("storeno","itemno","itemname","spec","itemseq","stocks","unit","ymd") >> SELECT "storeno","itemno","itemname","spec","itemseq","stocks","unit","ymd" >> FROM ext_gpload20160514_232419_14569 >> 2016-05-14 23:24:20|INFO|rows Inserted = 0 >> 2016-05-14 23:24:20|INFO|rows Updated = 0 >> 2016-05-14 23:24:20|INFO|data formatting errors = 0 >> 2016-05-14 23:24:20|INFO|gpload failed >> >> It was failed,even if there is only one line. >> >> ------------------------------ >> [email protected] >> > > > > ---------- Forwarded message ---------- > From: Gagan Brahmi <[email protected]> > To: [email protected] > Cc: dev <[email protected]> > Date: Sat, 14 May 2016 20:19:14 -0700 > Subject: Re: no partition for partitioning key make gpload failed. > > Looks like you've a date in ymd column which falls outside the defined > range for the two partitions. > > You can define a partition which can hold all the data which falls before > 20150101 > with something like below: > > ...partition pre2015 end ('20150101'::date)... > > > Regards, > > Gagan Brahmi > On May 14, 2016 08:34, "[email protected]" <[email protected]> wrote: > >> Hi,all: >> I am using hawq 1.1.0,When I using gpload there is a problem: >> >> [gpadmin@stars71 ~]$ psql -d postgres >> postgres=# create table stocks >> postgres-# ( >> postgres(# storeno bigint, >> postgres(# itemno bigint, >> postgres(# itemname text, >> postgres(# spec text, >> postgres(# itemseq bigint, >> postgres(# stocks bigint, >> postgres(# unit text, >> postgres(# ymd date >> postgres(# ) >> postgres-# distributed by (itemno) >> postgres-# partition by range(ymd) >> postgres-# ( >> >> postgres(# partition p2015 start ('20150101'::date) end ('20160101'::date), >> postgres(# partition p2016 start ('20160101'::date) end ('20170101'::date) >> postgres(# ); >> >> NOTICE: CREATE TABLE will create partition "stocks_1_prt_p2015" for table >> "stocks" >> >> NOTICE: CREATE TABLE will create partition "stocks_1_prt_p2016" for table >> "stocks" >> CREATE TABLE >> postgres=# \q >> >> [gpadmin@stars71 >> ctl]$ gpfdist -d /data/data/ -p 8081 -l /home/gpadmin/log & >> [gpadmin@stars71 ctl]$ cat stocks.ctl >> --- >> VERSION: 1.0.0.1 >> DATABASE: postgres >> USER: gpadmin >> HOST: stars71 >> PORT: 5432 >> GPLOAD: >> INPUT: >> - SOURCE: >> LOCAL_HOSTNAME: >> - stars71 >> PORT: 8081 >> FILE: >> - /data/data/* >> - COLUMNS: >> - storeno: bigint >> - itemno: bigint >> - itemname: text >> - spec: text >> - itemseq: bigint >> - stocks: bigint >> - unit: text >> - ymd: date >> - FORMAT: text >> - DELIMITER: E'\t' >> - ERROR_LIMIT: 25 >> - ERROR_TABLE: stocks_err >> OUTPUT: >> - TABLE: stocks >> - MODE: INSERT >> [gpadmin@stars71 ctl]$ gpload -f stocks.ctl >> 2016-05-14 23:24:19|INFO|gpload session started 2016-05-14 23:24:19 >> 2016-05-14 23:24:19|INFO|setting schema 'public' for table 'stocks' >> >> 2016-05-14 23:24:20|INFO|started gpfdist -p 8081 -P 8082 -f "/data/data/*" >> -t 30 >> >> 2016-05-14 23:24:20|WARN|A gpload control file processing warning occurred. >> A delimiter must have a length of one. Special characters must be quoted. >> gpload will assume this is a sql escape character sequence. >> 2016-05-14 23:24:20|ERROR|ERROR: *no partition for partitioning key* >> (seg5 localhost:40000 pid=14591) >> >> encountered while running INSERT INTO public."stocks" >> ("storeno","itemno","itemname","spec","itemseq","stocks","unit","ymd") >> SELECT "storeno","itemno","itemname","spec","itemseq","stocks","unit","ymd" >> FROM ext_gpload20160514_232419_14569 >> 2016-05-14 23:24:20|INFO|rows Inserted = 0 >> 2016-05-14 23:24:20|INFO|rows Updated = 0 >> 2016-05-14 23:24:20|INFO|data formatting errors = 0 >> 2016-05-14 23:24:20|INFO|gpload failed >> >> It was failed,even if there is only one line. >> >> ------------------------------ >> [email protected] >> > > > ---------- Forwarded message ---------- > From: "[email protected]" <[email protected]> > To: user <[email protected]> > Cc: dev <[email protected]> > Date: Sun, 15 May 2016 12:10:32 +0800 > Subject: Re: Re: no partition for partitioning key make gpload failed. > The point is I set error limits, It should be skip the bad lines,Let the > others go. > > ------------------------------ > [email protected] > > > *From:* Gagan Brahmi <[email protected]> > *Date:* 2016-05-15 11:19 > *To:* user <[email protected]> > *CC:* dev <[email protected]> > *Subject:* Re: no partition for partitioning key make gpload failed. > > Looks like you've a date in ymd column which falls outside the defined > range for the two partitions. > > You can define a partition which can hold all the data which falls before > 20150101 > with something like below: > > ...partition pre2015 end ('20150101'::date)... > > > Regards, > > Gagan Brahmi > On May 14, 2016 08:34, "[email protected]" <[email protected]> wrote: > >> Hi,all: >> I am using hawq 1.1.0,When I using gpload there is a problem: >> >> [gpadmin@stars71 ~]$ psql -d postgres >> postgres=# create table stocks >> postgres-# ( >> postgres(# storeno bigint, >> postgres(# itemno bigint, >> postgres(# itemname text, >> postgres(# spec text, >> postgres(# itemseq bigint, >> postgres(# stocks bigint, >> postgres(# unit text, >> postgres(# ymd date >> postgres(# ) >> postgres-# distributed by (itemno) >> postgres-# partition by range(ymd) >> postgres-# ( >> >> postgres(# partition p2015 start ('20150101'::date) end ('20160101'::date), >> postgres(# partition p2016 start ('20160101'::date) end ('20170101'::date) >> postgres(# ); >> >> NOTICE: CREATE TABLE will create partition "stocks_1_prt_p2015" for table >> "stocks" >> >> NOTICE: CREATE TABLE will create partition "stocks_1_prt_p2016" for table >> "stocks" >> CREATE TABLE >> postgres=# \q >> >> [gpadmin@stars71 >> ctl]$ gpfdist -d /data/data/ -p 8081 -l /home/gpadmin/log & >> [gpadmin@stars71 ctl]$ cat stocks.ctl >> --- >> VERSION: 1.0.0.1 >> DATABASE: postgres >> USER: gpadmin >> HOST: stars71 >> PORT: 5432 >> GPLOAD: >> INPUT: >> - SOURCE: >> LOCAL_HOSTNAME: >> - stars71 >> PORT: 8081 >> FILE: >> - /data/data/* >> - COLUMNS: >> - storeno: bigint >> - itemno: bigint >> - itemname: text >> - spec: text >> - itemseq: bigint >> - stocks: bigint >> - unit: text >> - ymd: date >> - FORMAT: text >> - DELIMITER: E'\t' >> - ERROR_LIMIT: 25 >> - ERROR_TABLE: stocks_err >> OUTPUT: >> - TABLE: stocks >> - MODE: INSERT >> [gpadmin@stars71 ctl]$ gpload -f stocks.ctl >> 2016-05-14 23:24:19|INFO|gpload session started 2016-05-14 23:24:19 >> 2016-05-14 23:24:19|INFO|setting schema 'public' for table 'stocks' >> >> 2016-05-14 23:24:20|INFO|started gpfdist -p 8081 -P 8082 -f "/data/data/*" >> -t 30 >> >> 2016-05-14 23:24:20|WARN|A gpload control file processing warning occurred. >> A delimiter must have a length of one. Special characters must be quoted. >> gpload will assume this is a sql escape character sequence. >> 2016-05-14 23:24:20|ERROR|ERROR: *no partition for partitioning key* >> (seg5 localhost:40000 pid=14591) >> >> encountered while running INSERT INTO public."stocks" >> ("storeno","itemno","itemname","spec","itemseq","stocks","unit","ymd") >> SELECT "storeno","itemno","itemname","spec","itemseq","stocks","unit","ymd" >> FROM ext_gpload20160514_232419_14569 >> 2016-05-14 23:24:20|INFO|rows Inserted = 0 >> 2016-05-14 23:24:20|INFO|rows Updated = 0 >> 2016-05-14 23:24:20|INFO|data formatting errors = 0 >> 2016-05-14 23:24:20|INFO|gpload failed >> >> It was failed,even if there is only one line. >> >> ------------------------------ >> [email protected] >> > > > ---------- Forwarded message ---------- > From: Wen Lin <[email protected]> > To: [email protected] > Cc: [email protected] > Date: Tue, 17 May 2016 10:59:16 +0800 > Subject: Re: HAWQ YARN RPC Errors > Hi, Gagan, > > Where is the log? There is no attachment in your email. > > Thanks! > > On Sun, May 15, 2016 at 1:24 AM, Gagan Brahmi <[email protected]> > wrote: > >> Hi Wen, >> >> Please find attached logs which has a few instances of the occurrence >> of the error. >> >> >> Regards, >> Gagan Brahmi >> >> On Thu, May 12, 2016 at 7:33 PM, Wen Lin <[email protected]> wrote: >> > Hi, Gagan, >> > >> > It seems a sync failure between QD and Resource Manager. Not related to >> > libyarn 's RPC. >> > Would you like to attach the master's log file? Thanks! >> > >> > On Fri, May 13, 2016 at 12:58 AM, Gagan Brahmi <[email protected]> >> > wrote: >> > >> >> Hi Team, >> >> >> >> Do we have some recommended tuning for the RPC warning/errors >> >> encountered intermittently? >> >> >> >> The error which is seen is the following: >> >> >> >> WARNING: Sync RPC framework (inet) finds exception raised. >> >> ERROR: failed to return resource to resource manager, failed to >> >> receive content (pquery.c:991) >> >> >> >> This error however, disappears when we retry the query. There are >> >> cases when the query is to be retried more than once. >> >> >> >> The error looks to be invoked when COMM2RM_CLIENT_FAIL_RECV is >> encountered. >> >> >> >> The setup is using YARN resource manager. And the following is the >> >> yarn-client configuration used: >> >> >> >> <configuration> >> >> >> >> <property> >> >> <name>hadoop.security.authentication</name> >> >> <value>kerberos</value> >> >> </property> >> >> >> >> <property> >> >> <name>rpc.client.connect.retry</name> >> >> <value>10</value> >> >> </property> >> >> >> >> <property> >> >> <name>rpc.client.connect.tcpnodelay</name> >> >> <value>true</value> >> >> </property> >> >> >> >> <property> >> >> <name>rpc.client.connect.timeout</name> >> >> <value>600000</value> >> >> </property> >> >> >> >> <property> >> >> <name>rpc.client.max.idle</name> >> >> <value>10000</value> >> >> </property> >> >> >> >> <property> >> >> <name>rpc.client.ping.interval</name> >> >> <value>10000</value> >> >> </property> >> >> >> >> <property> >> >> <name>rpc.client.read.timeout</name> >> >> <value>3600000</value> >> >> </property> >> >> >> >> <property> >> >> <name>rpc.client.socket.linger.timeout</name> >> >> <value>-1</value> >> >> </property> >> >> >> >> <property> >> >> <name>rpc.client.timeout</name> >> >> <value>3600000</value> >> >> </property> >> >> >> >> <property> >> >> <name>rpc.client.write.timeout</name> >> >> <value>3600000</value> >> >> </property> >> >> >> >> <property> >> >> <name>yarn.client.failover.max.attempts</name> >> >> <value>15</value> >> >> </property> >> >> >> >> </configuration> >> >> >> >> I would appreciate some recommendations. >> >> >> >> >> >> Regards, >> >> Gagan Brahmi >> >> >> > > > -- *Shailesh Doshi | Data Engineering | Pivotal* *Mobile: 513-604-9746 **| Email <[email protected]>**: **[email protected]* <[email protected]> Big Data - Scale out & Open Source <http://pivotal.io/big-data/pivotal-big-data-suite>
