Leon,thank you for your reply, yes, i have set the driver memory to 55G and the java heap memory to 8G. And this user case is run successfully in the impala that installed on the same cluster. But in drill , the cmd always show the follow errors: Error: RESOURCE ERROR: One or more nodes ran out of memory while executing the query.
Failure allocating buffer. Fragment 3:6 [Error Id: d623b6b2-279d-4d24-af4d-0b62554b440c on bigdata2:31010] (state=,code=0) i think i miss some configuration. -- thanks for your regards. At 2016-05-10 13:51:12, "Leon Clayton" <[email protected]> wrote: >did you increase the memory setting for Drill from the default? > >https://drill.apache.org/docs/configuring-drill-memory/ ><https://drill.apache.org/docs/configuring-drill-memory/> > > >> On 10 May 2016, at 02:25, [email protected] wrote: >> >> >> hi: >> i run join operation in the drill, i use broadcast and put the small table >> in the right. The small table has 32000000 rows. I have set the >> planner.broadcast_threshold to 100000000. The cluster has three nodes and >> every node has 64G memory. when join is running, the memory is increasing >> untill the driilbit process exit. But the same query is run successful in >> the impala and they are in the same cluster. >> here is the query plan. >> >> 00-00 Screen : rowType = RecordType(VARCHAR(65535) sourceIP, DOUBLE >> totalRevenue, ANY avgPageRank): rowcount = 1.0, cumulative cost = >> {5.294005617599999E8 rows, 6.356723058846001E10 cpu, 0.0 io, >> 1.4803953770495996E11 network, 9.1066982688E8 memory}, id = 5015 >> 00-01 Project(sourceIP=[$0], totalRevenue=[$1], avgPageRank=[$2]) : >> rowType = RecordType(VARCHAR(65535) sourceIP, DOUBLE totalRevenue, ANY >> avgPageRank): rowcount = 1.0, cumulative cost = {5.2940056165999985E8 rows, >> 6.356723058836001E10 cpu, 0.0 io, 1.4803953770495996E11 network, >> 9.1066982688E8 memory}, id = 5014 >> 00-02 SelectionVectorRemover : rowType = RecordType(VARCHAR(65535) >> sourceIP, DOUBLE totalRevenue, ANY avgPageRank): rowcount = 1.0, cumulative >> cost = {5.2940056165999985E8 rows, 6.356723058836001E10 cpu, 0.0 io, >> 1.4803953770495996E11 network, 9.1066982688E8 memory}, id = 5013 >> 00-03 Limit(fetch=[1]) : rowType = RecordType(VARCHAR(65535) >> sourceIP, DOUBLE totalRevenue, ANY avgPageRank): rowcount = 1.0, cumulative >> cost = {5.2940056065999985E8 rows, 6.356723058736001E10 cpu, 0.0 io, >> 1.4803953770495996E11 network, 9.1066982688E8 memory}, id = 5012 >> 00-04 SingleMergeExchange(sort0=[1 DESC]) : rowType = >> RecordType(VARCHAR(65535) sourceIP, DOUBLE totalRevenue, ANY avgPageRank): >> rowcount = 457983.77999999997, cumulative cost = {5.2940055965999985E8 rows, >> 6.356723058336001E10 cpu, 0.0 io, 1.4803953770495996E11 network, >> 9.1066982688E8 memory}, id = 5011 >> 01-01 SelectionVectorRemover : rowType = >> RecordType(VARCHAR(65535) sourceIP, DOUBLE totalRevenue, ANY avgPageRank): >> rowcount = 457983.77999999997, cumulative cost = {5.289425758799999E8 rows, >> 6.356173477800001E10 cpu, 0.0 io, 1.4241183301631998E11 network, >> 9.1066982688E8 memory}, id = 5010 >> 01-02 TopN(limit=[1]) : rowType = RecordType(VARCHAR(65535) >> sourceIP, DOUBLE totalRevenue, ANY avgPageRank): rowcount = >> 457983.77999999997, cumulative cost = {5.284845920999999E8 rows, >> 6.356127679422001E10 cpu, 0.0 io, 1.4241183301631998E11 network, >> 9.1066982688E8 memory}, id = 5009 >> 01-03 Project(sourceIP=[$0], totalRevenue=[$1], >> avgPageRank=[$2]) : rowType = RecordType(VARCHAR(65535) sourceIP, DOUBLE >> totalRevenue, ANY avgPageRank): rowcount = 457983.77999999997, cumulative >> cost = {5.2802660831999993E8 rows, 6.356127679422001E10 cpu, 0.0 io, >> 1.4241183301631998E11 network, 9.1066982688E8 memory}, id = 5008 >> 01-04 HashToRandomExchange(dist0=[[$1]]) : rowType = >> RecordType(VARCHAR(65535) sourceIP, DOUBLE totalRevenue, ANY avgPageRank, >> ANY E_X_P_R_H_A_S_H_F_I_E_L_D): rowcount = 457983.77999999997, cumulative >> cost = {5.2802660831999993E8 rows, 6.356127679422001E10 cpu, 0.0 io, >> 1.4241183301631998E11 network, 9.1066982688E8 memory}, id = 5007 >> 02-01 UnorderedMuxExchange : rowType = >> RecordType(VARCHAR(65535) sourceIP, DOUBLE totalRevenue, ANY avgPageRank, >> ANY E_X_P_R_H_A_S_H_F_I_E_L_D): rowcount = 457983.77999999997, cumulative >> cost = {5.2756862453999996E8 rows, 6.3553949053740005E10 cpu, 0.0 io, >> 1.3490822676479999E11 network, 9.1066982688E8 memory}, id = 5006 >> 03-01 Project(sourceIP=[$0], totalRevenue=[$1], >> avgPageRank=[$2], E_X_P_R_H_A_S_H_F_I_E_L_D=[hash32AsDouble($1)]) : rowType >> = RecordType(VARCHAR(65535) sourceIP, DOUBLE totalRevenue, ANY avgPageRank, >> ANY E_X_P_R_H_A_S_H_F_I_E_L_D): rowcount = 457983.77999999997, cumulative >> cost = {5.2711064076E8 rows, 6.355349106996001E10 cpu, 0.0 io, >> 1.3490822676479999E11 network, 9.1066982688E8 memory}, id = 5005 >> 03-02 Project(sourceIP=[$0], >> totalRevenue=[CASE(=($4, 0), null, $3)], >> avgPageRank=[CAST(/(CastHigh(CASE(=($2, 0), null, $1)), $2)):ANY NOT NULL]) >> : rowType = RecordType(VARCHAR(65535) sourceIP, DOUBLE totalRevenue, ANY >> avgPageRank): rowcount = 457983.77999999997, cumulative cost = >> {5.2665265698E8 rows, 6.3551659134840004E10 cpu, 0.0 io, >> 1.3490822676479999E11 network, 9.1066982688E8 memory}, id = 5004 >> 03-03 HashAgg(group=[{0}], agg#0=[$SUM0($1)], >> agg#1=[$SUM0($2)], agg#2=[$SUM0($3)], agg#3=[$SUM0($4)]) : rowType = >> RecordType(VARCHAR(65535) sourceIP, INTEGER $f1, BIGINT $f2, DOUBLE $f3, >> BIGINT $f4): rowcount = 457983.77999999997, cumulative cost = >> {5.2619467320000005E8 rows, 6.3547995264600006E10 cpu, 0.0 io, >> 1.3490822676479999E11 network, 9.1066982688E8 memory}, id = 5003 >> 03-04 Project(sourceIP=[$0], $f1=[$1], >> $f2=[$2], $f3=[$3], $f4=[$4]) : rowType = RecordType(VARCHAR(65535) >> sourceIP, INTEGER $f1, BIGINT $f2, DOUBLE $f3, BIGINT $f4): rowcount = >> 4579837.8, cumulative cost = {5.2161483540000004E8 rows, 6.32915243478E10 >> cpu, 0.0 io, 1.3490822676479999E11 network, 8.300646816E8 memory}, id = 5002 >> 03-05 HashToRandomExchange(dist0=[[$0]]) : >> rowType = RecordType(VARCHAR(65535) sourceIP, INTEGER $f1, BIGINT $f2, >> DOUBLE $f3, BIGINT $f4, ANY E_X_P_R_H_A_S_H_F_I_E_L_D): rowcount = >> 4579837.8, cumulative cost = {5.2161483540000004E8 rows, 6.32915243478E10 >> cpu, 0.0 io, 1.3490822676479999E11 network, 8.300646816E8 memory}, id = 5001 >> 04-01 UnorderedMuxExchange : rowType = >> RecordType(VARCHAR(65535) sourceIP, INTEGER $f1, BIGINT $f2, DOUBLE $f3, >> BIGINT $f4, ANY E_X_P_R_H_A_S_H_F_I_E_L_D): rowcount = 4579837.8, cumulative >> cost = {5.170349976E8 rows, 6.3218246943E10 cpu, 0.0 io, 2.2354132992E10 >> network, 8.300646816E8 memory}, id = 5000 >> 05-01 Project(sourceIP=[$0], $f1=[$1], >> $f2=[$2], $f3=[$3], $f4=[$4], >> E_X_P_R_H_A_S_H_F_I_E_L_D=[hash32AsDouble($0)]) : rowType = >> RecordType(VARCHAR(65535) sourceIP, INTEGER $f1, BIGINT $f2, DOUBLE $f3, >> BIGINT $f4, ANY E_X_P_R_H_A_S_H_F_I_E_L_D): rowcount = 4579837.8, cumulative >> cost = {5.124551598E8 rows, 6.32136671052E10 cpu, 0.0 io, 2.2354132992E10 >> network, 8.300646816E8 memory}, id = 4999 >> 05-02 HashAgg(group=[{0}], >> agg#0=[$SUM0($1)], agg#1=[COUNT($1)], agg#2=[$SUM0($2)], agg#3=[COUNT($2)]) >> : rowType = RecordType(VARCHAR(65535) sourceIP, INTEGER $f1, BIGINT $f2, >> DOUBLE $f3, BIGINT $f4): rowcount = 4579837.8, cumulative cost = >> {5.07875322E8 rows, 6.3195347754E10 cpu, 0.0 io, 2.2354132992E10 network, >> 8.300646816E8 memory}, id = 4998 >> 05-03 Project(sourceIP=[$0], >> pagerank=[$5], adrevenue=[$3]) : rowType = RecordType(VARCHAR(65535) >> sourceIP, INTEGER pagerank, DOUBLE adrevenue): rowcount = 4.5798378E7, >> cumulative cost = {4.62076944E8 rows, 6.0630638586E10 cpu, 0.0 io, >> 2.2354132992E10 network, 2.40132288E7 memory}, id = 4997 >> 05-04 HashJoin(condition=[=($4, >> $1)], joinType=[inner]) : rowType = RecordType(VARCHAR(65535) sourceip, >> VARCHAR(65535) desturl, VARCHAR(65535) visitdate, DOUBLE adrevenue, >> VARCHAR(65535) pageurl, INTEGER pagerank): rowcount = 4.5798378E7, >> cumulative cost = {4.62076944E8 rows, 6.0630638586E10 cpu, 0.0 io, >> 2.2354132992E10 network, 2.40132288E7 memory}, id = 4996 >> 05-06 SelectionVectorRemover : >> rowType = RecordType(VARCHAR(65535) sourceip, VARCHAR(65535) desturl, >> VARCHAR(65535) visitdate, DOUBLE adrevenue): rowcount = 4.5798378E7, >> cumulative cost = {4.12185402E8 rows, 5.9950076802E10 cpu, 0.0 io, 0.0 >> network, 0.0 memory}, id = 4993 >> 05-07 >> Filter(condition=[AND(>=(DATEDIFF($2, '1980-01-01'), 0), <=(DATEDIFF($2, >> '1980-04-01'), 0))]) : rowType = RecordType(VARCHAR(65535) sourceip, >> VARCHAR(65535) desturl, VARCHAR(65535) visitdate, DOUBLE adrevenue): >> rowcount = 4.5798378E7, cumulative cost = {3.66387024E8 rows, >> 5.9904278424E10 cpu, 0.0 io, 0.0 network, 0.0 memory}, id = 4992 >> 05-08 >> Scan(groupscan=[HiveScan [table=Table(dbName:default, >> tableName:uservisits_copy_huge), columns=[`sourceip`, `desturl`, >> `visitdate`, `adrevenue`], numPartitions=0, partitions= null, >> inputDirectories=[hdfs://nameservice1/HiBench/huge/Join/Input/uservisits]]]) >> : rowType = RecordType(VARCHAR(65535) sourceip, VARCHAR(65535) desturl, >> VARCHAR(65535) visitdate, DOUBLE adrevenue): rowcount = 1.83193512E8, >> cumulative cost = {1.83193512E8 rows, 5.862192384E10 cpu, 0.0 io, 0.0 >> network, 0.0 memory}, id = 4991 >> 05-05 BroadcastExchange : rowType >> = RecordType(VARCHAR(65535) pageurl, INTEGER pagerank): rowcount = >> 1364388.0, cumulative cost = {2728776.0 rows, 1.20066144E8 cpu, 0.0 io, >> 2.2354132992E10 network, 0.0 memory}, id = 4995 >> 06-01 Scan(groupscan=[HiveScan >> [table=Table(dbName:default, tableName:rankings_huge), columns=[`pageurl`, >> `pagerank`], numPartitions=0, partitions= null, >> inputDirectories=[hdfs://nameservice1/HiBench/huge/Join/Input/rankings]]]) : >> rowType = RecordType(VARCHAR(65535) pageurl, INTEGER pagerank): rowcount = >> 1364388.0, cumulative cost = {1364388.0 rows, 1.0915104E8 cpu, 0.0 io, 0.0 >> network, 0.0 memory}, id = 4994 >> -- >> >> thanks for your regards. >> >> >> >
