If you've already set the limit at 4GB, there might be something else going on. We'll take a look at this in more detail, but you shouldn't expect a solution soon. (probably 1.14.0, since 1.13.0 is already on its way out for release).
For now, bumping up the limit as you've done, reducing the "planner.width.max_per_node" from the current value (so as to make more memory available to each fragment) are the options. THanks for sharing this information! On Fri, Mar 16, 2018 at 4:29 AM, Anup Tiwari <[email protected]> wrote: > Hi Kunal, > First of all, thanks for such a good explanation, it really helped me > understanding few things.But as you have mentioned that in case of failure > "Drillbits capped at around 1.2GB" and suggested to "increase the > memory-per-query-per-node from the current 2GB to a higher level". > Are you saying to increase this "planner.memory.max_query_memory_per_node" > from > 2GB? If yes then just wanted to mention that i have already set > planner.memory.max_query_memory_per_node = 4G(mentioned in trail mail). > Let me know if i have misinterpreted anything. > > > > > > On Fri, Mar 16, 2018 3:08 AM, Kunal Khatua [email protected] wrote: > Anup > > > > > If you look at the successful run's major fragment overview, you can see > > the amount of memory consumed across the 5 nodes and the average per node > > (shown in the table): > > +----------------+-----------------+-------------+---------- > ---+-------------+--------+-----------------+-------------------+ > > | Major Fragment | Minor Fragments | Min Runtime | Avg Runtime | Max > > Runtime | % Busy | Max Peak Memory | Total Peak Memory | > > +----------------+-----------------+-------------+---------- > ---+-------------+--------+-----------------+-------------------+ > > | 00-xx-xx | 1 / 1 | 2m18s | 2m18s | 2m18s > > | 0.28% | 3MB | 3MB | > > | 01-xx-xx | 30 / 30 | 2m17s | 2m17s | 2m18s > > | 0.19% | 55MB | 1650MB | > > | 02-xx-xx | 5 / 5 | 2m4s | 2m11s | 2m17s > > | 0.07% | 9MB | 45MB | > > | 03-xx-xx | 30 / 30 | 1m18s | 2m1s | 2m17s > > | 70.89% | 200MB | 6000MB | > > | 04-xx-xx | 2 / 2 | 2.608s | 2.627s | 2.646s > > | 5.52% | 10MB | 20MB | > > | | | | | > > | | Avg/Node => | 1544MB | > > +----------------+-----------------+-------------+---------- > ---+-------------+--------+-----------------+-------------------+ > > > > > So, we're looking at 1.5GB average. > > > > > The failed run was > > +----------------+-----------------+-------------+---------- > ---+-------------+--------+-----------------+------------+ > > | Major Fragment | Minor Fragments | Min Runtime | Avg Runtime | Max > > Runtime | % Busy | Max Peak Memory | Total Peak | > > +----------------+-----------------+-------------+---------- > ---+-------------+--------+-----------------+------------+ > > | 00-xx-xx | 0 / 1 | | | > > | | | #N/A | > > | 01-xx-xx | 30 / 30 | 1m30s | 1m33s | 1m34s > > | 0.00% | 8MB | 240MB | > > | 02-xx-xx | 5-May | 1m29s | 1m31s | 1m33s > > | 0.03% | 6MB | 30MB | > > | 03-xx-xx | 30 / 30 | 1m17s | 1m30s | 1m34s > > | 31.25% | 195MB | 5850MB | > > | 04-xx-xx | 2-Feb | 2.673s | 2.674s | 2.676s > > | 2.39% | 10MB | 20MB | > > | | | | | > > | | Avg/Node => | 1228MB | > > +----------------+-----------------+-------------+---------- > ---+-------------+--------+-----------------+------------+ > > > > > You can see that the Drillbits here capped at around 1.2GB before failing. > > > > > Drill 1.12 currently does not have the Spill-to-disk for the HashJoin > > operator, but there is memory usage cap being enforced for HashAgg. > > Enabling the `fallback` option removes this cap. > > > > > The option here would be to increase the memory-per-query-per-node from the > > current 2GB to a higher level. > > > > > The option you're looking for is: > > planner.memory.max_query_memory_per_node > > and/or > > planner.memory.percent_per_query > > > > > > > > Read this for a better understanding of all your options: > > https://drill.apache.org/docs/configuring-drill-memory/ > > > > > ~ Kunal > > > > > > > > > > > > > > On Tue, Mar 13, 2018 at 11:41 PM, Anup Tiwari <[email protected]> > > wrote: > > > > > > Hi Kunal, > > > Please find below cluster/platform details :- > > > Number of Nodes : 5 > > > RAM/Node : 32GBCore/Node : 8DRILL_MAX_DIRECT_MEMORY="20G" > > > DRILL_HEAP="8G"DRILL > > > VERSION = 1.12.0HADOOP VERSION = 2.7.3ZOOKEEPER VERSION = 3.4.8(Installed > > > in > > > Distributed Mode on 3 nodes)planner.memory.max_query_memory_per_node = > 4G > > > Please find below query which was giving error before setting "alter > > > session set > > > `drill.exec.hashagg.fallback.enabled`=TRUE;" and its json profile for > > > success(variable is set) and failure(variable is not set). Please note > this > > > query was working fine in Drill 1.10.0 in same environment. > > > Query :- > > > create table a_games_log_visit_utm asselectdistinctglv.sessionid,case > when > > > (UFG('utms=', glv.url, '&') <> 'null') then UFG('utms=', glv.url, '&') > else > > > UFG('utms=',glv.hcookie, '&') end as utms, > > > case when (UFG('utms=', glv.url, '&') <> 'null') then UFG('utmm=', > > > glv.url, '&') > > > else UFG('utmm=',glv.hcookie, '&') end as utmm, > > > case when (UFG('utms=', glv.url, '&') <> 'null') then UFG('utmc=', > > > glv.url, '&') > > > else UFG('utmc=',glv.hcookie, '&') end as utmc, > > > case when (UFG('utms=', glv.url, '&') <> 'null') then UFG('utmt=', > > > glv.url, '&') > > > else UFG('utmt=',glv.hcookie, '&') end as utmt, > > > case when (UFG('utms=', glv.url, '&') <> 'null') then UFG('utmcmpn=', > > > glv.url, > > > '&') else UFG('utmcmpn=',glv.hcookie, '&') end as utmcmpn, > > > case when (UFG('utms=', glv.url, '&') <> 'null') then UFG('utmplnt=', > > > glv.url, > > > '&') else UFG('utmplnt=',glv.hcookie, '&') end as utmplnt, > > > case when (UFG('utms=', glv.url, '&') <> 'null') then UFG('uid=', > glv.url, > > > '&') > > > else UFG('uid=', glv.hcookie, '&') end as uid, > > > case when (UFG('utms=', glv.url, '&') <> 'null') then UFG('cid=', > glv.url, > > > '&') > > > else UFG('cid=', glv.hcookie, '&') end as cid, > > > case when (UFG('utms=', glv.url, '&') <> 'null') then UFG('gclid=', > > > glv.url, > > > '&') else UFG('gclid=', glv.hcookie, '&') end as gclid > > > fromglv_utm_1 a,a_games_log_visit_base glvwherea.sessionid = > > > glv.sessionidand > > > a.max_serverTime = glv.serverTimeand (STRPOS(glv.url,'utms=') > 0 OR > > > STRPOS(glv.hcookie,'utms=') > 0) and glv.url is not null ; > > > > > > JSON Profile when Failed :- > > > {"id":{"part1":2690698323716856156,"part2":- > 8447970446519047466},"type":1, > > > "start":1521006624202,"end":1521006722666,"query":"create > > > table a_games_log_visit_utm as\nselect\ndistinct\nglv.sessionid,\ncase > > > when > > > (UFG('utms=', glv.url, '&') <> 'null') then UFG('utms=', glv.url, '&') > else > > > UFG('utms=',glv.hcookie, '&') end as utms,\n\ncase when (UFG('utms=', > > > glv.url, > > > '&') <> 'null') then UFG('utmm=', glv.url, '&') else > > > UFG('utmm=',glv.hcookie, > > > '&') end as utmm,\n\ncase when (UFG('utms=', glv.url, '&') <> 'null') > then > > > UFG('utmc=', glv.url, '&') else UFG('utmc=',glv.hcookie, '&') end as > > > utmc,\n\ncase when (UFG('utms=', glv.url, '&') <> 'null') then > UFG('utmt=', > > > glv.url, '&') else UFG('utmt=',glv.hcookie, '&') end as utmt,\n\ncase > when > > > (UFG('utms=', glv.url, '&') <> 'null') then UFG('utmcmpn=', glv.url, '&') > > > else > > > UFG('utmcmpn=',glv.hcookie, '&') end as utmcmpn,\n\ncase when > (UFG('utms=', > > > glv.url, '&') <> 'null') then UFG('utmplnt=', glv.url, '&') else > > > UFG('utmplnt=',glv.hcookie, '&') end as utmplnt,\n\ncase when > (UFG('utms=', > > > glv.url, '&') <> 'null') then UFG('uid=', glv.url, '&') else UFG('uid=', > > > glv.hcookie, '&') end as uid,\n\ncase when (UFG('utms=', glv.url, '&') <> > > > 'null') then UFG('cid=', glv.url, '&') else UFG('cid=', glv.hcookie, '&') > > > end as > > > cid,\n\ncase when (UFG('utms=', glv.url, '&') <> 'null') then > UFG('gclid=', > > > glv.url, '&') else UFG('gclid=', glv.hcookie, '&') end as > > > gclid\n\nfrom\nglv_utm_1 a,\na_games_log_visit_base > > > glv\nwhere\na.sessionid = > > > glv.sessionid\nand a.max_serverTime = glv.serverTime\nand > > > (STRPOS(glv.url,'utms=') > 0 OR STRPOS(glv.hcookie,'utms=') > 0)\nand > > > glv.url is > > > not null ","plan":"00-00 Screen : rowType = RecordType(VARCHAR(255) > > > Fragment, > > > BIGINT Number of records written): rowcount = 402043.18275, cumulative > > > cost = > > > {5.354375750937748E8 rows, 7.821369551257525E9 cpu, 0.0 io, > > > 1.88842215292928E11 > > > network, 4.283794390722E9 memory}, id = 7755274\n00-01 > > > Project(Fragment=[$0], > > > Number of records written=[$1]) : rowType = RecordType(VARCHAR(255) > > > Fragment, > > > BIGINT Number of records written): rowcount = 402043.18275, cumulative > > > cost = > > > {5.353973707754998E8 rows, 7.82132934693925E9 cpu, 0.0 io, > > > 1.88842215292928E11 > > > network, 4.283794390722E9 memory}, id = 7755273\n00-02 UnionExchange : > > > rowType = > > > RecordType(VARCHAR(255) Fragment, BIGINT Number of records written): > > > rowcount = > > > 402043.18275, cumulative cost = {5.353973707754998E8 rows, > > > 7.82132934693925E9 > > > cpu, 0.0 io, 1.88842215292928E11 network, 4.283794390722E9 memory}, id = > > > 7755272\n01-01 Writer : rowType = RecordType(VARCHAR(255) Fragment, > BIGINT > > > Number of records written): rowcount = 402043.18275, cumulative cost = > > > {5.3499532759274983E8 rows, 7.81811300147725E9 cpu, 0.0 io, > > > 1.8554867753984E11 > > > network, 4.283794390722E9 memory}, id = 7755271\n01-02 > > > ProjectAllowDup(sessionid=[$0], utms=[$1], utmm=[$2], utmc=[$3], > > > utmt=[$4], > > > utmcmpn=[$5], utmplnt=[$6], uid=[$7], cid=[$8], gclid=[$9]) : rowType = > > > RecordType(ANY sessionid, ANY utms, ANY utmm, ANY utmc, ANY utmt, ANY > > > utmcmpn, > > > ANY utmplnt, ANY uid, ANY cid, ANY gclid): rowcount = 402043.18275, > > > cumulative > > > cost = {5.3459328440999985E8 rows, 7.8177109582945E9 cpu, 0.0 io, > > > 1.8554867753984E11 network, 4.283794390722E9 memory}, id = 7755270\n01-03 > > > Project(sessionid=[$0], utms=[$1], utmm=[$2], utmc=[$3], utmt=[$4], > > > utmcmpn=[$5], utmplnt=[$6], uid=[$7], cid=[$8], gclid=[$9]) : rowType = > > > RecordType(ANY sessionid, ANY utms, ANY utmm, ANY utmc, ANY utmt, ANY > > > utmcmpn, > > > ANY utmplnt, ANY uid, ANY cid, ANY gclid): rowcount = 402043.18275, > > > cumulative > > > cost = {5.3459328440999985E8 rows, 7.8177109582945E9 cpu, 0.0 io, > > > 1.8554867753984E11 network, 4.283794390722E9 memory}, id = 7755269\n01-04 > > > HashAgg(group=[{0, 1, 2, 3, 4, 5, 6, 7, 8, 9}]) : rowType = > RecordType(ANY > > > sessionid, ANY utms, ANY utmm, ANY utmc, ANY utmt, ANY utmcmpn, ANY > > > utmplnt, ANY > > > uid, ANY cid, ANY gclid): rowcount = 402043.18275, cumulative cost = > > > {5.3459328440999985E8 rows, 7.8177109582945E9 cpu, 0.0 io, > > > 1.8554867753984E11 > > > network, 4.283794390722E9 memory}, id = 7755268\n01-05 > > > Project(sessionid=[$0], > > > utms=[$1], utmm=[$2], utmc=[$3], utmt=[$4], utmcmpn=[$5], utmplnt=[$6], > > > uid=[$7], cid=[$8], gclid=[$9]) : rowType = RecordType(ANY sessionid, ANY > > > utms, > > > ANY utmm, ANY utmc, ANY utmt, ANY utmcmpn, ANY utmplnt, ANY uid, ANY cid, > > > ANY > > > gclid): rowcount = 4020431.8274999997, cumulative cost = > > > {5.3057285258249986E8 > > > rows, 7.496076412094501E9 cpu, 0.0 io, 1.8554867753984E11 network, > > > 3.89461658982E9 memory}, id = 7755267\n01-06 > HashToRandomExchange(dist0=[[$ > > > 0]], > > > dist1=[[$1]], dist2=[[$2]], dist3=[[$3]], dist4=[[$4]], dist5=[[$5]], > > > dist6=[[$6]], dist7=[[$7]], dist8=[[$8]], dist9=[[$9]]) : rowType = > > > RecordType(ANY sessionid, ANY utms, ANY utmm, ANY utmc, ANY utmt, ANY > > > utmcmpn, > > > ANY utmplnt, ANY uid, ANY cid, ANY gclid, ANY E_X_P_R_H_A_S_H_F_I_E_L_D): > > > rowcount = 4020431.8274999997, cumulative cost = {5.3057285258249986E8 > > > rows, > > > 7.496076412094501E9 cpu, 0.0 io, 1.8554867753984E11 network, > > > 3.89461658982E9 > > > memory}, id = 7755266\n02-01 UnorderedMuxExchange : rowType = > > > RecordType(ANY > > > sessionid, ANY utms, ANY utmm, ANY utmc, ANY utmt, ANY utmcmpn, ANY > > > utmplnt, ANY > > > uid, ANY cid, ANY gclid, ANY E_X_P_R_H_A_S_H_F_I_E_L_D): rowcount = > > > 4020431.8274999997, cumulative cost = {5.265524207549999E8 rows, > > > 7.460696612012501E9 cpu, 0.0 io, 4.40410112E9 network, 3.89461658982E9 > > > memory}, > > > id = 7755265\n03-01 Project(sessionid=[$0], utms=[$1], utmm=[$2], > > > utmc=[$3], > > > utmt=[$4], utmcmpn=[$5], utmplnt=[$6], uid=[$7], cid=[$8], gclid=[$9], > > > E_X_P_R_H_A_S_H_F_I_E_L_D=[hash32AsDouble($9, hash32AsDouble($8, > > > hash32AsDouble($7, hash32AsDouble($6, hash32AsDouble($5, > hash32AsDouble($4, > > > hash32AsDouble($3, hash32AsDouble($2, hash32AsDouble($1, > hash32AsDouble($0, > > > 1301011))))))))))]) : rowType = RecordType(ANY sessionid, ANY utms, ANY > > > utmm, > > > ANY utmc, ANY utmt, ANY utmcmpn, ANY utmplnt, ANY uid, ANY cid, ANY > gclid, > > > ANY > > > E_X_P_R_H_A_S_H_F_I_E_L_D): rowcount = 4020431.8274999997, cumulative > cost > > > = > > > {5.225319889274999E8 rows, 7.456676180185E9 cpu, 0.0 io, 4.40410112E9 > > > network, > > > 3.89461658982E9 memory}, id = 7755264\n03-02 HashAgg(group=[{0, 1, 2, 3, > > > 4, 5, > > > 6, 7, 8, 9}]) : rowType = RecordType(ANY sessionid, ANY utms, ANY utmm, > ANY > > > utmc, ANY utmt, ANY utmcmpn, ANY utmplnt, ANY uid, ANY cid, ANY gclid): > > > rowcount > > > = 4020431.8274999997, cumulative cost = {5.185115570999999E8 rows, > > > 7.440594452875E9 cpu, 0.0 io, 4.40410112E9 network, 3.89461658982E9 > > > memory}, id > > > = 7755263\n03-03 Project(sessionid=[$2], utms=[CASE(<>(UFG('utms=', $4, > > > '&'), > > > 'null'), UFG('utms=', $4, '&'), UFG('utms=', $5, '&'))], > > > utmm=[CASE(<>(UFG('utms=', $4, '&'), 'null'), UFG('utmm=', $4, '&'), > > > UFG('utmm=', $5, '&'))], utmc=[CASE(<>(UFG('utms=', $4, '&'), 'null'), > > > UFG('utmc=', $4, '&'), UFG('utmc=', $5, '&'))], > utmt=[CASE(<>(UFG('utms=', > > > $4, > > > '&'), 'null'), UFG('utmt=', $4, '&'), UFG('utmt=', $5, '&'))], > > > utmcmpn=[CASE(<>(UFG('utms=', $4, '&'), 'null'), UFG('utmcmpn=', $4, > '&'), > > > UFG('utmcmpn=', $5, '&'))], utmplnt=[CASE(<>(UFG('utms=', $4, '&'), > > > 'null'), > > > UFG('utmplnt=', $4, '&'), UFG('utmplnt=', $5, '&'))], > > > uid=[CASE(<>(UFG('utms=', > > > $4, '&'), 'null'), UFG('uid=', $4, '&'), UFG('uid=', $5, '&'))], > > > cid=[CASE(<>(UFG('utms=', $4, '&'), 'null'), UFG('cid=', $4, '&'), > > > UFG('cid=', > > > $5, '&'))], gclid=[CASE(<>(UFG('utms=', $4, '&'), 'null'), UFG('gclid=', > > > $4, > > > '&'), UFG('gclid=', $5, '&'))]) : rowType = RecordType(ANY sessionid, ANY > > > utms, > > > ANY utmm, ANY utmc, ANY utmt, ANY utmcmpn, ANY utmplnt, ANY uid, ANY cid, > > > ANY > > > gclid): rowcount = 4.0204318275E7, cumulative cost = {4.783072388249999E8 > > > rows, > > > 4.224248990875E9 cpu, 0.0 io, 4.40410112E9 network, 2838580.8000000003 > > > memory}, > > > id = 7755262\n03-04 Project(sessionid=[$4], max_serverTime=[$5], > > > sessionid0=[$0], serverTime=[$1], url=[$2], hcookie=[$3]) : rowType = > > > RecordType(ANY sessionid, ANY max_serverTime, ANY sessionid0, ANY > > > serverTime, > > > ANY url, ANY hcookie): rowcount = 4.0204318275E7, cumulative cost = > > > {4.3810292054999995E8 rows, 2.776893532975E9 cpu, 0.0 io, 4.40410112E9 > > > network, > > > 2838580.8000000003 memory}, id = 7755261\n03-05 > > > HashJoin(condition=[AND(=($4, > > > $0), =($5, $1))], joinType=[inner]) : rowType = RecordType(ANY sessionid, > > > ANY > > > serverTime, ANY url, ANY hcookie, ANY sessionid0, ANY max_serverTime): > > > rowcount > > > = 4.0204318275E7, cumulative cost = {4.3810292054999995E8 rows, > > > 2.776893532975E9 > > > cpu, 0.0 io, 4.40410112E9 network, 2838580.8000000003 memory}, id = > > > 7755260\n03-07 SelectionVectorRemover : rowType = RecordType(ANY > > > sessionid, ANY > > > serverTime, ANY url, ANY hcookie): rowcount = 4.0204318275E7, cumulative > > > cost = > > > {3.97576036275E8 rows, 1.809194322375E9 cpu, 0.0 io, 0.0 network, 0.0 > > > memory}, > > > id = 7755256\n03-09 Filter(condition=[AND(OR(>(STRPOS($2, 'utms='), 0), > > > >(STRPOS($3, 'utms='), 0)), IS NOT NULL($2))]) : rowType = RecordType(ANY > > > sessionid, ANY serverTime, ANY url, ANY hcookie): rowcount = > > > 4.0204318275E7, > > > cumulative cost = {3.57371718E8 rows, 1.7689900041E9 cpu, 0.0 io, 0.0 > > > network, > > > 0.0 memory}, id = 7755255\n03-10 Scan(groupscan=[ParquetGroupScan > > > [entries=[ReadEntryWithPath > > > [path=hdfs://prod-hadoop-xx1.bom-prod.com:9000/usr/drill/ > > > warehouse/a_games_log_visit_base]], > > > selectionRoot=hdfs://prod-hadoop-xx1.bom-prod.com:9000/ > > > usr/drill/warehouse/a_games_log_visit_base, > > > numFiles=1, numRowGroups=232, usedMetadataFile=false, > columns=[`sessionid`, > > > `serverTime`, `url`, `hcookie`]]]) : rowType = RecordType(ANY sessionid, > > > ANY > > > serverTime, ANY url, ANY hcookie): rowcount = 1.78685859E8, cumulative > > > cost = > > > {1.78685859E8 rows, 7.14743436E8 cpu, 0.0 io, 0.0 network, 0.0 memory}, > id > > > = > > > 7755254\n03-06 Project(sessionid0=[$0], max_serverTime=[$1]) : rowType = > > > RecordType(ANY sessionid0, ANY max_serverTime): rowcount = 107522.0, > > > cumulative > > > cost = {215044.0 rows, 1075220.0 cpu, 0.0 io, 4.40410112E9 network, 0.0 > > > memory}, > > > id = 7755259\n03-08 BroadcastExchange : rowType = RecordType(ANY > > > sessionid, ANY > > > max_serverTime): rowcount = 107522.0, cumulative cost = {215044.0 rows, > > > 1075220.0 cpu, 0.0 io, 4.40410112E9 network, 0.0 memory}, id = > > > 7755258\n04-01 > > > Scan(groupscan=[ParquetGroupScan [entries=[ReadEntryWithPath > > > [path=hdfs://prod-hadoop-xx1.bom-prod.com:9000/usr/drill/ > > > warehouse/glv_utm_1]], > > > selectionRoot=hdfs://prod-hadoop-xx1.bom-prod.com:9000/ > > > usr/drill/warehouse/glv_utm_1, > > > numFiles=1, numRowGroups=30, usedMetadataFile=false, > columns=[`sessionid`, > > > `max_serverTime`]]]) : rowType = RecordType(ANY sessionid, ANY > > > max_serverTime): > > > rowcount = 107522.0, cumulative cost = {107522.0 rows, 215044.0 cpu, 0.0 > > > io, 0.0 > > > network, 0.0 memory}, id = > > > 7755257\n","foreman":{"address":"prod-hadoop-xx1.bom-prod.com > > > ","userPort":31010,"controlPort":31011,"dataPort": > > > 31012,"version":"1.12.0","state":0},"state":4,"totalFragments":68," > > > finishedFragments":42,"fragmentProfile":[{"majorFragmentId":0," > > > minorFragmentProfile":[{"state":0,"minorFragmentId":0," > > > endpoint":{"address":"prod-hadoop-xx1.bom-prod.com"," > > > userPort":31010,"controlPort":31011,"dataPort":31012," > > > version":"1.12.0","state":0},"lastUpdate":1521006628333," > > > lastProgress":1521006628333}]},{"majorFragmentId":3," > > > minorFragmentProfile":[{"state":3,"minorFragmentId":0," > > > operatorProfile":[{"inputProfile":[{"records":6186060,"batches":200," > > > schemas":1}],"operatorId":10,"operatorType":21,"setupNanos": > > > 0,"processNanos":17802732925,"peakLocalMemoryAllocated": > > > 101089280,"metric":[{"metricId":0,"longValue":0},{" > > > metricId":5,"longValue":0},{"metricId":7,"longValue":0},{" > > > metricId":4,"longValue":8010},{"metricId":11,"longValue": > > > 11687847},{"metricId":6,"longValue":0},{"metricId":12," > > > longValue":0},{"metricId":16,"longValue":0},{"metricId":8," > > > longValue":1569776733},{"metricId":19,"longValue": > > > 81395966725},{"metricId":10,"longValue":63696348545},{" > > > metricId":15,"longValue":63696348545},{"metricId":9," > > > longValue":0},{"metricId":2,"longValue":8010},{"metricId": > > > 13,"longValue":0},{"metricId":1,"longValue":8010},{" > > > metricId":14,"longValue":4099331644},{"metricId":3," > > > longValue":0},{"metricId":17,"longValue":70718},{"metricId": > > > 18,"longValue":81395481538}],"waitNanos":63690635089},{" > > > inputProfile":[{"records":6186060,"batches":192," > > > schemas":1}],"operatorId":9,"operatorType":2,"setupNanos": > > > 2167676,"processNanos":9481199735,"peakLocalMemoryAllocated": > > > 59965440,"waitNanos":0},{"inputProfile":[{"records": > > > 47601,"batches":192,"schemas":1}],"operatorId":7," > > > operatorType":14,"setupNanos":376514,"processNanos":961402725," > > > peakLocalMemoryAllocated":1896448,"waitNanos":0},{" > > > inputProfile":[{"records":107522,"batches":30,"schemas": > > > 1}],"operatorId":8,"operatorType":11,"setupNanos": > > > 0,"processNanos":796948,"peakLocalMemoryAllocated":0," > > > metric":[{"metricId":0,"longValue":0},{"metricId":1," > > > longValue":2}],"waitNanos":1972510452},{"inputProfile":[{ > > > "records":107522,"batches":30,"schemas":1}],"operatorId":6," > > > operatorType":10,"setupNanos":94400,"processNanos":256436," > > > peakLocalMemoryAllocated":262144,"waitNanos":0},{" > > > inputProfile":[{"records":47601,"batches":192,"schemas": > > > 1},{"records":107522,"batches":30,"schemas":1}],"operatorId" > > > :5,"operatorType":4,"setupNanos":0,"processNanos":106350538," > > > peakLocalMemoryAllocated":26697728,"metric":[{"metricId" > > > :0,"longValue":262144},{"metricId":1,"longValue":107522},{"metricId":2," > > > longValue":2},{"metricId":3,"longValue":14}],"waitNanos":0} > > > ,{"inputProfile":[{"records":3468,"batches":2,"schemas":1}] > > > ,"operatorId":4,"operatorType":10,"setupNanos":69408," > > > processNanos":59257,"peakLocalMemoryAllocated":6021120,"waitNanos":0},{" > > > inputProfile":[{"records":3468,"batches":2,"schemas":1}] > > > ,"operatorId":3,"operatorType":10,"setupNanos":13988583," > > > processNanos":96946739,"peakLocalMemoryAllocated": > > > 1024000,"waitNanos":0},{"inputProfile":[{"records": > > > 3468,"batches":2,"schemas":1}],"operatorId":2,"operatorType" > > > :3,"setupNanos":163742,"processNanos":24203075," > peakLocalMemoryAllocated": > > > 48627712,"metric":[{"metricId":0,"longValue":65536},{" > > > metricId":2,"longValue":0},{"metricId":3,"longValue":0},{" > > > metricId":7,"longValue":0},{"metricId":4,"longValue":1},{" > > > metricId":6,"longValue":0},{"metricId":1,"longValue":3462}] > > > ,"waitNanos":0},{"inputProfile":[{"records": > 3463,"batches":3,"schemas":1}] > > > ,"operatorId":1,"operatorType":10,"setupNanos":2865181," > > > processNanos":2660709,"peakLocalMemoryAllocated": > > > 47857664,"waitNanos":0},{"inputProfile":[{"records":0," > > > batches":0,"schemas":0}],"operatorId":0,"operatorType": > 0,"setupNanos":0," > > > processNanos":0,"peakLocalMemoryAllocated":47857664,"waitNanos":0},{" > > > inputProfile":[{"records":3463,"batches":3,"schemas":1}] > > > ,"operatorId":0,"operatorType":0,"setupNanos":0,"processNanos":216584," > > > peakLocalMemoryAllocated":47857664,"metric":[{"metricId" > > > :0,"longValue":730877}],"waitNanos":1256571}]," > startTime":1521006628339," > > > endTime":1521006722561,"memoryUsed":0,"maxMemoryUsed": > > > 159717376,"endpoint":{"address":"prod-hadoop-xx5.bom-prod.com > > > ","userPort":31010,"controlPort":31011,"dataPort": > > > 31012,"version":"1.12.0","state":1},"lastUpdate": > > > 1521006722561,"lastProgress":1521006722561},{"state":2," > > > minorFragmentId":2,"operatorProfile":[{"inputProfile":[{"records": > > > 5847418,"batches":189,"schemas":1}],"operatorId":10," > > > operatorType":21,"setupNanos":0,"processNanos":17568225997," > > > peakLocalMemoryAllocated":101089280,"metric":[{" > > > metricId":0,"longValue":0},{"metricId":17,"longValue": > > > 56174},{"metricId":18,"longValue":75411719920},{" > > > metricId":13,"longValue":0},{"metricId":1,"longValue":6969}, > > > {"metricId":15,"longValue":59830547344},{"metricId":16," > > > longValue":0},{"metricId":11,"longValue":7388846},{" > > > metricId":14,"longValue":3521451380},{"metricId":3," > > > longValue":0},{"metricId":5,"longValue":0},{"metricId":10," > > > longValue":59830547344},{"metricId":9,"longValue":0},{" > > > metricId":8,"longValue":1412404599},{"metricId":6," > > > longValue":0},{"metricId":12,"longValue":0},{"metricId":7," > > > longValue":0},{"metricId":4,"longValue":6969},{"metricId": > > > 2,"longValue":6969},{"metricId":19,"longValue": > 75412289227}],"waitNanos": > > > 64058389010},{"inputProfile":[{"records":5847418,"batches": > > > 182,"schemas":1}],"operatorId":9,"operatorType":2,"setupNanos":1753035," > > > processNanos":9140686073,"peakLocalMemoryAllocated": > > > 59965440,"waitNanos":0},{"inputProfile":[{"records": > > > 49978,"batches":181,"schemas":1}],"operatorId":7," > > > operatorType":14,"setupNanos":270460,"processNanos":729011104," > > > peakLocalMemoryAllocated":3346432,"waitNanos":0},{" > > > inputProfile":[{"records":107522,"batches":30,"schemas": > > > 1}],"operatorId":8,"operatorType":11,"setupNanos": > > > 0,"processNanos":671832,"peakLocalMemoryAllocated":0," > > > metric":[{"metricId":0,"longValue":0},{"metricId":1," > > > longValue":2}],"waitNanos":1746848211},{"inputProfile":[{ > > > "records":107522,"batches":30,"schemas":1}],"operatorId":6," > > > operatorType":10,"setupNanos":76821,"processNanos":237739," > > > peakLocalMemoryAllocated":262144,"waitNanos":0},{" > > > inputProfile":[{"records":49978,"batches":181,"schemas": > > > 1},{"records":107522,"batches":30,"schemas":1}],"operatorId" > > > :5,"operatorType":4,"setupNanos":0,"processNanos":98954835," > > > peakLocalMemoryAllocated":24862720,"metric":[{"metricId" > > > :0,"longValue":262144},{"metricId":3,"longValue":8},{" > > > metricId":2,"longValue":2},{"metricId":1,"longValue": > > > 107522}],"waitNanos":0},{"inputProfile":[{"records":0," > > > batches":1,"schemas":1}],"operatorId":4,"operatorType": > > > 10,"setupNanos":67901,"processNanos":19445,"peakLocalMemoryAllocated": > > > 319488,"waitNanos":0},{"inputProfile":[{"records":0," > > > batches":1,"schemas":1}],"operatorId":3,"operatorType": > > > 10,"setupNanos":14114279,"processNanos":114198," > peakLocalMemoryAllocated": > > > 532480,"waitNanos":0},{"inputProfile":[{"records":0," > > > batches":1,"schemas":1}],"operatorId":2,"operatorType": > > > 3,"setupNanos":169547,"processNanos":88422," > peakLocalMemoryAllocated":100, > > > "waitNanos":0},{"inputProfile":[{"records":0,"batches":1," > > > schemas":1}],"operatorId":1,"operatorType":10,"setupNanos": > > > 2801259,"processNanos":31428,"peakLocalMemoryAllocated":100, > > > "waitNanos":0},{"inputProfile":[{"records":0,"batches":0," > > > schemas":0}],"operatorId":0,"operatorType":0,"setupNanos": > > > 0,"processNanos":0,"peakLocalMemoryAllocated":0," > > > waitNanos":0},{"inputProfile":[{"records":0,"batches":1," > > > schemas":1}],"operatorId":0,"operatorType":0,"setupNanos": > > > 0,"processNanos":37696,"peakLocalMemoryAllocated":0," > > > metric":[{"metricId":0,"longValue":0}],"waitNanos":22253}],"startTime": > > > 1521006628336,"endTime":1521006721768,"memoryUsed": > > > 107174464,"maxMemoryUsed":156047360,"endpoint":{"address":" > > > prod-hadoop-xx2.bom-prod.com","userPort":31010," > > > controlPort":31011,"dataPort":31012,"version":"1.12.0"," > > > state":1},"lastUpdate":1521006721772,"lastProgress": > > > 1521006721772},{"state":3,"minorFragmentId":7,"operatorProfile":[{" > > > inputProfile":[{"records":5436947,"batches":178," > > > schemas":1}],"operatorId":10,"operatorType":21,"setupNanos": > > > 0,"processNanos":16076368516,"peakLocalMemoryAllocated": > > > 101089280,"metric":[{"metricId":0,"longValue":0},{" > > > metricId":16,"longValue":0},{"metricId":7,"longValue":0},{" > > > metricId":13,"longValue":0},{"metricId":5,"longValue":0},{" > > > metricId":9,"longValue":0},{"metricId":4,"longValue":7050}, > > > {"metricId":1,"longValue":7050},{"metricId":17," > > > longValue":59090},{"metricId":3,"longValue":0},{"metricId": > > > 14,"longValue":3532399855},{"metricId":19,"longValue": > > > 79685657618},{"metricId":8,"longValue":1390856453},{" > > > metricId":2,"longValue":7050},{"metricId":12,"longValue":0}, > > > {"metricId":10,"longValue":63782980256},{"metricId":11," > > > longValue":8093990},{"metricId":15,"longValue": > > > 63782980256},{"metricId":6,"longValue":0},{"metricId":18," > > > longValue":79685280894}],"waitNanos":63778234790},{" > > > inputProfile":[{"records":5436947,"batches":170," > > > schemas":1}],"operatorId":9,"operatorType":2,"setupNanos": > > > 1909071,"processNanos":9762620840,"peakLocalMemoryAllocated": > > > 59965440,"waitNanos":0},{"inputProfile":[{"records": > > > 49331,"batches":170,"schemas":1}],"operatorId":7," > > > operatorType":14,"setupNanos":387561,"processNanos":881571738," > > > peakLocalMemoryAllocated":1777664,"waitNanos":0},{" > > > inputProfile":[{"records":107522,"batches":30,"schemas": > > > 1}],"operatorId":8,"operatorType":11,"setupNanos": > > > 0,"processNanos":846055,"peakLocalMemoryAllocated":0," > > > metric":[{"metricId":0,"longValue":0},{"metricId":1," > > > longValue":2}],"waitNanos":2284223832},{"inputProfile":[{ > > > "records":107522,"batches":30,"schemas":1}],"operatorId":6," > > > operatorType":10,"setupNanos":93530,"processNanos":507491," > > > peakLocalMemoryAllocated":262144,"waitNanos":0},{" > > > inputProfile":[{"records":49331,"batches":170,"schemas": > > > 1},{"records":107522,"batches":30,"schemas":1}],"operatorId" > > > :5,"operatorType":4,"setupNanos":0,"processNanos":116026703," > > > peakLocalMemoryAllocated":23486464,"metric":[{"metricId" > > > :0,"longValue":262144},{"metricId":2,"longValue":2},{" > > > metricId":1,"longValue":107522},{"metricId":3," > > > longValue":7}],"waitNanos":0},{"inputProfile":[{"records": > > > 3471,"batches":2,"schemas":1}],"operatorId":4,"operatorType" > > > :10,"setupNanos":112653,"processNanos":69506,"peakLocalMemoryAllocated": > > > 6021120,"waitNanos":0},{"inputProfile":[{"records": > > > 3471,"batches":2,"schemas":1}],"operatorId":3,"operatorType" > > > :10,"setupNanos":13982159,"processNanos":58947356," > > > peakLocalMemoryAllocated":1024000,"waitNanos":0},{" > > > inputProfile":[{"records":3471,"batches":2,"schemas":1}] > > > ,"operatorId":2,"operatorType":3,"setupNanos":158821," > > > processNanos":61630100,"peakLocalMemoryAllocated": > > > 48627712,"metric":[{"metricId":0,"longValue":65536},{" > > > metricId":4,"longValue":1},{"metricId":7,"longValue":0},{" > > > metricId":6,"longValue":0},{"metricId":2,"longValue":0},{" > > > metricId":1,"longValue":3464},{"metricId":3,"longValue":0}], > > > "waitNanos":0},{"inputProfile":[{"records":3465,"batches":3, > > > "schemas":1}],"operatorId":1,"operatorType":10,"setupNanos": > > > 2899498,"processNanos":1936582,"peakLocalMemoryAllocated": > > > 47857664,"waitNanos":0},{"inputProfile":[{"records":0," > > > batches":0,"schemas":0}],"operatorId":0,"operatorType": > 0,"setupNanos":0," > > > processNanos":0,"peakLocalMemoryAllocated":47857664,"waitNanos":0},{" > > > inputProfile":[{"records":3465,"batches":3,"schemas":1}] > > > ,"operatorId":0,"operatorType":0,"setupNanos":0,"processNanos":209657," > > > peakLocalMemoryAllocated":47857664,"metric":[{"metricId" > > > :0,"longValue":727943}],"waitNanos":3824128}]," > startTime":1521006628337," > > > endTime":1521006721438,"memoryUsed":0,"maxMemoryUsed": > > > 156833792,"endpoint":{"address":"prod-hadoop-xx3.bom-prod.com > > > ","userPort":31010,"controlPort":31011,"dataPort": > > > 31012,"version":"1.12.0","state":1},"lastUpdate": > > > 1521006721441,"lastProgress":1521006721441},{"state":2," > > > minorFragmentId":13,"operatorProfile":[{"inputProfile":[{"records": > > > 4913490,"batches":159,"schemas":1}],"operatorId":10," > > > operatorType":21,"setupNanos":0,"processNanos":16161055409," > > > peakLocalMemoryAllocated":114720768,"metric":[{" > > > metricId":0,"longValue":0},{"metricId":3,"longValue":0},{" > > > metricId":7,"longValue":0},{"metricId":13,"longValue":0},{" > > > metricId":14,"longValue":3059465534},{"metricId":6," > > > longValue":0},{"metricId":10,"longValue":60896126241},{" > > > metricId":16,"longValue":0},{"metricId":17,"longValue": > > > 68704},{"metricId":19,"longValue":75593545859},{" > metricId":15,"longValue": > > > 60896126241},{"metricId":9,"longValue":0},{"metricId":5," > > > longValue":0},{"metricId":8,"longValue":1204159762},{" > > > metricId":18,"longValue":75593164897},{"metricId":1," > > > longValue":6151},{"metricId":4,"longValue":6151},{" > > > metricId":2,"longValue":6151},{"metricId":11,"longValue": > > > 7817449},{"metricId":12,"longValue":0}],"waitNanos": > > > 65849770288},{"inputProfile":[{"records":4913490,"batches": > > > 153,"schemas":1}],"operatorId":9,"operatorType":2,"setupNanos":1945569," > > > processNanos":7872627200,"peakLocalMemoryAllocated": > > > 59965440,"waitNanos":0},{"inputProfile":[{"records": > > > 42830,"batches":153,"schemas":1}],"operatorId":7," > > > operatorType":14,"setupNanos":362199,"processNanos":762036527," > > > peakLocalMemoryAllocated":1896448,"waitNanos":0},{" > > > inputProfile":[{"records":107522,"batches":30,"schemas": > > > 1}],"operatorId":8,"operatorType":11,"setupNanos": > > > 0,"processNanos":813849,"peakLocalMemoryAllocated":0," > > > metric":[{"metricId":0,"longValue":0},{"metricId":1," > > > longValue":2}],"waitNanos":398149836},{"inputProfile":[{" > > > records":107522,"batches":30,"schemas":1}],"operatorId":6," > > > operatorType":10,"setupNanos":91516,"processNanos":275118," > > > peakLocalMemoryAllocated":262144,"waitNanos":0},{" > > > inputProfile":[{"records":42830,"batches":153,"schemas": > > > 1},{"records":107522,"batches":30,"schemas":1}],"operatorId" > > > :5,"operatorType":4,"setupNanos":0,"processNanos":116159411," > > > peakLocalMemoryAllocated":22700032,"metric":[{"metricId" > > > :0,"longValue":262144},{"metricId":2,"longValue":2},{" > > > metricId":3,"longValue":14},{"metricId":1,"longValue": > > > 107522}],"waitNanos":0},{"inputProfile":[{"records":0," > > > batches":1,"schemas":1}],"operatorId":4,"operatorType": > > > 10,"setupNanos":72039,"processNanos":21450,"peakLocalMemoryAllocated": > > > 319488,"waitNanos":0},{"inputProfile":[{"records":0," > > > batches":1,"schemas":1}],"operatorId":3,"operatorType": > > > 10,"setupNanos":14467684,"processNanos":138210," > peakLocalMemoryAllocated": > > > 532480,"waitNanos":0},{"inputProfile":[{"records":0," > > > batches":1,"schemas":1}],"operatorId":2,"operatorType": > > > 3,"setupNanos":197149,"processNanos":90764," > peakLocalMemoryAllocated":100, > > > "waitNanos":0},{"inputProfile":[{"records":0,"batches":1," > > > schemas":1}],"operatorId":1,"operatorType":10,"setupNanos": > > > 2996530,"processNanos":37511,"peakLocalMemoryAllocated":100, > > > "waitNanos":0},{"inputProfile":[{"records":0,"batches":0," > > > schemas":0}],"operatorId":0,"operatorType":0,"setupNanos": > > > 0,"processNanos":0,"peakLocalMemoryAllocated":0," > > > waitNanos":0},{"inputProfile":[{"records":0,"batches":1," > > > schemas":1}],"operatorId":0,"operatorType":0,"setupNanos": > > > 0,"processNanos":42664,"peakLocalMemoryAllocated":0," > > > metric":[{"metricId":0,"longValue":0}],"waitNanos":26896}],"startTime": > > > 1521006628337,"endTime":1521006719596,"memoryUsed": > > > 75192896,"maxMemoryUsed":166860800,"endpoint":{"address":" > > > prod-hadoop-xx4.bom-prod.com","userPort":31010," > > > controlPort":31011,"dataPort":31012,"version":"1.12.0"," > > > state":1},"lastUpdate":1521006719599,"lastProgress": > > > 1521006719599},{"state":3,"minorFragmentId":20,"operatorProfile":[{" > > > inputProfile":[{"records":5758732,"batches":186," > > > schemas":1}],"operatorId":10,"operatorType":21,"setupNanos": > > > 0,"processNanos":14753064057,"peakLocalMemoryAllocated": > > > 102531072,"metric":[{"metricId":0,"longValue":0},{" > > > metricId":3,"longValue":0},{"metricId":17,"longValue": > > > 76240},{"metricId":1,"longValue":7442},{"metricId": > 2,"longValue":7442},{" > > > metricId":13,"longValue":0},{"metricId":16,"longValue":0},{" > > > metricId":6,"longValue":0},{"metricId":5,"longValue":0},{" > > > metricId":19,"longValue":76042440626},{"metricId":8," > > > longValue":1487463051},{"metricId":11,"longValue": > > > 10346359},{"metricId":18,"longValue":76041985564},{" > > > metricId":9,"longValue":0},{"metricId":4,"longValue":7442}, > > > {"metricId":14,"longValue":3688806289},{"metricId":15," > > > longValue":61377397640},{"metricId":10,"longValue": > > > 61377397640},{"metricId":7,"longValue":0},{"metricId":12," > > > longValue":0}],"waitNanos":61372362497},{"inputProfile":[ > > > {"records":5758732,"batches":179,"schemas":1}],"operatorId" > > > :9,"operatorType":2,"setupNanos":2345259,"processNanos":8343301523," > > > peakLocalMemoryAllocated":59965440,"waitNanos":0},{" > > > inputProfile":[{"records":47638,"batches":179,"schemas": > > > 1}],"operatorId":7,"operatorType":14,"setupNanos":330337,"processNanos": > > > 598639052,"peakLocalMemoryAllocated":1851392,"waitNanos":0},{" > > > inputProfile":[{"records":107522,"batches":30,"schemas": > > > 1}],"operatorId":8,"operatorType":11,"setupNanos": > > > 0,"processNanos":744028,"peakLocalMemoryAllocated":0," > > > metric":[{"metricId":0,"longValue":0},{"metricId":1," > > > longValue":2}],"waitNanos":2018401193},{"inputProfile":[{ > > > "records":107522,"batches":30,"schemas":1}],"operatorId":6," > > > operatorType":10,"setupNanos":87917,"processNanos":273785," > > > peakLocalMemoryAllocated":262144,"waitNanos":0},{" > > > inputProfile":[{"records":47638,"batches":179,"schemas": > > > 1},{"records":107522,"batches":30,"schemas":1}],"operatorId" > > > :5,"operatorType":4,"setupNanos":0,"processNanos":98771363," > > > peakLocalMemoryAllocated":23748608,"metric":[{"metricId" > > > :0,"longValue":262144},{"metricId":1,"longValue":107522},{"metricId":2," > > > longValue":2},{"metricId":3,"longValue":10}],"waitNanos":0} > > > ,{"inputProfile":[{"records":3380,"batches":2,"schemas":1}] > > > ,"operatorId":4,"operatorType":10,"setupNanos":70049," > > > processNanos":48246,"peakLocalMemoryAllocated":6021120,"waitNanos":0},{" > > > inputProfile":[{"records":3380,"batches":2,"schemas":1}] > > > ,"operatorId":3,"operatorType":10,"setupNanos":78700463," > > > processNanos":52297956,"peakLocalMemoryAllocated": > > > 1024000,"waitNanos":0},{"inputProfile":[{"records": > > > 3380,"batches":2,"schemas":1}],"operatorId":2,"operatorType" > > > :3,"setupNanos":185515,"processNanos":14679198," > peakLocalMemoryAllocated": > > > 48627712,"metric":[{"metricId":0,"longValue":65536},{" > > > metricId":6,"longValue":0},{"metricId":1,"longValue":3371}, > > > {"metricId":4,"longValue":1},{"metricId":3,"longValue":0},{" > > > metricId":2,"longValue":0},{"metricId":7,"longValue":0}]," > > > waitNanos":0},{"inputProfile":[{"records":3372,"batches":3," > > > schemas":1}],"operatorId":1,"operatorType":10,"setupNanos": > > > 2972145,"processNanos":1503086,"peakLocalMemoryAllocated": > > > 47857664,"waitNanos":0},{"inputProfile":[{"records":0," > > > batches":0,"schemas":0}],"operatorId":0,"operatorType": > 0,"setupNanos":0," > > > processNanos":0,"peakLocalMemoryAllocated":47857664,"waitNanos":0},{" > > > inputProfile":[{"records":3372,"batches":3,"schemas":1}] > > > ,"operatorId":0,"operatorType":0,"setupNanos":0,"processNanos":160236," > > > peakLocalMemoryAllocated":47857664,"metric":[{"metricId" > > > :0,"longValue":703268}],"waitNanos":1145163}]," > startTime":1521006628339," > > > endTime":1521006715738,"memoryUsed":0,"maxMemoryUsed": > > > 158472192,"endpoint":{"address":"prod-hadoop-xx5.bom-prod.com > > > ","userPort":31010,"controlPort":31011,"dataPort": > > > 31012,"version":"1.12.0","state":1},"lastUpdate": > > > 1521006715738,"lastProgress":1521006715738},{"state":3," > > > minorFragmentId":16,"operatorProfile":[{"inputProfile":[{"records": > > > 5745026,"batches":187,"schemas":1}],"operatorId":10," > > > operatorType":21,"setupNanos":0,"processNanos":14973716148," > > > peakLocalMemoryAllocated":101089280,"metric":[{" > > > metricId":0,"longValue":0},{"metricId":15,"longValue": > > > 58734209864},{"metricId":18,"longValue":73616025152},{" > > > metricId":19,"longValue":73616468495},{"metricId":9," > > > longValue":0},{"metricId":11,"longValue":10021869},{" > > > metricId":2,"longValue":7488},{"metricId":12,"longValue":0}, > > > {"metricId":16,"longValue":0},{"metricId":14,"longValue": > > > 3555357260},{"metricId":13,"longValue":0},{"metricId":6," > > > longValue":0},{"metricId":5,"longValue":0},{"metricId":10," > > > longValue":58734209864},{"metricId":17,"longValue": > 65222},{"metricId":4," > > > longValue":7488},{"metricId":7,"longValue":0},{"metricId": > > > 3,"longValue":0},{"metricId":1,"longValue":7488},{" > > > metricId":8,"longValue":1420991420}],"waitNanos": > > > 58729371283},{"inputProfile":[{"records":5745026,"batches": > > > 179,"schemas":1}],"operatorId":9,"operatorType":2,"setupNanos":2077524," > > > processNanos":8819158792,"peakLocalMemoryAllocated": > > > 59965440,"waitNanos":0},{"inputProfile":[{"records": > > > 49255,"batches":179,"schemas":1}],"operatorId":7," > > > operatorType":14,"setupNanos":322279,"processNanos":288516387," > > > peakLocalMemoryAllocated":1851392,"waitNanos":0},{" > > > inputProfile":[{"records":107522,"batches":30,"schemas": > > > 1}],"operatorId":8,"operatorType":11,"setupNanos": > > > 0,"processNanos":663778,"peakLocalMemoryAllocated":0," > > > metric":[{"metricId":0,"longValue":0},{"metricId":1," > > > longValue":2}],"waitNanos":1540398648},{"inputProfile":[{ > > > "records":107522,"batches":30,"schemas":1}],"operatorId":6," > > > operatorType":10,"setupNanos":96144,"processNanos":141315," > > > peakLocalMemoryAllocated":262144,"waitNanos":0},{" > > > inputProfile":[{"records":49255,"batches":179,"schemas": > > > 1},{"records":107522,"batches":30,"schemas":1}],"operatorId" > > > :5,"operatorType":4,"setupNanos":0,"processNanos":99997088," > > > peakLocalMemoryAllocated":23224320,"metric":[{"metricId" > > > :0,"longValue":262144},{"metricId":3,"longValue":10},{" > > > metricId":2,"longValue":2},{"metricId":1,"longValue": > > > 107522}],"waitNanos":0},{"inputProfile":[{"records": > > > 3555,"batches":2,"schemas":1}],"operatorId":4,"operatorType" > > > :10,"setupNanos":67416,"processNanos":58380,"peakLocalMemoryAllocated": > > > 6021120,"waitNanos":0},{"inputProfile":[{"records": > > > 3555,"batches":2,"schemas":1}],"operatorId":3,"operatorType" > > > :10,"setupNanos":14020547,"processNanos":86441801," > > > peakLocalMemoryAllocated":1056768,"waitNanos":0},{" > > > inputProfile":[{"records":3555,"batches":2,"schemas":1}] > > > ,"operatorId":2,"operatorType":3,"setupNanos":168666," > > > processNanos":15267452,"peakLocalMemoryAllocated": > > > 48627712,"metric":[{"metricId":0,"longValue":65536},{" > > > metricId":6,"longValue":0},{"metricId":2,"longValue":0},{" > > > metricId":3,"longValue":0},{"metricId":1,"longValue":3546}, > > > {"metricId":7,"longValue":0},{"metricId":4,"longValue":1}]," > > > waitNanos":0},{"inputProfile":[{"records":3547,"batches":3," > > > schemas":1}],"operatorId":1,"operatorType":10,"setupNanos": > > > 2823840,"processNanos":1584526,"peakLocalMemoryAllocated": > > > 47857664,"waitNanos":0},{"inputProfile":[{"records":0," > > > batches":0,"schemas":0}],"operatorId":0,"operatorType": > 0,"setupNanos":0," > > > processNanos":0,"peakLocalMemoryAllocated":47857664,"waitNanos":0},{" > > > inputProfile":[{"records":3547,"batches":3,"schemas":1}] > > > ,"operatorId":0,"operatorType":0,"setupNanos":0,"processNanos":162364," > > > peakLocalMemoryAllocated":47857664,"metric":[{"metricId" > > > :0,"longValue":743221}],"waitNanos":1594361}]," > startTime":1521006628339," > > > endTime":1521006712979,"memoryUsed":0,"maxMemoryUsed": > > > 156571648,"endpoint":{"address":"prod-hadoop-xx5.bom-prod.com > > > ","userPort":31010,"controlPort":31011,"dataPort": > > > 31012,"version":"1.12.0","state":1},"lastUpdate": > > > 1521006712979,"lastProgress":1521006712979},{"state":2," > > > minorFragmentId":28,"operatorProfile":[{"inputProfile":[{"records": > > > 3541038,"batches":113,"schemas":1}],"operatorId":10," > > > operatorType":21,"setupNanos":0,"processNanos":26134913852," > > > peakLocalMemoryAllocated":102531072,"metric":[{" > > > metricId":0,"longValue":0},{"metricId":17,"longValue": > > > 37365},{"metricId":5,"longValue":0},{"metricId":15," > > > longValue":47192448048},{"metricId":2,"longValue":4250}, > > > {"metricId":13,"longValue":0},{"metricId":12,"longValue":0}, > > > {"metricId":19,"longValue":71047212271},{"metricId":8," > > > longValue":852912272},{"metricId":1,"longValue":4250}, > > > {"metricId":11,"longValue":4648324},{"metricId":4," > > > longValue":4250},{"metricId":3,"longValue":0},{"metricId": > > > 16,"longValue":0},{"metricId":6,"longValue":0},{"metricId": > > > 18,"longValue":71046942098},{"metricId":9,"longValue":0},{" > > > metricId":10,"longValue":47192448048},{"metricId":7," > > > longValue":0},{"metricId":14,"longValue":2246289869}]," > > > waitNanos":54214989547},{"inputProfile":[{"records": > > > 3541038,"batches":109,"schemas":1}],"operatorId":9," > > > operatorType":2,"setupNanos":1397588,"processNanos":6083637353," > > > peakLocalMemoryAllocated":59965440,"waitNanos":0},{" > > > inputProfile":[{"records":30993,"batches":109,"schemas": > > > 1}],"operatorId":7,"operatorType":14,"setupNanos":281141,"processNanos": > > > 2792123565,"peakLocalMemoryAllocated":1765376,"waitNanos":0},{" > > > inputProfile":[{"records":107522,"batches":30,"schemas": > > > 1}],"operatorId":8,"operatorType":11,"setupNanos": > > > 0,"processNanos":725054,"peakLocalMemoryAllocated":0," > > > metric":[{"metricId":0,"longValue":0},{"metricId":1," > > > longValue":2}],"waitNanos":872268018},{"inputProfile":[{" > > > records":107522,"batches":30,"schemas":1}],"operatorId":6," > > > operatorType":10,"setupNanos":73730,"processNanos":146704," > > > peakLocalMemoryAllocated":262144,"waitNanos":0},{" > > > inputProfile":[{"records":30572,"batches":108,"schemas": > > > 1},{"records":107522,"batches":30,"schemas":1}],"operatorId" > > > :5,"operatorType":4,"setupNanos":0,"processNanos":90412977," > > > peakLocalMemoryAllocated":23224320,"metric":[{"metricId" > > > :0,"longValue":262144},{"metricId":1,"longValue":107522},{"metricId":2," > > > longValue":2},{"metricId":3,"longValue":9}],"waitNanos":0}, > > > {"inputProfile":[{"records":0,"batches":1,"schemas":1}]," > > > operatorId":4,"operatorType":10,"setupNanos":66142," > processNanos":20082," > > > peakLocalMemoryAllocated":319488,"waitNanos":0},{" > > > inputProfile":[{"records":0,"batches":1,"schemas":1}]," > > > operatorId":3,"operatorType":10,"setupNanos":14426188," > > > processNanos":161709,"peakLocalMemoryAllocated":532480,"waitNanos":0},{" > > > inputProfile":[{"records":0,"batches":1,"schemas":1}]," > > > operatorId":2,"operatorType":3,"setupNanos":143054," > processNanos":87937," > > > peakLocalMemoryAllocated":100,"waitNanos":0},{"inputProfile" > > > :[{"records":0,"batches":1,"schemas":1}],"operatorId":1," > > > operatorType":10,"setupNanos":2951276,"processNanos":33516," > > > peakLocalMemoryAllocated":100,"waitNanos":0},{"inputProfile" > > > :[{"records":0,"batches":0,"schemas":0}],"operatorId":0," > > > operatorType":0,"setupNanos":0,"processNanos":0," > > > peakLocalMemoryAllocated":0,"waitNanos":0},{"inputProfile": > > > [{"records":0,"batches":1,"schemas":1}],"operatorId":0," > > > operatorType":0,"setupNanos":0,"processNanos":34621," > > > peakLocalMemoryAllocated":0,"metric":[{"metricId":0," > > > longValue":0}],"waitNanos":16681}],"startTime":1521006628338,"endTime": > > > 1521006718677,"memoryUsed":44556288,"maxMemoryUsed": > > > 155719680,"endpoint":{"address":"prod-hadoop-xx1.bom-prod.com > > > ","userPort":31010,"controlPort":31011,"dataPort": > > > 31012,"version":"1.12.0","state":1},"lastUpdate": > > > 1521006718677,"lastProgress":1521006718677},{"state":2," > > > minorFragmentId":15,"operatorProfile":[{"inputProfile":[{"records": > > > 5592132,"batches":180,"schemas":1}],"operatorId":10," > > > operatorType":21,"setupNanos":0,"processNanos":15256777986," > > > peakLocalMemoryAllocated":102400000,"metric":[{" > > > metricId":0,"longValue":0},{"metricId":4,"longValue":6549}, > > > {"metricId":16,"longValue":0},{"metricId":10,"longValue": > > > 60264671479},{"metricId":1,"longValue":6549},{"metricId": > > > 19,"longValue":72560534277},{"metricId":3,"longValue":0},{" > > > metricId":5,"longValue":0},{"metricId":13,"longValue":0},{" > > > metricId":12,"longValue":0},{"metricId":9,"longValue":0},{" > > > metricId":8,"longValue":1271594981},{"metricId":15," > > > longValue":60264671479},{"metricId":14,"longValue": > > > 3204221039},{"metricId":17,"longValue":54496},{"metricId": > > > 2,"longValue":6549},{"metricId":6,"longValue":0},{" > > > metricId":7,"longValue":0},{"metricId":11,"longValue": > > > 7452940},{"metricId":18,"longValue":72560207199}]," > > > waitNanos":64273784113},{"inputProfile":[{"records": > > > 5592132,"batches":174,"schemas":1}],"operatorId":9," > > > operatorType":2,"setupNanos":2138431,"processNanos":9302016956," > > > peakLocalMemoryAllocated":59965440,"waitNanos":0},{" > > > inputProfile":[{"records":45618,"batches":174,"schemas": > > > 1}],"operatorId":7,"operatorType":14,"setupNanos":388783,"processNanos": > > > 811697156,"peakLocalMemoryAllocated":1851392,"waitNanos":0},{" > > > inputProfile":[{"records":107522,"batches":30,"schemas": > > > 1}],"operatorId >
