Here is the exact resource_manager_pool_status /tmp/resource_manager_respool_status HOST_ID(id=0:hostname:segment06.gcp.test-domain.com) HOST_INFO(FTSTotalMemoryMB=24576:FTSTotalCore=8:GRMTotalMemoryMB=98304:GRMTotalCore=12) HOST_AVAILABLITY(HAWQAvailable=true)
HOST_RESOURCE(AllocatedMemory=0:AllocatedCores=0.000000:AvailableMemory=0:AvailableCores=0.000000:IOBytesWorkload=0:SliceWorkload=0:LastUpdateTime=21595756349163:RUAlivePending=false) HOST_RESOURCE_CONTAINERSET(ratio=8192:AllocatedMemory=0:AvailableMemory=0:AllocatedCore=0.000000:AvailableCore:0.000000) HOST_ID(id=1:hostname:segment11.gcp.test-domain.com) HOST_INFO(FTSTotalMemoryMB=24576:FTSTotalCore=8:GRMTotalMemoryMB=98304:GRMTotalCore=12) HOST_AVAILABLITY(HAWQAvailable=true) HOST_RESOURCE(AllocatedMemory=0:AllocatedCores=0.000000:AvailableMemory=0:AvailableCores=0.000000:IOBytesWorkload=0:SliceWorkload=0:LastUpdateTime=21595756544067:RUAlivePending=false) HOST_RESOURCE_CONTAINERSET(ratio=8192:AllocatedMemory=0:AvailableMemory=0:AllocatedCore=0.000000:AvailableCore:0.000000) HOST_ID(id=2:hostname:segment12.gcp.test-domain.com) HOST_INFO(FTSTotalMemoryMB=24576:FTSTotalCore=8:GRMTotalMemoryMB=98304:GRMTotalCore=12) HOST_AVAILABLITY(HAWQAvailable=true) HOST_RESOURCE(AllocatedMemory=0:AllocatedCores=0.000000:AvailableMemory=0:AvailableCores=0.000000:IOBytesWorkload=0:SliceWorkload=0:LastUpdateTime=21595756749623:RUAlivePending=false) HOST_RESOURCE_CONTAINERSET(ratio=8192:AllocatedMemory=0:AvailableMemory=0:AllocatedCore=0.000000:AvailableCore:0.000000) HOST_ID(id=3:hostname:segment05.gcp.test-domain.com) HOST_INFO(FTSTotalMemoryMB=24576:FTSTotalCore=8:GRMTotalMemoryMB=98304:GRMTotalCore=12) HOST_AVAILABLITY(HAWQAvailable=true) HOST_RESOURCE(AllocatedMemory=0:AllocatedCores=0.000000:AvailableMemory=0:AvailableCores=0.000000:IOBytesWorkload=0:SliceWorkload=0:LastUpdateTime=21595756246999:RUAlivePending=false) HOST_RESOURCE_CONTAINERSET(ratio=8192:AllocatedMemory=0:AvailableMemory=0:AllocatedCore=0.000000:AvailableCore:0.000000) HOST_ID(id=4:hostname:segment10.gcp.test-domain.com) HOST_INFO(FTSTotalMemoryMB=24576:FTSTotalCore=8:GRMTotalMemoryMB=98304:GRMTotalCore=12) HOST_AVAILABLITY(HAWQAvailable=true) HOST_RESOURCE(AllocatedMemory=0:AllocatedCores=0.000000:AvailableMemory=0:AvailableCores=0.000000:IOBytesWorkload=0:SliceWorkload=0:LastUpdateTime=21595757552955:RUAlivePending=false) HOST_RESOURCE_CONTAINERSET(ratio=8192:AllocatedMemory=0:AvailableMemory=0:AllocatedCore=0.000000:AvailableCore:0.000000) HOST_ID(id=5:hostname:segment03.gcp.test-domain.com) HOST_INFO(FTSTotalMemoryMB=24576:FTSTotalCore=8:GRMTotalMemoryMB=98304:GRMTotalCore=12) HOST_AVAILABLITY(HAWQAvailable=true) HOST_RESOURCE(AllocatedMemory=0:AllocatedCores=0.000000:AvailableMemory=0:AvailableCores=0.000000:IOBytesWorkload=0:SliceWorkload=0:LastUpdateTime=21595758356042:RUAlivePending=false) HOST_RESOURCE_CONTAINERSET(ratio=8192:AllocatedMemory=0:AvailableMemory=0:AllocatedCore=0.000000:AvailableCore:0.000000) HOST_ID(id=6:hostname:segment09.gcp.test-domain.com) HOST_INFO(FTSTotalMemoryMB=24576:FTSTotalCore=8:GRMTotalMemoryMB=98304:GRMTotalCore=12) HOST_AVAILABLITY(HAWQAvailable=true) HOST_RESOURCE(AllocatedMemory=0:AllocatedCores=0.000000:AvailableMemory=0:AvailableCores=0.000000:IOBytesWorkload=0:SliceWorkload=0:LastUpdateTime=21595759258665:RUAlivePending=false) HOST_RESOURCE_CONTAINERSET(ratio=8192:AllocatedMemory=0:AvailableMemory=0:AllocatedCore=0.000000:AvailableCore:0.000000) HOST_ID(id=7:hostname:segment04.gcp.test-domain.com) HOST_INFO(FTSTotalMemoryMB=24576:FTSTotalCore=8:GRMTotalMemoryMB=98304:GRMTotalCore=12) HOST_AVAILABLITY(HAWQAvailable=true) HOST_RESOURCE(AllocatedMemory=0:AllocatedCores=0.000000:AvailableMemory=0:AvailableCores=0.000000:IOBytesWorkload=0:SliceWorkload=0:LastUpdateTime=21595754750573:RUAlivePending=false) HOST_RESOURCE_CONTAINERSET(ratio=8192:AllocatedMemory=0:AvailableMemory=0:AllocatedCore=0.000000:AvailableCore:0.000000) HOST_ID(id=8:hostname:segment02.gcp.test-domain.com) HOST_INFO(FTSTotalMemoryMB=24576:FTSTotalCore=8:GRMTotalMemoryMB=98304:GRMTotalCore=12) HOST_AVAILABLITY(HAWQAvailable=true) HOST_RESOURCE(AllocatedMemory=0:AllocatedCores=0.000000:AvailableMemory=0:AvailableCores=0.000000:IOBytesWorkload=0:SliceWorkload=0:LastUpdateTime=21595758451056:RUAlivePending=false) HOST_RESOURCE_CONTAINERSET(ratio=8192:AllocatedMemory=0:AvailableMemory=0:AllocatedCore=0.000000:AvailableCore:0.000000) HOST_ID(id=9:hostname:segment13.gcp.test-domain.com) HOST_INFO(FTSTotalMemoryMB=24576:FTSTotalCore=8:GRMTotalMemoryMB=98304:GRMTotalCore=12) HOST_AVAILABLITY(HAWQAvailable=true) HOST_RESOURCE(AllocatedMemory=0:AllocatedCores=0.000000:AvailableMemory=0:AvailableCores=0.000000:IOBytesWorkload=0:SliceWorkload=0:LastUpdateTime=21595758958073:RUAlivePending=false) HOST_RESOURCE_CONTAINERSET(ratio=8192:AllocatedMemory=0:AvailableMemory=0:AllocatedCore=0.000000:AvailableCore:0.000000) HOST_ID(id=10:hostname:segment08.gcp.test-domain.com) HOST_INFO(FTSTotalMemoryMB=24576:FTSTotalCore=8:GRMTotalMemoryMB=98304:GRMTotalCore=12) HOST_AVAILABLITY(HAWQAvailable=true) HOST_RESOURCE(AllocatedMemory=0:AllocatedCores=0.000000:AvailableMemory=0:AvailableCores=0.000000:IOBytesWorkload=0:SliceWorkload=0:LastUpdateTime=21595759458370:RUAlivePending=false) HOST_RESOURCE_CONTAINERSET(ratio=8192:AllocatedMemory=0:AvailableMemory=0:AllocatedCore=0.000000:AvailableCore:0.000000) HOST_ID(id=11:hostname:segment07.gcp.test-domain.com) HOST_INFO(FTSTotalMemoryMB=24576:FTSTotalCore=8:GRMTotalMemoryMB=98304:GRMTotalCore=12) HOST_AVAILABLITY(HAWQAvailable=true) HOST_RESOURCE(AllocatedMemory=0:AllocatedCores=0.000000:AvailableMemory=0:AvailableCores=0.000000:IOBytesWorkload=0:SliceWorkload=0:LastUpdateTime=21595741128453:RUAlivePending=false) HOST_RESOURCE_CONTAINERSET(ratio=8192:AllocatedMemory=0:AvailableMemory=0:AllocatedCore=0.000000:AvailableCore:0.000000) HOST_ID(id=12:hostname:segment01.gcp.test-domain.com) HOST_INFO(FTSTotalMemoryMB=24576:FTSTotalCore=8:GRMTotalMemoryMB=98304:GRMTotalCore=12) HOST_AVAILABLITY(HAWQAvailable=true) HOST_RESOURCE(AllocatedMemory=0:AllocatedCores=0.000000:AvailableMemory=0:AvailableCores=0.000000:IOBytesWorkload=0:SliceWorkload=0:LastUpdateTime=21595737023537:RUAlivePending=false) HOST_RESOURCE_CONTAINERSET(ratio=8192:AllocatedMemory=0:AvailableMemory=0:AllocatedCore=0.000000:AvailableCore:0.000000) HOST_ID(id=13:hostname:segment14.gcp.test-domain.com) HOST_INFO(FTSTotalMemoryMB=24576:FTSTotalCore=8:GRMTotalMemoryMB=98304:GRMTotalCore=12) HOST_AVAILABLITY(HAWQAvailable=true) HOST_RESOURCE(AllocatedMemory=0:AllocatedCores=0.000000:AvailableMemory=0:AvailableCores=0.000000:IOBytesWorkload=0:SliceWorkload=0:LastUpdateTime=21595735919485:RUAlivePending=false) HOST_RESOURCE_CONTAINERSET(ratio=8192:AllocatedMemory=0:AvailableMemory=0:AllocatedCore=0.000000:AvailableCore:0.000000) On Mon, Nov 13, 2017 at 8:10 PM, Shubham Sharma <[email protected]> wrote: Hi Wen, > > I thought the same but when I look at - resource_manager_respool_status > using - SELECT * FROM dump_resource_manager_status(2); All segments are > registered and each has configuration 98304MB and 12.000000 CORE > > On Mon, Nov 13, 2017 at 7:40 PM, Wen Lin <[email protected]> wrote: > >> Hi, Shubham, >> >> The second message means that, this YARN cluster's total resource is >> 1376256 >> MB, 168.000000 CORE. >> And HAWQ clusters have the resource: 98304MB, 12.000000 CORE of the whole >> YARN cluster now. >> The other resources in YARN haven't been allocated for HAWQ. >> Maybe there are no HAWQ segments running on these segments at the moment, >> or other reasons. >> >> Thanks! >> >> >> On Tue, Nov 14, 2017 at 9:25 AM, Shubham Sharma <[email protected]> >> wrote: >> >> > While YARN mode is enabled, resource broker logs two things - >> > >> > - YARN cluster total resource >> > - HAWQ's total resource per node. >> > >> > Following messages are logged >> > >> > 2017-11-11 23:21:40.944904 >> > UTC,,,p549330,th900077856,,,,0,con4,,seg-10000,,,,,"LOG","00 >> 000","Resource >> > manager YARN resource broker counted YARN cluster having total >> > resource (1376256 MB, 168.000000 >> > CORE).",,,,,,,0,,"resourcebroker_LIBYARN.c",776, >> > >> > 2017-11-11 23:21:40.944921 >> > UTC,,,p549330,th900077856,,,,0,con4,,seg-10000,,,,,"LOG","00 >> 000","Resource >> > manager YARN resource broker counted HAWQ cluster now having (98304 >> > MB, 12.000000 CORE) in a YARN cluster of total resource (1376256 MB, >> > 168.000000 CORE).",,,,,,,0,,"resourcebroker_LIBYARN.c",785, >> > >> > The second message shown above is ambiguous, After reading the sentence >> > below it looks like that complete Hawq cluster in whole has only 98304 >> MB >> > and 12 cores. However according to the configuration it should be 98304 >> MB >> > and 12 cores per segment server. >> > >> > Resource manager YARN resource broker counted HAWQ cluster now having >> > (98304 MB, 12.000000 CORE) in a YARN cluster of total resource >> > (1376256 MB, 168.000000 CORE). >> > >> > Either the wrong variables are printed or we can correct the message to >> > represent that the resources logged are per node. As this can confuse >> the >> > user into thinking that hawq cluster does not have enough resources. >> > >> > Is this understanding correct ? >> > >> > Regards, >> > Shubham Sharma >> > >> > > > > -- > Regards, > Shubham Sharma > Staff Customer Engineer > Pivotal Global Support Services > [email protected] > Direct Tel: +1(510)-304-8201 <(510)%20304-8201> > Office Hours: Mon-Fri 9:00 am to 5:00 pm PDT > Out of Office Hours Contact +1 877-477-2269 <(877)%20477-2269> > -- Regards, Shubham Sharma Staff Customer Engineer Pivotal Global Support Services [email protected] Direct Tel: +1(510)-304-8201 <(510)%20304-8201> Office Hours: Mon-Fri 9:00 am to 5:00 pm PDT Out of Office Hours Contact +1 877-477-2269 <(877)%20477-2269>
