Thanks Jagadish for replying. Yes I am treating it as a opaque durable I'd to execute some code on a container specific container.
Just for my understanding can you please provide a bit more details or point me to the code how samza is mapping yarn container id to durable sequential integers? On Sat, 24 Apr 2021, 01:44 Jagadish Venkatraman, <jagadish1...@gmail.com> wrote: > Hi Debraj, > > Clarifying a bit on Yi’s response, since it was referring to the physical > Yarn container id.. > > If there are N Yarn containers, samza.container.ids are generated > sequentially from 0 to N-1. This ID is meant to be durable - ie., if a > particular container fails, the Samza AM will restart it with the same ID. > > Having said that, you should just treat it as an opaque key that uniquely > identifies a container within a Samza job. > > Could you share some details on how you intend to use this? > > On Friday, April 23, 2021, Debraj Manna <subharaj.ma...@gmail.com> wrote: > > > Thanks, Yi for replying. > > > > I also checked that class. But Container#getId().toString() > > returna s string like container_e02_1619095810959_0006_10_000004. But I > am > > seeing samza.container.id is an integer like 0, 1 that is getting set as > > system var. Can you let me know how Container#getId().toString() is > getting > > mapped to an integer? > > > > For example, below is the output of ps -ef for samza yarn container and I > > am seeing *-Dsamza.container.id <http://Dsamza.container.id>=2* & > > *-Dsamza.container.name > > <http://Dsamza.container.name>=samza-container-2* > > > > yarn 7706 7704 7 Apr22 ? 02:08:23 > > /usr/lib/jvm/zulu-11-amd64/bin/java -Xmx8820M > > -XX:-OmitStackTraceInFastThrow -XX:NewRatio=8 -Xss256K > > -XX:+ExitOnOutOfMemoryError -XX:+HeapDumpOnOutOfMemoryError > > -XX:HeapDumpPath=/var/lib/heap-dumps/samzajobs > > -XX:NativeMemoryTracking=summary -Dio.netty.allocator.type=unpooled > > -Dio.grpc.netty.shaded.io.netty.allocator.type=unpooled -server > > *-Dsamza.container.id > > <http://Dsamza.container.id>=2 -Dsamza.container.name > > <http://Dsamza.container.name>=samza-container-2* > > -DisThreadContextMapInheritable=true > > -Dlog4j.configuration=file:/var/lib/hadoop-yarn/cache/ > > yarn/nm-local-dir/usercache/ubuntu/appcache/application_ > > 1619095810959_0006/container_e02_1619095810959_0006_10_ > > 000004/__package/lib/log4j.xml > > > -Dsamza.log.dir=/var/log/hadoop-yarn/containers/application_1619095810959_ > > 0006/container_e02_1619095810959_0006_10_000004 > > -Djava.io.tmpdir=/var/lib/hadoop-yarn/cache/yarn/nm- > > local-dir/usercache/ubuntu/appcache/application_ > > 1619095810959_0006/container_e02_1619095810959_0006_10_ > > 000004/__package/tmp > > -cp > > /etc/hadoop/conf::/var/lib/hadoop-yarn/cache/yarn/nm- > > local-dir/usercache/ubuntu/appcache/application_ > > 1619095810959_0006/container_e02_1619095810959_0006_10_ > > 000004/__package/lib/activation-1.1.jar:/var/lib/ > > hadoop-yarn/cache/yarn/nm-local-dir/usercache/ubuntu/appcache/ > > > > > > On Fri, Apr 23, 2021 at 11:09 AM Yi Pan <nickpa...@gmail.com> wrote: > > > > > Hi, Debraj, > > > > > > In YARN environment, Samza uses YARN generated containerIds as > > > environmental variables to set each container process's > > samza.container.id > > > . > > > i.e. when containers are requested by Samza AM process in YARN, YARN RM > > > will reply with a set of allocated container objects, which is of class > > > org.apache.hadoop.yarn.api.records.Container. That's the resource class > > to > > > uniquely identify a container in YARN and Container#getId().toString() > is > > > the container ID string we set to samza.container.id. > > > > > > Best, > > > > > > -Yi > > > > > > On Wed, Apr 21, 2021 at 11:28 PM Debraj Manna < > subharaj.ma...@gmail.com> > > > wrote: > > > > > > > The same has been asked in stackoverflow > > > > < > > > > > > > https://stackoverflow.com/questions/67207850/how-does- > > samza-generate-the-container-id-when-the-application-is-deployed-in-yar > > > > > > > > > also. Anyone any thoughts on this? > > > > > > > > > > > > > > > https://stackoverflow.com/questions/67207850/how-does- > > samza-generate-the-container-id-when-the-application-is-deployed-in-yar > > > > > > > > On Wed, Apr 21, 2021 at 6:08 PM Debraj Manna < > subharaj.ma...@gmail.com > > > > > > > wrote: > > > > > > > > > Hi > > > > > > > > > > Can someone let me know how is "samza.container.id" generated > when a > > > > > samza app is running in yarn? > > > > > > > > > > Thanks, > > > > > > > > > > > > > > > > > > > > > > -- > Jagadish >