You have the correct understanding about the "yarn.resources.*" configuration, and your question is a good one. Currently, the implementation is that Samza will look in a specific place on the file system (i.e. <current working directory>/__samzaFrameworkApi and <current working directory>/__samzaFrameworkInfrastructure) to get the API/infrastructure classpaths. I have a TODO in the code to make the file system location configurable (or specified through an environment variable). The configuration or environment variable for the file system location would not be YARN-specific, and it would be applicable to any execution environment.
On Wed, Mar 11, 2020 at 10:54 PM Yi Pan <nickpa...@gmail.com> wrote: > OK. If I understand correctly, your answer is the following: > yarn.resources.* configuration variables are used by YARN localizer to make > API and infrastructure classpath available, together with the application's > own classpath, which is also determined by the YARN localizer. > The question here is: how do we let the container JVM know the > API/infrastructure classpaths when launching the container processes? If > the API and infrastructure classpaths (i.e. installation path determined by > the localizer) are customizable, then we would need to tell the container > JVM those API/infra classpaths via some configuration variables as well, > right? Hence, those configuration variable names need to be understood by > the Samza application's code (which is run within the container) as well. > If not, what's the mechanism that we will use to let the container JVM > process to know where the YARN localizer has put API/infra classpaths? > > Thanks! > > -Yi > > > > On Wed, Mar 11, 2020 at 8:09 PM Cameron Lee <cameronlee...@gmail.com> > wrote: > > > The configuration variables are only used by the YARN localizer. The > Samza > > application will look for the framework resources in certain places in > the > > application's working directory when it needs to access them. My aim is > to > > do something similar to how "yarn.package.path" works. In other execution > > environments, it is my understanding that "yarn.package.path" would get > > replaced by a different environment-specific configuration key/value. > > I agree that we should not use "yarn.resources.*" if the configurations > are > > not YARN-specific. Do you think that these resource localization configs > > are generalizable to arbitrary environments? If so, does that mean > > "yarn.package.path" is also generalizable? For example, what if some > > execution environment does not use URLs to specify resource locations > > (although maybe this isn't a reasonable concern to worry about?)? > > > > Thanks, > > Cameron > > > > On Wed, Mar 11, 2020 at 4:43 PM Yi Pan <nickpa...@gmail.com> wrote: > > > > > Hi, Cameron, > > > > > > Thanks for the quick responses! Appreciate it. > > > > > > I am still having a concern on a): are those configuration variables > used > > > by YARN localizer or by Samza applications? If those are used only by > the > > > YARN localizer, I agree that we should keep those as yarn specific. > > > Otherwise, I think that would still be better to name those as > > > cluster.based.resources.*. The reason being: Samza applications are > > > supposed to be able to run on different execution environments. > Ideally, > > > when we are deploying the same Samza application on YARN vs Mesos or > > > managed K8s clusters, we should only need to change the configure > values, > > > not the configuration variable names and values. Does it make sense? > > > Otherwise, we can schedule a conf call to clarify that. > > > > > > Thanks! > > > > > > -Yi > > > > > > On Tue, Mar 10, 2020 at 3:25 PM Cameron Lee <cameronlee...@gmail.com> > > > wrote: > > > > > > > a) The "yarn.resources.*" configs are for localizing the necessary > > > > resources into the working directory for the process. I felt that the > > > > specific configuration format to specify these resources might be > > > > YARN-specific (e.g. YARN has type and visibility configs for each of > > its > > > > resources), so a generic format might not apply. In a non-YARN case, > > the > > > > localization configs would need to be specified according to the > > > technology > > > > being used. > > > > b) It is correct that the Avro version will need to be compatible > with > > > the > > > > version that is used by the infrastructure, if infrastructure needs > to > > > use > > > > Avro and pass the Avro object to the application. This is the case > with > > > any > > > > serde technology that needs to be used. For the job coordinator, it > is > > > not > > > > much of a concern anyways, since it is not doing serde of Avro > > messages. > > > > This may be more of a concern for general split deployment, which > will > > > > impact the processing containers, and will be a separate SEP. > > > > c) It should work to leave infrastructure serdes in the > infrastructure > > > > classpath. The infrastructure serdes just see generic types (which > are > > > > java.lang.Object at runtime) for the messages, and they don't do > > anything > > > > with the concrete types, so in the infrastructure classes, the > messages > > > get > > > > passed around as Object, but their concrete classes can still be > loaded > > > > from the application. As with (b), this is more of a concern for > > general > > > > split deployment, since the job coordinator doesn't do message > serde. I > > > > have run some tests regarding this classloading pattern, but we will > do > > > > further verification for general split deployment. > > > > d) Yes, you are correct. Good catch. It should be "described above at > > > > Application classloader". > > > > > > > > Thanks for all of your questions. I will clarify some details in the > > doc > > > > regarding your questions. > > > > > > > > Cameron > > > > > > > > On Mon, Mar 9, 2020 at 12:07 PM Yi Pan <nickpa...@gmail.com> wrote: > > > > > > > > > Hi, Cameron, > > > > > > > > > > Sorry to chime in late. Overall, looks great! I do have a few > > > > > suggestions/questions before I can cast my vote here: > > > > > a) for the configuration variable names, why are we limiting > > ourselves > > > to > > > > > yarn.resource.*? We have changed some of the configuration > variables > > > from > > > > > yarn specific to non-yarn specific. I would love to keep that > > > consistent > > > > > (i.e. gradually moving all our yarn-specific configuration > variables > > to > > > > > non-yarn-specifc names) > > > > > b) for the avro case as referred to in the delegation case in the > > > > > Infrastructure classloader, if we delegate the object > deserialization > > > > class > > > > > to the application classloader, would it be possible that the > > > application > > > > > provides an non-compatible version of avro class than the ones used > > > > within > > > > > the "infrastructure plugins" and hence causing runtime exception in > > the > > > > > infrastructure plugin? Or is the solution being: do not directly > use > > > > serde > > > > > classes in the infrastructure code? > > > > > c) following the description of infrastructure classloader flow, > > where > > > > > should we expect the serde classes? In the application classpath, I > > > > guess? > > > > > So, does that mean that we should exclude serde classes (including > > > > > SerializableSerde and JsonSerdeV2) in the Samza infrastructure > > package, > > > > and > > > > > tell the users to package them in application package? > > > > > d) I am a bit confused about the description on "multiple" > > application > > > > > classloaders on the job coordinator: one is for the describe flow > and > > > the > > > > > other is in the "Application" classloader, instead of "API" > > > classloader, > > > > > right? > > > > > > > > > > Best, > > > > > > > > > > -Yi > > > > > > > > > > > > > > > On Wed, Mar 4, 2020 at 11:32 AM Ke Wu <ke.wu...@gmail.com> wrote: > > > > > > > > > > > +1. > > > > > > > > > > > > Thanks for driving this effort. > > > > > > > > > > > > Best, > > > > > > Ke > > > > > > > > > > > > > On Mar 3, 2020, at 6:28 PM, Jagadish Venkatraman < > > > > > jagadish1...@gmail.com> > > > > > > wrote: > > > > > > > > > > > > > > +1 binding. > > > > > > > > > > > > > > Thanks Cameron. I look forward to this feature taking our > "Stream > > > > > > > Processing as a service" offering to the next level. > > > > > > > > > > > > > > Cheers > > > > > > > > > > > > > > On Tuesday, March 3, 2020, Prateek Maheshwari < > > prate...@utexas.edu > > > > > > > > > > wrote: > > > > > > > > > > > > > >> +1 (binding) from me. Thanks for contributing this feature. > > > Looking > > > > > > forward > > > > > > >> to having dependency isolation and to the ability to upgrade > the > > > > > > framework > > > > > > >> independently from an application. > > > > > > >> > > > > > > >> Thanks, > > > > > > >> Prateek > > > > > > >> > > > > > > >> On Fri, Feb 28, 2020 at 10:48 AM Cameron Lee < > > > > cameronlee...@gmail.com > > > > > > > > > > > > >> wrote: > > > > > > >> > > > > > > >>> Hi all, > > > > > > >>> > > > > > > >>> This is a call for a vote on SEP-24: Cluster-based Job > > > Coordinator > > > > > > >>> Dependency Isolation. Thanks to everyone who reviewed the > > > proposal > > > > > and > > > > > > >>> provided feedback. > > > > > > >>> > > > > > > >>> I have addressed comments on the SEP, and I am not aware of > any > > > > > further > > > > > > >>> major questions or objections, so I am starting this vote. > > > > > > >>> > > > > > > >>> SEP link: > > > > > > >>> > > > > > > >>> https://cwiki.apache.org/confluence/display/SAMZA/SEP- > > > > > > >> 24%3A+Cluster-based+Job+Coordinator+Dependency+Isolation > > > > > > >>> > > > > > > >>> Discuss thread: > > > > > > >>> > > > > > > >>> > > > https://mail-archives.apache.org/mod_mbox/samza-dev/202001.mbox/% > > > > > > >> 3cCAMja7KeGcRZ3H95Rxk5XE= > > > > 60zxm6jxjkjuwwxmgmadpfbyk...@mail.gmail.com > > > > > %3e > > > > > > >>> There was also some discussion through comments on the SEP > page > > > > (see > > > > > > >>> Resolved Comments). > > > > > > >>> > > > > > > >>> Please vote: > > > > > > >>> [ ] +1 approve > > > > > > >>> [ ] +0 no opinion > > > > > > >>> [ ] -1 disapprove (and reason why) > > > > > > >>> > > > > > > >>> Thank you, > > > > > > >>> Cameron > > > > > > >>> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > Jagadish > > > > > > > > > > > > > > > > > > > > > > > > > > >
