[CONF] Apache Samza > SEP-24: Cluster-based Job Coordinator Dependency Isolation

[Cameron Lee (Confluence)]

Title: Message Title

There's 1 new comment and 1 new edit on this page

SEP-24: Cluster-based Job Coordinator Dependency Isolation 1 new comment Daniel Nishimura Any test plan guidance for this SEP? Cameron Lee Good catch. Added a new section. View comment • Reply • Like 1 new edit Cameron Lee edited this page Here's what changed: ... Generating the Samza API whitelist In order to load the Samza API classes from the API classloader, we need to tell cytodynamics what those classes are. We can do this by providing a whitelist of packages/classes when building the cytodynamics classloader. All public interfaces/classes inside of samza-api should be considered an API class. One way to generate this whitelist is to use a Gradle task to find all the classes from samza-api and put that list in a file. Then, that file can be read by Samza when constructing the cytodynamics classloader. The Gradle task should also include classes from samza-kv. ... Once we have general split deployment, this will no longer be a problem, because the version of Samza used across all parts of the application will be consistent. Testing Local testing We can use samza-hello-samza to test this locally. It has scripts to set up Zookeeper, Kafka, and YARN locally. The local YARN deployment will give the process isolation necessary to test the AM. Locally build the framework tarballs for API and infrastructure. It would be useful to put an example somewhere for how to build those tarballs. Deploy Zookeeper, Kafka, and YARN locally (https://samza.apache.org/startup/hello-samza/latest/). Fill in certain configs (see New configs above). These will go into the properties file passed to the run-app.sh script. Create the tarball for the application (https://samza.apache.org/startup/hello-samza/latest/). For testing local changes, remember to run the "publishToMavenLocal" command. Automated integration test Build API and infrastructure framework artifacts Build a simple test job with dependency isolation enabled This will require multiple configs, including the location of the framework artifacts for YARN resources (see New configs above). Use the integration test framework (which uses real YARN) to check that the job runs successfully Alternative solutions Alternative solutions for SEP-24 Go to page history View page

Stop watching space • Manage notifications This message was sent by Atlassian Confluence 6.15.8