If I understand this correctly. Is LivyClient and PiJob are separate classes within the same Jar such that LivyClient will have the main method? or LivyClient and PiJob are separate classes in separate Jars ? If so which class will have the main method?
I believe Spark also does this but in a much more easy fashion public static String[] jarOfClass(Class<?> cls) https://spark.apache.org/docs/2.1.1/api/java/index.html?org/apache/spark/api/java/JavaSparkContext.html On Fri, Dec 1, 2017 at 7:45 AM, Stefan Miklosovic <mikloso...@gmail.com> wrote: > The last paragraph holds if your job class is in src/test/java as mine > is, I am using Livy for submitting my jobs programmatically as a part > of mvn test. > > On Fri, Dec 1, 2017 at 4:42 PM, Stefan Miklosovic <mikloso...@gmail.com> > wrote: > > As a former Red Hatter, I am using ShrinkWrap and ShrinkWrap Maven > > resolver for it. > > > > It basically looks like this: > > > > public static final class JarBuilder { > > > > public File buildJobJar(File destinationJar) { > > > > final JavaArchive javaArchive = > > ShrinkWrap.create(MavenImporter.class) > > .loadPomFromFile("pom.xml") > > .importBuildOutput() > > .as(JavaArchive.class) > > .addClass(MyLivyJob.class); > > > > logger.info(javaArchive.toString(true)); > > > > javaArchive.as(ZipExporter.class).exportTo(destinationJar, > true); > > > > return destinationJar; > > } > > > > > > // for every other dependency in the runtime which is not > > bundled into the build jar > > // you would need to make something similar as below > > > > public File buildHttpClientJar() { > > return Maven.configureResolver() > > .withMavenCentralRepo(true) > > .resolve(HTTP_CLIENT_COORDINATES) > > .withoutTransitivity() > > .asSingleFile(); > > } > > > > public File buildJacksonDatatyeJodaJar() { > > return Maven.configureResolver() > > .withMavenCentralRepo(true) > > .resolve(JACKSON_DATATYPE_JODA_COORDINATES) > > .withoutTransitivity() > > .asSingleFile(); > > } > > } > > > > After that, you just use uploadJar on livy client instance. After > > that, you can submit your job via submit method on Livy client. > > > > Notice that I made a jar and I added that job class into that. If you > > dont add it there and you upload it, it will not know how to > > deserialize it because that compiled class would not be there. > > > > On Fri, Dec 1, 2017 at 3:54 PM, Meisam Fathi <meisam.fa...@gmail.com> > wrote: > >> You should compile and package PiJar before running this code snippet. > It > >> does not need to be a separate app/project. You can put the PiJob code > right > >> next to the code snippet to run it. MVN/sbt/gradle can create the jar > for > >> you and I assume there is a way to call them programmatically, but that > is > >> not needed. You can use the path to the jar file as piJar. > >> > >> I hope this answers your question. > >> > >> Thanks, > >> Meisam > >> > >> import org.apache.spark.api.java.function.*; > >> > >> import org.apache.livy.*; > >> > >> public class PiJob implements Job<Double>, Function<Integer, Integer>, > >> Function2<Integer, Integer, Integer> { > >> > >> private final int samples; > >> > >> public PiJob(int samples) { > >> this.samples = samples; > >> } > >> > >> @Override > >> public Double call(JobContext ctx) throws Exception { > >> List<Integer> sampleList = new ArrayList<Integer>(); > >> for (int i = 0; i < samples; i++) { > >> sampleList.add(i + 1); > >> } > >> > >> return 4.0d * ctx.sc().parallelize(sampleList).map(this).reduce(this) > / > >> samples; > >> } > >> > >> @Override > >> public Integer call(Integer v1) { > >> double x = Math.random(); > >> double y = Math.random(); > >> return (x*x + y*y < 1) ? 1 : 0; > >> } > >> > >> @Override > >> public Integer call(Integer v1, Integer v2) { > >> return v1 + v2; > >> } > >> > >> } > >> > >> > >> On Fri, Dec 1, 2017 at 1:09 AM kant kodali <kanth...@gmail.com> wrote: > >>> > >>> Hi All, > >>> > >>> I am looking at the following snippet of code and I wonder where and > how > >>> do I create piJar ? can I create programmatically if so how? is there a > >>> complete hello world example somewhere where I can follow steps and > see how > >>> this works? > >>> > >>> Concerning line > >>> > >>> client.uploadJar(new File(piJar)).get(); > >>> > >>> > >>> > >>> Code snippet > >>> > >>> LivyClient client = new LivyClientBuilder() > >>> .setURI(new URI(livyUrl)) > >>> .build(); > >>> > >>> try { > >>> System.err.printf("Uploading %s to the Spark context...\n", piJar); > >>> client.uploadJar(new File(piJar)).get(); > >>> > >>> System.err.printf("Running PiJob with %d samples...\n", samples); > >>> double pi = client.submit(new PiJob(samples)).get(); > >>> > >>> System.out.println("Pi is roughly: " + pi); > >>> } finally { > >>> client.stop(true); > >>> } > > > > > > > > -- > > Stefan Miklosovic > > > > -- > Stefan Miklosovic >
