can we just do sparkConf.setJars(JavaSparkContext.jarOfClass(PiJob.class)); ?
On Fri, Dec 1, 2017 at 7:51 AM, kant kodali <kanth...@gmail.com> wrote: > > If I understand this correctly. Is LivyClient and PiJob are separate > classes within the same Jar such that LivyClient will have the main > method? or LivyClient and PiJob are separate classes in separate Jars ? If > so which class will have the main method? > > I believe Spark also does this but in a much more easy fashion > > public static String[] jarOfClass(Class<?> cls) > > https://spark.apache.org/docs/2.1.1/api/java/index.html?org/apache/spark/api/java/JavaSparkContext.html > > > On Fri, Dec 1, 2017 at 7:45 AM, Stefan Miklosovic <mikloso...@gmail.com> > wrote: > >> The last paragraph holds if your job class is in src/test/java as mine >> is, I am using Livy for submitting my jobs programmatically as a part >> of mvn test. >> >> On Fri, Dec 1, 2017 at 4:42 PM, Stefan Miklosovic <mikloso...@gmail.com> >> wrote: >> > As a former Red Hatter, I am using ShrinkWrap and ShrinkWrap Maven >> > resolver for it. >> > >> > It basically looks like this: >> > >> > public static final class JarBuilder { >> > >> > public File buildJobJar(File destinationJar) { >> > >> > final JavaArchive javaArchive = >> > ShrinkWrap.create(MavenImporter.class) >> > .loadPomFromFile("pom.xml") >> > .importBuildOutput() >> > .as(JavaArchive.class) >> > .addClass(MyLivyJob.class); >> > >> > logger.info(javaArchive.toString(true)); >> > >> > javaArchive.as(ZipExporter.class).exportTo(destinationJar, >> true); >> > >> > return destinationJar; >> > } >> > >> > >> > // for every other dependency in the runtime which is not >> > bundled into the build jar >> > // you would need to make something similar as below >> > >> > public File buildHttpClientJar() { >> > return Maven.configureResolver() >> > .withMavenCentralRepo(true) >> > .resolve(HTTP_CLIENT_COORDINATES) >> > .withoutTransitivity() >> > .asSingleFile(); >> > } >> > >> > public File buildJacksonDatatyeJodaJar() { >> > return Maven.configureResolver() >> > .withMavenCentralRepo(true) >> > .resolve(JACKSON_DATATYPE_JODA_COORDINATES) >> > .withoutTransitivity() >> > .asSingleFile(); >> > } >> > } >> > >> > After that, you just use uploadJar on livy client instance. After >> > that, you can submit your job via submit method on Livy client. >> > >> > Notice that I made a jar and I added that job class into that. If you >> > dont add it there and you upload it, it will not know how to >> > deserialize it because that compiled class would not be there. >> > >> > On Fri, Dec 1, 2017 at 3:54 PM, Meisam Fathi <meisam.fa...@gmail.com> >> wrote: >> >> You should compile and package PiJar before running this code snippet. >> It >> >> does not need to be a separate app/project. You can put the PiJob code >> right >> >> next to the code snippet to run it. MVN/sbt/gradle can create the jar >> for >> >> you and I assume there is a way to call them programmatically, but >> that is >> >> not needed. You can use the path to the jar file as piJar. >> >> >> >> I hope this answers your question. >> >> >> >> Thanks, >> >> Meisam >> >> >> >> import org.apache.spark.api.java.function.*; >> >> >> >> import org.apache.livy.*; >> >> >> >> public class PiJob implements Job<Double>, Function<Integer, Integer>, >> >> Function2<Integer, Integer, Integer> { >> >> >> >> private final int samples; >> >> >> >> public PiJob(int samples) { >> >> this.samples = samples; >> >> } >> >> >> >> @Override >> >> public Double call(JobContext ctx) throws Exception { >> >> List<Integer> sampleList = new ArrayList<Integer>(); >> >> for (int i = 0; i < samples; i++) { >> >> sampleList.add(i + 1); >> >> } >> >> >> >> return 4.0d * ctx.sc().parallelize(sampleList).map(this).reduce(this) >> / >> >> samples; >> >> } >> >> >> >> @Override >> >> public Integer call(Integer v1) { >> >> double x = Math.random(); >> >> double y = Math.random(); >> >> return (x*x + y*y < 1) ? 1 : 0; >> >> } >> >> >> >> @Override >> >> public Integer call(Integer v1, Integer v2) { >> >> return v1 + v2; >> >> } >> >> >> >> } >> >> >> >> >> >> On Fri, Dec 1, 2017 at 1:09 AM kant kodali <kanth...@gmail.com> wrote: >> >>> >> >>> Hi All, >> >>> >> >>> I am looking at the following snippet of code and I wonder where and >> how >> >>> do I create piJar ? can I create programmatically if so how? is there >> a >> >>> complete hello world example somewhere where I can follow steps and >> see how >> >>> this works? >> >>> >> >>> Concerning line >> >>> >> >>> client.uploadJar(new File(piJar)).get(); >> >>> >> >>> >> >>> >> >>> Code snippet >> >>> >> >>> LivyClient client = new LivyClientBuilder() >> >>> .setURI(new URI(livyUrl)) >> >>> .build(); >> >>> >> >>> try { >> >>> System.err.printf("Uploading %s to the Spark context...\n", piJar); >> >>> client.uploadJar(new File(piJar)).get(); >> >>> >> >>> System.err.printf("Running PiJob with %d samples...\n", samples); >> >>> double pi = client.submit(new PiJob(samples)).get(); >> >>> >> >>> System.out.println("Pi is roughly: " + pi); >> >>> } finally { >> >>> client.stop(true); >> >>> } >> > >> > >> > >> > -- >> > Stefan Miklosovic >> >> >> >> -- >> Stefan Miklosovic >> > >
