Please find some of the answers inlline.
> SparkSession sparkSession = ctx.sparkSession();
>
> ctx.sc().getConf().setAppName("MyJob"); // *This app name is not getting
> set when I go http://localhost:8998/sessions <http://localhost:8998/sessions>*
>
> By this time the spark session is already created. You should set the
configs before starting SparkContext.
>
> * // I can see my query but Appid is
> always set to null*
>
> AppId should be given to Livy by YARN (if you are running on YARN). It may
take a while to get response if YARN is busy. If you are not getting an
appID at all, then your application was not submitted correctly. You may
want to check your cluster manager UI for more information
>
> * System.out.println("READING STREAM");*
>
> This will be executed on the driver node. If you are running Spark in
standalone mode or in client mode, the driver node is the same node that
runs Spark. If you are running Spark in cluster mode, the driver is a
random node on the cluster assigned by YARN.
>
>
> df.printSchema(); // *Where does these print statements go ?*
>
>
>
Same as above.
> awaitTermination(); // *This thing blocks forever and I don't want to set a
> timeout. *
>
>
> * // so what should I do to fire and
> forget a streaming job ? *
>
> I believe you can call .start()
> livy.submit(new StreamingJob()).get(); // *This will block forever*
>
> Livy.submit.get returns a value only if the job succeeds. You may want to
use
onJobFailed(JobHandle<T> job, Throwable cause) as well to handle errors and
get a better idea why the job is not returning.
> System.out.println("SUBMITTED JAR!"); // *The control will never get
> here so I can't submit another job.*\
>
> See above.
Thanks,
Meisam
