Hi I am new to Apache Livy. We are evaluating this product for a usecase to read data from object storage in a Spark cluster. Here are the detail of API and payload
- Apache Livy API: POST http://server:port /sessions/{sessionId}/statements - Request Payload: {"code":" val df = spark.read.option(\"header\", true).json(\"xxx://somebucket@somenamespace/xyz\"); \n df.toJSON.collect.mkString(\"[\", \",\", \"]\")"} I see the above working with postman without issues. We need to call this API from Java. When we tried to call the above API from Java App using a Jersey Client(version 2.30.1 on Java 11), Apache Livy server returns 'Internal Server Error' (500). The Jersey client code looks like as shown below JSONObject payLoad = new JSONObject(); String codeBlock = "val df = spark.read.option(\"header\", true).json(\"xxx://somebucket@somenamespace/xyz\"); \n df.toJSON.collect.mkString(\"[\", \",\", \"]\")"; payLoad.put("code", codeBlock); // In this e.g, session id: '0' is created before calling the below API and it works String apacheLivyUrl = "http://localhost:8998/sessions/0/statements";; Response response = ClientBuilder.newBuilder().build().target(apacheLivyUrl) .request(MediaType.APPLICATION_JSON) .accept(MediaType.APPLICATION_JSON) .post(Entity.json(payLoad.toString())); Any idea what is going wrong here? Thanks for your time Jobinesh
