Hello Jobinesh, From my experience, it usually takes few seconds for session to be up and running to accept incoming statements, although response indicates a success and a session id is returned. Since the session is quite possibly not in a ready state yet, you are seeing a 500 error code for subsequent calls. Please check if that is the case.
Thanks, Umesh Krishnaswamy From: Jobinesh Purushothaman <jobin...@gmail.com> Sent: Wednesday, June 24, 2020 2:56 PM To: user@livy.incubator.apache.org Subject: [EXT] Re: Calling Apache Livy statements REST API from Java fails with status 500 Hi Andras Thanks for looking into it. Look like it happens in the following scenario: We have a Java API to execute statement and it does the following: 1. Create a session by callin Livy /sessions API 2. Then create a statement by passing the session id from prev step. /<sessionId>/statements In this case, Step 2 gives 500 error code. By looking at the response for session api call, noticed a 'state' flag in response(it was state='starting' when it failed). Look like I need to check the status of the session creation API before proceeding with Step 2. - Is there a better way to handle the above approach(other than keep checking the 'state' flag by making a call to sessions/<sessionId>)? - How long a session is active by default? Again , thanks much for your time Jobinesh On Wed, Jun 24, 2020 at 10:22 AM Andras Beni <andrasb...@cloudera.com<mailto:andrasb...@cloudera.com>> wrote: Hi Jobinesh, I'm not familiar with Jersey, but I find it strange how the request body is put together. Are you sure you need to call .toString? My assumption is that this way you don post the document itself but a json string that contains the document. If this is not the problem, can you please give us some Livy logs to ease debugging? Thanks, Andras On Wed, Jun 24, 2020 at 6:43 PM Jobinesh Purushothaman <jobin...@gmail.com<mailto:jobin...@gmail.com>> wrote: Hi I am new to Apache Livy. We are evaluating this product for a usecase to read data from object storage in a Spark cluster. Here are the detail of API and payload - Apache Livy API: POST http://server:port/sessions/{sessionId}/statements<http://server:port/sessions/%7bsessionId%7d/statements> - Request Payload: {"code":" val df = spark.read.option(\"header\", true).json(\"xxx://somebucket@somenamespace/xyz\"); \n df.toJSON.collect.mkString(\"[\", \",\", \"]\")"} I see the above working with postman without issues. We need to call this API from Java. When we tried to call the above API from Java App using a Jersey Client(version 2.30.1 on Java 11), Apache Livy server returns 'Internal Server Error' (500). The Jersey client code looks like as shown below JSONObject payLoad = new JSONObject(); String codeBlock = "val df = spark.read.option(\"header\", true).json(\"xxx://somebucket@somenamespace/xyz\"); \n df.toJSON.collect.mkString(\"[\", \",\", \"]\")"; payLoad.put("code", codeBlock); // In this e.g, session id: '0' is created before calling the below API and it works String apacheLivyUrl = "http://localhost:8998/sessions/0/statements<http://secure-web.cisco.com/1kA4pHYBD3gML3KxeHjQXeAT376M4ZZuHshIkEizMlY1kOKL7CdqxWSHDZ0yG-IBPODre3ja4tcmTVZYQBubiK8Ya8k86K1pXskoFnYJ0c718y8hamDrCAnYU90kdbd5r5JForHlzS6cnyXrNr-Xz_J5M4PpIli9A-NtIRWnTkcME_BOe8Z9bn4TKr-pbzvQc9NMrY98kpC0Oc06c-AR0SKAgcw838WmnTFi-Rit9iEEFzDTme8uw-NFVdYswzoibZrCBwz2MGblipNRuuLwdtWhu6z7BrW0ecaIFQCE2wCYe4nsIa-jiYJFN6UAo0A3SJVC3RBGWCOHERxP3T60CZVw7uUDxTwls0wEPu19iO8t40XN5xbkc3UcT-LWld_6PP8qkSJXWaxQlB9nnaGgHH9VxJ0q6uXfFrwlPcDK0pjnjOo38Prw6h5pcaJZEsDL79-J5mp3iIi3JkkWrDjUdLJygXbVDjyNEE45O7SOku_59eD3tJLyla-hfD68eMpABtNTkxassoT4bBxZUqPkLgQ/http%3A%2F%2Flocalhost%3A8998%2Fsessions%2F0%2Fstatements>"; Response response = ClientBuilder.newBuilder().build().target(apacheLivyUrl) .request(MediaType.APPLICATION_JSON) .accept(MediaType.APPLICATION_JSON) .post(Entity.json(payLoad.toString())); Any idea what is going wrong here? Thanks for your time Jobinesh This message (including any attachments) contains confidential information intended for a specific individual and purpose, and is protected by law. If you are not the intended recipient, you should delete this message and any disclosure, copying, or distribution of this message, or the taking of any action based on it, by you is strictly prohibited. Deloitte refers to a Deloitte member firm, one of its related entities, or Deloitte Touche Tohmatsu Limited ("DTTL"). Each Deloitte member firm is a separate legal entity and a member of DTTL. DTTL does not provide services to clients. Please see www.deloitte.com/about to learn more. v.E.1
