RE: Calling Apache Livy statements REST API from Java fails with status 500
Hello Jobinesh,
Yes, that’s the sequence I had in my POC application. Any other users with
production application can chime in.
Thanks,
Umesh
From: Jobinesh Purushothaman
Sent: Wednesday, June 24, 2020 4:54 PM
To: Krishnaswamy, Umesh
Cc: user@livy.incubator.apache.org
Subject: [EXT] Re: Calling Apache Livy statements REST API from Java fails with
status 500
Thanks a lot Umesh !
Yes, you are right. A very basic question - How is the sequencing of steps
normally handled in such scenarios ? For instance, In my case, I need to
create a session, wait till the state changes from 'starting' and then make a
call to create a statement. Is there any recommended coding pattern to follow
in order to sequence this flow of calls?
Thanks
Jobinesh
On Wed, Jun 24, 2020 at 12:14 PM Krishnaswamy, Umesh
mailto:ukrishnasw...@deloitte.com>> wrote:
Hello Jobinesh,
From my experience, it usually takes few seconds for session to be up and
running to accept incoming statements, although response indicates a success
and a session id is returned.
Since the session is quite possibly not in a ready state yet, you are seeing a
500 error code for subsequent calls. Please check if that is the case.
Thanks,
Umesh Krishnaswamy
From: Jobinesh Purushothaman mailto:jobin...@gmail.com>>
Sent: Wednesday, June 24, 2020 2:56 PM
To: user@livy.incubator.apache.org<mailto:user@livy.incubator.apache.org>
Subject: [EXT] Re: Calling Apache Livy statements REST API from Java fails with
status 500
Hi Andras
Thanks for looking into it. Look like it happens in the following scenario: We
have a Java API to execute statement and it does the following:
1. Create a session by callin Livy /sessions API
2. Then create a statement by passing the session id from prev step.
//statements
In this case, Step 2 gives 500 error code. By looking at the response for
session api call, noticed a 'state' flag in response(it was state='starting'
when it failed). Look like I need to check the status of the session creation
API before proceeding with Step 2.
- Is there a better way to handle the above approach(other than keep checking
the 'state' flag by making a call to sessions/)?
- How long a session is active by default?
Again , thanks much for your time
Jobinesh
On Wed, Jun 24, 2020 at 10:22 AM Andras Beni
mailto:andrasb...@cloudera.com>> wrote:
Hi Jobinesh,
I'm not familiar with Jersey, but I find it strange how the request body is put
together. Are you sure you need to call .toString? My assumption is that this
way you don post the document itself but a json string that contains the
document.
If this is not the problem, can you please give us some Livy logs to ease
debugging?
Thanks,
Andras
On Wed, Jun 24, 2020 at 6:43 PM Jobinesh Purushothaman
mailto:jobin...@gmail.com>> wrote:
Hi
I am new to Apache Livy. We are evaluating this product for a usecase to read
data from object storage in a Spark cluster. Here are the detail of API and
payload
- Apache Livy API: POST
http://server:port/sessions/{sessionId}/statements<http://secure-web.cisco.com/1ezKMPu9jJ0MRte0MaeagXufPKmWNWtg440oW4xDTkyxVooXxiSEUYsTYRO32_sxkmdACtWzehdz_rEK3TZRAuSqrtNxCBzHLNJiRS4vSBLyHkoNeLlGmXNw8u9VdTzoYoLTpNg-ty06TFtfSu2Lu2zv9xU0hnpPhPTCsuWFOXJLVBgfZ97e7SNzYV2onWqgD7C2IFTt6Y9XFe0TrtwMLXffE6yWqJ2qmusMkU9MkVAMrl05YPlQxSD3namER767y16-o0Q6DJEq1VneV6Abbbo3CtceAagLCnuCKrkiw-JxwRIOSKTV6GYlcs2rynApi9PTyVnLCVUNV1scej-Kkp6PGmxFjnYbNI3YtKiGUmkIF7DOp77n1PbiLt76QK7lZy-T-ysTNXZt5mpbO54u_1VPFUkaxVCGuDgVkMy0TMO7-BQXqam0cZ-UD1barAVrzZLF09GrEGHBNpiAjgQU3cJm1TYv2t9nKL1HyzwgNIfpzmjVjl3qbRzH0ipDArDNzfsJr2NgZkONxBLt_3N6s4A/http%3A%2F%2Fserver%3Aport%2Fsessions%2F%257bsessionId%257d%2Fstatements>
- Request Payload:
{"code":" val df = spark.read.option(\"header\",
true).json(\"xxx://somebucket@somenamespace/xyz\"); \n
df.toJSON.collect.mkString(\"[\", \",\", \"]\")"}
I see the above working with postman without issues. We need to call this API
from Java. When we tried to call the above API from Java App using a Jersey
Client(version 2.30.1 on Java 11), Apache Livy server returns 'Internal Server
Error' (500). The Jersey client code looks like as shown below
JSONObject payLoad = new JSONObject();
String codeBlock = "val df = spark.read.option(\"header\",
true).json(\"xxx://somebucket@somenamespace/xyz\"); \n
df.toJSON.collect.mkString(\"[\", \",\", \"]\")";
payLoad.put("code", codeBlock);
// In this e.g, session id: '0' is created before calling the below API and it
works
String apacheLivyUrl =
"http://localhost:8998/sessions/0/statements<http://secure-web.cisco.com/1kA4pHYBD3gML3KxeHjQXeAT376M4ZZuHshIkEizMlY1kOKL7CdqxWSHDZ0yG-IBPODre3ja4tcmTVZYQBubiK8Ya8k86K1pXskoFnYJ0c718y8hamDrCAnYU90kdbd5
Re: Calling Apache Livy statements REST API from Java fails with status 500
Thanks a lot Umesh !
Yes, you are right. A very basic question - How is the sequencing of steps
normally handled in such scenarios ? For instance, In my case, I need to
create a session, wait till the state changes from 'starting' and then
make a call to create a statement. Is there any recommended coding pattern
to follow in order to sequence this flow of calls?
Thanks
Jobinesh
On Wed, Jun 24, 2020 at 12:14 PM Krishnaswamy, Umesh <
ukrishnasw...@deloitte.com> wrote:
> Hello Jobinesh,
>
>
>
> From my experience, it usually takes few seconds for session to be up and
> running to accept incoming statements, although response indicates a
> success and a session id is returned.
>
> Since the session is quite possibly not in a ready state yet, you are
> seeing a 500 error code for subsequent calls. Please check if that is the
> case.
>
>
>
> Thanks,
>
> Umesh Krishnaswamy
>
>
>
> *From:* Jobinesh Purushothaman
> *Sent:* Wednesday, June 24, 2020 2:56 PM
> *To:* user@livy.incubator.apache.org
> *Subject:* [EXT] Re: Calling Apache Livy statements REST API from Java
> fails with status 500
>
>
>
> Hi Andras
>
> Thanks for looking into it. Look like it happens in the following
> scenario: We have a Java API to execute statement and it does the following:
>
> 1. Create a session by callin Livy /sessions API
>
> 2. Then create a statement by passing the session id from prev step.
> //statements
>
> In this case, Step 2 gives 500 error code. By looking at the response for
> session api call, noticed a 'state' flag in response(it was
> state='starting' when it failed). Look like I need to check the status of
> the session creation API before proceeding with Step 2.
>
>
>
> - Is there a better way to handle the above approach(other than keep
> checking the 'state' flag by making a call to sessions/)?
>
> - How long a session is active by default?
>
>
>
> Again , thanks much for your time
>
> Jobinesh
>
>
>
> On Wed, Jun 24, 2020 at 10:22 AM Andras Beni
> wrote:
>
> Hi Jobinesh,
>
>
>
> I'm not familiar with Jersey, but I find it strange how the request body
> is put together. Are you sure you need to call .toString? My assumption is
> that this way you don post the document itself but a json string that
> contains the document.
>
> If this is not the problem, can you please give us some Livy logs
> to ease debugging?
>
>
>
> Thanks,
>
> Andras
>
>
>
> On Wed, Jun 24, 2020 at 6:43 PM Jobinesh Purushothaman
> wrote:
>
> Hi
>
> I am new to Apache Livy. We are evaluating this product for a usecase to
> read data from object storage in a Spark cluster. Here are the detail of
> API and payload
>
>
> - Apache Livy API: POST
> http://server:port/sessions/{sessionId}/statements
>
> - Request Payload:
>
> {"code":" val df = spark.read.option(\"header\",
> true).json(\"xxx://somebucket@somenamespace/xyz\"); \n
> df.toJSON.collect.mkString(\"[\", \",\", \"]\")"}
>
>
>
> I see the above working with postman without issues. We need to call this
> API from Java. When we tried to call the above API from Java App using a
> Jersey Client(version 2.30.1 on Java 11), Apache Livy server returns
> 'Internal Server Error' (500). The Jersey client code looks like as shown
> below
>
>
> JSONObject payLoad = new JSONObject();
> String codeBlock = "val df = spark.read.option(\"header\",
> true).json(\"xxx://somebucket@somenamespace/xyz\"); \n
> df.toJSON.collect.mkString(\"[\", \",\", \"]\")";
> payLoad.put("code", codeBlock);
>
> // In this e.g, session id: '0' is created before calling the below API
> and it works
> String apacheLivyUrl = "http://localhost:8998/sessions/0/statements
> <http://secure-web.cisco.com/1kA4pHYBD3gML3KxeHjQXeAT376M4ZZuHshIkEizMlY1kOKL7CdqxWSHDZ0yG-IBPODre3ja4tcmTVZYQBubiK8Ya8k86K1pXskoFnYJ0c718y8hamDrCAnYU90kdbd5r5JForHlzS6cnyXrNr-Xz_J5M4PpIli9A-NtIRWnTkcME_BOe8Z9bn4TKr-pbzvQc9NMrY98kpC0Oc06c-AR0SKAgcw838WmnTFi-Rit9iEEFzDTme8uw-NFVdYswzoibZrCBwz2MGblipNRuuLwdtWhu6z7BrW0ecaIFQCE2wCYe4nsIa-jiYJFN6UAo0A3SJVC3RBGWCOHERxP3T60CZVw7uUDxTwls0wEPu19iO8t40XN5xbkc3UcT-LWld_6PP8qkSJXWaxQlB9nnaGgHH9VxJ0q6uXfFrwlPcDK0pjnjOo38Prw6h5pcaJZEsDL79-J5mp3iIi3JkkWrDjUdLJygXbVDjyNEE45O7SOku_59eD3tJLyla-hfD68eMpABtNTkxassoT4bBxZUqPkLgQ/http%3A%2F%2Flocalhost%3A8998%2Fsessions%2F0%2Fstatements>
> ";
> Response response =
> ClientBuilder.newBuilder().build().target(apacheLivyUrl)
> .request(MediaType.APPLICATION_JSON)
>
RE: Calling Apache Livy statements REST API from Java fails with status 500
Hello Jobinesh,
From my experience, it usually takes few seconds for session to be up and
running to accept incoming statements, although response indicates a success
and a session id is returned.
Since the session is quite possibly not in a ready state yet, you are seeing a
500 error code for subsequent calls. Please check if that is the case.
Thanks,
Umesh Krishnaswamy
From: Jobinesh Purushothaman
Sent: Wednesday, June 24, 2020 2:56 PM
To: user@livy.incubator.apache.org
Subject: [EXT] Re: Calling Apache Livy statements REST API from Java fails with
status 500
Hi Andras
Thanks for looking into it. Look like it happens in the following scenario: We
have a Java API to execute statement and it does the following:
1. Create a session by callin Livy /sessions API
2. Then create a statement by passing the session id from prev step.
//statements
In this case, Step 2 gives 500 error code. By looking at the response for
session api call, noticed a 'state' flag in response(it was state='starting'
when it failed). Look like I need to check the status of the session creation
API before proceeding with Step 2.
- Is there a better way to handle the above approach(other than keep checking
the 'state' flag by making a call to sessions/)?
- How long a session is active by default?
Again , thanks much for your time
Jobinesh
On Wed, Jun 24, 2020 at 10:22 AM Andras Beni
mailto:andrasb...@cloudera.com>> wrote:
Hi Jobinesh,
I'm not familiar with Jersey, but I find it strange how the request body is put
together. Are you sure you need to call .toString? My assumption is that this
way you don post the document itself but a json string that contains the
document.
If this is not the problem, can you please give us some Livy logs to ease
debugging?
Thanks,
Andras
On Wed, Jun 24, 2020 at 6:43 PM Jobinesh Purushothaman
mailto:jobin...@gmail.com>> wrote:
Hi
I am new to Apache Livy. We are evaluating this product for a usecase to read
data from object storage in a Spark cluster. Here are the detail of API and
payload
- Apache Livy API: POST
http://server:port/sessions/{sessionId}/statements<http://server:port/sessions/%7bsessionId%7d/statements>
- Request Payload:
{"code":" val df = spark.read.option(\"header\",
true).json(\"xxx://somebucket@somenamespace/xyz\"); \n
df.toJSON.collect.mkString(\"[\", \",\", \"]\")"}
I see the above working with postman without issues. We need to call this API
from Java. When we tried to call the above API from Java App using a Jersey
Client(version 2.30.1 on Java 11), Apache Livy server returns 'Internal Server
Error' (500). The Jersey client code looks like as shown below
JSONObject payLoad = new JSONObject();
String codeBlock = "val df = spark.read.option(\"header\",
true).json(\"xxx://somebucket@somenamespace/xyz\"); \n
df.toJSON.collect.mkString(\"[\", \",\", \"]\")";
payLoad.put("code", codeBlock);
// In this e.g, session id: '0' is created before calling the below API and it
works
String apacheLivyUrl =
"http://localhost:8998/sessions/0/statements<http://secure-web.cisco.com/1kA4pHYBD3gML3KxeHjQXeAT376M4ZZuHshIkEizMlY1kOKL7CdqxWSHDZ0yG-IBPODre3ja4tcmTVZYQBubiK8Ya8k86K1pXskoFnYJ0c718y8hamDrCAnYU90kdbd5r5JForHlzS6cnyXrNr-Xz_J5M4PpIli9A-NtIRWnTkcME_BOe8Z9bn4TKr-pbzvQc9NMrY98kpC0Oc06c-AR0SKAgcw838WmnTFi-Rit9iEEFzDTme8uw-NFVdYswzoibZrCBwz2MGblipNRuuLwdtWhu6z7BrW0ecaIFQCE2wCYe4nsIa-jiYJFN6UAo0A3SJVC3RBGWCOHERxP3T60CZVw7uUDxTwls0wEPu19iO8t40XN5xbkc3UcT-LWld_6PP8qkSJXWaxQlB9nnaGgHH9VxJ0q6uXfFrwlPcDK0pjnjOo38Prw6h5pcaJZEsDL79-J5mp3iIi3JkkWrDjUdLJygXbVDjyNEE45O7SOku_59eD3tJLyla-hfD68eMpABtNTkxassoT4bBxZUqPkLgQ/http%3A%2F%2Flocalhost%3A8998%2Fsessions%2F0%2Fstatements>";
Response response = ClientBuilder.newBuilder().build().target(apacheLivyUrl)
.request(MediaType.APPLICATION_JSON)
.accept(MediaType.APPLICATION_JSON)
.post(Entity.json(payLoad.toString()));
Any idea what is going wrong here?
Thanks for your time
Jobinesh
This message (including any attachments) contains confidential information
intended for a specific individual and purpose, and is protected by law. If you
are not the intended recipient, you should delete this message and any
disclosure, copying, or distribution of this message, or the taking of any
action based on it, by you is strictly prohibited.
Deloitte refers to a Deloitte member firm, one of its related entities, or
Deloitte Touche Tohmatsu Limited ("DTTL"). Each Deloitte member firm is a
separate legal entity and a member of DTTL. DTTL does not provide services to
clients. Please see www.deloitte.com/about to learn more.
v.E.1
Re: Calling Apache Livy statements REST API from Java fails with status 500
Hi Andras
Thanks for looking into it. Look like it happens in the following
scenario: We have a Java API to execute statement and it does the following:
1. Create a session by callin Livy /sessions API
2. Then create a statement by passing the session id from prev step.
//statements
In this case, Step 2 gives 500 error code. By looking at the response for
session api call, noticed a 'state' flag in response(it was
state='starting' when it failed). Look like I need to check the status of
the session creation API before proceeding with Step 2.
- Is there a better way to handle the above approach(other than keep
checking the 'state' flag by making a call to sessions/)?
- How long a session is active by default?
Again , thanks much for your time
Jobinesh
On Wed, Jun 24, 2020 at 10:22 AM Andras Beni
wrote:
> Hi Jobinesh,
>
> I'm not familiar with Jersey, but I find it strange how the request body
> is put together. Are you sure you need to call .toString? My assumption is
> that this way you don post the document itself but a json string that
> contains the document.
> If this is not the problem, can you please give us some Livy logs
> to ease debugging?
>
> Thanks,
> Andras
>
> On Wed, Jun 24, 2020 at 6:43 PM Jobinesh Purushothaman
> wrote:
>
>> Hi
>> I am new to Apache Livy. We are evaluating this product for a usecase to
>> read data from object storage in a Spark cluster. Here are the detail of
>> API and payload
>>
>> - Apache Livy API: POST http://server:port
>> /sessions/{sessionId}/statements
>> - Request Payload:
>> {"code":" val df = spark.read.option(\"header\",
>> true).json(\"xxx://somebucket@somenamespace/xyz\"); \n
>> df.toJSON.collect.mkString(\"[\", \",\", \"]\")"}
>>
>> I see the above working with postman without issues. We need to call this
>> API from Java. When we tried to call the above API from Java App using a
>> Jersey Client(version 2.30.1 on Java 11), Apache Livy server returns
>> 'Internal Server Error' (500). The Jersey client code looks like as shown
>> below
>>
>> JSONObject payLoad = new JSONObject();
>> String codeBlock = "val df = spark.read.option(\"header\",
>> true).json(\"xxx://somebucket@somenamespace/xyz\"); \n
>> df.toJSON.collect.mkString(\"[\", \",\", \"]\")";
>> payLoad.put("code", codeBlock);
>>
>> // In this e.g, session id: '0' is created before calling the below API
>> and it works
>> String apacheLivyUrl = "http://localhost:8998/sessions/0/statements";;
>> Response response =
>> ClientBuilder.newBuilder().build().target(apacheLivyUrl)
>> .request(MediaType.APPLICATION_JSON)
>> .accept(MediaType.APPLICATION_JSON)
>> .post(Entity.json(payLoad.toString()));
>>
>> Any idea what is going wrong here?
>> Thanks for your time
>> Jobinesh
>>
>
Re: Calling Apache Livy statements REST API from Java fails with status 500
Hi Jobinesh,
I'm not familiar with Jersey, but I find it strange how the request body is
put together. Are you sure you need to call .toString? My assumption is
that this way you don post the document itself but a json string that
contains the document.
If this is not the problem, can you please give us some Livy logs
to ease debugging?
Thanks,
Andras
On Wed, Jun 24, 2020 at 6:43 PM Jobinesh Purushothaman
wrote:
> Hi
> I am new to Apache Livy. We are evaluating this product for a usecase to
> read data from object storage in a Spark cluster. Here are the detail of
> API and payload
>
> - Apache Livy API: POST http://server:port
> /sessions/{sessionId}/statements
> - Request Payload:
> {"code":" val df = spark.read.option(\"header\",
> true).json(\"xxx://somebucket@somenamespace/xyz\"); \n
> df.toJSON.collect.mkString(\"[\", \",\", \"]\")"}
>
> I see the above working with postman without issues. We need to call this
> API from Java. When we tried to call the above API from Java App using a
> Jersey Client(version 2.30.1 on Java 11), Apache Livy server returns
> 'Internal Server Error' (500). The Jersey client code looks like as shown
> below
>
> JSONObject payLoad = new JSONObject();
> String codeBlock = "val df = spark.read.option(\"header\",
> true).json(\"xxx://somebucket@somenamespace/xyz\"); \n
> df.toJSON.collect.mkString(\"[\", \",\", \"]\")";
> payLoad.put("code", codeBlock);
>
> // In this e.g, session id: '0' is created before calling the below API
> and it works
> String apacheLivyUrl = "http://localhost:8998/sessions/0/statements";;
> Response response =
> ClientBuilder.newBuilder().build().target(apacheLivyUrl)
> .request(MediaType.APPLICATION_JSON)
> .accept(MediaType.APPLICATION_JSON)
> .post(Entity.json(payLoad.toString()));
>
> Any idea what is going wrong here?
> Thanks for your time
> Jobinesh
>
