[jira] [Created] (LIVY-741) Unable to submit python code using Python Client
Shreyas Kudale created LIVY-741: --- Summary: Unable to submit python code using Python Client Key: LIVY-741 URL: https://issues.apache.org/jira/browse/LIVY-741 Project: Livy Issue Type: Question Components: API, Batch Affects Versions: 0.6.0 Environment: Spark - 2.3.0.2.6.5.1175-1 hdp - 2.6.5.1175-1 Reporter: Shreyas Kudale Fix For: 0.6.0 Attachments: CodeSnippet.PNG, error1.PNG, error2.png * We want to remotely trigger a pySpark job using rest call via Livy. For this, we tried the Python Http Client but getting various errors due to the python version. Since not much is clear through documentation, we wanted to know whether the Livy Code base would be updated for Python 3 or not since Python 2 has been already deprecated. * Http Client setup.py is still showing Python2 as a requirement in the setup.py file and hence not sure which version is to be used. We tried using both Python versions but it's not working out. * With Python2, we get the error shown in the error1.png file while with Python3, the error shown in error2.png occurs but gets resolved when upgraded to Python 3. The code snippet is also attached to the email. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (LIVY-740) Trying to access jar on s3 using livy
[
https://issues.apache.org/jira/browse/LIVY-740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
poonam chaudhari updated LIVY-740:
--
Description:
curl -X POST --data '{"file": "s3://livy-test/spark-examples_2.11-2.4.4.jar",
"className":
"org.apache.spark.examples.SparkPi","conf":{"spark.hadoop.fs.s3.awsAccessKeyId":"***","spark.hadoop.fs.s3.awsSecretAccessKey":"***","spark.executor.extraJavaOptions":"-Dcom.amazonaws.services.s3.enableV4=true","spark.driver.extraJavaOptions":"Dcom.amazonaws.services.s3.enableV4=true"}}'
-H "Content-Type: application/json" -H "X-Requested-By: admin"
[http://hostname:8999/batches|http://172.16.7.203:8999/batches]
I am trying to submit spark job to like this but it fails with below error:
!image-2020-01-10-09-56-27-060.png!
AM I missing something here? I have verified aws secret many times.The same
secret is working with aws cli. Can you please guide me here?
was:
curl -X POST --data '\{"file": "s3://livy-test/spark-examples_2.11-2.4.4.jar",
"className":
"org.apache.spark.examples.SparkPi","conf":{"spark.hadoop.fs.s3.awsAccessKeyId":"***","spark.hadoop.fs.s3.awsSecretAccessKey":"***","spark.executor.extraJavaOptions":"-Dcom.amazonaws.services.s3.enableV4=true","spark.driver.extraJavaOptions":"Dcom.amazonaws.services.s3.enableV4=true"}}'
-H "Content-Type: application/json" -H "X-Requested-By: admin"
[http://172.16.7.203:8999/batches]
I am trying to submit spark job to like this but it fails with below error:
!image-2020-01-10-09-56-27-060.png!
AM I missing something here? I have verified aws secret many times.The same
secret is working with aws cli. Can you please guide me here?
> Trying to access jar on s3 using livy
> -
>
> Key: LIVY-740
> URL: https://issues.apache.org/jira/browse/LIVY-740
> Project: Livy
> Issue Type: Question
> Components: API
> Environment: Spark - 2.3.0.2.6.5.1175-1
> hdp - 2.6.5.1175-1
> AWS jar versions in spark:
> /aws-java-sdk-core-1.10.6.jar
> /aws-java-sdk-kms-1.10.6.jar
> /aws-java-sdk-s3-1.10.6.jar
> /hadoop-aws-2.7.3.2.6.5.1175-1.jar
>Reporter: poonam chaudhari
>Priority: Major
> Labels: beginner
> Attachments: image-2020-01-10-09-56-27-060.png
>
>
> curl -X POST --data '{"file": "s3://livy-test/spark-examples_2.11-2.4.4.jar",
> "className":
> "org.apache.spark.examples.SparkPi","conf":{"spark.hadoop.fs.s3.awsAccessKeyId":"***","spark.hadoop.fs.s3.awsSecretAccessKey":"***","spark.executor.extraJavaOptions":"-Dcom.amazonaws.services.s3.enableV4=true","spark.driver.extraJavaOptions":"Dcom.amazonaws.services.s3.enableV4=true"}}'
> -H "Content-Type: application/json" -H "X-Requested-By: admin"
> [http://hostname:8999/batches|http://172.16.7.203:8999/batches]
>
> I am trying to submit spark job to like this but it fails with below error:
> !image-2020-01-10-09-56-27-060.png!
> AM I missing something here? I have verified aws secret many times.The same
> secret is working with aws cli. Can you please guide me here?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (LIVY-740) Trying to access jar on s3 using livy
poonam chaudhari created LIVY-740:
-
Summary: Trying to access jar on s3 using livy
Key: LIVY-740
URL: https://issues.apache.org/jira/browse/LIVY-740
Project: Livy
Issue Type: Question
Components: API
Environment: Spark - 2.3.0.2.6.5.1175-1
hdp - 2.6.5.1175-1
AWS jar versions in spark:
/aws-java-sdk-core-1.10.6.jar
/aws-java-sdk-kms-1.10.6.jar
/aws-java-sdk-s3-1.10.6.jar
/hadoop-aws-2.7.3.2.6.5.1175-1.jar
Reporter: poonam chaudhari
Attachments: image-2020-01-10-09-56-27-060.png
curl -X POST --data '\{"file": "s3://livy-test/spark-examples_2.11-2.4.4.jar",
"className":
"org.apache.spark.examples.SparkPi","conf":{"spark.hadoop.fs.s3.awsAccessKeyId":"***","spark.hadoop.fs.s3.awsSecretAccessKey":"***","spark.executor.extraJavaOptions":"-Dcom.amazonaws.services.s3.enableV4=true","spark.driver.extraJavaOptions":"Dcom.amazonaws.services.s3.enableV4=true"}}'
-H "Content-Type: application/json" -H "X-Requested-By: admin"
[http://172.16.7.203:8999/batches]
I am trying to submit spark job to like this but it fails with below error:
!image-2020-01-10-09-56-27-060.png!
AM I missing something here? I have verified aws secret many times.The same
secret is working with aws cli. Can you please guide me here?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (LIVY-718) Support multi-active high availability in Livy
[ https://issues.apache.org/jira/browse/LIVY-718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17011909#comment-17011909 ] Yiheng Wang commented on LIVY-718: -- Hi [~bikassaha] I agree that allows multiple servers to connect to one spark driver solution is more flexible and ideal. But I think the current one designated server solution is not conflicted with it. The difference is multiple server solution brings in the state consist problem, it needs extra effort to optimize the existing Livy server code. Other parts are compatible. As you comment, the ideal solution is 1. The user sends a request to a Livy cluster 2. The request may be handled by the current server or route to another server (based on configured strategy) 3. The server handles the request. It may take some time to hydrate the state if it's the first hit For step 2, the one designated server solution strategy is always route to one same server. It doesn't allow route to different servers to avoid the state consist problem. For step 3, we can change the time of initializing the session object and RPC connection in the server. We change it from doing it when server fail/new server join to session first hit, which also works in the one designated solution. I did some dig in these days to see how to optimize livy server code to fix the state consist problem. I list some work items to achieve it. ||Component||Object||Effort|| |Yarn Job|SparkYarnApp|SparkYarnApp contains yarn application log. It is stored in the original session create server memory, which may not be access by other servers. I think it is not suitable to put it in state store. Batch session also have this problem so we cannot push it to rsc driver.| |Session|last activity time|Push to driver for interactive session. Push it to state store for batch session as it always be start time in batch session| |Interactive session|operations(statement records)|push to driver| |Interactive session|operationCount|push to driver| |Session|appId|appId is empty when session metadata is persisted to state store frist time, it is changed after a while. We need to make sure the consist across servers| |Interactive Session|Rsc driver url|same with the above| |Session|session state|Session state is updated by SparkYarnApp thread. There's already some inconsistent between livy server and yarn. If there're multiple servers, the inconsistent will be amplified, as the thread check time is different across servers. One solution is query yarn but it will make many query much longer. Another solution is put it in state store. It also add overhead| |Thrift|Fetched row count|push to driver| |Thrift|mapping from thrift session to livy session|put to state store| |Thrift|operation states|push to driver, touch 1/2 thrift code| I think it's quite a lot of effort and touch existing code base which brings many risks. I suggest we make it as a separate goal. Another discussion is session id. I'm very worried the thing that it breaks the API compatible. We may need to upgrade the API version to V2. And it's a lot effort for people to migrate existing code. I strongly suggest to stick on the int session id and make it as a separate goal. Make a summary, my suggestion is: 1. Change the design to make it compatible with multi-server solution 2. Implement the one-designate solution 3. Make multi-servers solution as a separate goal and do it in another JIRA 3.1 make livy server behavior consist in the cluster(see the table) 3.2 add a new route strategy 4. Stick on int session id What do you think? > Support multi-active high availability in Livy > -- > > Key: LIVY-718 > URL: https://issues.apache.org/jira/browse/LIVY-718 > Project: Livy > Issue Type: Epic > Components: RSC, Server >Reporter: Yiheng Wang >Priority: Major > > In this JIRA we want to discuss how to implement multi-active high > availability in Livy. > Currently, Livy only supports single node recovery. This is not sufficient in > some production environments. In our scenario, the Livy server serves many > notebook and JDBC services. We want to make Livy service more fault-tolerant > and scalable. > There're already some proposals in the community for high availability. But > they're not so complete or just for active-standby high availability. So we > propose a multi-active high availability design to achieve the following > goals: > # One or more servers will serve the client requests at the same time. > # Sessions are allocated among different servers. > # When one node crashes, the affected sessions will be moved to other active > services. > Here's our design document, please review and comment: > https://docs.google.com/document/d/1bD3qYZpw14_NuCcSGUOfqQ0pqvSbCQsOLFuZp26Ohjc/edit?usp=sharing >
