Michael's solution might do it for you, but if it doesn't I would try the
pyFiles arg instead, I haven't used it but it's purpose may be to fix this
exact issue. I can research it more if needed.
Alex Bozarth
Software Engineer
Spark Technology Center
E-mail: [email protected]
GitHub: github.com/ajbozarth
505 Howard
Street
San Francisco, CA
94105
United
States
From: Michael Rhee <[email protected]>
To: "[email protected]"
<[email protected]>
Date: 11/03/2017 07:41 AM
Subject: Re: POST /batches failing with "Only local python files are
supported"
Hi Lucas,
I believe you need to modify your Livy configuration file to allow access
to a local directory on your master node. Something like the following:
livy.file.local-dir-whitelist =/home/hadoop-user/
Then use the file:/home/hadoop-user/file argument when you pass your
request to Livy. Hope that helps.
Best,
Michael
From: "Partridge, Lucas (GE Aviation)" <[email protected]>
Reply-To: "[email protected]" <[email protected]>
Date: Friday, November 3, 2017 at 5:34 AM
To: "[email protected]" <[email protected]>
Subject: POST /batches failing with "Only local python files are supported"
Thanks for the suggestion Alex. However whenever I try anything beginning
with file:/, file:// or file:/// or file://NNPRDHA/ I get this error:
org.springframework.web.client.RestClientException: Error running rest
call; nested exception is
org.springframework.web.client.HttpClientErrorException: 400 Bad Request
It sounds like from what you’ve said it’s a Spark error rather than a Livy
error, which I didn’t realise before. But whenever I put in a file argument
value without a file: in front it - Livy or Spark? - assumes it’s an HDFS
path and prepends it with hdfs://. Then Spark complains that only local
Python files are supported.
I’ve also tried copying the Python file from hdfs to the local file system
of the Spark Master node. But I can’t specify that path in my POST call
because if I use file: I get 400 Bad Request; and if I don’t use file: it
(Spark?) assumes it’s in hdfs! Should I use the files argument for
POST /batches too? Or the pyFiles argument, although I assumed that was for
Python libraries required by the main application. I’ve tried lots of
combinations but none have worked so far.
From: Alex Bozarth [mailto:[email protected]]
Sent: 02 November 2017 21:34
To: [email protected]
Subject: EXT: Re: POST /batches failing with "Only local python files are
supported"
It sounds like you're passing in a local past and it's being treated as an
HDFS path. Have you tired passing the path in with file:// at the front
(similar to hdfs://) that tells hdfs that the path is local, I've run into
this issue with Spark before.
Alex Bozarth
Software Engineer
Spark Technology Center
E-mail: [email protected]
GitHub: github.com/ajbozarth
505 Howard
Street
San Francisco, CA
94105
United
States
nactive hide details for "Partridge, Lucas (GE Aviation)" ---11/02/2017 0
"Partridge, Lucas (GE Aviation)" ---11/02/2017 07:14:58 AM---I want to use
Livy (0.3) to run a Python file that I've placed in HDFS. I'm invoking
POST /batches
From: "Partridge, Lucas (GE Aviation)" <[email protected]>
To: "[email protected]" <[email protected]>
Date: 11/02/2017 07:14 AM
Subject: POST /batches failing with "Only local python files are supported"
I want to use Livy (0.3) to run a Python file that I’ve placed in HDFS. I’m
invoking POST /batches from a Java REST client, passing in the path to the
HDFS file as the ‘file’ argument of the POST request’s body (
https://github.com/apache/incubator-livy/tree/branch-0.3#request-body-2).
The value I’m providing for ‘file’ is "/user/MyUserName/hello.py".
The POST response says the batch is in state ‘starting’ but when I query it
using GET /batches/{batchId} I see this:
Error: Only local python files are supported:
hdfs://NNPRDHA/user/MyUserName/hello.py
You can see the value has been altered from what I provided. How do I
successfully invoke the Python file please?
(I see someone raised a similar problem at
https://groups.google.com/a/cloudera.org/d/msg/livy-user/6AZeqtVwipg/U46tUjqNBwAJ
but it’s not clear if or how they solved it.)
Thanks, Lucas.
