Hello
Sorry to bother you. This is my first time using livy with Python in EMR.
My code is as simple as below:
```
import json, pprint, requests, textwrap, time
def lambda_handler(event, context):
start_time = time.time()
ip = event['ip']
host = 'http://{0}:8998'.format(ip)
data = {'kind': 'spark'}
headers = {'Content-Type': 'application/json'}
r = requests.post(host + '/sessions', data=json.dumps(data),
headers=headers)
pprint.pprint(r.json(), compact=True)
pprint.pprint(r.headers, compact=True)
session_url = host + r.headers['location']
ready = False
while not ready:
r = requests.get(session_url, headers=headers)
resp = r.json()
ready = resp['state'] == 'idle'
print('checking session creation')
pprint.pprint(r.json(), compact=True)
time.sleep(2)
session_ready = time.time() - start_time
print('{0} elapsed to create session'.format(time.strftime("%H:%M:%S",
time.gmtime(session_ready))))
```
I basically post create a session and repeatedly wait 2s to get its status.
I only got it working once. For the rest of times, the session got
shutting_down very early like below logs:
```
2024-03-19T02:50:51.218Z {'appId': None,
2024-03-19T02:50:51.218Z 'appInfo': {'driverLogUrl': None, 'sparkUiUrl':
None},
2024-03-19T02:50:51.218Z 'id': 3,
2024-03-19T02:50:51.218Z 'kind': 'spark',
2024-03-19T02:50:51.218Z 'log': ['\tat '
2024-03-19T02:50:51.218Z
'org.apache.spark.deploy.SparkSubmit.error(SparkSubmit.scala:1093)',
2024-03-19T02:50:51.218Z '\tat '
2024-03-19T02:50:51.218Z
'org.apache.spark.deploy.SparkSubmit.prepareSubmitEnvironment(SparkSubmit.scala:257)',
2024-03-19T02:50:51.218Z '\tat '
2024-03-19T02:50:51.218Z 'org.apache.spark.deploy.SparkSubmit.org
$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:1010)',
2024-03-19T02:50:51.218Z '\tat '
2024-03-19T02:50:51.218Z
'org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:194)',
2024-03-19T02:50:51.218Z '\tat '
2024-03-19T02:50:51.218Z
'org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:217)',
2024-03-19T02:50:51.218Z '\tat '
2024-03-19T02:50:51.218Z
'org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:91)',
2024-03-19T02:50:51.218Z '\tat '
2024-03-19T02:50:51.218Z
'org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1167)',
2024-03-19T02:50:51.218Z '\tat '
2024-03-19T02:50:51.218Z
'org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1176)',
2024-03-19T02:50:51.218Z '\tat
org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)',
2024-03-19T02:50:51.218Z '\nYARN Diagnostics: '],
2024-03-19T02:50:51.218Z 'name': None,
2024-03-19T02:50:51.218Z 'owner': None,
2024-03-19T02:50:51.218Z 'proxyUser': None,
2024-03-19T02:50:51.218Z
'state': 'shutting_down'}
```
Why does it sound like it wants me to provide runnable jars or pyFiles? How
can get a complete log? From examples in https://livy.apache.org/examples/
and
https://github.com/apache/incubator-livy/blob/f615f272e9130d02170024832ea308516b907195/dev/docker/README.md?plain=1#L74,
I thought the real job is submitted via statement endpoint.
Thanks.
