Cory Nguyen created SPARK-8024:
----------------------------------
Summary: Luigi triggering resolved Blockmanager bug
Key: SPARK-8024
URL: https://issues.apache.org/jira/browse/SPARK-8024
Project: Spark
Issue Type: Bug
Components: Block Manager
Affects Versions: 1.3.1
Reporter: Cory Nguyen
We are using Luigi with Spark to manage our jobs
However we run into a unique rare case with the following conditions that
trigger the resolved Block Manger Bug:
- Dataset is relatively large ~ 1.5TB
- Spark job is ran with Luigi
- save to local HDFS
The spark job would process data and mappings just fine, until the very end
when it proceeds to save the files to local hdfs this is when it triggers this
bug.
However, the job saves and complete data successfully if it was saved to s3://
location.
wondering what might cause this resolved bug to trigger when ran with luigi
saving to local hdfs but not trigger when saved to s3 with luigi or ran without
luigi?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
