[
https://issues.apache.org/jira/browse/SPARK-8024?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Owen resolved SPARK-8024.
------------------------------
Resolution: Duplicate
Target Version/s: (was: 1.3.1)
Please review
https://cwiki.apache.org/confluence/display/SPARK/Contributing+to+Spark
> Luigi triggering resolved Blockmanager bug
> ------------------------------------------
>
> Key: SPARK-8024
> URL: https://issues.apache.org/jira/browse/SPARK-8024
> Project: Spark
> Issue Type: Bug
> Components: Block Manager
> Affects Versions: 1.3.1
> Reporter: Cory Nguyen
>
> We are using Luigi with Spark to manage our jobs
> However we run into a unique rare case with the following conditions that
> trigger the resolved Block Manger Bug:
> - Dataset is relatively large ~ 1.5TB
> - Spark job is ran with Luigi
> - save to local HDFS
> The spark job would process data and mappings just fine, until the very end
> when it proceeds to save the files to local hdfs this is when it triggers
> this bug.
> However, the job saves and complete data successfully if it was saved to
> s3:// location.
> wondering what might cause this resolved bug to trigger when ran with luigi
> saving to local hdfs but not trigger when saved to s3 with luigi or ran
> without luigi?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
