[jira] [Created] (SPARK-2671) BlockObjectWriter should create parent directory when the directory doesn't exist

[Kousuke Saruta (JIRA)]

Kousuke Saruta created SPARK-2671:
-------------------------------------

             Summary: BlockObjectWriter should create parent directory when the 
directory doesn't exist
                 Key: SPARK-2671
                 URL: https://issues.apache.org/jira/browse/SPARK-2671
             Project: Spark
          Issue Type: Bug
    Affects Versions: 1.0.0
            Reporter: Kousuke Saruta
            Priority: Minor


BlockObjectWriter#open expects parent directory is present. 

{code}

  override def open(): BlockObjectWriter = {
    fos = new FileOutputStream(file, true)
    ts = new TimeTrackingOutputStream(fos)
    channel = fos.getChannel()
    lastValidPosition = initialPosition
    bs = compressStream(new BufferedOutputStream(ts, bufferSize))
    objOut = serializer.newInstance().serializeStream(bs)
    initialized = true
    this
  }
{code}

Normally, the parent directory is created by DiskBlockManager#createLocalDirs 
but, just in case, BlockObjectWriter#open should check the existence of the 
directory and create the directory if the directory does not exist.

I think, recoverable error should be recovered.



--
This message was sent by Atlassian JIRA
(v6.2#6252)