Github user cloud-fan commented on the issue:

    https://github.com/apache/spark/pull/20490
  
    This is the writing code I was talking about:
    ```
        // write the data and commit this writer.
        Utils.tryWithSafeFinallyAndFailureCallbacks(block = {
          iter.foreach(dataWriter.write)
          logInfo(s"Writer for partition ${context.partitionId()} is 
committing.")
          val msg = dataWriter.commit()
          logInfo(s"Writer for partition ${context.partitionId()} committed.")
          msg
        })(catchBlock = {
          // If there is an error, abort this writer
          logError(s"Writer for partition ${context.partitionId()} is 
aborting.")
          dataWriter.abort()
          logError(s"Writer for partition ${context.partitionId()} aborted.")
        })
    ```
    What we can probably do is to check job cancellation periodically during 
`iter.foreach(dataWriter.write)`, e.g. do a check for every 1k writes.
    
    Anyway let's merge this PR first. I'm only merging to master, let's 
backport it to 2.3 if RC3 fails(very likely to happen as there are already 
several regressions show up)


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

Reply via email to