Ramandeep Singh created SPARK-25982:
---------------------------------------
Summary: Dataframe write is non blocking in fair scheduling mode
Key: SPARK-25982
URL: https://issues.apache.org/jira/browse/SPARK-25982
Project: Spark
Issue Type: Bug
Components: Spark Core
Affects Versions: 2.3.1
Reporter: Ramandeep Singh
Hi,
I have noticed that expected behavior of dataframe write operation to block is
not working in fair scheduling mode.
Ideally when a dataframe write is occurring and a future is blocking on
AwaitResult, no other job should be started, but this is not the case. I have
noticed that other jobs are started when the partitions are being written.
Regards,
Ramandeep Singh
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
