I noticed that sometimes the spark cluster seems to restart the job completely.
In the Ambari UI (where I can check jobs/stages) everything that was done up to a certain point is removed, and the job is restarted. Does anyone know what the issue could be? Thank you,
