Github user mgaido91 commented on the issue:
https://github.com/apache/spark/pull/20410
I think that the problem is not SPARK-22797. The problem is that before
this PR, the Python API considers as Defined but not Set all the parameters
with a default value, while the Scala/Java class representing it considers as
Set all them.
This has come up in this case, but it can cause other problems in the
future and also now, because it creates an inconsistency between the Python API
and the representation in the JVM backend.
Thus I do believe that this PR is needed, and it is not only a fix for the
test failures. I think this is a first step and a second step would later be to
drop all the `setDefault` in the Python API, in favor of retrieving them from
the JVM backend. In this way, we will be sure there is no logical inconsistency
between the API and the backend.
Unfortunately, this second part is much bigger and has a large impact. So I
think it best would need a design doc or something similar.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
