[
https://issues.apache.org/jira/browse/SPARK-27751?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16844451#comment-16844451
]
Hyukjin Kwon commented on SPARK-27751:
--------------------------------------
{{FileFormat}} itself wasn't supported to be external for clarification.
{code}
I cannot understand why this public method was adding so much overhead, and it
would be great if you could elaborate a bit on that.
{code}
It is overhead.
1. if we accept this case, we should add other public method as well per
requested.
2. Once we start to keep the compatibility there, we should keep maintaining
it. For instance, if we change name of the class {{FileFormat}}, it breaks
compatibility again.
3. {{execution}} package is internal and private. Should we now then convert
this as an external package due to this one case?
{quote}
Besides, if we keep the method `FileFormat.buildReader` marked as protected,
would it make sense to mark the overrides in `CSVFileFormat`, `JsonFileFormat`,
`TextFileFormat`, `OrcFileFormat`, `SimpleTextSource`, `LibSVMOutputWriter` and
`AvroFileFormat` as protected as well?
{quote}
We can when needed and required. I won't change that with no reason. But
compatibility is not the concern.
> buildReader is now protected
> ----------------------------
>
> Key: SPARK-27751
> URL: https://issues.apache.org/jira/browse/SPARK-27751
> Project: Spark
> Issue Type: Question
> Components: Spark Core
> Affects Versions: 2.4.3
> Reporter: Geet Kumar
> Priority: Major
>
> I have recently upgraded to spark 2.4.0 and was relying on the `buildReader`
> method. It originally was public and now it is protected.
> What was the reason for this change?
> The only workaround I can see is to use `buildReaderWithPartitionValues`
> which remains public. Any plans to revert `buildReader` to be public again?
> The change was made here: [https://github.com/apache/spark/pull/17253/files]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
