I'd be okay removing that modifier, with one caveat. The code in org.apache.spark.sql.catalyst.* is purposefully excluded from published documentation and does not have the same compatibility guarantees as the rest of the Spark's Public APIs. We leave most of it not "private" so that advanced users can experiment.
So, you can write a custom encoder, but until we finalize the encoder API, you might need to update it each Spark release. On Fri, Sep 30, 2016 at 6:36 AM, Aleksander Eskilson < aleksander...@gmail.com> wrote: > Hi there, > > > > Currently Catalyst supports encoding custom classes represented as Java > Beans (among others). This Java Bean implementation depends internally on > Catalyst’s ObjectType extension of DataType. Currently, this class is > private to the sql package [1], which is sensible, as it is only necessary > for representing objects in the encoder. However, its private scope makes > it more difficult (if not impossible) to write full custom encoders for > other classes, themselves perhaps composed of additional objects. > > > > Could the definition of the ObjectType be made public in order to support > writing custom Encoders? I can't see any particular danger in this. > > In order to implement an encoder for Avro-specified classes, I have to > work off a fork where the ObjectType definition has be liberalized as > described. > > > > [1] -- https://github.com/apache/spark/blob/master/sql/catalyst/ > src/main/scala/org/apache/spark/sql/types/ObjectType.scala#L39 > > > > Thanks, > > Aleksander Eskilson >
