Sounds good to me, I'll keep that in mind and pay attention to updates. Shall I make a pull request from my fork, or would someone else prefer to change those function signatures themselves?
Thanks, Alek On Fri, Sep 30, 2016 at 2:25 PM Michael Armbrust <mich...@databricks.com> wrote: > I'd be okay removing that modifier, with one caveat. The code in > org.apache.spark.sql.catalyst.* is purposefully excluded from published > documentation and does not have the same compatibility guarantees as the > rest of the Spark's Public APIs. We leave most of it not "private" so that > advanced users can experiment. > > So, you can write a custom encoder, but until we finalize the encoder API, > you might need to update it each Spark release. > > On Fri, Sep 30, 2016 at 6:36 AM, Aleksander Eskilson < > aleksander...@gmail.com> wrote: > > Hi there, > > > > Currently Catalyst supports encoding custom classes represented as Java > Beans (among others). This Java Bean implementation depends internally on > Catalyst’s ObjectType extension of DataType. Currently, this class is > private to the sql package [1], which is sensible, as it is only necessary > for representing objects in the encoder. However, its private scope makes > it more difficult (if not impossible) to write full custom encoders for > other classes, themselves perhaps composed of additional objects. > > > > Could the definition of the ObjectType be made public in order to support > writing custom Encoders? I can't see any particular danger in this. > > In order to implement an encoder for Avro-specified classes, I have to > work off a fork where the ObjectType definition has be liberalized as > described. > > > > [1] -- > https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/types/ObjectType.scala#L39 > > > > Thanks, > > Aleksander Eskilson > > >