Re: which classes/methods are considered as private in Spark?

2018-11-13 Thread Reynold Xin
I used to, before each release during the RC phase, go through every single doc page to make sure we don’t unintentionally leave things public. I no longer have time to do that unfortunately. I find that very useful because I always catch some mistakes through organic development. > On Nov 13,

Re: which classes/methods are considered as private in Spark?

2018-11-13 Thread Wenchen Fan
> Could you clarify what you mean here? Mima has some known limitations such as not handling "private[blah]" very well Yes that's what I mean. What I want to know here is, which classes/methods we expect them to be private. I think things marked as "private[blabla]" are expected to be private

Re: which classes/methods are considered as private in Spark?

2018-11-13 Thread Marcelo Vanzin
On Tue, Nov 13, 2018 at 6:26 PM Wenchen Fan wrote: > Recently I updated the MiMa exclusion rules, and found MiMa tracks some > private classes/methods unexpectedly. Could you clarify what you mean here? Mima has some known limitations such as not handling "private[blah]" very well (because that

Re: which classes/methods are considered as private in Spark?

2018-11-13 Thread Sean Owen
You should find that 'surprisingly public' classes are there because of language technicalities. For example DummySerializerInstance is public because it's a Java class, and can't be used outside its package otherwise. LIkewise I think MiMa just looks at bytecode, and private[spark] classes are