I would like to make sure I am open for other options that can be considered situationally and based on the context. It's okay, and I don't target to restrict this here. For example, DSv2, I understand it's written in Java because Java interfaces arguably brings better performance. That's why vectorized readers are written in Java too.
Maybe the "general" wasn't explicit in my previous email. Adding APIs to return a Java instance is still rather rare in general given my few years monitoring. The problem I would more like to deal with is more about when we need to add one or a couple of user-facing Java-specific APIs to return Java instances, which is relatively more frequent compared to when we need a bunch of Java specific APIs. In this case, I think it should be guided to use 4. approach. There are pros and cons between 3. and 4., of course. But it looks to me 4. approach is closer to what Spark has targeted so far. 2020년 4월 28일 (화) 오전 8:34, Hyukjin Kwon <gurwls...@gmail.com>님이 작성: > > One thing we could do here is use Java collections internally and make > the Scala API a thin wrapper around Java -- like how Python works. > > Then adding a method to the Scala API would require adding it to the > Java API and we would keep the two more in sync. > > I think it can be an appropriate idea for when we have to deal with this > case a lot but I don't think there are so many > user-facing APIs to return a Java collections, it's rather rare. Also, the > Java users are relatively less than Scala users. > This case is slightly different from Python in a way that there are so > many differences to deal with in PySpark case. > > Also, in case of `Seq`, actually we can just use `Array` instead for both > Scala and Java side simply. I don't find such cases notably awkward. > This problematic cases might be specific to few Java collections or > instances, and I would like to avoid an overkill here. > > Of course, if there is a place to consider other options, let's do. I > don't like to say this is the only required option. > > > > > > 2020년 4월 28일 (화) 오전 1:18, Ryan Blue <rb...@netflix.com.invalid>님이 작성: > >> I think the right choice here depends on how the object is used. For >> developer and internal APIs, I think standardizing on Java collections >> makes the most sense. >> >> For user-facing APIs, it is awkward to return Java collections to Scala >> code -- I think that's the motivation for Tom's comment. For user APIs, I >> think most methods should return Scala collections, and I don't have a >> strong opinion about whether the conversion (or lack thereof) is done in a >> separate object (#1) or in parallel methods (#3). >> >> Both #1 and #3 seem like about the same amount of work and have the same >> likelihood that a developer will leave out a Java method version. One thing >> we could do here is use Java collections internally and make the Scala API >> a thin wrapper around Java -- like how Python works. Then adding a method >> to the Scala API would require adding it to the Java API and we would keep >> the two more in sync. It would also help avoid Scala collections leaking >> into internals. >> >> On Mon, Apr 27, 2020 at 8:49 AM Hyukjin Kwon <gurwls...@gmail.com> wrote: >> >>> Let's stick to the less maintenance efforts then rather than we leave it >>> undecided and delay with leaving this inconsistency. >>> >>> I dont think we can have some very meaningful data about this soon given >>> that we don't hear much complaints about this in general so far. >>> >>> The point of this thread is to make a call rather then defer to the >>> future. >>> >>> On Mon, 27 Apr 2020, 23:15 Wenchen Fan, <cloud0...@gmail.com> wrote: >>> >>>> IIUC We are moving away from having 2 classes for Java and Scala, like >>>> JavaRDD and RDD. It's much simpler to maintain and use with a single class. >>>> >>>> I don't have a strong preference over option 3 or 4. We may need to >>>> collect more data points from actual users. >>>> >>>> On Mon, Apr 27, 2020 at 9:50 PM Hyukjin Kwon <gurwls...@gmail.com> >>>> wrote: >>>> >>>>> Scala users are arguably more prevailing compared to Java users, yes. >>>>> Using the Java instances in Scala side is legitimate, and they are >>>>> already being used in multiple please. I don't believe Scala >>>>> users find this not Scala friendly as it's legitimate and already >>>>> being used. I personally find it's more trouble some to let Java >>>>> users to search which APIs to call. Yes, I understand the pros and >>>>> cons - we should also find the balance considering the actual usage. >>>>> >>>>> One more argument from me is, though, I think one of the goals in >>>>> Spark APIs is the unified API set up to my knowledge >>>>> e.g., JavaRDD <> RDD vs DataFrame. >>>>> If either way is not particularly preferred over the other, I would >>>>> just choose the one to have the unified API set. >>>>> >>>>> >>>>> >>>>> 2020년 4월 27일 (월) 오후 10:37, Tom Graves <tgraves...@yahoo.com>님이 작성: >>>>> >>>>>> I agree a general guidance is good so we keep consistent in the apis. >>>>>> I don't necessarily agree that 4 is the best solution though. I agree >>>>>> its >>>>>> nice to have one api, but it is less friendly for the scala side. >>>>>> Searching for the equivalent Java api shouldn't be hard as it should be >>>>>> very close in the name and if we make it a general rule users should >>>>>> understand it. I guess one good question is what API do most of our >>>>>> users >>>>>> use between Java and Scala and what is the ratio? I don't know the >>>>>> answer >>>>>> to that. I've seen more using Scala over Java. If the majority use Scala >>>>>> then I think the API should be more friendly to that. >>>>>> >>>>>> Tom >>>>>> >>>>>> On Monday, April 27, 2020, 04:04:28 AM CDT, Hyukjin Kwon < >>>>>> gurwls...@gmail.com> wrote: >>>>>> >>>>>> >>>>>> Hi all, >>>>>> >>>>>> I would like to discuss Java specific APIs and which design we will >>>>>> choose. >>>>>> This has been discussed in multiple places so far, for example, at >>>>>> https://github.com/apache/spark/pull/28085#discussion_r407334754 >>>>>> >>>>>> >>>>>> *The problem:* >>>>>> >>>>>> In short, I would like us to have clear guidance on how we support >>>>>> Java specific APIs when >>>>>> it requires to return a Java instance. The problem is simple: >>>>>> >>>>>> def requests: Map[String, ExecutorResourceRequest] = ... >>>>>> def requestsJMap: java.util.Map[String, ExecutorResourceRequest] = ... >>>>>> >>>>>> vs >>>>>> >>>>>> def requests: java.util.Map[String, ExecutorResourceRequest] = ... >>>>>> >>>>>> >>>>>> *Current codebase:* >>>>>> >>>>>> My understanding so far was that the latter is preferred and more >>>>>> consistent and prevailing in the >>>>>> existing codebase, for example, see StateOperatorProgress and >>>>>> StreamingQueryProgress in Structured Streaming. >>>>>> However, I realised that we also have other approaches in the current >>>>>> codebase. There look >>>>>> four approaches to deal with Java specifics in general: >>>>>> >>>>>> 1. Java specific classes such as JavaRDD and JavaSparkContext. >>>>>> 2. Java specific methods with the same name that overload its >>>>>> parameters, see functions.scala. >>>>>> 3. Java specific methods with a different name that needs to >>>>>> return a different type such as TaskContext.resourcesJMap vs >>>>>> TaskContext.resources. >>>>>> 4. One method that returns a Java instance for both Scala and >>>>>> Java sides. see StateOperatorProgress and StreamingQueryProgress. >>>>>> >>>>>> >>>>>> *Analysis on the current codebase:* >>>>>> >>>>>> I agree with 2. approach because the corresponding cases give you a >>>>>> consistent API usage across >>>>>> other language APIs in general. Approach 1. is from the old world >>>>>> when we didn't have unified APIs. >>>>>> This might be the worst approach. >>>>>> >>>>>> 3. and 4. are controversial. >>>>>> >>>>>> For 3., if you have to use Java APIs, then, you should search if >>>>>> there is a variant of that API >>>>>> every time specifically for Java APIs. But yes, it gives you >>>>>> Java/Scala friendly instances. >>>>>> >>>>>> For 4., having one API that returns a Java instance makes you able to >>>>>> use it in both Scala and Java APIs >>>>>> sides although it makes you call asScala in Scala side specifically. >>>>>> But you don’t >>>>>> have to search if there’s a variant of this API and it gives you a >>>>>> consistent API usage across languages. >>>>>> >>>>>> Also, note that calling Java in Scala is legitimate but the opposite >>>>>> case is not, up to my best knowledge. >>>>>> In addition, you should have a method that returns a Java instance >>>>>> for PySpark or SparkR to support. >>>>>> >>>>>> >>>>>> *Proposal:* >>>>>> >>>>>> I would like to have a general guidance on this that the Spark dev >>>>>> agrees upon: Do 4. approach. If not possible, do 3. Avoid 1 almost at all >>>>>> cost. >>>>>> >>>>>> Note that this isn't a hard requirement but *a general guidance*; >>>>>> therefore, the decision might be up to >>>>>> the specific context. For example, when there are some strong >>>>>> arguments to have a separate Java specific API, that’s fine. >>>>>> Of course, we won’t change the existing methods given Micheal’s >>>>>> rubric added before. I am talking about new >>>>>> methods in unreleased branches. >>>>>> >>>>>> Any concern or opinion on this? >>>>>> >>>>> >> >> -- >> Ryan Blue >> Software Engineer >> Netflix >> >
