On Mon, Jun 12, 2017 at 6:21 PM, Matt Sicker <boa...@gmail.com> wrote:
> Last time I used Spring, they had an @Async annotation you could use which > would automatically execute in an executor service (all handled via bean > proxies as usual). > Yeah, for higher-level stuff like Spring Batch, it's a little more awkward but, hey, it works (split and flow elements). Gary > On 12 June 2017 at 19:22, Gary Gregory <garydgreg...@gmail.com> wrote: > > > Hi All, > > > > I think it would be most helpful to note the distinction between the > > parallelism aspect and the bridge to domain classes aspect (currently > done > > with reflection in the proposed github repo.) > > > > It seems (to me) that in between the ForkJoin framework already in Java > (a > > low-level library) and up to Apache Spark (an lowel-level set of classes > > and high-level application-server-like code base), there are a ton of > > options already out there. I am not sure what yet another framework would > > do that is not already there. > > > > Maybe the distinguishing factor here is the use of reflection? What about > > annotations? That seems to be more modern approach (Java 5! :-) than the > > typing of method names in code (as currently done in the repo) which is a > > nightmare to maintain especially when you are in an evolving code base > and > > refactoring all the time. > > > > Maybe an interesting angle would be decorating domain classes with > > annotations and submitting those to fork/join. Just thinkin' aloud... > > > > Gary > > > > On Mon, Jun 12, 2017 at 4:29 PM, Matt Sicker <boa...@gmail.com> wrote: > > > > > I'd be interested to see where this leads to. It could end up as a sort > > of > > > Commons Parallel library. Besides providing an execution API, there > could > > > be plenty of support utilities that tend to be found in all the > > > *Util(s)/*Helper classes in projects like all the ones I mentioned > > earlier > > > (basically all sorts of Hadoop-related projects and other distributed > > > systems here). > > > > > > Really, there's so many ways that such a project could head, I'd like > to > > > hear more ideas on what to focus on. > > > > > > On 12 June 2017 at 18:19, Gary Gregory <garydgreg...@gmail.com> wrote: > > > > > > > The upshot is that there has to be a way to do this with some custom > > code > > > > to at least have the ability to 'fast path' the code without > > reflection. > > > > Using lambdas should make this fairly syntactically unobtrusive. > > > > > > > > On Mon, Jun 12, 2017 at 4:02 PM, Arun Mohan <strider90a...@gmail.com > > > > > > wrote: > > > > > > > > > Yes, reflection is not very performant but I don't think I have any > > > other > > > > > choice since the library has to inspect the object supplied by the > > > client > > > > > at runtime to pick out the methods to be invoked using > > > CompletableFuture. > > > > > But the performance penalty paid for using reflection will be more > > than > > > > > offset by the savings of parallel method execution, more so as the > no > > > of > > > > > methods executed in parallel increases. > > > > > > > > > > On Mon, Jun 12, 2017 at 3:21 PM, Gary Gregory < > > garydgreg...@gmail.com> > > > > > wrote: > > > > > > > > > > > On a lower-level, if you want to use this for lower-level > services > > > > (where > > > > > > there is no network latency for example), you will need to avoid > > > using > > > > > > reflection to get the best performance. > > > > > > > > > > > > Gary > > > > > > > > > > > > On Mon, Jun 12, 2017 at 3:15 PM, Arun Mohan < > > strider90a...@gmail.com > > > > > > > > > > wrote: > > > > > > > > > > > > > Hi Gary, > > > > > > > > > > > > > > Thanks for your response. You have some valid and interesting > > > points > > > > > :-) > > > > > > > Of course you are right that Spark is much more mature. Thanks > > for > > > > your > > > > > > > insight. > > > > > > > It will be interesting indeed to find out if the core > > > parallelization > > > > > > > engine of Spark can be isolated like you suggest. > > > > > > > > > > > > > > I started working on this project because I felt that there was > > no > > > > good > > > > > > > library for parallelizing method calls which can be plugged in > > > easily > > > > > > into > > > > > > > an existing java project. Ultimately, if such a solution can be > > > > > > > incorporated in the Apache Commons, it would be a useful > addition > > > to > > > > > the > > > > > > > Commons repository. > > > > > > > > > > > > > > Thanks, > > > > > > > Arun > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Mon, Jun 12, 2017 at 3:01 PM, Gary Gregory < > > > > garydgreg...@gmail.com> > > > > > > > wrote: > > > > > > > > > > > > > > > Hi Arun, > > > > > > > > > > > > > > > > Sure, and that is to be expected, Spark is more mature than a > > > four > > > > > > class > > > > > > > > prototype. What I am trying to get to is that in order for > the > > > > > library > > > > > > to > > > > > > > > be useful, you will end up with more in a first release, and > > > after > > > > a > > > > > > > couple > > > > > > > > more releases, there will be more and more. Would Spark not > > have > > > in > > > > > its > > > > > > > > guts the same kind of code your are proposing here? By > > extension, > > > > > will > > > > > > > you > > > > > > > > not end up with more framework-like (Spark-like) code and > > > solutions > > > > > as > > > > > > > > found in Spark? I am just playing devil's advocate here ;-) > > > > > > > > > > > > > > > > > > > > > > > > What would be interesting would be to find out if there is a > > core > > > > > part > > > > > > of > > > > > > > > Spark that is separable and ex tractable into a Commons > > > component. > > > > > > Since > > > > > > > > Spark has a proven track record, it is more likely, that > such a > > > > > library > > > > > > > > would be generally useful than one created from scratch that > > does > > > > not > > > > > > > > integrate with anything else. Again, please do not take any > of > > > this > > > > > > > > personally, I am just playing here :-) > > > > > > > > > > > > > > > > Gary > > > > > > > > > > > > > > > > > > > > > > > > On Mon, Jun 12, 2017 at 2:29 PM, Matt Sicker < > boa...@gmail.com > > > > > > > > wrote: > > > > > > > > > > > > > > > > > I already see a huge difference here: Spark requires a > bunch > > of > > > > > > > > > infrastructure to be set up, while this library is just a > > > > library. > > > > > > > > Similar > > > > > > > > > to Kafka Streams versus Spark Streaming or Flink or Storm > or > > > > Samza > > > > > or > > > > > > > the > > > > > > > > > others. > > > > > > > > > > > > > > > > > > On 12 June 2017 at 16:28, Gary Gregory < > > garydgreg...@gmail.com > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > On Mon, Jun 12, 2017 at 2:26 PM, Arun Mohan < > > > > > > strider90a...@gmail.com > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > Hi All, > > > > > > > > > > > > > > > > > > > > > > Good afternoon. > > > > > > > > > > > > > > > > > > > > > > I have been working on a java generic parallel > execution > > > > > library > > > > > > > > which > > > > > > > > > > will > > > > > > > > > > > allow clients to execute methods in parallel > irrespective > > > of > > > > > the > > > > > > > > number > > > > > > > > > > of > > > > > > > > > > > method arguments, type of method arguments, return type > > of > > > > the > > > > > > > method > > > > > > > > > > etc. > > > > > > > > > > > > > > > > > > > > > > Here is the link to the source code: > > > > > > > > > > > https://github.com/striderarun/parallel- > execution-engine > > > > > > > > > > > > > > > > > > > > > > The project is in a nascent state and I am the only > > > > contributor > > > > > > so > > > > > > > > > far. I > > > > > > > > > > > am new to the Apache community and I would like to > bring > > > this > > > > > > > project > > > > > > > > > > into > > > > > > > > > > > Apache and improve, expand and build a developer > > community > > > > > around > > > > > > > it. > > > > > > > > > > > > > > > > > > > > > > I think this project can be a sub project of Apache > > Commons > > > > > since > > > > > > > it > > > > > > > > > > > provides generic components for parallelizing any kind > of > > > > > > methods. > > > > > > > > > > > > > > > > > > > > > > Can somebody please guide me or suggest what other > > options > > > I > > > > > can > > > > > > > > > explore > > > > > > > > > > ? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi Arun, > > > > > > > > > > > > > > > > > > > > Thank you for your proposal. > > > > > > > > > > > > > > > > > > > > How would this be different from Apache Spark? > > > > > > > > > > > > > > > > > > > > Thank you, > > > > > > > > > > Gary > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > Arun > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > Matt Sicker <boa...@gmail.com> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > Matt Sicker <boa...@gmail.com> > > > > > > > > > -- > Matt Sicker <boa...@gmail.com> >
