Re: Will you accept contributions in Scala?

2018-03-16 Thread Otto Fowler
Sorry for the late response, but the point about the Extension Repo making
this a moot point is an important one.
The idea that you need to get your processor into the main codebase to have
it be known and useful is something
that is easier to overcome than a language war.


On February 13, 2018 at 12:15:23, Russell Bateman (r...@windofkeltia.com)
wrote:

I completely second Mike's sentiment here as well as cautionary
statements by other contributors over this thread's history (Matt
Burgess' post in particular). Besides fueling flame wars and religious
inquisition, you cannot open the door to Scala without opening it to
other (JVM languages at least) and there's no umbrella that will shield
you from the chaos doing so will unleash rendering NiFi difficult to
develop to (and maybe even to deploy). As it is presently, all
JVM-language developers are able to consume NiFi easily for development
and to deal with their own deployment issues.

Russ

On 02/13/2018 10:06 AM, Mike Thomsen wrote:
> Milan,
>
> I don't think you can do that without creating a lot of fuel for a flame
> war. I have personally never met a Scala developer who was incapable of
> writing decent Java. The same is true of Groovy. I think anyone who finds
> the requirement to use Java over their language of choice to be a deal
> breaker on contributing is probably someone unlikely to be more help than
> trouble anyway.
>
> Mike
>
> On Tue, Feb 13, 2018 at 11:35 AM, Milan Das  wrote:
>
>> I think we should not add blindly any language but should be open to add
>> couple of language like Scala.
>> In Bigdata world Scala/Python/Java are widely accepted.
>>
>> Thanks,
>> Milan Das
>> Interset
>>
>> On 2/13/18, 10:20 AM, "Weiss, Adam"  wrote:
>>
>> I think it makes the most sense to me for us to publish a separate
>> repo with a module and nar build for now and post when it's available in
>> the users group.
>>
>> Thanks for the discussion everyone, hopefully we can start making some
>> helpful contributions soon.
>>
>> -Adam
>>
>>
>> On 2018/02/10 23:43:31, Tony Kurc > gmail.com>> wrote:
>> > It is like Matt read my mind.>
>> >
>> > On Sat, Feb 10, 2018 at 6:26 PM, Matt Burgess > > wrote:>
>> >
>> > > I'm fine with a vote, but I'll be voting to keep Java as the
>> single>
>> > > language for the (non-test) code. I share the same concerns as
>> many of>
>> > > the other folks as far as accepting other languages, it's mainly
>> the>
>> > > "slippery slope" argument that I don't want to turn into a>
>> > > JVM-language flame war. If Scala, why not Groovy? Certainly the>
>> > > syntax is closer to Java, and the community has accepted it as a
>> valid>
>> > > language for writing unit tests, although we stopped short for>
>> > > allowing it for the deployable NiFi codebase, for the same reasons>
>> > > IIRC. If Scala and/or Groovy, why not Kotlin? The same argument>
>> > > (albeit more tenuous) goes for Clojure and just about every other
>> JVM>
>> > > language (although I don't expect a call for LuaJ processors lol).>
>> > >>
>> > > Whether we decide to support various languages ad-hoc or not, I
>> would>
>> > > strenuously object to multiple/hybrid build systems for the
>> deployed>
>> > > artifacts. If I could switch NiFi completely to Gradle I would,
>> but I>
>> > > realize there are good reasons for not doing so (yet?) in the
>> Apache>
>> > > NiFi community, and I would never want any hybrid Maven/Gradle
>> build>
>> > > for the deployable code, likewise for SBT, Leiningen, etc. With a>
>> > > custom Mojo for Maven NAR builds, and the complexity for hybrid
>> builds>
>> > > in general, I think this would create a maintenance nightmare.>
>> > >>
>> > > The language thing is a tough decision though, it's not awesome
>> that>
>> > > specifying a single language can be a barrier to a more diverse>
>> > > community, certainly Scala-based bundles would be more than
>> welcome in>
>> > > the overall NiFi ecosystem, I just think the cons outweigh the
>> pros>
>> > > for the baseline code. I've written Groovy processors/NARs using>
>> > > Gradle as the build system, and I'm good with keeping them in my
>> own>
>> > > repo, especially when the Extension Registry becomes a thing. I
>> can>
>> > > see the Extension Registry perhaps making this a moot point, but>
>> > > clearly we need to have this discussion in the meantime.>
>> > >>
>> > > Regards,>
>> > > Matt>
>> > >>
>> > >>
>> > > On Sat, Feb 10, 2018 at 5:23 PM, Andrew Grande > > wrote:>
>> > > > Wasn't there a warning trigger about the NiFi distro size from
>> Apache>
>> > > > recently? IMO, before talking alternative languages, solve the
>> modularity>
>> > > > and NAR distribution problem. I think the implementation of a
>> module>
>> > > won't>
>> > > > matter much then, the point being not everything has to go in
>> the core,>
>> > > > 

Re: Will you accept contributions in Scala?

2018-02-13 Thread Russell Bateman
I completely second Mike's sentiment here as well as cautionary 
statements by other contributors over this thread's history (Matt 
Burgess' post in particular). Besides fueling flame wars and religious 
inquisition, you cannot open the door to Scala without opening it to 
other (JVM languages at least) and there's no umbrella that will shield 
you from the chaos doing so will unleash rendering NiFi difficult to 
develop to (and maybe even to deploy). As it is presently, all 
JVM-language developers are able to consume NiFi easily for development 
and to deal with their own deployment issues.


Russ

On 02/13/2018 10:06 AM, Mike Thomsen wrote:

Milan,

I don't think you can do that without creating a lot of fuel for a flame
war. I have personally never met a Scala developer who was incapable of
writing decent Java. The same is true of Groovy. I think anyone who finds
the requirement to use Java over their language of choice to be a deal
breaker on contributing is probably someone unlikely to be more help than
trouble anyway.

Mike

On Tue, Feb 13, 2018 at 11:35 AM, Milan Das  wrote:


I think we should not add blindly any language but should be open to add
couple of language like Scala.
In Bigdata world Scala/Python/Java are widely accepted.

Thanks,
Milan Das
Interset

On 2/13/18, 10:20 AM, "Weiss, Adam"  wrote:

 I think it makes the most sense to me for us to publish a separate
repo with a module and nar build for now and post when it's available in
the users group.

 Thanks for the discussion everyone, hopefully we can start making some
helpful contributions soon.

 -Adam


 On 2018/02/10 23:43:31, Tony Kurc > wrote:
 > It is like Matt read my mind.>
 >
 > On Sat, Feb 10, 2018 at 6:26 PM, Matt Burgess > wrote:>
 >
 > > I'm fine with a vote, but I'll be voting to keep Java as the
single>
 > > language for the (non-test) code. I share the same concerns as
many of>
 > > the other folks as far as accepting other languages, it's mainly
the>
 > > "slippery slope" argument that I don't want to turn into a>
 > > JVM-language flame war.  If Scala, why not Groovy? Certainly the>
 > > syntax is closer to Java, and the community has accepted it as a
valid>
 > > language for writing unit tests, although we stopped short for>
 > > allowing it for the deployable NiFi codebase, for the same reasons>
 > > IIRC.  If Scala and/or Groovy, why not Kotlin?  The same argument>
 > > (albeit more tenuous) goes for Clojure and just about every other
JVM>
 > > language (although I don't expect a call for LuaJ processors lol).>
 > >>
 > > Whether we decide to support various languages ad-hoc or not, I
would>
 > > strenuously object to multiple/hybrid build systems for the
deployed>
 > > artifacts. If I could switch NiFi completely to Gradle I would,
but I>
 > > realize there are good reasons for not doing so (yet?) in the
Apache>
 > > NiFi community, and I would never want any hybrid Maven/Gradle
build>
 > > for the deployable code, likewise for SBT, Leiningen, etc. With a>
 > > custom Mojo for Maven NAR builds, and the complexity for hybrid
builds>
 > > in general, I think this would create a maintenance nightmare.>
 > >>
 > > The language thing is a tough decision though, it's not awesome
that>
 > > specifying a single language can be a barrier to a more diverse>
 > > community, certainly Scala-based bundles would be more than
welcome in>
 > > the overall NiFi ecosystem, I just think the cons outweigh the
pros>
 > > for the baseline code. I've written Groovy processors/NARs using>
 > > Gradle as the build system, and I'm good with keeping them in my
own>
 > > repo, especially when the Extension Registry becomes a thing. I
can>
 > > see the Extension Registry perhaps making this a moot point, but>
 > > clearly we need to have this discussion in the meantime.>
 > >>
 > > Regards,>
 > > Matt>
 > >>
 > >>
 > > On Sat, Feb 10, 2018 at 5:23 PM, Andrew Grande > wrote:>
 > > > Wasn't there a warning trigger about the NiFi distro size from
Apache>
 > > > recently? IMO, before talking alternative languages, solve the
modularity>
 > > > and NAR distribution problem. I think the implementation of a
module>
 > > won't>
 > > > matter much then, the point being not everything has to go in
the core,>
 > > > base distribution, but can still be easily sourced from a known
repo, for>
 > > > example.>
 > > >>
 > > > I have a feeling NiFi 1.6+ can be approaching 2GB distro size
soon :)>
 > > >>
 > > > Andrew>
 > > >>
 > > > On Sat, Feb 10, 2018, 5:12 PM Joey Frazee >>
 > > wrote:>
 > > >>
 > > >> This 

Re: Will you accept contributions in Scala?

2018-02-13 Thread Mike Thomsen
Milan,

I don't think you can do that without creating a lot of fuel for a flame
war. I have personally never met a Scala developer who was incapable of
writing decent Java. The same is true of Groovy. I think anyone who finds
the requirement to use Java over their language of choice to be a deal
breaker on contributing is probably someone unlikely to be more help than
trouble anyway.

Mike

On Tue, Feb 13, 2018 at 11:35 AM, Milan Das  wrote:

> I think we should not add blindly any language but should be open to add
> couple of language like Scala.
> In Bigdata world Scala/Python/Java are widely accepted.
>
> Thanks,
> Milan Das
> Interset
>
> On 2/13/18, 10:20 AM, "Weiss, Adam"  wrote:
>
> I think it makes the most sense to me for us to publish a separate
> repo with a module and nar build for now and post when it's available in
> the users group.
>
> Thanks for the discussion everyone, hopefully we can start making some
> helpful contributions soon.
>
> -Adam
>
>
> On 2018/02/10 23:43:31, Tony Kurc  gmail.com>> wrote:
> > It is like Matt read my mind.>
> >
> > On Sat, Feb 10, 2018 at 6:26 PM, Matt Burgess  > wrote:>
> >
> > > I'm fine with a vote, but I'll be voting to keep Java as the
> single>
> > > language for the (non-test) code. I share the same concerns as
> many of>
> > > the other folks as far as accepting other languages, it's mainly
> the>
> > > "slippery slope" argument that I don't want to turn into a>
> > > JVM-language flame war.  If Scala, why not Groovy? Certainly the>
> > > syntax is closer to Java, and the community has accepted it as a
> valid>
> > > language for writing unit tests, although we stopped short for>
> > > allowing it for the deployable NiFi codebase, for the same reasons>
> > > IIRC.  If Scala and/or Groovy, why not Kotlin?  The same argument>
> > > (albeit more tenuous) goes for Clojure and just about every other
> JVM>
> > > language (although I don't expect a call for LuaJ processors lol).>
> > >>
> > > Whether we decide to support various languages ad-hoc or not, I
> would>
> > > strenuously object to multiple/hybrid build systems for the
> deployed>
> > > artifacts. If I could switch NiFi completely to Gradle I would,
> but I>
> > > realize there are good reasons for not doing so (yet?) in the
> Apache>
> > > NiFi community, and I would never want any hybrid Maven/Gradle
> build>
> > > for the deployable code, likewise for SBT, Leiningen, etc. With a>
> > > custom Mojo for Maven NAR builds, and the complexity for hybrid
> builds>
> > > in general, I think this would create a maintenance nightmare.>
> > >>
> > > The language thing is a tough decision though, it's not awesome
> that>
> > > specifying a single language can be a barrier to a more diverse>
> > > community, certainly Scala-based bundles would be more than
> welcome in>
> > > the overall NiFi ecosystem, I just think the cons outweigh the
> pros>
> > > for the baseline code. I've written Groovy processors/NARs using>
> > > Gradle as the build system, and I'm good with keeping them in my
> own>
> > > repo, especially when the Extension Registry becomes a thing. I
> can>
> > > see the Extension Registry perhaps making this a moot point, but>
> > > clearly we need to have this discussion in the meantime.>
> > >>
> > > Regards,>
> > > Matt>
> > >>
> > >>
> > > On Sat, Feb 10, 2018 at 5:23 PM, Andrew Grande  > wrote:>
> > > > Wasn't there a warning trigger about the NiFi distro size from
> Apache>
> > > > recently? IMO, before talking alternative languages, solve the
> modularity>
> > > > and NAR distribution problem. I think the implementation of a
> module>
> > > won't>
> > > > matter much then, the point being not everything has to go in
> the core,>
> > > > base distribution, but can still be easily sourced from a known
> repo, for>
> > > > example.>
> > > >>
> > > > I have a feeling NiFi 1.6+ can be approaching 2GB distro size
> soon :)>
> > > >>
> > > > Andrew>
> > > >>
> > > > On Sat, Feb 10, 2018, 5:12 PM Joey Frazee  >>
> > > wrote:>
> > > >>
> > > >> This probably necessitates a vote, yeah?>
> > > >>>
> > > >> Frankly, I’m usually happier writing Scala, and I’ve not
> encountered any>
> > > >> problems using processors written in Scala, but I think it’ll
> be>
> > > important>
> > > >> to tread lightly.>
> > > >>>
> > > >> There’s a few things that pop into my head:>
> > > >>>
> > > >> - Maintainability and reviewability. A very very good Java
> developer>
> > > need>
> > > >> not, by definition, either know how to 

Re: Will you accept contributions in Scala?

2018-02-13 Thread Milan Das
I think we should not add blindly any language but should be open to add couple 
of language like Scala.
In Bigdata world Scala/Python/Java are widely accepted.

Thanks,
Milan Das
Interset 

On 2/13/18, 10:20 AM, "Weiss, Adam"  wrote:

I think it makes the most sense to me for us to publish a separate repo 
with a module and nar build for now and post when it's available in the users 
group.

Thanks for the discussion everyone, hopefully we can start making some 
helpful contributions soon.

-Adam


On 2018/02/10 23:43:31, Tony Kurc > 
wrote:
> It is like Matt read my mind.>
>
> On Sat, Feb 10, 2018 at 6:26 PM, Matt Burgess 
> wrote:>
>
> > I'm fine with a vote, but I'll be voting to keep Java as the single>
> > language for the (non-test) code. I share the same concerns as many of>
> > the other folks as far as accepting other languages, it's mainly the>
> > "slippery slope" argument that I don't want to turn into a>
> > JVM-language flame war.  If Scala, why not Groovy? Certainly the>
> > syntax is closer to Java, and the community has accepted it as a valid>
> > language for writing unit tests, although we stopped short for>
> > allowing it for the deployable NiFi codebase, for the same reasons>
> > IIRC.  If Scala and/or Groovy, why not Kotlin?  The same argument>
> > (albeit more tenuous) goes for Clojure and just about every other JVM>
> > language (although I don't expect a call for LuaJ processors lol).>
> >>
> > Whether we decide to support various languages ad-hoc or not, I would>
> > strenuously object to multiple/hybrid build systems for the deployed>
> > artifacts. If I could switch NiFi completely to Gradle I would, but I>
> > realize there are good reasons for not doing so (yet?) in the Apache>
> > NiFi community, and I would never want any hybrid Maven/Gradle build>
> > for the deployable code, likewise for SBT, Leiningen, etc. With a>
> > custom Mojo for Maven NAR builds, and the complexity for hybrid builds>
> > in general, I think this would create a maintenance nightmare.>
> >>
> > The language thing is a tough decision though, it's not awesome that>
> > specifying a single language can be a barrier to a more diverse>
> > community, certainly Scala-based bundles would be more than welcome in>
> > the overall NiFi ecosystem, I just think the cons outweigh the pros>
> > for the baseline code. I've written Groovy processors/NARs using>
> > Gradle as the build system, and I'm good with keeping them in my own>
> > repo, especially when the Extension Registry becomes a thing. I can>
> > see the Extension Registry perhaps making this a moot point, but>
> > clearly we need to have this discussion in the meantime.>
> >>
> > Regards,>
> > Matt>
> >>
> >>
> > On Sat, Feb 10, 2018 at 5:23 PM, Andrew Grande 
> wrote:>
> > > Wasn't there a warning trigger about the NiFi distro size from Apache>
> > > recently? IMO, before talking alternative languages, solve the 
modularity>
> > > and NAR distribution problem. I think the implementation of a module>
> > won't>
> > > matter much then, the point being not everything has to go in the 
core,>
> > > base distribution, but can still be easily sourced from a known repo, 
for>
> > > example.>
> > >>
> > > I have a feeling NiFi 1.6+ can be approaching 2GB distro size soon :)>
> > >>
> > > Andrew>
> > >>
> > > On Sat, Feb 10, 2018, 5:12 PM Joey Frazee 
>>
> > wrote:>
> > >>
> > >> This probably necessitates a vote, yeah?>
> > >>>
> > >> Frankly, I’m usually happier writing Scala, and I’ve not encountered 
any>
> > >> problems using processors written in Scala, but I think it’ll be>
> > important>
> > >> to tread lightly.>
> > >>>
> > >> There’s a few things that pop into my head:>
> > >>>
> > >> - Maintainability and reviewability. A very very good Java developer>
> > need>
> > >> not, by definition, either know how to write or identify good Scala 
or>
> > spot>
> > >> problems and bugs.>
> > >> - Every Scala processor would either end up with a 5MB 
scala-lang.jar>
> > >> packaged into the .nar or we’d have to start including it in the 
core>
> > >> somewhere, if it’s not. It’s possible it might have already gotten>
> > pulled>
> > >> up from other dependencies.>
> > >> - Style. There’s a tremendous amount of variation in Scala style 
because>
> > >> of its type system, implicits, macros, and functional nature. There 
are>
> > >> very good people out there that can write good Scala that isn’t>
> > readable by>
> > >> the 99%.>

Re: Will you accept contributions in Scala?

2018-02-13 Thread Weiss, Adam
I think it makes the most sense to me for us to publish a separate repo with a 
module and nar build for now and post when it's available in the users group.

Thanks for the discussion everyone, hopefully we can start making some helpful 
contributions soon.

-Adam


On 2018/02/10 23:43:31, Tony Kurc > wrote:
> It is like Matt read my mind.>
>
> On Sat, Feb 10, 2018 at 6:26 PM, Matt Burgess 
> > wrote:>
>
> > I'm fine with a vote, but I'll be voting to keep Java as the single>
> > language for the (non-test) code. I share the same concerns as many of>
> > the other folks as far as accepting other languages, it's mainly the>
> > "slippery slope" argument that I don't want to turn into a>
> > JVM-language flame war.  If Scala, why not Groovy? Certainly the>
> > syntax is closer to Java, and the community has accepted it as a valid>
> > language for writing unit tests, although we stopped short for>
> > allowing it for the deployable NiFi codebase, for the same reasons>
> > IIRC.  If Scala and/or Groovy, why not Kotlin?  The same argument>
> > (albeit more tenuous) goes for Clojure and just about every other JVM>
> > language (although I don't expect a call for LuaJ processors lol).>
> >>
> > Whether we decide to support various languages ad-hoc or not, I would>
> > strenuously object to multiple/hybrid build systems for the deployed>
> > artifacts. If I could switch NiFi completely to Gradle I would, but I>
> > realize there are good reasons for not doing so (yet?) in the Apache>
> > NiFi community, and I would never want any hybrid Maven/Gradle build>
> > for the deployable code, likewise for SBT, Leiningen, etc. With a>
> > custom Mojo for Maven NAR builds, and the complexity for hybrid builds>
> > in general, I think this would create a maintenance nightmare.>
> >>
> > The language thing is a tough decision though, it's not awesome that>
> > specifying a single language can be a barrier to a more diverse>
> > community, certainly Scala-based bundles would be more than welcome in>
> > the overall NiFi ecosystem, I just think the cons outweigh the pros>
> > for the baseline code. I've written Groovy processors/NARs using>
> > Gradle as the build system, and I'm good with keeping them in my own>
> > repo, especially when the Extension Registry becomes a thing. I can>
> > see the Extension Registry perhaps making this a moot point, but>
> > clearly we need to have this discussion in the meantime.>
> >>
> > Regards,>
> > Matt>
> >>
> >>
> > On Sat, Feb 10, 2018 at 5:23 PM, Andrew Grande 
> > > wrote:>
> > > Wasn't there a warning trigger about the NiFi distro size from Apache>
> > > recently? IMO, before talking alternative languages, solve the modularity>
> > > and NAR distribution problem. I think the implementation of a module>
> > won't>
> > > matter much then, the point being not everything has to go in the core,>
> > > base distribution, but can still be easily sourced from a known repo, for>
> > > example.>
> > >>
> > > I have a feeling NiFi 1.6+ can be approaching 2GB distro size soon :)>
> > >>
> > > Andrew>
> > >>
> > > On Sat, Feb 10, 2018, 5:12 PM Joey Frazee 
> > > >>
> > wrote:>
> > >>
> > >> This probably necessitates a vote, yeah?>
> > >>>
> > >> Frankly, I’m usually happier writing Scala, and I’ve not encountered any>
> > >> problems using processors written in Scala, but I think it’ll be>
> > important>
> > >> to tread lightly.>
> > >>>
> > >> There’s a few things that pop into my head:>
> > >>>
> > >> - Maintainability and reviewability. A very very good Java developer>
> > need>
> > >> not, by definition, either know how to write or identify good Scala or>
> > spot>
> > >> problems and bugs.>
> > >> - Every Scala processor would either end up with a 5MB scala-lang.jar>
> > >> packaged into the .nar or we’d have to start including it in the core>
> > >> somewhere, if it’s not. It’s possible it might have already gotten>
> > pulled>
> > >> up from other dependencies.>
> > >> - Style. There’s a tremendous amount of variation in Scala style because>
> > >> of its type system, implicits, macros, and functional nature. There are>
> > >> very good people out there that can write good Scala that isn’t>
> > readable by>
> > >> the 99%.>
> > >> - Binary compatibility. Scala tends to be a little more brazen about>
> > >> breaking binary compatibility in major releases and those happen a bit>
> > more>
> > >> often than with Java. That’s not a problem for any potential source>
> > code in>
> > >> the project, but it could present some dependency issues someday.>
> > >> - Testing. There’s N > 1 test frameworks and testing styles within>
> > those,>
> > >> so there’s a lot of options for introducing more variability into the>
> > tests.>
> > >> - NiFi uses a lot of statics in setting up properties and relationships>
> > >> and the like, and 

Re: Will you accept contributions in Scala?

2018-02-10 Thread Tony Kurc
It is like Matt read my mind.

On Sat, Feb 10, 2018 at 6:26 PM, Matt Burgess  wrote:

> I'm fine with a vote, but I'll be voting to keep Java as the single
> language for the (non-test) code. I share the same concerns as many of
> the other folks as far as accepting other languages, it's mainly the
> "slippery slope" argument that I don't want to turn into a
> JVM-language flame war.  If Scala, why not Groovy? Certainly the
> syntax is closer to Java, and the community has accepted it as a valid
> language for writing unit tests, although we stopped short for
> allowing it for the deployable NiFi codebase, for the same reasons
> IIRC.  If Scala and/or Groovy, why not Kotlin?  The same argument
> (albeit more tenuous) goes for Clojure and just about every other JVM
> language (although I don't expect a call for LuaJ processors lol).
>
> Whether we decide to support various languages ad-hoc or not, I would
> strenuously object to multiple/hybrid build systems for the deployed
> artifacts. If I could switch NiFi completely to Gradle I would, but I
> realize there are good reasons for not doing so (yet?) in the Apache
> NiFi community, and I would never want any hybrid Maven/Gradle build
> for the deployable code, likewise for SBT, Leiningen, etc. With a
> custom Mojo for Maven NAR builds, and the complexity for hybrid builds
> in general, I think this would create a maintenance nightmare.
>
> The language thing is a tough decision though, it's not awesome that
> specifying a single language can be a barrier to a more diverse
> community, certainly Scala-based bundles would be more than welcome in
> the overall NiFi ecosystem, I just think the cons outweigh the pros
> for the baseline code. I've written Groovy processors/NARs using
> Gradle as the build system, and I'm good with keeping them in my own
> repo, especially when the Extension Registry becomes a thing. I can
> see the Extension Registry perhaps making this a moot point, but
> clearly we need to have this discussion in the meantime.
>
> Regards,
> Matt
>
>
> On Sat, Feb 10, 2018 at 5:23 PM, Andrew Grande  wrote:
> > Wasn't there a warning trigger about the NiFi distro size from Apache
> > recently? IMO, before talking alternative languages, solve the modularity
> > and NAR distribution problem. I think the implementation of a module
> won't
> > matter much then, the point being not everything has to go in the core,
> > base distribution, but can still be easily sourced from a known repo, for
> > example.
> >
> > I have a feeling NiFi 1.6+ can be approaching 2GB distro size soon :)
> >
> > Andrew
> >
> > On Sat, Feb 10, 2018, 5:12 PM Joey Frazee 
> wrote:
> >
> >> This probably necessitates a vote, yeah?
> >>
> >> Frankly, I’m usually happier writing Scala, and I’ve not encountered any
> >> problems using processors written in Scala, but I think it’ll be
> important
> >> to tread lightly.
> >>
> >> There’s a few things that pop into my head:
> >>
> >> - Maintainability and reviewability. A very very good Java developer
> need
> >> not, by definition, either know how to write or identify good Scala or
> spot
> >> problems and bugs.
> >> - Every Scala processor would either end up with a 5MB scala-lang.jar
> >> packaged into the .nar or we’d have to start including it in the core
> >> somewhere, if it’s not. It’s possible it might have already gotten
> pulled
> >> up from other dependencies.
> >> - Style. There’s a tremendous amount of variation in Scala style because
> >> of its type system, implicits, macros, and functional nature. There are
> >> very good people out there that can write good Scala that isn’t
> readable by
> >> the 99%.
> >> - Binary compatibility. Scala tends to be a little more brazen about
> >> breaking binary compatibility in major releases and those happen a bit
> more
> >> often than with Java. That’s not a problem for any potential source
> code in
> >> the project, but it could present some dependency issues someday.
> >> - Testing. There’s N > 1 test frameworks and testing styles within
> those,
> >> so there’s a lot of options for introducing more variability into the
> tests.
> >> - NiFi uses a lot of statics in setting up properties and relationships
> >> and the like, and idiomatic Scala approaches that stuff a bit
> differently,
> >> so it’ll be necessary to impose some style guidelines so there isn’t too
> >> much variation.
> >>
> >> That said, there are some things that won’t be problematic:
> >>
> >> - As mentioned, processors written in Scala do just work.
> >> - The scala-maven-plugin works just fine allowing mixed Java-Scala
> >> projects (btw, it’d probably be not super great to do mixed Java-Scala
> and
> >> mixed Maven-SBT though).
> >> - A lot of the above concerns could be addressed by having clear style
> >> guidelines.
> >>
> >> Another thing: most of the projects I see deliver separate jars for
> Scala
> >> components are delivering 

Re: Will you accept contributions in Scala?

2018-02-10 Thread Matt Burgess
I'm fine with a vote, but I'll be voting to keep Java as the single
language for the (non-test) code. I share the same concerns as many of
the other folks as far as accepting other languages, it's mainly the
"slippery slope" argument that I don't want to turn into a
JVM-language flame war.  If Scala, why not Groovy? Certainly the
syntax is closer to Java, and the community has accepted it as a valid
language for writing unit tests, although we stopped short for
allowing it for the deployable NiFi codebase, for the same reasons
IIRC.  If Scala and/or Groovy, why not Kotlin?  The same argument
(albeit more tenuous) goes for Clojure and just about every other JVM
language (although I don't expect a call for LuaJ processors lol).

Whether we decide to support various languages ad-hoc or not, I would
strenuously object to multiple/hybrid build systems for the deployed
artifacts. If I could switch NiFi completely to Gradle I would, but I
realize there are good reasons for not doing so (yet?) in the Apache
NiFi community, and I would never want any hybrid Maven/Gradle build
for the deployable code, likewise for SBT, Leiningen, etc. With a
custom Mojo for Maven NAR builds, and the complexity for hybrid builds
in general, I think this would create a maintenance nightmare.

The language thing is a tough decision though, it's not awesome that
specifying a single language can be a barrier to a more diverse
community, certainly Scala-based bundles would be more than welcome in
the overall NiFi ecosystem, I just think the cons outweigh the pros
for the baseline code. I've written Groovy processors/NARs using
Gradle as the build system, and I'm good with keeping them in my own
repo, especially when the Extension Registry becomes a thing. I can
see the Extension Registry perhaps making this a moot point, but
clearly we need to have this discussion in the meantime.

Regards,
Matt


On Sat, Feb 10, 2018 at 5:23 PM, Andrew Grande  wrote:
> Wasn't there a warning trigger about the NiFi distro size from Apache
> recently? IMO, before talking alternative languages, solve the modularity
> and NAR distribution problem. I think the implementation of a module won't
> matter much then, the point being not everything has to go in the core,
> base distribution, but can still be easily sourced from a known repo, for
> example.
>
> I have a feeling NiFi 1.6+ can be approaching 2GB distro size soon :)
>
> Andrew
>
> On Sat, Feb 10, 2018, 5:12 PM Joey Frazee  wrote:
>
>> This probably necessitates a vote, yeah?
>>
>> Frankly, I’m usually happier writing Scala, and I’ve not encountered any
>> problems using processors written in Scala, but I think it’ll be important
>> to tread lightly.
>>
>> There’s a few things that pop into my head:
>>
>> - Maintainability and reviewability. A very very good Java developer need
>> not, by definition, either know how to write or identify good Scala or spot
>> problems and bugs.
>> - Every Scala processor would either end up with a 5MB scala-lang.jar
>> packaged into the .nar or we’d have to start including it in the core
>> somewhere, if it’s not. It’s possible it might have already gotten pulled
>> up from other dependencies.
>> - Style. There’s a tremendous amount of variation in Scala style because
>> of its type system, implicits, macros, and functional nature. There are
>> very good people out there that can write good Scala that isn’t readable by
>> the 99%.
>> - Binary compatibility. Scala tends to be a little more brazen about
>> breaking binary compatibility in major releases and those happen a bit more
>> often than with Java. That’s not a problem for any potential source code in
>> the project, but it could present some dependency issues someday.
>> - Testing. There’s N > 1 test frameworks and testing styles within those,
>> so there’s a lot of options for introducing more variability into the tests.
>> - NiFi uses a lot of statics in setting up properties and relationships
>> and the like, and idiomatic Scala approaches that stuff a bit differently,
>> so it’ll be necessary to impose some style guidelines so there isn’t too
>> much variation.
>>
>> That said, there are some things that won’t be problematic:
>>
>> - As mentioned, processors written in Scala do just work.
>> - The scala-maven-plugin works just fine allowing mixed Java-Scala
>> projects (btw, it’d probably be not super great to do mixed Java-Scala and
>> mixed Maven-SBT though).
>> - A lot of the above concerns could be addressed by having clear style
>> guidelines.
>>
>> Another thing: most of the projects I see deliver separate jars for Scala
>> components are delivering idiomatic APIs wrapping Java (or vice versa). I
>> think publishing   a separate set of jars/nars for stuff written in Scala
>> would be odd since here it’d mostly be processors with new functionality
>> and not functionality for using Scala. I could imagine a lib of implicits,
>> traits, classes that could make the 

Re: Will you accept contributions in Scala?

2018-02-10 Thread Andrew Grande
Wasn't there a warning trigger about the NiFi distro size from Apache
recently? IMO, before talking alternative languages, solve the modularity
and NAR distribution problem. I think the implementation of a module won't
matter much then, the point being not everything has to go in the core,
base distribution, but can still be easily sourced from a known repo, for
example.

I have a feeling NiFi 1.6+ can be approaching 2GB distro size soon :)

Andrew

On Sat, Feb 10, 2018, 5:12 PM Joey Frazee  wrote:

> This probably necessitates a vote, yeah?
>
> Frankly, I’m usually happier writing Scala, and I’ve not encountered any
> problems using processors written in Scala, but I think it’ll be important
> to tread lightly.
>
> There’s a few things that pop into my head:
>
> - Maintainability and reviewability. A very very good Java developer need
> not, by definition, either know how to write or identify good Scala or spot
> problems and bugs.
> - Every Scala processor would either end up with a 5MB scala-lang.jar
> packaged into the .nar or we’d have to start including it in the core
> somewhere, if it’s not. It’s possible it might have already gotten pulled
> up from other dependencies.
> - Style. There’s a tremendous amount of variation in Scala style because
> of its type system, implicits, macros, and functional nature. There are
> very good people out there that can write good Scala that isn’t readable by
> the 99%.
> - Binary compatibility. Scala tends to be a little more brazen about
> breaking binary compatibility in major releases and those happen a bit more
> often than with Java. That’s not a problem for any potential source code in
> the project, but it could present some dependency issues someday.
> - Testing. There’s N > 1 test frameworks and testing styles within those,
> so there’s a lot of options for introducing more variability into the tests.
> - NiFi uses a lot of statics in setting up properties and relationships
> and the like, and idiomatic Scala approaches that stuff a bit differently,
> so it’ll be necessary to impose some style guidelines so there isn’t too
> much variation.
>
> That said, there are some things that won’t be problematic:
>
> - As mentioned, processors written in Scala do just work.
> - The scala-maven-plugin works just fine allowing mixed Java-Scala
> projects (btw, it’d probably be not super great to do mixed Java-Scala and
> mixed Maven-SBT though).
> - A lot of the above concerns could be addressed by having clear style
> guidelines.
>
> Another thing: most of the projects I see deliver separate jars for Scala
> components are delivering idiomatic APIs wrapping Java (or vice versa). I
> think publishing   a separate set of jars/nars for stuff written in Scala
> would be odd since here it’d mostly be processors with new functionality
> and not functionality for using Scala. I could imagine a lib of implicits,
> traits, classes that could make the Scala development more enjoyable. That
> probably would make sense to deliver that way.
>
> -joey
>
> On Feb 10, 2018, 10:33 AM -0600, Bryan Bende , wrote:
> > I agree more with Andy about sticking with Java. The more varying
> languages
> > used, the more challenging it is to maintain. Once the code is part of
> the
> > Apache NiFi git repo, it is now the responsibility of the committers and
> > PMC members to maintain it.
> >
> > I’d even say I am somewhat against the groovy/Spock test code that Andy
> > mentioned. I have frequently spent hours trying to fix a Spock test that
> > broke from something I was working on. Every committer is familiar with
> > JUnit, but only a couple know Spock. Just using this as an example that
> > every committer knows Java, but only a couple probably know Scala,
> Clojure,
> > etc.
> >
> > On Sat, Feb 10, 2018 at 10:25 AM Jeff  wrote:
> >
> > > +1 to Joe's response. If you can develop a component in Groovy or Scala
> > > (or Clojure!) more quickly/comfortably, or if allowing components
> written
> > > in other languages would encourage people to contribute more, I'm all
> for
> > > it.
> > >
> > > On Sat, Feb 10, 2018 at 7:42 AM Joe Witt  wrote:
> > >
> > > > i personally would be ok with it for an extension/processor provided
> it
> > > > integrates well with the build.
> > > >
> > > > i would agree with andys view for core framework stuff but for
> > > extensions i
> > > > think we can do it like mikethomsen suggested.
> > > >
> > > > others?
> > > >
> > > > thanks
> > > > joe
> > > >
> > > > On Feb 10, 2018 7:30 AM, "Mike Thomsen" 
> wrote:
> > > >
> > > > > I'm just a community contributor, so take that FWIW, but a
> compromise
> > > > might
> > > > > be to publish the Scala code as separate maven modules to maven
> central
> > > > and
> > > > > then submit a thoroughly tested processor written in Java. As long
> as
> > > you
> > > > > have enough unit and integration tests to give 

Re: Will you accept contributions in Scala?

2018-02-10 Thread Joey Frazee
This probably necessitates a vote, yeah?

Frankly, I’m usually happier writing Scala, and I’ve not encountered any 
problems using processors written in Scala, but I think it’ll be important to 
tread lightly.

There’s a few things that pop into my head:

- Maintainability and reviewability. A very very good Java developer need not, 
by definition, either know how to write or identify good Scala or spot problems 
and bugs.
- Every Scala processor would either end up with a 5MB scala-lang.jar packaged 
into the .nar or we’d have to start including it in the core somewhere, if it’s 
not. It’s possible it might have already gotten pulled up from other 
dependencies.
- Style. There’s a tremendous amount of variation in Scala style because of its 
type system, implicits, macros, and functional nature. There are very good 
people out there that can write good Scala that isn’t readable by the 99%.
- Binary compatibility. Scala tends to be a little more brazen about breaking 
binary compatibility in major releases and those happen a bit more often than 
with Java. That’s not a problem for any potential source code in the project, 
but it could present some dependency issues someday.
- Testing. There’s N > 1 test frameworks and testing styles within those, so 
there’s a lot of options for introducing more variability into the tests.
- NiFi uses a lot of statics in setting up properties and relationships and the 
like, and idiomatic Scala approaches that stuff a bit differently, so it’ll be 
necessary to impose some style guidelines so there isn’t too much variation.

That said, there are some things that won’t be problematic:

- As mentioned, processors written in Scala do just work.
- The scala-maven-plugin works just fine allowing mixed Java-Scala projects 
(btw, it’d probably be not super great to do mixed Java-Scala and mixed 
Maven-SBT though).
- A lot of the above concerns could be addressed by having clear style 
guidelines.

Another thing: most of the projects I see deliver separate jars for Scala 
components are delivering idiomatic APIs wrapping Java (or vice versa). I think 
publishing   a separate set of jars/nars for stuff written in Scala would be 
odd since here it’d mostly be processors with new functionality and not 
functionality for using Scala. I could imagine a lib of implicits, traits, 
classes that could make the Scala development more enjoyable. That probably 
would make sense to deliver that way.

-joey

On Feb 10, 2018, 10:33 AM -0600, Bryan Bende , wrote:
> I agree more with Andy about sticking with Java. The more varying languages
> used, the more challenging it is to maintain. Once the code is part of the
> Apache NiFi git repo, it is now the responsibility of the committers and
> PMC members to maintain it.
>
> I’d even say I am somewhat against the groovy/Spock test code that Andy
> mentioned. I have frequently spent hours trying to fix a Spock test that
> broke from something I was working on. Every committer is familiar with
> JUnit, but only a couple know Spock. Just using this as an example that
> every committer knows Java, but only a couple probably know Scala, Clojure,
> etc.
>
> On Sat, Feb 10, 2018 at 10:25 AM Jeff  wrote:
>
> > +1 to Joe's response. If you can develop a component in Groovy or Scala
> > (or Clojure!) more quickly/comfortably, or if allowing components written
> > in other languages would encourage people to contribute more, I'm all for
> > it.
> >
> > On Sat, Feb 10, 2018 at 7:42 AM Joe Witt  wrote:
> >
> > > i personally would be ok with it for an extension/processor provided it
> > > integrates well with the build.
> > >
> > > i would agree with andys view for core framework stuff but for
> > extensions i
> > > think we can do it like mikethomsen suggested.
> > >
> > > others?
> > >
> > > thanks
> > > joe
> > >
> > > On Feb 10, 2018 7:30 AM, "Mike Thomsen"  wrote:
> > >
> > > > I'm just a community contributor, so take that FWIW, but a compromise
> > > might
> > > > be to publish the Scala code as separate maven modules to maven central
> > > and
> > > > then submit a thoroughly tested processor written in Java. As long as
> > you
> > > > have enough unit and integration tests to give strong coverage, I
> > > wouldn't
> > > > imagine anyone here would have issues reviewing it. If the tests fail
> > > > because of code issues in the external dependencies, the obvious answer
> > > is
> > > > to just hold the PR until the tests pass.
> > > >
> > > > On Fri, Feb 9, 2018 at 9:00 AM, Weiss, Adam <
> > adam.we...@perkinelmer.com
> > > > wrote:
> > > >
> > > > > Devs,
> > > > >
> > > > > I have some interns starting with my team and we use Scala internally
> > > for
> > > > > our work.
> > > > > If I wanted to have them work to contribute some new processors,
> > would
> > > > > they have to be Java to be included with the base distribution or
> > could
> > > > > they use Scala as 

Re: Will you accept contributions in Scala?

2018-02-10 Thread Bryan Bende
I agree more with Andy about sticking with Java. The more varying languages
used, the more challenging it is to maintain. Once the code is part of the
Apache NiFi git repo, it is now the responsibility of the committers and
PMC members to maintain it.

I’d even say I am somewhat against the groovy/Spock test code that Andy
mentioned. I have frequently spent hours trying to fix a Spock test that
broke from something I was working on. Every committer is familiar with
JUnit, but only a couple know Spock. Just using this as an example that
every committer knows Java, but only a couple probably know Scala, Clojure,
etc.

On Sat, Feb 10, 2018 at 10:25 AM Jeff  wrote:

> +1 to Joe's response.  If you can develop a component in Groovy or Scala
> (or Clojure!) more quickly/comfortably, or if allowing components written
> in other languages would encourage people to contribute more, I'm all for
> it.
>
> On Sat, Feb 10, 2018 at 7:42 AM Joe Witt  wrote:
>
> > i personally would be ok with it for an extension/processor provided it
> > integrates well with the build.
> >
> > i would agree with andys view for core framework stuff but for
> extensions i
> > think we can do it like mikethomsen suggested.
> >
> > others?
> >
> > thanks
> > joe
> >
> > On Feb 10, 2018 7:30 AM, "Mike Thomsen"  wrote:
> >
> > > I'm just a community contributor, so take that FWIW, but a compromise
> > might
> > > be to publish the Scala code as separate maven modules to maven central
> > and
> > > then submit a thoroughly tested processor written in Java. As long as
> you
> > > have enough unit and integration tests to give strong coverage, I
> > wouldn't
> > > imagine anyone here would have issues reviewing it. If the tests fail
> > > because of code issues in the external dependencies, the obvious answer
> > is
> > > to just hold the PR until the tests pass.
> > >
> > > On Fri, Feb 9, 2018 at 9:00 AM, Weiss, Adam <
> adam.we...@perkinelmer.com>
> > > wrote:
> > >
> > > > Devs,
> > > >
> > > > I have some interns starting with my team and we use Scala internally
> > for
> > > > our work.
> > > > If I wanted to have them work to contribute some new processors,
> would
> > > > they have to be Java to be included with the base distribution or
> could
> > > > they use Scala as long as it fit within your current build and test
> > > process?
> > > >
> > > > Thanks,
> > > > -Adam
> > > >
> > >
> >
>
-- 
Sent from Gmail Mobile


Re: Will you accept contributions in Scala?

2018-02-10 Thread Jeff
+1 to Joe's response.  If you can develop a component in Groovy or Scala
(or Clojure!) more quickly/comfortably, or if allowing components written
in other languages would encourage people to contribute more, I'm all for
it.

On Sat, Feb 10, 2018 at 7:42 AM Joe Witt  wrote:

> i personally would be ok with it for an extension/processor provided it
> integrates well with the build.
>
> i would agree with andys view for core framework stuff but for extensions i
> think we can do it like mikethomsen suggested.
>
> others?
>
> thanks
> joe
>
> On Feb 10, 2018 7:30 AM, "Mike Thomsen"  wrote:
>
> > I'm just a community contributor, so take that FWIW, but a compromise
> might
> > be to publish the Scala code as separate maven modules to maven central
> and
> > then submit a thoroughly tested processor written in Java. As long as you
> > have enough unit and integration tests to give strong coverage, I
> wouldn't
> > imagine anyone here would have issues reviewing it. If the tests fail
> > because of code issues in the external dependencies, the obvious answer
> is
> > to just hold the PR until the tests pass.
> >
> > On Fri, Feb 9, 2018 at 9:00 AM, Weiss, Adam 
> > wrote:
> >
> > > Devs,
> > >
> > > I have some interns starting with my team and we use Scala internally
> for
> > > our work.
> > > If I wanted to have them work to contribute some new processors, would
> > > they have to be Java to be included with the base distribution or could
> > > they use Scala as long as it fit within your current build and test
> > process?
> > >
> > > Thanks,
> > > -Adam
> > >
> >
>


Re: Will you accept contributions in Scala?

2018-02-10 Thread Joe Witt
i personally would be ok with it for an extension/processor provided it
integrates well with the build.

i would agree with andys view for core framework stuff but for extensions i
think we can do it like mikethomsen suggested.

others?

thanks
joe

On Feb 10, 2018 7:30 AM, "Mike Thomsen"  wrote:

> I'm just a community contributor, so take that FWIW, but a compromise might
> be to publish the Scala code as separate maven modules to maven central and
> then submit a thoroughly tested processor written in Java. As long as you
> have enough unit and integration tests to give strong coverage, I wouldn't
> imagine anyone here would have issues reviewing it. If the tests fail
> because of code issues in the external dependencies, the obvious answer is
> to just hold the PR until the tests pass.
>
> On Fri, Feb 9, 2018 at 9:00 AM, Weiss, Adam 
> wrote:
>
> > Devs,
> >
> > I have some interns starting with my team and we use Scala internally for
> > our work.
> > If I wanted to have them work to contribute some new processors, would
> > they have to be Java to be included with the base distribution or could
> > they use Scala as long as it fit within your current build and test
> process?
> >
> > Thanks,
> > -Adam
> >
>


Re: Will you accept contributions in Scala?

2018-02-10 Thread Mike Thomsen
I'm just a community contributor, so take that FWIW, but a compromise might
be to publish the Scala code as separate maven modules to maven central and
then submit a thoroughly tested processor written in Java. As long as you
have enough unit and integration tests to give strong coverage, I wouldn't
imagine anyone here would have issues reviewing it. If the tests fail
because of code issues in the external dependencies, the obvious answer is
to just hold the PR until the tests pass.

On Fri, Feb 9, 2018 at 9:00 AM, Weiss, Adam 
wrote:

> Devs,
>
> I have some interns starting with my team and we use Scala internally for
> our work.
> If I wanted to have them work to contribute some new processors, would
> they have to be Java to be included with the base distribution or could
> they use Scala as long as it fit within your current build and test process?
>
> Thanks,
> -Adam
>


Re: Will you accept contributions in Scala?

2018-02-09 Thread Andy LoPresto
Hi Adam,

It’s great that your team wants to contribute to NiFi. I think there should be 
some discussion from other community members on this, but I’ll give my view 
here. At this time, I believe we should only accept core contributions in Java. 
It’s not because I have a predisposition against Scala or any other language 
(frankly, I would prefer to write my processors in Groovy), but because Apache 
prioritizes community over code, and in order to make the code accessible to 
the community at large, Java is the standard for this project.

That said, I think there are some important ways you can contribute in other 
languages:

* Testing — while we try to ensure the “production” code is pure Java, the test 
frameworks around NiFi can use other languages. We currently have a number of 
tests in Groovy/Spock.
* Custom repositories — you are welcome to publish processors written in Scala 
on GitHub/Maven and anyone in the community who is interested can add that 
functionality into their installations very easily.

Hopefully this helps guide your instruction to your interns. If others in the 
community feel strongly about this, I hope they weigh in.


Andy LoPresto
alopre...@apache.org
alopresto.apa...@gmail.com
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69

> On Feb 9, 2018, at 6:00 AM, Weiss, Adam  wrote:
> 
> Devs,
> 
> I have some interns starting with my team and we use Scala internally for our 
> work.
> If I wanted to have them work to contribute some new processors, would they 
> have to be Java to be included with the base distribution or could they use 
> Scala as long as it fit within your current build and test process?
> 
> Thanks,
> -Adam



signature.asc
Description: Message signed with OpenPGP using GPGMail


Will you accept contributions in Scala?

2018-02-09 Thread Weiss, Adam
Devs,

I have some interns starting with my team and we use Scala internally for our 
work.
If I wanted to have them work to contribute some new processors, would they 
have to be Java to be included with the base distribution or could they use 
Scala as long as it fit within your current build and test process?

Thanks,
-Adam