Re: [DISCUSS] MiNiFi C++ 0.1.0 Release

[Joe Witt]

Regarding OpenLDAP Public License

This does not appear to be listed under the approved apache licenses
to use for source or binary dependencies.  However, it looks like it
is probably considered a variant of the BSD 3-Clause license which
would be ok as category a.  It would be helpful to validate this with
a question to legal though.

Regarding LevelDB vs Anything

Right now we have something we can discuss and validate the pros and
cons of.  If there are other implementations that allow us to do that
let's bring them forth and do that.  I don't think we're at the phase
now where any decision is necessarily reflective of a long term intent
so plenty of opportunity to choose the best tradeoffs.

This thread:

We should probably root this thread on the release discussion and fork
it out to its own thread to discuss some of these points raised.

Thanks
Joe

On Tue, Nov 22, 2016 at 4:03 PM, Andy LoPresto  wrote:
> Daniel,
>
> I think one reason that MiNiFi Java has been successful, both as a prototype
> and tool deployed in real environments today, is that it was able to
> leverage much of the existing NiFi Java codebase. This greatly reduced the
> "time-to-market” and allowed the development team to get working code out
> into the ecosystem and start to build a community and refine the
> application. Part of that community is gathering feedback from interested
> parties, which you are helpfully providing here. However, I do believe there
> is legitimate value to MiNiFi Java. Developers are hard at work on both the
> Java and C++ versions and are pushing for feature parity so that choice of
> language is not a barrier to feature availability. We welcome community
> focus on various aspects of the project and would not try to direct you to
> get involved with MiNiFi Java if that’s not what you are interested in.
>
> For now, this list is the best place to have conversations about MiNiFi C++
> topics. The wiki does have room for feature requests, roadmap evolution,
> etc. As the MiNiFi community grows, we may split the mailing lists, but that
> would likely not be for some time.
>
>
> Andy LoPresto
> alopre...@apache.org
> alopresto.apa...@gmail.com
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>
> On Nov 22, 2016, at 11:23 AM, Daniel Cave  wrote:
>
> For me personally, I don't see a value add of MiNiFi Java.  The needs that
> NiFi can't address MiNiFi Java can't either, so my focus is MiNiFi C++ as
> that is the hole that needs fixing, again in my opinion, so that is where my
> MiNiFi focus is going to be.
>
> As I go through things I am sure I will have more questions about choices
> that have been made so far regarding MiNiFi C++ (as with all things, we all
> have different views on how do things and there isn't necessarily a
> right/wrong answer).  If there is a better forum to address these more
> specific to MiNiFi C++, please let me know.  My most pressing question is
> the choice to use LevelDB for the provenance repository rather than LMDB.  A
> core tenant of NiFi is fault tolerance in near all cases (as well as full
> data provenance).  As LevelDB is vulnerable to corruption during write
> operations due to unexpected application interruptions, would not something
> more fault tolerant such as LMDB (covered under OpenLDAP Public License) be
> preferable?  The question of fault tolerance applies to the flowfile
> repository as well.
>
>
>
> --
> View this message in context:
> http://apache-nifi-developer-list.39713.n7.nabble.com/DISCUSS-MiNiFi-C-0-1-0-Release-tp13956p13959.html
> Sent from the Apache NiFi Developer List mailing list archive at Nabble.com.
>
>

Re: [DISCUSS] MiNiFi C++ 0.1.0 Release

[Andy LoPresto]

Daniel,

I think one reason that MiNiFi Java has been successful, both as a prototype 
and tool deployed in real environments today, is that it was able to leverage 
much of the existing NiFi Java codebase. This greatly reduced the 
"time-to-market” and allowed the development team to get working code out into 
the ecosystem and start to build a community and refine the application. Part 
of that community is gathering feedback from interested parties, which you are 
helpfully providing here. However, I do believe there is legitimate value to 
MiNiFi Java. Developers are hard at work on both the Java and C++ versions and 
are pushing for feature parity so that choice of language is not a barrier to 
feature availability. We welcome community focus on various aspects of the 
project and would not try to direct you to get involved with MiNiFi Java if 
that’s not what you are interested in.

For now, this list is the best place to have conversations about MiNiFi C++ 
topics. The wiki does have room for feature requests, roadmap evolution, etc. 
As the MiNiFi community grows, we may split the mailing lists, but that would 
likely not be for some time.


Andy LoPresto
alopre...@apache.org
alopresto.apa...@gmail.com
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69

> On Nov 22, 2016, at 11:23 AM, Daniel Cave  wrote:
> 
> For me personally, I don't see a value add of MiNiFi Java.  The needs that
> NiFi can't address MiNiFi Java can't either, so my focus is MiNiFi C++ as
> that is the hole that needs fixing, again in my opinion, so that is where my
> MiNiFi focus is going to be.
> 
> As I go through things I am sure I will have more questions about choices
> that have been made so far regarding MiNiFi C++ (as with all things, we all
> have different views on how do things and there isn't necessarily a
> right/wrong answer).  If there is a better forum to address these more
> specific to MiNiFi C++, please let me know.  My most pressing question is
> the choice to use LevelDB for the provenance repository rather than LMDB.  A
> core tenant of NiFi is fault tolerance in near all cases (as well as full
> data provenance).  As LevelDB is vulnerable to corruption during write
> operations due to unexpected application interruptions, would not something
> more fault tolerant such as LMDB (covered under OpenLDAP Public License) be
> preferable?  The question of fault tolerance applies to the flowfile
> repository as well.
> 
> 
> 
> --
> View this message in context: 
> http://apache-nifi-developer-list.39713.n7.nabble.com/DISCUSS-MiNiFi-C-0-1-0-Release-tp13956p13959.html
> Sent from the Apache NiFi Developer List mailing list archive at Nabble.com.



signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: [DISCUSS] MiNiFi C++ 0.1.0 Release

[Daniel Cave]

For me personally, I don't see a value add of MiNiFi Java.  The needs that
NiFi can't address MiNiFi Java can't either, so my focus is MiNiFi C++ as
that is the hole that needs fixing, again in my opinion, so that is where my
MiNiFi focus is going to be.

As I go through things I am sure I will have more questions about choices
that have been made so far regarding MiNiFi C++ (as with all things, we all
have different views on how do things and there isn't necessarily a
right/wrong answer).  If there is a better forum to address these more
specific to MiNiFi C++, please let me know.  My most pressing question is
the choice to use LevelDB for the provenance repository rather than LMDB.  A
core tenant of NiFi is fault tolerance in near all cases (as well as full
data provenance).  As LevelDB is vulnerable to corruption during write
operations due to unexpected application interruptions, would not something
more fault tolerant such as LMDB (covered under OpenLDAP Public License) be
preferable?  The question of fault tolerance applies to the flowfile
repository as well.



--
View this message in context: 
http://apache-nifi-developer-list.39713.n7.nabble.com/DISCUSS-MiNiFi-C-0-1-0-Release-tp13956p13959.html
Sent from the Apache NiFi Developer List mailing list archive at Nabble.com.

Re: [DISCUSS] MiNiFi C++ 0.1.0 Release

[Aldrin Piri]

Hi Daniel,

Thanks for mailing and getting involved with MiNiFi!

There is no inherent coupling of versions rather, as has been the custom
within the NiFi community, we use, roughly, semantic versioning.  As a
result, our initial release of MiNiFi C++ started as 0.0.1 and since there
have been some new features, our next release is considered a minor one,
putting us at 0.1.0.  There is no correlation of the C++ version with that
of the Java codebase, rather both being in their relative infancy are
starting from a common place.  Instead, this release is predicated on
keeping a regular cadence of making the community's efforts available for
general consumption.  Additionally, the release is also being suggested as
we've reached a point with the filed issues where those items not scheduled
for the next release require considerable effort and another period of
development to incorporate.

In terms of roadmap, there is nothing rigidly defined and is very much open
to the direction of the community and its will.  There is a rough outline
that was captured on the wiki which I'd imagine you have seen, but is
additionally provided as context for anyone else reading along [1].
Please feel free to share thoughts on specifics in terms of milestones for
challenges you are looking to address on the Wiki and we can look to grow
and evolve that information.

>From my perspective, I feel the most important area is that of chasing down
framework parity with the core features and building blocks of all that our
Java equivalents have to offer.  In addition to this, extending the
flexibility of the user experience in dataflow to our MiNiFi instances as
outlined with some of the feature proposals such as Command & Control
(C2)[2] are areas that can be worked on in a parallel fashion.  In general,
my instincts lead me to believe that the C++ version will be a "generation"
behind Java for the immediate future.  MiNiFi Java allows us to make use of
much of the effort done in NiFi to try new things out, iterate, and expand
functionality quickly.   Once established, we can then perform the same in
C++.

In terms of your view of the hub and spoke model, I believe this is the
shared vision of the community long term; the efforts of (C2) would allow
users to more effectively execute such architectures more efficiently,
extending the core principles of NiFi beyond the traditional data center.

We have ensured initial connectivity to core NiFi via the raw socket Site
to Site implementation in C++.  Later, this will be further extended with
the HTTP equivalent, which will make its debut in the MiNiFi realm with the
next release of MiNiFi Java.

Please let us know if there are additional questions you would like to
discuss and appreciate your feedback on the process and look forward to
your continued engagement.

Thanks!

[1] https://cwiki.apache.org/confluence/display/MINIFI/Proposed+Roadmap
[2]
https://cwiki.apache.org/confluence/display/MINIFI/MiNiFi+Command+and+Control

On Tue, Nov 22, 2016 at 1:29 PM, Daniel Cave  wrote:

> Having been out of touch since MiNiFi C++ got added and just getting into
> it,
> is there a reason the C++ version is trying to follow closely the MiNiFi
> Java version rather than just insuring connectivity with NiFi?  I have not
> been able to find alot of details regarding the roadmap for MiNiFi C++.
>
> It seems to me that this tight coupling is coming at the cost of the
> efficiency that should be gained through a C++ version.  MiNiFi C++ should
> lend itself to a hub and spoke model with MiNiFi C++ acting as the spoke
> clients and NiFi as the hub.  This only works, however, if maximum
> efficiency is maintained as spoke needs may range from servers to embedded.
> Additional to embedded advantages, MiNiFi C++ also has the ability to run
> natively as a Windows service with direct interaction with the Windows API
> which is also difficult at best with the Java version.
>
> Can you please provide some clarity on where things are headed?  For
> reference, I have been through the wiki, JIRA, Confluence, Git, etc.
>
>
>
> --
> View this message in context: http://apache-nifi-developer-
> list.39713.n7.nabble.com/DISCUSS-MiNiFi-C-0-1-0-Release-tp13956p13957.html
> Sent from the Apache NiFi Developer List mailing list archive at
> Nabble.com.
>

Re: [DISCUSS] MiNiFi C++ 0.1.0 Release

[Daniel Cave]

Having been out of touch since MiNiFi C++ got added and just getting into it,
is there a reason the C++ version is trying to follow closely the MiNiFi
Java version rather than just insuring connectivity with NiFi?  I have not
been able to find alot of details regarding the roadmap for MiNiFi C++.

It seems to me that this tight coupling is coming at the cost of the
efficiency that should be gained through a C++ version.  MiNiFi C++ should
lend itself to a hub and spoke model with MiNiFi C++ acting as the spoke
clients and NiFi as the hub.  This only works, however, if maximum
efficiency is maintained as spoke needs may range from servers to embedded. 
Additional to embedded advantages, MiNiFi C++ also has the ability to run
natively as a Windows service with direct interaction with the Windows API
which is also difficult at best with the Java version.

Can you please provide some clarity on where things are headed?  For
reference, I have been through the wiki, JIRA, Confluence, Git, etc.



--
View this message in context: 
http://apache-nifi-developer-list.39713.n7.nabble.com/DISCUSS-MiNiFi-C-0-1-0-Release-tp13956p13957.html
Sent from the Apache NiFi Developer List mailing list archive at Nabble.com.

[DISCUSS] MiNiFi C++ 0.1.0 Release

[Aldrin Piri]

Hey folks,

There have been some nice improvements increasing stability and another
 step toward framework parity with its Java counterparts.  There was also
work done to update build tooling to provide a better foundation to tackle
specific environments and flexibility in generating differing builds.

To capture this progress, I would like to generate another release of
MiNiFi C++ as we start establishing the basis for higher level capabilities
such as command and control.

I think there may be a couple of other issues that may be candidates for
simple fixes to make this release.  I will scope out the issues registered
in JIRA and collect those.

I will volunteer to act as RM for this release if no one else has any
interest.

Thanks!