Implementation of ListFile's Primary Node only in a cluster

[Sivaprasanna]

I was going through ListFile processor's code and found out that in the
documentation
,
it is mentioned that "this processor is designed to run on Primary Node
only in a cluster". I want to understand what "designed" stands for here.
Does that mean the processor was built in a way that it only runs on the
Primary node regardless of the "Execution Nodes" strategy set to otherwise
or does it mean that dataflow manager/developer is expected to set the
'Execution Nodes' strategy to "Primary Node" at the time of flow design? If
it is of the former case, how is it handled in the code? If it is handled,
it should be in the framework side but I don't see any annotation
indicating anything related to such mechanism in the processor code and
more over a related JIRA NIFI-543
 is also open so I want
clear my doubt.

-
Sivaprasanna

Re: NiFi 1.5.0 HBase_1_1_2_ClientService performance bug

[Koji Kawamura]

Hi,

The PR is ready for review. I confirmed that performance issue is addressed.
https://github.com/apache/nifi/pull/2464

I was also testing to see if the
nifi-hbase_1_1_2-client-service-nar-1.6.0-SNAPSHOT.nar can be used in
NiFi 1.5.0 env. But unfortunately it doesn't seem we can put it as it
is.
A validation error occurs saying, 'HBase_1_1_2_ClientService
-1.6.0-SNAPSHOT from org.apache.nifi -
nifi-hbase_1_1_2-client-service-nar is not compatible with
HBaseClientService -1.5.0 from org.apache.nifi -
nifi-standard-services-api-nar'.
It looks like nifi-standard-services needs to be updated, too, but I
think that's a bit risky, it may affect other services.

So, I've wrote a Gist to work around this, with
nifi-hbase_1_1_2-client-service-nar-1.5.0_nifi-4866.nar built with
1.5.0 released commit with cherry-picked performance fix.
https://gist.github.com/ijokarumawak/85db60ca71f1825f543c18c62bf7c3fd

Thanks,
Koji



On Sat, Feb 10, 2018 at 10:37 AM, Koji Kawamura  wrote:
> Hi Adam,
>
> Thank you very much for reporting the performance issue.
> I created NIFI-4866 and started fixing the issue by moving the
> problematic code block to createConnection.
> After confirming that addresses performance issue, I will send a PR to
> get it merged.
>
> Koji
>
>
> On Sat, Feb 10, 2018 at 9:25 AM, Joe Witt  wrote:
>> adam
>>
>> you should also be able to put the old hbase nar in and switch to that
>> version.
>>
>> we now support multiple versions of the same component.
>>
>> thanks
>>
>> On Feb 9, 2018 7:10 PM, "Mike Thomsen"  wrote:
>>
>>> Adam,
>>>
>>> If you're doing bulk ingestion of JSON, I would recommend using
>>> PutHBaseRecord. I wrote it/contributed it when my team ran into similar
>>> limitations doing genomic data ingestion (several 10s of billions of Puts
>>> from the 1000 genomes project). If you run into problems with it, just post
>>> them and poke me.
>>>
>>> Mike
>>>
>>> On Fri, Feb 9, 2018 at 6:56 PM, Joe Witt  wrote:
>>>
>>> > adam
>>> >
>>> > thanks for reporting and if you can do a contrib that would be great!
>>> >
>>> > thanks
>>> > joe
>>> >
>>> > On Feb 9, 2018 6:56 PM, "Martini, Adam"  wrote:
>>> >
>>> > > Hello NiFi Dev Community,
>>> > >
>>> > > This commit hash (part of the NiFi 1.5.0 release) created serious
>>> > > performance issues for HBase Put operations: "
>>> > > 116c8463428c1fb51bfb7a8adfcf23c32fded964".
>>> > >
>>> > > The override of the “toTransitUri” method makes a call to
>>> > > “connection.getAdmin().getClusterStatus().getMaster()
>>> .getHostAndPort()”
>>> > > upon every flow file transfer, which essentially doubles the traffic
>>> > > through the HBase connector.  The performance of our PutHBaseJSON
>>> > processor
>>> > > dropped to 1/3 after deploying NiFi 1.5.0.
>>> > >
>>> > > Please let us know a timeline for a fix.  We are building and testing
>>> our
>>> > > own tar ball in the interim to fix the issue and are happy to
>>> contribute
>>> > > our code back to the project if you would like.
>>> > >
>>> > > All the best and thank you.
>>> > >
>>> > > Adam Martini
>>> > > Senior Developer, Nike Digital
>>> > >
>>> > >
>>> > >
>>> >
>>>

Re: NiFi 1.5.0 HBase_1_1_2_ClientService performance bug

[Koji Kawamura]

Hi Adam,

Thank you very much for reporting the performance issue.
I created NIFI-4866 and started fixing the issue by moving the
problematic code block to createConnection.
After confirming that addresses performance issue, I will send a PR to
get it merged.

Koji


On Sat, Feb 10, 2018 at 9:25 AM, Joe Witt  wrote:
> adam
>
> you should also be able to put the old hbase nar in and switch to that
> version.
>
> we now support multiple versions of the same component.
>
> thanks
>
> On Feb 9, 2018 7:10 PM, "Mike Thomsen"  wrote:
>
>> Adam,
>>
>> If you're doing bulk ingestion of JSON, I would recommend using
>> PutHBaseRecord. I wrote it/contributed it when my team ran into similar
>> limitations doing genomic data ingestion (several 10s of billions of Puts
>> from the 1000 genomes project). If you run into problems with it, just post
>> them and poke me.
>>
>> Mike
>>
>> On Fri, Feb 9, 2018 at 6:56 PM, Joe Witt  wrote:
>>
>> > adam
>> >
>> > thanks for reporting and if you can do a contrib that would be great!
>> >
>> > thanks
>> > joe
>> >
>> > On Feb 9, 2018 6:56 PM, "Martini, Adam"  wrote:
>> >
>> > > Hello NiFi Dev Community,
>> > >
>> > > This commit hash (part of the NiFi 1.5.0 release) created serious
>> > > performance issues for HBase Put operations: "
>> > > 116c8463428c1fb51bfb7a8adfcf23c32fded964".
>> > >
>> > > The override of the “toTransitUri” method makes a call to
>> > > “connection.getAdmin().getClusterStatus().getMaster()
>> .getHostAndPort()”
>> > > upon every flow file transfer, which essentially doubles the traffic
>> > > through the HBase connector.  The performance of our PutHBaseJSON
>> > processor
>> > > dropped to 1/3 after deploying NiFi 1.5.0.
>> > >
>> > > Please let us know a timeline for a fix.  We are building and testing
>> our
>> > > own tar ball in the interim to fix the issue and are happy to
>> contribute
>> > > our code back to the project if you would like.
>> > >
>> > > All the best and thank you.
>> > >
>> > > Adam Martini
>> > > Senior Developer, Nike Digital
>> > >
>> > >
>> > >
>> >
>>

Re: NiFi 1.5.0 HBase_1_1_2_ClientService performance bug

[Joe Witt]

adam

you should also be able to put the old hbase nar in and switch to that
version.

we now support multiple versions of the same component.

thanks

On Feb 9, 2018 7:10 PM, "Mike Thomsen"  wrote:

> Adam,
>
> If you're doing bulk ingestion of JSON, I would recommend using
> PutHBaseRecord. I wrote it/contributed it when my team ran into similar
> limitations doing genomic data ingestion (several 10s of billions of Puts
> from the 1000 genomes project). If you run into problems with it, just post
> them and poke me.
>
> Mike
>
> On Fri, Feb 9, 2018 at 6:56 PM, Joe Witt  wrote:
>
> > adam
> >
> > thanks for reporting and if you can do a contrib that would be great!
> >
> > thanks
> > joe
> >
> > On Feb 9, 2018 6:56 PM, "Martini, Adam"  wrote:
> >
> > > Hello NiFi Dev Community,
> > >
> > > This commit hash (part of the NiFi 1.5.0 release) created serious
> > > performance issues for HBase Put operations: "
> > > 116c8463428c1fb51bfb7a8adfcf23c32fded964".
> > >
> > > The override of the “toTransitUri” method makes a call to
> > > “connection.getAdmin().getClusterStatus().getMaster()
> .getHostAndPort()”
> > > upon every flow file transfer, which essentially doubles the traffic
> > > through the HBase connector.  The performance of our PutHBaseJSON
> > processor
> > > dropped to 1/3 after deploying NiFi 1.5.0.
> > >
> > > Please let us know a timeline for a fix.  We are building and testing
> our
> > > own tar ball in the interim to fix the issue and are happy to
> contribute
> > > our code back to the project if you would like.
> > >
> > > All the best and thank you.
> > >
> > > Adam Martini
> > > Senior Developer, Nike Digital
> > >
> > >
> > >
> >
>

Re: NiFi 1.5.0 HBase_1_1_2_ClientService performance bug

[Mike Thomsen]

Adam,

If you're doing bulk ingestion of JSON, I would recommend using
PutHBaseRecord. I wrote it/contributed it when my team ran into similar
limitations doing genomic data ingestion (several 10s of billions of Puts
from the 1000 genomes project). If you run into problems with it, just post
them and poke me.

Mike

On Fri, Feb 9, 2018 at 6:56 PM, Joe Witt  wrote:

> adam
>
> thanks for reporting and if you can do a contrib that would be great!
>
> thanks
> joe
>
> On Feb 9, 2018 6:56 PM, "Martini, Adam"  wrote:
>
> > Hello NiFi Dev Community,
> >
> > This commit hash (part of the NiFi 1.5.0 release) created serious
> > performance issues for HBase Put operations: "
> > 116c8463428c1fb51bfb7a8adfcf23c32fded964".
> >
> > The override of the “toTransitUri” method makes a call to
> > “connection.getAdmin().getClusterStatus().getMaster().getHostAndPort()”
> > upon every flow file transfer, which essentially doubles the traffic
> > through the HBase connector.  The performance of our PutHBaseJSON
> processor
> > dropped to 1/3 after deploying NiFi 1.5.0.
> >
> > Please let us know a timeline for a fix.  We are building and testing our
> > own tar ball in the interim to fix the issue and are happy to contribute
> > our code back to the project if you would like.
> >
> > All the best and thank you.
> >
> > Adam Martini
> > Senior Developer, Nike Digital
> >
> >
> >
>

Re: NiFi 1.5.0 HBase_1_1_2_ClientService performance bug

[Joe Witt]

adam

thanks for reporting and if you can do a contrib that would be great!

thanks
joe

On Feb 9, 2018 6:56 PM, "Martini, Adam"  wrote:

> Hello NiFi Dev Community,
>
> This commit hash (part of the NiFi 1.5.0 release) created serious
> performance issues for HBase Put operations: "
> 116c8463428c1fb51bfb7a8adfcf23c32fded964".
>
> The override of the “toTransitUri” method makes a call to
> “connection.getAdmin().getClusterStatus().getMaster().getHostAndPort()”
> upon every flow file transfer, which essentially doubles the traffic
> through the HBase connector.  The performance of our PutHBaseJSON processor
> dropped to 1/3 after deploying NiFi 1.5.0.
>
> Please let us know a timeline for a fix.  We are building and testing our
> own tar ball in the interim to fix the issue and are happy to contribute
> our code back to the project if you would like.
>
> All the best and thank you.
>
> Adam Martini
> Senior Developer, Nike Digital
>
>
>

NiFi 1.5.0 HBase_1_1_2_ClientService performance bug

[Martini, Adam]

Hello NiFi Dev Community,

This commit hash (part of the NiFi 1.5.0 release) created serious performance 
issues for HBase Put operations: "116c8463428c1fb51bfb7a8adfcf23c32fded964".

The override of the “toTransitUri” method makes a call to 
“connection.getAdmin().getClusterStatus().getMaster().getHostAndPort()” upon 
every flow file transfer, which essentially doubles the traffic through the 
HBase connector.  The performance of our PutHBaseJSON processor dropped to 1/3 
after deploying NiFi 1.5.0.

Please let us know a timeline for a fix.  We are building and testing our own 
tar ball in the interim to fix the issue and are happy to contribute our code 
back to the project if you would like.

All the best and thank you.

Adam Martini
Senior Developer, Nike Digital

Re: Trouble with setting up SSL connecting MiNiFi to NiFi

[Marc]

Jonah,
  Sorry for the double reply.
https://issues.apache.org/jira/browse/MINIFICPP-396 is the ticket
corresponding to that PR. It may address your issue depending no whether or
not you are currently using the Context service to configure the RPG. It
may also address another issue preventing proper setup, but the CA trust
issue is likely not addressed with the PR I sent a few minutes ago.
  Thanks,
  Marc


On Fri, Feb 9, 2018 at 5:44 PM, Marc  wrote:

> Jonah,
>
>   There is a pull request to address some of the configuration of these
> objects since there were two routes to configure it ( Context Service and
> minifi.properties ) : https://github.com/apache/nifi-minifi-cpp/pull/263
> ; however,
>   I'm not sure this is your issues since the curl_easy_perform implies
> that the peer certificate cannot be authenticated -- "Peer certificate
> cannot be authenticated with given CA certificates"
>
>   Are you setting an SSL context service? Does the CA certificate path
> contain the entire trust chain?
>
>   Thanks,
>   Marc
>
> On Fri, Feb 9, 2018 at 5:20 PM, Jonah Husson 
> wrote:
>
>> Hey All,
>>
>> Figured I'd shoot off an email before looking into issue reporting, in
>> case
>> this is a product of my own stupidity rather than an actual bug.
>>
>> I'm trying to get MiNiFi communicating with a NiFi cluster on an internal
>> network running with SSL.  I'm able to connect to NiFi from a web browser
>> after importing the correct certificates, but attempting to actually
>> transfer a file with minifi produces the following result:
>>
>> [2018-02-09 15:45:55.136] [main] [info] MiNiFi started
>> [2018-02-09 15:45:57.923] [org::apache::nifi::minifi::pr
>> ocessors::GetFile]
>> [info] GetFile process
>> /home/jonah/optimus/data/dynamic/ready_logs/testcopy2.txt
>> [2018-02-09 15:45:58.339] [org::apache::nifi::minifi::utils::HTTPClient]
>> [debug] Setting callback for
>> [2018-02-09 15:45:58.401] [org::apache::nifi::minifi::co
>> re::ProcessSession]
>> [info] Transferring 3b086abc-0dda-11e8-ab09-c85b769e9522 from GetFile to
>> relationship success
>> [2018-02-09 15:45:58.438]
>> [org::apache::nifi::minifi::RemoteProcessorGroupPort] [debug] Refreshing
>> the peer list since there are none configured.
>> [2018-02-09 15:45:58.439] [org::apache::nifi::minifi::utils::HTTPClient]
>> [debug] https://rs0.internal.optimusride.com:9093/nifi-api/site-to-site
>> is
>> a secure url
>> [2018-02-09 15:45:58.439] [org::apache::nifi::minifi::utils::HTTPClient]
>> [debug] Submitting to
>> https://rs0.internal.optimusride.com:9093/nifi-api/site-to-site
>> [2018-02-09 15:45:58.553] [org::apache::nifi::minifi::utils::HTTPClient]
>> [error] curl_easy_perform() failed Peer certificate cannot be
>> authenticated
>> with given CA certificates
>>
>> [2018-02-09 15:45:58.553]
>> [org::apache::nifi::minifi::RemoteProcessorGroupPort] [error]
>> ProcessGroup::refreshRemoteSite2SiteInfo -- curl_easy_perform() failed
>>
>> [2018-02-09 15:45:58.553]
>> [org::apache::nifi::minifi::RemoteProcessorGroupPort] [debug] Obtained
>> protocol from available_protocols_
>> [2018-02-09 15:45:58.553]
>> [org::apache::nifi::minifi::RemoteProcessorGroupPort] [info] no protocol,
>> yielding
>> [2018-02-09 15:46:01.541] [org::apache::nifi::minifi::utils::HTTPClient]
>> [debug] Setting callback for
>> [2018-02-09 15:46:02.701] [org::apache::nifi::minifi::FlowController]
>> [info] Stop Flow Controller
>> [2018-02-09 15:46:04.748] [org::apache::nifi::minifi::utils::HTTPClient]
>> [debug] Setting callback for
>> [2018-02-09 15:46:05.137] [org::apache::nifi::minifi::FlowController]
>> [info] Unload Flow Controller
>> [2018-02-09 15:46:05.137] [main] [info] MiNiFi exit
>>
>> I'm not entirely sure what I'm doing wrong here, I'm fairly sure my
>> certificate setup is fine.  NiFi's keystore has a cert/private key signed
>> by a CA i self-signed and created locally, it's truststore has the public
>> key that corresponds to the client certificate and private key on MiNiFi,
>> and MiNiFi has the certificate used to sign the one in NiFi's keystore.
>>
>> The best guess I have stems from the fact that the log message  is a
>> secure url occurs, but the message that configure_secure_connection(CURL
>> *http_session) should play if it runs doesn't come up, which leads me to
>> beleive that ssl_context_service_ isn't initialized properly for some
>> reason (see line 129 in /extensions/http-curl/HTTPClient.cpp, only place
>> I
>> see either of those methods called).
>>
>> Let me know if you have any insight on this, frankly I haven't found much
>> documentation on the SSL setup for MiNiFi at all, so it's totally possible
>> I did something horribly wrong there.  I'd also be happy to make a bug
>> report if y'all think this warrants one.
>>
>> Best,
>> Jonah.
>>
>
>

Re: Trouble with setting up SSL connecting MiNiFi to NiFi

[Marc]

Jonah,

  There is a pull request to address some of the configuration of these
objects since there were two routes to configure it ( Context Service and
minifi.properties ) : https://github.com/apache/nifi-minifi-cpp/pull/263 ;
however,
  I'm not sure this is your issues since the curl_easy_perform implies that
the peer certificate cannot be authenticated -- "Peer certificate cannot be
authenticated with given CA certificates"

  Are you setting an SSL context service? Does the CA certificate path
contain the entire trust chain?

  Thanks,
  Marc

On Fri, Feb 9, 2018 at 5:20 PM, Jonah Husson  wrote:

> Hey All,
>
> Figured I'd shoot off an email before looking into issue reporting, in case
> this is a product of my own stupidity rather than an actual bug.
>
> I'm trying to get MiNiFi communicating with a NiFi cluster on an internal
> network running with SSL.  I'm able to connect to NiFi from a web browser
> after importing the correct certificates, but attempting to actually
> transfer a file with minifi produces the following result:
>
> [2018-02-09 15:45:55.136] [main] [info] MiNiFi started
> [2018-02-09 15:45:57.923] [org::apache::nifi::minifi::processors::GetFile]
> [info] GetFile process
> /home/jonah/optimus/data/dynamic/ready_logs/testcopy2.txt
> [2018-02-09 15:45:58.339] [org::apache::nifi::minifi::utils::HTTPClient]
> [debug] Setting callback for
> [2018-02-09 15:45:58.401] [org::apache::nifi::minifi::
> core::ProcessSession]
> [info] Transferring 3b086abc-0dda-11e8-ab09-c85b769e9522 from GetFile to
> relationship success
> [2018-02-09 15:45:58.438]
> [org::apache::nifi::minifi::RemoteProcessorGroupPort] [debug] Refreshing
> the peer list since there are none configured.
> [2018-02-09 15:45:58.439] [org::apache::nifi::minifi::utils::HTTPClient]
> [debug] https://rs0.internal.optimusride.com:9093/nifi-api/site-to-site is
> a secure url
> [2018-02-09 15:45:58.439] [org::apache::nifi::minifi::utils::HTTPClient]
> [debug] Submitting to
> https://rs0.internal.optimusride.com:9093/nifi-api/site-to-site
> [2018-02-09 15:45:58.553] [org::apache::nifi::minifi::utils::HTTPClient]
> [error] curl_easy_perform() failed Peer certificate cannot be authenticated
> with given CA certificates
>
> [2018-02-09 15:45:58.553]
> [org::apache::nifi::minifi::RemoteProcessorGroupPort] [error]
> ProcessGroup::refreshRemoteSite2SiteInfo -- curl_easy_perform() failed
>
> [2018-02-09 15:45:58.553]
> [org::apache::nifi::minifi::RemoteProcessorGroupPort] [debug] Obtained
> protocol from available_protocols_
> [2018-02-09 15:45:58.553]
> [org::apache::nifi::minifi::RemoteProcessorGroupPort] [info] no protocol,
> yielding
> [2018-02-09 15:46:01.541] [org::apache::nifi::minifi::utils::HTTPClient]
> [debug] Setting callback for
> [2018-02-09 15:46:02.701] [org::apache::nifi::minifi::FlowController]
> [info] Stop Flow Controller
> [2018-02-09 15:46:04.748] [org::apache::nifi::minifi::utils::HTTPClient]
> [debug] Setting callback for
> [2018-02-09 15:46:05.137] [org::apache::nifi::minifi::FlowController]
> [info] Unload Flow Controller
> [2018-02-09 15:46:05.137] [main] [info] MiNiFi exit
>
> I'm not entirely sure what I'm doing wrong here, I'm fairly sure my
> certificate setup is fine.  NiFi's keystore has a cert/private key signed
> by a CA i self-signed and created locally, it's truststore has the public
> key that corresponds to the client certificate and private key on MiNiFi,
> and MiNiFi has the certificate used to sign the one in NiFi's keystore.
>
> The best guess I have stems from the fact that the log message  is a
> secure url occurs, but the message that configure_secure_connection(CURL
> *http_session) should play if it runs doesn't come up, which leads me to
> beleive that ssl_context_service_ isn't initialized properly for some
> reason (see line 129 in /extensions/http-curl/HTTPClient.cpp, only place I
> see either of those methods called).
>
> Let me know if you have any insight on this, frankly I haven't found much
> documentation on the SSL setup for MiNiFi at all, so it's totally possible
> I did something horribly wrong there.  I'd also be happy to make a bug
> report if y'all think this warrants one.
>
> Best,
> Jonah.
>

Trouble with setting up SSL connecting MiNiFi to NiFi

[Jonah Husson]

Hey All,

Figured I'd shoot off an email before looking into issue reporting, in case
this is a product of my own stupidity rather than an actual bug.

I'm trying to get MiNiFi communicating with a NiFi cluster on an internal
network running with SSL.  I'm able to connect to NiFi from a web browser
after importing the correct certificates, but attempting to actually
transfer a file with minifi produces the following result:

[2018-02-09 15:45:55.136] [main] [info] MiNiFi started
[2018-02-09 15:45:57.923] [org::apache::nifi::minifi::processors::GetFile]
[info] GetFile process
/home/jonah/optimus/data/dynamic/ready_logs/testcopy2.txt
[2018-02-09 15:45:58.339] [org::apache::nifi::minifi::utils::HTTPClient]
[debug] Setting callback for
[2018-02-09 15:45:58.401] [org::apache::nifi::minifi::core::ProcessSession]
[info] Transferring 3b086abc-0dda-11e8-ab09-c85b769e9522 from GetFile to
relationship success
[2018-02-09 15:45:58.438]
[org::apache::nifi::minifi::RemoteProcessorGroupPort] [debug] Refreshing
the peer list since there are none configured.
[2018-02-09 15:45:58.439] [org::apache::nifi::minifi::utils::HTTPClient]
[debug] https://rs0.internal.optimusride.com:9093/nifi-api/site-to-site is
a secure url
[2018-02-09 15:45:58.439] [org::apache::nifi::minifi::utils::HTTPClient]
[debug] Submitting to
https://rs0.internal.optimusride.com:9093/nifi-api/site-to-site
[2018-02-09 15:45:58.553] [org::apache::nifi::minifi::utils::HTTPClient]
[error] curl_easy_perform() failed Peer certificate cannot be authenticated
with given CA certificates

[2018-02-09 15:45:58.553]
[org::apache::nifi::minifi::RemoteProcessorGroupPort] [error]
ProcessGroup::refreshRemoteSite2SiteInfo -- curl_easy_perform() failed

[2018-02-09 15:45:58.553]
[org::apache::nifi::minifi::RemoteProcessorGroupPort] [debug] Obtained
protocol from available_protocols_
[2018-02-09 15:45:58.553]
[org::apache::nifi::minifi::RemoteProcessorGroupPort] [info] no protocol,
yielding
[2018-02-09 15:46:01.541] [org::apache::nifi::minifi::utils::HTTPClient]
[debug] Setting callback for
[2018-02-09 15:46:02.701] [org::apache::nifi::minifi::FlowController]
[info] Stop Flow Controller
[2018-02-09 15:46:04.748] [org::apache::nifi::minifi::utils::HTTPClient]
[debug] Setting callback for
[2018-02-09 15:46:05.137] [org::apache::nifi::minifi::FlowController]
[info] Unload Flow Controller
[2018-02-09 15:46:05.137] [main] [info] MiNiFi exit

I'm not entirely sure what I'm doing wrong here, I'm fairly sure my
certificate setup is fine.  NiFi's keystore has a cert/private key signed
by a CA i self-signed and created locally, it's truststore has the public
key that corresponds to the client certificate and private key on MiNiFi,
and MiNiFi has the certificate used to sign the one in NiFi's keystore.

The best guess I have stems from the fact that the log message  is a
secure url occurs, but the message that configure_secure_connection(CURL
*http_session) should play if it runs doesn't come up, which leads me to
beleive that ssl_context_service_ isn't initialized properly for some
reason (see line 129 in /extensions/http-curl/HTTPClient.cpp, only place I
see either of those methods called).

Let me know if you have any insight on this, frankly I haven't found much
documentation on the SSL setup for MiNiFi at all, so it's totally possible
I did something horribly wrong there.  I'd also be happy to make a bug
report if y'all think this warrants one.

Best,
Jonah.

Re: Will you accept contributions in Scala?

[Andy LoPresto]

Hi Adam,

It’s great that your team wants to contribute to NiFi. I think there should be 
some discussion from other community members on this, but I’ll give my view 
here. At this time, I believe we should only accept core contributions in Java. 
It’s not because I have a predisposition against Scala or any other language 
(frankly, I would prefer to write my processors in Groovy), but because Apache 
prioritizes community over code, and in order to make the code accessible to 
the community at large, Java is the standard for this project.

That said, I think there are some important ways you can contribute in other 
languages:

* Testing — while we try to ensure the “production” code is pure Java, the test 
frameworks around NiFi can use other languages. We currently have a number of 
tests in Groovy/Spock.
* Custom repositories — you are welcome to publish processors written in Scala 
on GitHub/Maven and anyone in the community who is interested can add that 
functionality into their installations very easily.

Hopefully this helps guide your instruction to your interns. If others in the 
community feel strongly about this, I hope they weigh in.


Andy LoPresto
alopre...@apache.org
alopresto.apa...@gmail.com
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69

> On Feb 9, 2018, at 6:00 AM, Weiss, Adam  wrote:
> 
> Devs,
> 
> I have some interns starting with my team and we use Scala internally for our 
> work.
> If I wanted to have them work to contribute some new processors, would they 
> have to be Java to be included with the base distribution or could they use 
> Scala as long as it fit within your current build and test process?
> 
> Thanks,
> -Adam



signature.asc
Description: Message signed with OpenPGP using GPGMail

Will you accept contributions in Scala?

[Weiss, Adam]

Devs,

I have some interns starting with my team and we use Scala internally for our 
work.
If I wanted to have them work to contribute some new processors, would they 
have to be Java to be included with the base distribution or could they use 
Scala as long as it fit within your current build and test process?

Thanks,
-Adam