Unused X-ProxyContextPath header in ApplicationResource

[Martini, Adam]

Hello all,

I am wondering if there is any intention of using the X-ProxyContextPath header 
in the org.apache.nifi.web.api.ApplicationResource class?

The class defines the static:
public static final String PROXY_CONTEXT_PATH_HTTP_HEADER = 
"X-ProxyContextPath";

However, it is unused and perhaps there is a good reason for this. Please let 
me know if this is on our development radar. Also, let me know if there is a 
different use for this header than I am aware of.

Thanks!

Adam Martini
Senior Developer, Nike Digital

Re: Re: NiFi 1.5.0 HBase_1_1_2_ClientService performance bug

[Martini, Adam]

Hi Koji and all other who responded,

Thanks for getting this PR out so quickly.  Your original response chain was 
sent to the CC address (Maxwell Eng) so I only now got this email.

I made on comment on your PR, but otw it was the same fix I made to the AMI we 
are testing now with great performance.

Thanks again!

Adam

On 2/12/18, 2:16 PM, "Eng, Maxwell" <maxwell@nike.com> wrote:



On 2/9/18, 7:29 PM, "Koji Kawamura" <ijokaruma...@gmail.com> wrote:

Hi,

The PR is ready for review. I confirmed that performance issue is 
addressed.

https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_nifi_pull_2464=DwIFaQ=7DfhQjPWzR3PmWBQVpi-kw=NutwvZ9ElvhWvhi1rBJ0mGwi4rVfiCUP7ys98E-FZMk=DShcVudbDkrAz_aYA4C4uklUFAqfg8A6UmvK2Y3-Nv8=awjLBwtc0H5JFLdSB_3R-Acp5xEwRvrhnk0WRMQH3uk=

I was also testing to see if the
nifi-hbase_1_1_2-client-service-nar-1.6.0-SNAPSHOT.nar can be used in
NiFi 1.5.0 env. But unfortunately it doesn't seem we can put it as it
is.
A validation error occurs saying, 'HBase_1_1_2_ClientService
-1.6.0-SNAPSHOT from org.apache.nifi -
nifi-hbase_1_1_2-client-service-nar is not compatible with
HBaseClientService -1.5.0 from org.apache.nifi -
nifi-standard-services-api-nar'.
It looks like nifi-standard-services needs to be updated, too, but I
think that's a bit risky, it may affect other services.

So, I've wrote a Gist to work around this, with
nifi-hbase_1_1_2-client-service-nar-1.5.0_nifi-4866.nar built with
1.5.0 released commit with cherry-picked performance fix.

https://urldefense.proofpoint.com/v2/url?u=https-3A__gist.github.com_ijokarumawak_85db60ca71f1825f543c18c62bf7c3fd=DwIFaQ=7DfhQjPWzR3PmWBQVpi-kw=NutwvZ9ElvhWvhi1rBJ0mGwi4rVfiCUP7ys98E-FZMk=DShcVudbDkrAz_aYA4C4uklUFAqfg8A6UmvK2Y3-Nv8=1-xmJiTPwyx15tjdJdDzNM7W2amJTzkovP2OuP10b08=

Thanks,
Koji



On Sat, Feb 10, 2018 at 10:37 AM, Koji Kawamura 
<ijokaruma...@gmail.com> wrote:
> Hi Adam,
>
> Thank you very much for reporting the performance issue.
> I created NIFI-4866 and started fixing the issue by moving the
> problematic code block to createConnection.
> After confirming that addresses performance issue, I will send a PR to
> get it merged.
>
> Koji
>
>
> On Sat, Feb 10, 2018 at 9:25 AM, Joe Witt <joe.w...@gmail.com> wrote:
>> adam
>>
>> you should also be able to put the old hbase nar in and switch to 
that
>> version.
>>
>> we now support multiple versions of the same component.
>>
>> thanks
>>
>> On Feb 9, 2018 7:10 PM, "Mike Thomsen" <mikerthom...@gmail.com> 
wrote:
>>
>>> Adam,
>>>
>>> If you're doing bulk ingestion of JSON, I would recommend using
>>> PutHBaseRecord. I wrote it/contributed it when my team ran into 
similar
>>> limitations doing genomic data ingestion (several 10s of billions 
of Puts
>>> from the 1000 genomes project). If you run into problems with it, 
just post
>>> them and poke me.
>>>
>>> Mike
>>>
>>> On Fri, Feb 9, 2018 at 6:56 PM, Joe Witt <joe.w...@gmail.com> wrote:
>>>
>>> > adam
>>> >
>>> > thanks for reporting and if you can do a contrib that would be 
great!
>>> >
>>> > thanks
>>> > joe
>>> >
>>> > On Feb 9, 2018 6:56 PM, "Martini, Adam" <adam.mart...@nike.com> 
wrote:
>>> >
>>> > > Hello NiFi Dev Community,
>>> > >
>>> > > This commit hash (part of the NiFi 1.5.0 release) created 
serious
>>> > > performance issues for HBase Put operations: "
>>> > > 116c8463428c1fb51bfb7a8adfcf23c32fded964".
>>> > >
>>> > > The override of the “toTransitUri” method makes a call to
>>> > > “connection.getAdmin().getClusterStatus().getMaster()
>>> .getHostAndPort()”
>>> > > upon every flow file transfer, which essentially doubles the 
traffic
>>> > > through the HBase connector.  The performance of our 
PutHBaseJSON
>>> > processor
>>> > > dropped to 1/3 after deploying NiFi 1.5.0.
>>> > >
>>> > > Please let us know a timeline for a fix.  We are building and 
testing
>>> our
>>> > > own tar ball in the interim to fix the issue and are happy to
>>> contribute
>>> > > our code back to the project if you would like.
>>> > >
>>> > > All the best and thank you.
>>> > >
>>> > > Adam Martini
>>> > > Senior Developer, Nike Digital
>>> > >
>>> > >
>>> > >
>>> >
>>>

NiFi 1.5.0 HBase_1_1_2_ClientService performance bug

[Martini, Adam]

Hello NiFi Dev Community,

This commit hash (part of the NiFi 1.5.0 release) created serious performance 
issues for HBase Put operations: "116c8463428c1fb51bfb7a8adfcf23c32fded964".

The override of the “toTransitUri” method makes a call to 
“connection.getAdmin().getClusterStatus().getMaster().getHostAndPort()” upon 
every flow file transfer, which essentially doubles the traffic through the 
HBase connector.  The performance of our PutHBaseJSON processor dropped to 1/3 
after deploying NiFi 1.5.0.

Please let us know a timeline for a fix.  We are building and testing our own 
tar ball in the interim to fix the issue and are happy to contribute our code 
back to the project if you would like.

All the best and thank you.

Adam Martini
Senior Developer, Nike Digital

Re: Re: Pheonix client jar required for HBase conn

[Martini, Adam]

Bryan,

Yes, an HBase client upgrade makes sense for the Java 10 upgrade path. However, 
the NoClassDefFoundError is more mysterious and does concern me.

Thanks,
Adam

On 8/23/18, 12:12 PM, "Bryan Bende"  wrote:

Adam,

Yes for now Java 8 is what is fully supported.

Hopefully the remaining issues can be resolved in the not too distant
future to become compatible with Java 9+.

I know in the HBase case it requires changing to a newer version of
the client because the 1.1.2 client has the code that caused the issue
you ran into with "Unexpected version format: 10.0.1".

Thanks,

Bryan

On Thu, Aug 23, 2018 at 2:54 PM, Martini, Adam  
wrote:
> Bryan,
>
> We are running HBase 1.2.  I am confused as well about the missing 
ClassSize class.  Does seem completely unrelated to Pheonix and I do not 
understand why adding the client would help resolve the class.
>
> Would you advise reverting to Java 8 until Java 10 is fully supported?
>
> Thanks,
>
> Adam
>
> On 8/23/18, 11:27 AM, "Bryan Bende"  wrote:
>
> Adam,
>
> The ClassSize class comes from hbase-common [1] so I'm not sure how
> that would related to the Phoenix client JAR. What version of HBase
> was this against?
>
> The only case I know of that needs the Phoenix client jar is when
> Phoenix has been installed which then modifies the HBase config files
> to specify some class names that come from phoenix client JAR.
>
> As far as Java 10, there has been an on-going effort to become
> compatible with Java 9+, but it is a non-trivial effort and not
> complete yet [2].
>
> Thanks,
>
> Bryan
>
> [1] 
https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_hbase_blob_rel_1.1.2_hbase-2Dcommon_src_main_java_org_apache_hadoop_hbase_util_ClassSize.java=DwIBaQ=7DfhQjPWzR3PmWBQVpi-kw=Sb-vyl3zZFvHwGIFN-QEmQCWZUYgyH4UBZPmcWFaTC8=bnetA9Y56Fx_Tiq-PCdevSuGr34t8ixQz5sDQMViOmc=XVjke9vTfRooHYCqZCNp1AS5kUbiGaJxHHF6OyNcnfQ=
> [2] 
https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_browse_NIFI-2D5174=DwIBaQ=7DfhQjPWzR3PmWBQVpi-kw=Sb-vyl3zZFvHwGIFN-QEmQCWZUYgyH4UBZPmcWFaTC8=bnetA9Y56Fx_Tiq-PCdevSuGr34t8ixQz5sDQMViOmc=dHD8gEEW65OaoGA1DC-FE3i7EJRLatqydkeFpaHB1IA=
>
>
> On Thu, Aug 23, 2018 at 1:47 PM, Martini, Adam 
 wrote:
> > Hello all,
> >
> > We were struggling with our HBase client connection in the NiFi 
1.7.1 release. We are not using Phoenix and our client was erroring with a 
missing class exception:
> >
> > Caused by: java.lang.NoClassDefFoundError: Could not initialize 
class org.apache.hadoop.hbase.util.ClassSize
> > at 
org.apache.hadoop.hbase.ipc.IPCUtil.(IPCUtil.java:72)
> > at 
org.apache.hadoop.hbase.ipc.AbstractRpcClient.(AbstractRpcClient.java:91)
> > at 
org.apache.hadoop.hbase.ipc.RpcClientImpl.(RpcClientImpl.java:1113)
> > at 
org.apache.hadoop.hbase.ipc.RpcClientImpl.(RpcClientImpl.java:1139)
> > ... 33 common frames omitted
> >
> > Adding a Phoenix client (which we do not need) helps resolve this 
class dependency, but fails if NiFi is running on Java 10:
> >
> > Caused by: java.lang.RuntimeException: Unexpected version format: 
10.0.1
> > at 
org.apache.hadoop.hbase.util.ClassSize.(ClassSize.java:119)
> > ... 37 common frames omitted
> >
> > Starting NiFi with Java 9 fixes this error, but complicates our 
devops as Java 9 is no longer supported and cannot be installed via package 
management.
> >
> > Please fix this dependency issue so that a Phoenix client is not 
required to resolve the org.apache.hadoop.hbase.util.ClassSize class. Also, 
please migrate to Java 10 as soon as possible.
> >
> > Let me know if I can help with this process.
> >
> > Thanks,
> >
> > Adam Martini
> > Senior Software Dev, Nike Digital
> >
>
>

Re: Re: Pheonix client jar required for HBase conn

[Martini, Adam]

Bryan,

We are running HBase 1.2.  I am confused as well about the missing ClassSize 
class.  Does seem completely unrelated to Pheonix and I do not understand why 
adding the client would help resolve the class.

Would you advise reverting to Java 8 until Java 10 is fully supported?

Thanks,

Adam

On 8/23/18, 11:27 AM, "Bryan Bende"  wrote:

Adam,

The ClassSize class comes from hbase-common [1] so I'm not sure how
that would related to the Phoenix client JAR. What version of HBase
was this against?

The only case I know of that needs the Phoenix client jar is when
Phoenix has been installed which then modifies the HBase config files
to specify some class names that come from phoenix client JAR.

As far as Java 10, there has been an on-going effort to become
compatible with Java 9+, but it is a non-trivial effort and not
complete yet [2].

Thanks,

Bryan

[1] 
https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_hbase_blob_rel_1.1.2_hbase-2Dcommon_src_main_java_org_apache_hadoop_hbase_util_ClassSize.java=DwIBaQ=7DfhQjPWzR3PmWBQVpi-kw=Sb-vyl3zZFvHwGIFN-QEmQCWZUYgyH4UBZPmcWFaTC8=bnetA9Y56Fx_Tiq-PCdevSuGr34t8ixQz5sDQMViOmc=XVjke9vTfRooHYCqZCNp1AS5kUbiGaJxHHF6OyNcnfQ=
[2] 
https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_browse_NIFI-2D5174=DwIBaQ=7DfhQjPWzR3PmWBQVpi-kw=Sb-vyl3zZFvHwGIFN-QEmQCWZUYgyH4UBZPmcWFaTC8=bnetA9Y56Fx_Tiq-PCdevSuGr34t8ixQz5sDQMViOmc=dHD8gEEW65OaoGA1DC-FE3i7EJRLatqydkeFpaHB1IA=


On Thu, Aug 23, 2018 at 1:47 PM, Martini, Adam  
wrote:
> Hello all,
>
> We were struggling with our HBase client connection in the NiFi 1.7.1 
release. We are not using Phoenix and our client was erroring with a missing 
class exception:
>
> Caused by: java.lang.NoClassDefFoundError: Could not initialize class 
org.apache.hadoop.hbase.util.ClassSize
> at 
org.apache.hadoop.hbase.ipc.IPCUtil.(IPCUtil.java:72)
> at 
org.apache.hadoop.hbase.ipc.AbstractRpcClient.(AbstractRpcClient.java:91)
> at 
org.apache.hadoop.hbase.ipc.RpcClientImpl.(RpcClientImpl.java:1113)
> at 
org.apache.hadoop.hbase.ipc.RpcClientImpl.(RpcClientImpl.java:1139)
> ... 33 common frames omitted
>
> Adding a Phoenix client (which we do not need) helps resolve this class 
dependency, but fails if NiFi is running on Java 10:
>
> Caused by: java.lang.RuntimeException: Unexpected version format: 10.0.1
> at org.apache.hadoop.hbase.util.ClassSize.(ClassSize.java:119)
> ... 37 common frames omitted
>
> Starting NiFi with Java 9 fixes this error, but complicates our devops as 
Java 9 is no longer supported and cannot be installed via package management.
>
> Please fix this dependency issue so that a Phoenix client is not required 
to resolve the org.apache.hadoop.hbase.util.ClassSize class. Also, please 
migrate to Java 10 as soon as possible.
>
> Let me know if I can help with this process.
>
> Thanks,
>
> Adam Martini
> Senior Software Dev, Nike Digital
>

GetKinesis processor

[Martini, Adam]

Hello all,

Is anyone actively working on a GetKinesis Processor? This is something that we 
could really use, but do not want to duplicate effort.

If not, is there a timeline for this work and/or is there any chance of getting 
this in the 1.6.0 release?

Thanks!

Adam Martini
Senior Dev, Nike Digital

NiFi Registry over HTTPS

[Martini, Adam]

Hello all,

We have NiFi Registry 0.2.0 spun up with an nginx proxy and SSL termination 
such that our services is being served over https without using NiFi’s builtin 
security configurations.

We are able to add the registry service to NiFi using our HTTPS endpoint and 
everything works perfectly.  However, we see errors when we restart NiFi:
org.apache.nifi.controller.serialization.FlowSynchronizationException: 
java.lang.IllegalStateException: Failed to create Flow Registry for URI 
https://nifi-registry.test.streams.nikecloud.com/ because this NiFi is not 
configured with a Keystore/Truststore, so it is not capable of communicating 
with a secure Registry. Please populate NiFi's Keystore/Truststore properties 
or connect to a NiFi Registry over http instead of https.

Is there a work around that will allow us to use this nginx proxy architecture 
with NiFi Registry? HTTPS is historically an important requirement for us but 
we do not need, or desire, the complexity of a NiFi’s builtin security.

Thanks,

Adam Martini

Senior Software Engineer
Nike Digital
adam.mart...@nike.com