Re: Custom Processor Upgrade

[Bryan Bende]

Where is application-context.xml in your NAR?

And how are you trying to load it in
com.thinkbiganalytics.nifi.processor.AbstractNiFiProcessor
?

I would expect it to be packaged into the jar that contains your
processors, most likely in src/main/resources of the processors module
 which then ends up at the root of the jar.

On Wed, Aug 14, 2019 at 5:36 PM Bimal Mehta  wrote:

> Ahh, seems like a Springboot error.
> Is it to do with upgraded Jetty server ?
>
> Caused by: org.springframework.beans.factory.BeanDefinitionStoreException:
> Unexpected exception parsing XML document from class path resource
> [application-context.xml]; nested exception is
> org.springframework.beans.FatalBeanException: Class
> [org.springframework.context.config.ContextNamespaceHandler] for namespace [
> http://www.springframework.org/schema/context] does not implement the
> [org.springframework.beans.factory.xml.NamespaceHandler] interface
> at
> org.springframework.beans.factory.xml.XmlBeanDefinitionReader.doLoadBeanDefinitions(XmlBeanDefinitionReader.java:414)
> at
> org.springframework.beans.factory.xml.XmlBeanDefinitionReader.loadBeanDefinitions(XmlBeanDefinitionReader.java:336)
> at
> org.springframework.beans.factory.xml.XmlBeanDefinitionReader.loadBeanDefinitions(XmlBeanDefinitionReader.java:304)
> at
> org.springframework.beans.factory.support.AbstractBeanDefinitionReader.loadBeanDefinitions(AbstractBeanDefinitionReader.java:181)
> at
> org.springframework.beans.factory.support.AbstractBeanDefinitionReader.loadBeanDefinitions(AbstractBeanDefinitionReader.java:217)
> at
> org.springframework.beans.factory.support.AbstractBeanDefinitionReader.loadBeanDefinitions(AbstractBeanDefinitionReader.java:188)
> at
> org.springframework.beans.factory.support.AbstractBeanDefinitionReader.loadBeanDefinitions(AbstractBeanDefinitionReader.java:252)
> at
> org.springframework.context.support.AbstractXmlApplicationContext.loadBeanDefinitions(AbstractXmlApplicationContext.java:127)
> at
> org.springframework.context.support.AbstractXmlApplicationContext.loadBeanDefinitions(AbstractXmlApplicationContext.java:93)
> at
> org.springframework.context.support.AbstractRefreshableApplicationContext.refreshBeanFactory(AbstractRefreshableApplicationContext.java:129)
> at
> org.springframework.context.support.AbstractApplicationContext.obtainFreshBeanFactory(AbstractApplicationContext.java:609)
> at
> org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:510)
> at
> org.springframework.context.support.ClassPathXmlApplicationContext.(ClassPathXmlApplicationContext.java:139)
> at
> org.springframework.context.support.ClassPathXmlApplicationContext.(ClassPathXmlApplicationContext.java:83)
> at
> com.thinkbiganalytics.nifi.processor.AbstractNiFiProcessor.init(AbstractNiFiProcessor.java:48)
> at
> org.apache.nifi.processor.AbstractSessionFactoryProcessor.initialize(AbstractSessionFactoryProcessor.java:63)
> at
> org.apache.nifi.controller.ExtensionBuilder.createLoggableProcessor(ExtensionBuilder.java:421)
> ... 50 common frames omitted
> Caused by: org.springframework.beans.FatalBeanException: Class
> [org.springframework.context.config.ContextNamespaceHandler] for namespace [
> http://www.springframework.org/schema/context] does not implement the
> [org.springframework.beans.factory.xml.NamespaceHandler] interface
> at
> org.springframework.beans.factory.xml.DefaultNamespaceHandlerResolver.resolve(DefaultNamespaceHandlerResolver.java:128)
> at
> org.springframework.beans.factory.xml.BeanDefinitionParserDelegate.parseCustomElement(BeanDefinitionParserDelegate.java:1406)
> at
> org.springframework.beans.factory.xml.BeanDefinitionParserDelegate.parseCustomElement(BeanDefinitionParserDelegate.java:1401)
> at
> org.springframework.beans.factory.xml.DefaultBeanDefinitionDocumentReader.parseBeanDefinitions(DefaultBeanDefinitionDocumentReader.java:168)
> at
> org.springframework.beans.factory.xml.DefaultBeanDefinitionDocumentReader.doRegisterBeanDefinitions(DefaultBeanDefinitionDocumentReader.java:138)
> at
> org.springframework.beans.factory.xml.DefaultBeanDefinitionDocumentReader.registerBeanDefinitions(DefaultBeanDefinitionDocumentReader.java:94)
> at
> org.springframework.beans.factory.xml.XmlBeanDefinitionReader.registerBeanDefinitions(XmlBeanDefinitionReader.java:508)
> at
> org.springframework.beans.factory.xml.XmlBeanDefinitionReader.doLoadBeanDefinitions(XmlBeanDefinitionReader.java:392)
> ... 66 common frames omitted
>
> On Wed, Aug 14, 2019 at 4:44 PM Bryan Bende  wrote:
>
>> You have to add another instance of the processor which should
>> generate the same stracktrace you sent earlier, except this time there
>> should be a second part to it with "Caused by " and then more of
>> the stacktrace that wasn't there before.
>>
>> On Wed, Aug 14, 2019 at 4:41 PM Bimal Mehta  wrote:
>> >
>> > Hi Bryan,
>> >
>> > I did what you said.
>> > This is what I got
>> >
>> > 2019-08-14 20:16:18,948 DEBUG 

Re: [EXT] Specifying formatters at a record field level

[Mike Thomsen]

Hypothetically, I think the path of least resistance might be to refactor
the readers and writers to allow custom properties like
"timestamp.format.1...N" and have those properties evaluated until one is
found that can parse the incoming text. Thoughts?

On Wed, Aug 14, 2019 at 12:34 PM Peter Wicks (pwicks) 
wrote:

> Not that I’m aware of. We implemented something custom that lets you
> specify it with attributes on the FlowFile (something like
> data.field.#.format=….), we do the same thing for binary/hex fields. But we
> didn’t contribute it as it’s part of a custom record processing processor
> that’s application specific.
>
>
>
> Thanks,
>
>   Peter
>
>
>
> *From:* Mike Thomsen 
> *Sent:* Wednesday, August 14, 2019 8:35 AM
> *To:* users@nifi.apache.org
> *Subject:* [EXT] Specifying formatters at a record field level
>
>
>
> If there any way to specify a timestamp format string on each field that
> is a TIMESTAMP (long, logical type timestamp-millis)? We have a case where
> we would need at least three, possibly half a dozen timestamp formats to
> read a record set.
>
>
>
> Thanks,
>
>
>
> Mike
>

Re: Custom Processor Upgrade

[Bimal Mehta]

Ahh, seems like a Springboot error.
Is it to do with upgraded Jetty server ?

Caused by: org.springframework.beans.factory.BeanDefinitionStoreException:
Unexpected exception parsing XML document from class path resource
[application-context.xml]; nested exception is
org.springframework.beans.FatalBeanException: Class
[org.springframework.context.config.ContextNamespaceHandler] for namespace [
http://www.springframework.org/schema/context] does not implement the
[org.springframework.beans.factory.xml.NamespaceHandler] interface
at
org.springframework.beans.factory.xml.XmlBeanDefinitionReader.doLoadBeanDefinitions(XmlBeanDefinitionReader.java:414)
at
org.springframework.beans.factory.xml.XmlBeanDefinitionReader.loadBeanDefinitions(XmlBeanDefinitionReader.java:336)
at
org.springframework.beans.factory.xml.XmlBeanDefinitionReader.loadBeanDefinitions(XmlBeanDefinitionReader.java:304)
at
org.springframework.beans.factory.support.AbstractBeanDefinitionReader.loadBeanDefinitions(AbstractBeanDefinitionReader.java:181)
at
org.springframework.beans.factory.support.AbstractBeanDefinitionReader.loadBeanDefinitions(AbstractBeanDefinitionReader.java:217)
at
org.springframework.beans.factory.support.AbstractBeanDefinitionReader.loadBeanDefinitions(AbstractBeanDefinitionReader.java:188)
at
org.springframework.beans.factory.support.AbstractBeanDefinitionReader.loadBeanDefinitions(AbstractBeanDefinitionReader.java:252)
at
org.springframework.context.support.AbstractXmlApplicationContext.loadBeanDefinitions(AbstractXmlApplicationContext.java:127)
at
org.springframework.context.support.AbstractXmlApplicationContext.loadBeanDefinitions(AbstractXmlApplicationContext.java:93)
at
org.springframework.context.support.AbstractRefreshableApplicationContext.refreshBeanFactory(AbstractRefreshableApplicationContext.java:129)
at
org.springframework.context.support.AbstractApplicationContext.obtainFreshBeanFactory(AbstractApplicationContext.java:609)
at
org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:510)
at
org.springframework.context.support.ClassPathXmlApplicationContext.(ClassPathXmlApplicationContext.java:139)
at
org.springframework.context.support.ClassPathXmlApplicationContext.(ClassPathXmlApplicationContext.java:83)
at
com.thinkbiganalytics.nifi.processor.AbstractNiFiProcessor.init(AbstractNiFiProcessor.java:48)
at
org.apache.nifi.processor.AbstractSessionFactoryProcessor.initialize(AbstractSessionFactoryProcessor.java:63)
at
org.apache.nifi.controller.ExtensionBuilder.createLoggableProcessor(ExtensionBuilder.java:421)
... 50 common frames omitted
Caused by: org.springframework.beans.FatalBeanException: Class
[org.springframework.context.config.ContextNamespaceHandler] for namespace [
http://www.springframework.org/schema/context] does not implement the
[org.springframework.beans.factory.xml.NamespaceHandler] interface
at
org.springframework.beans.factory.xml.DefaultNamespaceHandlerResolver.resolve(DefaultNamespaceHandlerResolver.java:128)
at
org.springframework.beans.factory.xml.BeanDefinitionParserDelegate.parseCustomElement(BeanDefinitionParserDelegate.java:1406)
at
org.springframework.beans.factory.xml.BeanDefinitionParserDelegate.parseCustomElement(BeanDefinitionParserDelegate.java:1401)
at
org.springframework.beans.factory.xml.DefaultBeanDefinitionDocumentReader.parseBeanDefinitions(DefaultBeanDefinitionDocumentReader.java:168)
at
org.springframework.beans.factory.xml.DefaultBeanDefinitionDocumentReader.doRegisterBeanDefinitions(DefaultBeanDefinitionDocumentReader.java:138)
at
org.springframework.beans.factory.xml.DefaultBeanDefinitionDocumentReader.registerBeanDefinitions(DefaultBeanDefinitionDocumentReader.java:94)
at
org.springframework.beans.factory.xml.XmlBeanDefinitionReader.registerBeanDefinitions(XmlBeanDefinitionReader.java:508)
at
org.springframework.beans.factory.xml.XmlBeanDefinitionReader.doLoadBeanDefinitions(XmlBeanDefinitionReader.java:392)
... 66 common frames omitted

On Wed, Aug 14, 2019 at 4:44 PM Bryan Bende  wrote:

> You have to add another instance of the processor which should
> generate the same stracktrace you sent earlier, except this time there
> should be a second part to it with "Caused by " and then more of
> the stacktrace that wasn't there before.
>
> On Wed, Aug 14, 2019 at 4:41 PM Bimal Mehta  wrote:
> >
> > Hi Bryan,
> >
> > I did what you said.
> > This is what I got
> >
> > 2019-08-14 20:16:18,948 DEBUG [Validate Components Thread-3]
> o.a.n.controller.AbstractComponentNode Computed validation errors with
> Validation Context StandardValidationContext[componentId=
> 6fbe2407-7799-3908-f4c4-bf2f8940bf1e ,
> properties={PropertyDescriptor[Header Line Count]=1,
> PropertyDescriptor[Enable processing]=${searchTerm}}]; results = ['Missing
> Processor' validated against 'Any Property' is invalid because Processor is
> of type  org.apache.nifi.init.InitiateScan, but this is not a valid
> Processor type]
> > 2019-08-14 20:16:18,946 

Re: Custom Processor Upgrade

[Bryan Bende]

You have to add another instance of the processor which should
generate the same stracktrace you sent earlier, except this time there
should be a second part to it with "Caused by " and then more of
the stacktrace that wasn't there before.

On Wed, Aug 14, 2019 at 4:41 PM Bimal Mehta  wrote:
>
> Hi Bryan,
>
> I did what you said.
> This is what I got
>
> 2019-08-14 20:16:18,948 DEBUG [Validate Components Thread-3] 
> o.a.n.controller.AbstractComponentNode Computed validation errors with 
> Validation Context StandardValidationContext[componentId= 
> 6fbe2407-7799-3908-f4c4-bf2f8940bf1e , properties={PropertyDescriptor[Header 
> Line Count]=1, PropertyDescriptor[Enable processing]=${searchTerm}}]; results 
> = ['Missing Processor' validated against 'Any Property' is invalid because 
> Processor is of type  org.apache.nifi.init.InitiateScan, but this is not a 
> valid Processor type]
> 2019-08-14 20:16:18,946 DEBUG [Timer-Driven Process Thread-6] 
> o.a.n.c.r.m.SecondPrecisionEventContainer Updated bin 39. Did NOT replace.
>
> On Wed, Aug 14, 2019 at 1:42 PM Bryan Bende  wrote:
>>
>> Can you edit logback.xml and add the following, the get the stacktrace again?
>>
>> 
>>
>> This should include a root cause exception which we are missing right now.
>>
>> I think it takes about 20-30 seconds for logback to pick up the edits
>> to logback.xml.
>>
>> On Wed, Aug 14, 2019 at 12:53 PM Bimal Mehta  wrote:
>> >
>> > For the custom processor we have, we are extending the  
>> > AbstractNifiProcessor.java.
>> > The processor is used to scan metadata of an incoming flow file.
>> > The error we get in logs is as below.
>> > 2019-08-13 23:21:21,529 ERROR [main] o.a.nifi.controller.ExtensionBuilder 
>> > Could not create Processor of type org.apache.nifi.init.InitiateScan for 
>> > ID 6fbe2407-7799-3908-f4c4-bf2f8940bf1e; creating "Ghost" implementation
>> > org.apache.nifi.controller.exception.ProcessorInstantiationException:  
>> > org.apache.nifi.init.InitiateScan
>> > at 
>> > org.apache.nifi.controller.ExtensionBuilder.createLoggableProcessor(ExtensionBuilder.java:425)
>> > at 
>> > org.apache.nifi.controller.ExtensionBuilder.buildProcessor(ExtensionBuilder.java:191)
>> > at 
>> > org.apache.nifi.controller.flow.StandardFlowManager.createProcessor(StandardFlowManager.java:298)
>> > at 
>> > org.apache.nifi.controller.flow.StandardFlowManager.createProcessor(StandardFlowManager.java:274)
>> > at 
>> > org.apache.nifi.controller.StandardFlowSynchronizer.addProcessGroup(StandardFlowSynchronizer.java:1262)
>> > at 
>> > org.apache.nifi.controller.StandardFlowSynchronizer.addProcessGroup(StandardFlowSynchronizer.java:1389)
>> > at 
>> > org.apache.nifi.controller.StandardFlowSynchronizer.addProcessGroup(StandardFlowSynchronizer.java:1389)
>> > at 
>> > org.apache.nifi.controller.StandardFlowSynchronizer.sync(StandardFlowSynchronizer.java:362)
>> > at 
>> > org.apache.nifi.controller.FlowController.synchronize(FlowController.java:1296)
>> > at 
>> > org.apache.nifi.persistence.StandardXMLFlowConfigurationDAO.load(StandardXMLFlowConfigurationDAO.java:88)
>> > at 
>> > org.apache.nifi.controller.StandardFlowService.loadFromBytes(StandardFlowService.java:812)
>> > at 
>> > org.apache.nifi.controller.StandardFlowService.load(StandardFlowService.java:557)
>> > at 
>> > org.apache.nifi.web.contextlistener.ApplicationStartupContextListener.contextInitialized(ApplicationStartupContextListener.java:72)
>> > at 
>> > org.eclipse.jetty.server.handler.ContextHandler.callContextInitialized(ContextHandler.java:953)
>> > at 
>> > org.eclipse.jetty.servlet.ServletContextHandler.callContextInitialized(ServletContextHandler.java:558)
>> > at 
>> > org.eclipse.jetty.server.handler.ContextHandler.startContext(ContextHandler.java:918)
>> > at 
>> > org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:370)
>> > at 
>> > org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1497)
>> > at 
>> > org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1459)
>> > at 
>> > org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:848)
>> > at 
>> > org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:287)
>> > at 
>> > org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:545)
>> > at 
>> > org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
>> > at 
>> > org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:138)
>> > at 
>> > org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117)
>> > at 
>> > org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:113)
>> > at 
>> > 

Re: Custom Processor Upgrade

[Bimal Mehta]

Hi Bryan,

I did what you said.
This is what I got

2019-08-14 20:16:18,948 DEBUG [Validate Components Thread-3]
o.a.n.controller.AbstractComponentNode Computed validation errors with
Validation Context StandardValidationContext[componentId=
6fbe2407-7799-3908-f4c4-bf2f8940bf1e ,
properties={PropertyDescriptor[Header Line Count]=1,
PropertyDescriptor[Enable processing]=${searchTerm}}]; results = ['Missing
Processor' validated against 'Any Property' is invalid because Processor is
of type  org.apache.nifi.init.InitiateScan, but this is not a valid
Processor type]
2019-08-14 20:16:18,946 DEBUG [Timer-Driven Process Thread-6]
o.a.n.c.r.m.SecondPrecisionEventContainer Updated bin 39. Did NOT replace.

On Wed, Aug 14, 2019 at 1:42 PM Bryan Bende  wrote:

> Can you edit logback.xml and add the following, the get the stacktrace
> again?
>
> 
>
> This should include a root cause exception which we are missing right now.
>
> I think it takes about 20-30 seconds for logback to pick up the edits
> to logback.xml.
>
> On Wed, Aug 14, 2019 at 12:53 PM Bimal Mehta  wrote:
> >
> > For the custom processor we have, we are extending the
> AbstractNifiProcessor.java.
> > The processor is used to scan metadata of an incoming flow file.
> > The error we get in logs is as below.
> > 2019-08-13 23:21:21,529 ERROR [main]
> o.a.nifi.controller.ExtensionBuilder Could not create Processor of type
> org.apache.nifi.init.InitiateScan for ID
> 6fbe2407-7799-3908-f4c4-bf2f8940bf1e; creating "Ghost" implementation
> > org.apache.nifi.controller.exception.ProcessorInstantiationException:
> org.apache.nifi.init.InitiateScan
> > at
> org.apache.nifi.controller.ExtensionBuilder.createLoggableProcessor(ExtensionBuilder.java:425)
> > at
> org.apache.nifi.controller.ExtensionBuilder.buildProcessor(ExtensionBuilder.java:191)
> > at
> org.apache.nifi.controller.flow.StandardFlowManager.createProcessor(StandardFlowManager.java:298)
> > at
> org.apache.nifi.controller.flow.StandardFlowManager.createProcessor(StandardFlowManager.java:274)
> > at
> org.apache.nifi.controller.StandardFlowSynchronizer.addProcessGroup(StandardFlowSynchronizer.java:1262)
> > at
> org.apache.nifi.controller.StandardFlowSynchronizer.addProcessGroup(StandardFlowSynchronizer.java:1389)
> > at
> org.apache.nifi.controller.StandardFlowSynchronizer.addProcessGroup(StandardFlowSynchronizer.java:1389)
> > at
> org.apache.nifi.controller.StandardFlowSynchronizer.sync(StandardFlowSynchronizer.java:362)
> > at
> org.apache.nifi.controller.FlowController.synchronize(FlowController.java:1296)
> > at
> org.apache.nifi.persistence.StandardXMLFlowConfigurationDAO.load(StandardXMLFlowConfigurationDAO.java:88)
> > at
> org.apache.nifi.controller.StandardFlowService.loadFromBytes(StandardFlowService.java:812)
> > at
> org.apache.nifi.controller.StandardFlowService.load(StandardFlowService.java:557)
> > at
> org.apache.nifi.web.contextlistener.ApplicationStartupContextListener.contextInitialized(ApplicationStartupContextListener.java:72)
> > at
> org.eclipse.jetty.server.handler.ContextHandler.callContextInitialized(ContextHandler.java:953)
> > at
> org.eclipse.jetty.servlet.ServletContextHandler.callContextInitialized(ServletContextHandler.java:558)
> > at
> org.eclipse.jetty.server.handler.ContextHandler.startContext(ContextHandler.java:918)
> > at
> org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:370)
> > at
> org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1497)
> > at
> org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1459)
> > at
> org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:848)
> > at
> org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:287)
> > at
> org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:545)
> > at
> org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
> > at
> org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:138)
> > at
> org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117)
> > at
> org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:113)
> > at
> org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
> > at
> org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:138)
> > at
> org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:108)
> > at
> org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:113)
> > at
> org.eclipse.jetty.server.handler.gzip.GzipHandler.doStart(GzipHandler.java:403)
> > at
> 

Re: Custom Processor Upgrade

[Bryan Bende]

Can you edit logback.xml and add the following, the get the stacktrace again?



This should include a root cause exception which we are missing right now.

I think it takes about 20-30 seconds for logback to pick up the edits
to logback.xml.

On Wed, Aug 14, 2019 at 12:53 PM Bimal Mehta  wrote:
>
> For the custom processor we have, we are extending the  
> AbstractNifiProcessor.java.
> The processor is used to scan metadata of an incoming flow file.
> The error we get in logs is as below.
> 2019-08-13 23:21:21,529 ERROR [main] o.a.nifi.controller.ExtensionBuilder 
> Could not create Processor of type org.apache.nifi.init.InitiateScan for ID 
> 6fbe2407-7799-3908-f4c4-bf2f8940bf1e; creating "Ghost" implementation
> org.apache.nifi.controller.exception.ProcessorInstantiationException:  
> org.apache.nifi.init.InitiateScan
> at 
> org.apache.nifi.controller.ExtensionBuilder.createLoggableProcessor(ExtensionBuilder.java:425)
> at 
> org.apache.nifi.controller.ExtensionBuilder.buildProcessor(ExtensionBuilder.java:191)
> at 
> org.apache.nifi.controller.flow.StandardFlowManager.createProcessor(StandardFlowManager.java:298)
> at 
> org.apache.nifi.controller.flow.StandardFlowManager.createProcessor(StandardFlowManager.java:274)
> at 
> org.apache.nifi.controller.StandardFlowSynchronizer.addProcessGroup(StandardFlowSynchronizer.java:1262)
> at 
> org.apache.nifi.controller.StandardFlowSynchronizer.addProcessGroup(StandardFlowSynchronizer.java:1389)
> at 
> org.apache.nifi.controller.StandardFlowSynchronizer.addProcessGroup(StandardFlowSynchronizer.java:1389)
> at 
> org.apache.nifi.controller.StandardFlowSynchronizer.sync(StandardFlowSynchronizer.java:362)
> at 
> org.apache.nifi.controller.FlowController.synchronize(FlowController.java:1296)
> at 
> org.apache.nifi.persistence.StandardXMLFlowConfigurationDAO.load(StandardXMLFlowConfigurationDAO.java:88)
> at 
> org.apache.nifi.controller.StandardFlowService.loadFromBytes(StandardFlowService.java:812)
> at 
> org.apache.nifi.controller.StandardFlowService.load(StandardFlowService.java:557)
> at 
> org.apache.nifi.web.contextlistener.ApplicationStartupContextListener.contextInitialized(ApplicationStartupContextListener.java:72)
> at 
> org.eclipse.jetty.server.handler.ContextHandler.callContextInitialized(ContextHandler.java:953)
> at 
> org.eclipse.jetty.servlet.ServletContextHandler.callContextInitialized(ServletContextHandler.java:558)
> at 
> org.eclipse.jetty.server.handler.ContextHandler.startContext(ContextHandler.java:918)
> at 
> org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:370)
> at 
> org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1497)
> at 
> org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1459)
> at 
> org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:848)
> at 
> org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:287)
> at 
> org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:545)
> at 
> org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
> at 
> org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:138)
> at 
> org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117)
> at 
> org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:113)
> at 
> org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
> at 
> org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:138)
> at 
> org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:108)
> at 
> org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:113)
> at 
> org.eclipse.jetty.server.handler.gzip.GzipHandler.doStart(GzipHandler.java:403)
> at 
> org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
> at 
> org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:138)
> at 
> org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117)
> at 
> org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:113)
> at 
> org.eclipse.jetty.server.handler.ContextHandlerCollection.doStart(ContextHandlerCollection.java:167)
> at 
> org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
> at 
> org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:138)
> at 
> org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117)
> at 
> 

Re: Custom Processor Upgrade

[Bimal Mehta]

For the custom processor we have, we are extending the
AbstractNifiProcessor.java.
The processor is used to scan metadata of an incoming flow file.
The error we get in logs is as below.
2019-08-13 23:21:21,529 ERROR [main] o.a.nifi.controller.ExtensionBuilder
Could not create Processor of type org.apache.nifi.init.InitiateScan for ID
6fbe2407-7799-3908-f4c4-bf2f8940bf1e; creating "Ghost" implementation
org.apache.nifi.controller.exception.ProcessorInstantiationException:
org.apache.nifi.init.InitiateScan
at
org.apache.nifi.controller.ExtensionBuilder.createLoggableProcessor(ExtensionBuilder.java:425)
at
org.apache.nifi.controller.ExtensionBuilder.buildProcessor(ExtensionBuilder.java:191)
at
org.apache.nifi.controller.flow.StandardFlowManager.createProcessor(StandardFlowManager.java:298)
at
org.apache.nifi.controller.flow.StandardFlowManager.createProcessor(StandardFlowManager.java:274)
at
org.apache.nifi.controller.StandardFlowSynchronizer.addProcessGroup(StandardFlowSynchronizer.java:1262)
at
org.apache.nifi.controller.StandardFlowSynchronizer.addProcessGroup(StandardFlowSynchronizer.java:1389)
at
org.apache.nifi.controller.StandardFlowSynchronizer.addProcessGroup(StandardFlowSynchronizer.java:1389)
at
org.apache.nifi.controller.StandardFlowSynchronizer.sync(StandardFlowSynchronizer.java:362)
at
org.apache.nifi.controller.FlowController.synchronize(FlowController.java:1296)
at
org.apache.nifi.persistence.StandardXMLFlowConfigurationDAO.load(StandardXMLFlowConfigurationDAO.java:88)
at
org.apache.nifi.controller.StandardFlowService.loadFromBytes(StandardFlowService.java:812)
at
org.apache.nifi.controller.StandardFlowService.load(StandardFlowService.java:557)
at
org.apache.nifi.web.contextlistener.ApplicationStartupContextListener.contextInitialized(ApplicationStartupContextListener.java:72)
at
org.eclipse.jetty.server.handler.ContextHandler.callContextInitialized(ContextHandler.java:953)
at
org.eclipse.jetty.servlet.ServletContextHandler.callContextInitialized(ServletContextHandler.java:558)
at
org.eclipse.jetty.server.handler.ContextHandler.startContext(ContextHandler.java:918)
at
org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:370)
at
org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1497)
at
org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1459)
at
org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:848)
at
org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:287)
at
org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:545)
at
org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at
org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:138)
at
org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117)
at
org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:113)
at
org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at
org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:138)
at
org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:108)
at
org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:113)
at
org.eclipse.jetty.server.handler.gzip.GzipHandler.doStart(GzipHandler.java:403)
at
org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at
org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:138)
at
org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117)
at
org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:113)
at
org.eclipse.jetty.server.handler.ContextHandlerCollection.doStart(ContextHandlerCollection.java:167)
at
org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at
org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:138)
at
org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117)
at
org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:113)
at
org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at
org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:138)
at org.eclipse.jetty.server.Server.start(Server.java:419)
at
org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:108)
at
org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:113)

RE: [EXT] Specifying formatters at a record field level

[Peter Wicks (pwicks)]

Not that I’m aware of. We implemented something custom that lets you specify it 
with attributes on the FlowFile (something like data.field.#.format=….), we do 
the same thing for binary/hex fields. But we didn’t contribute it as it’s part 
of a custom record processing processor that’s application specific.

Thanks,
  Peter

From: Mike Thomsen 
Sent: Wednesday, August 14, 2019 8:35 AM
To: users@nifi.apache.org
Subject: [EXT] Specifying formatters at a record field level

If there any way to specify a timestamp format string on each field that is a 
TIMESTAMP (long, logical type timestamp-millis)? We have a case where we would 
need at least three, possibly half a dozen timestamp formats to read a record 
set.

Thanks,

Mike

FlowFile Repository can't checkpoint, out of heap space.

[Peter Wicks (pwicks)]

I have a node in a cluster whose FlowFile repository grew so fast that it 
exceeded the amount of available heap space and now can't checkpoint. Or that 
is my interpretation of the error.

"Cannot update journal file flowfile_repository/journals/.journal because 
this journal  has already encountered a failure when attempting to write to the 
file."
Additionally, on restart, we see NiFi failed to restart because it ran out of 
heap space while doing a SchemaRecordReader.readFieldValue.  Feeling a bit 
stuck on where to go from here.

Based on metrics we collect, we see a large increase in FlowFile's on that node 
right before it crashed, and in linux we see the following:
94G ./journals/overflow-569618072
356G./journals/overflow-569892338

Oh, and a 280 GB checkpoint file

There are a few queues/known FlowFile's that are probably the problem, and I'm 
OK with dropping them, but there is plenty of other data in there too that I 
don't want to lose...

Thanks,
  Peter

Specifying formatters at a record field level

[Mike Thomsen]

If there any way to specify a timestamp format string on each field that is
a TIMESTAMP (long, logical type timestamp-millis)? We have a case where we
would need at least three, possibly half a dozen timestamp formats to read
a record set.

Thanks,

Mike

Re: My nifi no more serve admin interface

[Edward Armes]

Hmm, I wonder if there's a change that could be made to expose this error
so its a bit more obvious, maybe one for the Dev mailing list?

Edward

On Wed, Aug 14, 2019 at 3:12 PM Pierre Villard 
wrote:

> Glad you sorted it out and thanks for letting us know!
> In case you missed it, you might be interested by the NiFi toolkit [1]
> containing a TLS toolkit to help you with certificates [2].
>
> [1] https://nifi.apache.org/download.html
> [2]
> https://nifi.apache.org/docs/nifi-docs/html/toolkit-guide.html#tls_toolkit
>
> Le mer. 14 août 2019 à 15:54, Nicolas Delsaux  a
> écrit :
>
>> Oh damn
>>
>> It appeared (after a long search) that my keystore was incorrectly built.
>>
>> Indeed, it contained the server certificate as a trusted certificate,
>> where it should had been a key pair (with both private and public keys in)
>> as is explained in Jetty documentation (
>> https://www.eclipse.org/jetty/documentation/9.4.19.v20190610/configuring-ssl.html#understanding-certificates-and-keys
>> - see part Layout of keystore and truststore). And this happened because
>> I'm really bad at certificates.
>>
>> Sorry to have consumed some of your time, you all.
>> Le 13/08/2019 à 16:21, Nicolas Delsaux a écrit :
>>
>> oh, sorry, I forgot to mention i use the nifi docker image, with
>> configuration
>> services:
>> nifi-runner:
>> hostname: nifi-psh.adeo.com
>> image: apache/nifi:1.9.2
>> ports:
>> - "38080:8443"
>> - "5000:8000"
>> volumes:
>> -
>> ${project.basedir}/target/docker-compose/includes/nifi/node/conf:/opt/nifi/nifi-current/conf
>> -
>> ${project.basedir}/target/docker-compose/includes/nifi/node/cacerts.jks:/opt/certs/cacerts.jks
>> -
>> ${project.basedir}/target/docker-compose/includes/nifi/node/https_certificates.pkcs:/opt/certs/https_certificates.pkcs
>>
>> And port 8443 is standard http port, I guess (the port 8000 is the
>> standard debug one)
>>
>>
>> Le 13/08/2019 à 16:10, Pierre Villard a écrit :
>>
>> Might be a dumb question but I'm wondering why you're trying with port
>> 38080? Did you change the configuration to use that specific port with a
>> secured instance?
>>
>> Pierre
>>
>> Le mar. 13 août 2019 à 16:00, Nicolas Delsaux  a
>> écrit :
>>
>>> To go a little further, a test with openssl s_client gives the following
>>>
>>> nicolas-delsaux@NICOLASDELSAUX C:\Users\nicolas-delsaux
>>> $ openssl s_client -host localhost -port 38080
>>> CONNECTED(0164)
>>> 416:error:14094410:SSL routines:ssl3_read_bytes:sslv3 alert handshake
>>> failure:ssl\record\rec_layer_s3.c:1399:SSL alert number 40
>>> ---
>>> no peer certificate available
>>> ---
>>> No client certificate CA names sent
>>> ---
>>> SSL handshake has read 7 bytes and written 176 bytes
>>> Verification: OK
>>> ---
>>> New, (NONE), Cipher is (NONE)
>>> Secure Renegotiation IS NOT supported
>>> Compression: NONE
>>> Expansion: NONE
>>> No ALPN negotiated
>>> SSL-Session:
>>>  Protocol  : TLSv1.2
>>>  Cipher: 
>>>  Session-ID:
>>>  Session-ID-ctx:
>>>  Master-Key:
>>>  PSK identity: None
>>>  PSK identity hint: None
>>>  SRP username: None
>>>  Start Time: 1565704262
>>>  Timeout   : 7200 (sec)
>>>  Verify return code: 0 (ok)
>>>  Extended master secret: no
>>> ---
>>>
>>>
>>> Which i weird considering nifi outputs in its startup log the lines
>>>
>>> nifi-runner_1  | 2019-08-13 13:37:52,315 INFO [main]
>>> o.e.jetty.server.handler.ContextHandler Started
>>> o.e.j.w.WebAppContext@7cb81ae{nifi-error,/,
>>> file:///opt/nifi/nifi-current/work/jetty/nifi-web-error-1.9.2.war/webapp/,AVAILABLE
>>> }{./work/nar/framework/nifi-framework-nar-1.9.2.nar-unpacked/NAR-INF/bundled-dependencies/nifi-web-error-1.9.2.war}
>>> nifi-runner_1  | 2019-08-13 13:37:52,490 INFO [main]
>>> o.e.jetty.util.ssl.SslContextFactory
>>> x509=X509@3d94d7f3(nifi-psh.adeo.com (adeo
>>> ca),h=[nifi-psh.adeo.com],w=[]) for
>>> SslContextFactory@da1abd6[provider=null,keyStore=
>>> file:///opt/certs/https_certificates.pkcs,trustStore=file:///opt/certs/cacerts.jks
>>> ]
>>> nifi-runner_1  | 2019-08-13 13:37:52,510 INFO [main]
>>> o.eclipse.jetty.server.AbstractConnector Started
>>> ServerConnector@2066f0d3{SSL,[ssl, http/1.1]}{0.0.0.0:8443}
>>>
>>>
>>> which seems to indicate Jetty is able to listen for https connections on
>>> port 8443 using certificates described in SslContextFactory. No ?
>>>
>>> Le 13/08/2019 à 15:40, Nicolas Delsaux a écrit :
>>> > I'm currently trying to implement ldap user group authorization in
>>> nifi.
>>> >
>>> > For that, I've deployed nifi docker image with configuration files
>>> > containing required config elements (a ldap identity provider, a ldap
>>> > user group provider).
>>> >
>>> > I've also configured https with a keystore/truststore that are injected
>>> > into docker container through volumes.
>>> >
>>> > Once all is configured, i've taken the time to do some debug session to
>>> > make sure tue FileAccessPolicyProvider correctly loads my user from
>>> > ldap, and it works ok.
>>> >

Re: My nifi no more serve admin interface

[Pierre Villard]

Glad you sorted it out and thanks for letting us know!
In case you missed it, you might be interested by the NiFi toolkit [1]
containing a TLS toolkit to help you with certificates [2].

[1] https://nifi.apache.org/download.html
[2]
https://nifi.apache.org/docs/nifi-docs/html/toolkit-guide.html#tls_toolkit

Le mer. 14 août 2019 à 15:54, Nicolas Delsaux  a
écrit :

> Oh damn
>
> It appeared (after a long search) that my keystore was incorrectly built.
>
> Indeed, it contained the server certificate as a trusted certificate,
> where it should had been a key pair (with both private and public keys in)
> as is explained in Jetty documentation (
> https://www.eclipse.org/jetty/documentation/9.4.19.v20190610/configuring-ssl.html#understanding-certificates-and-keys
> - see part Layout of keystore and truststore). And this happened because
> I'm really bad at certificates.
>
> Sorry to have consumed some of your time, you all.
> Le 13/08/2019 à 16:21, Nicolas Delsaux a écrit :
>
> oh, sorry, I forgot to mention i use the nifi docker image, with
> configuration
> services:
> nifi-runner:
> hostname: nifi-psh.adeo.com
> image: apache/nifi:1.9.2
> ports:
> - "38080:8443"
> - "5000:8000"
> volumes:
> -
> ${project.basedir}/target/docker-compose/includes/nifi/node/conf:/opt/nifi/nifi-current/conf
> -
> ${project.basedir}/target/docker-compose/includes/nifi/node/cacerts.jks:/opt/certs/cacerts.jks
> -
> ${project.basedir}/target/docker-compose/includes/nifi/node/https_certificates.pkcs:/opt/certs/https_certificates.pkcs
>
> And port 8443 is standard http port, I guess (the port 8000 is the
> standard debug one)
>
>
> Le 13/08/2019 à 16:10, Pierre Villard a écrit :
>
> Might be a dumb question but I'm wondering why you're trying with port
> 38080? Did you change the configuration to use that specific port with a
> secured instance?
>
> Pierre
>
> Le mar. 13 août 2019 à 16:00, Nicolas Delsaux  a
> écrit :
>
>> To go a little further, a test with openssl s_client gives the following
>>
>> nicolas-delsaux@NICOLASDELSAUX C:\Users\nicolas-delsaux
>> $ openssl s_client -host localhost -port 38080
>> CONNECTED(0164)
>> 416:error:14094410:SSL routines:ssl3_read_bytes:sslv3 alert handshake
>> failure:ssl\record\rec_layer_s3.c:1399:SSL alert number 40
>> ---
>> no peer certificate available
>> ---
>> No client certificate CA names sent
>> ---
>> SSL handshake has read 7 bytes and written 176 bytes
>> Verification: OK
>> ---
>> New, (NONE), Cipher is (NONE)
>> Secure Renegotiation IS NOT supported
>> Compression: NONE
>> Expansion: NONE
>> No ALPN negotiated
>> SSL-Session:
>>  Protocol  : TLSv1.2
>>  Cipher: 
>>  Session-ID:
>>  Session-ID-ctx:
>>  Master-Key:
>>  PSK identity: None
>>  PSK identity hint: None
>>  SRP username: None
>>  Start Time: 1565704262
>>  Timeout   : 7200 (sec)
>>  Verify return code: 0 (ok)
>>  Extended master secret: no
>> ---
>>
>>
>> Which i weird considering nifi outputs in its startup log the lines
>>
>> nifi-runner_1  | 2019-08-13 13:37:52,315 INFO [main]
>> o.e.jetty.server.handler.ContextHandler Started
>> o.e.j.w.WebAppContext@7cb81ae{nifi-error,/,
>> file:///opt/nifi/nifi-current/work/jetty/nifi-web-error-1.9.2.war/webapp/,AVAILABLE
>> }{./work/nar/framework/nifi-framework-nar-1.9.2.nar-unpacked/NAR-INF/bundled-dependencies/nifi-web-error-1.9.2.war}
>> nifi-runner_1  | 2019-08-13 13:37:52,490 INFO [main]
>> o.e.jetty.util.ssl.SslContextFactory
>> x509=X509@3d94d7f3(nifi-psh.adeo.com (adeo
>> ca),h=[nifi-psh.adeo.com],w=[]) for
>> SslContextFactory@da1abd6[provider=null,keyStore=
>> file:///opt/certs/https_certificates.pkcs,trustStore=file:///opt/certs/cacerts.jks
>> ]
>> nifi-runner_1  | 2019-08-13 13:37:52,510 INFO [main]
>> o.eclipse.jetty.server.AbstractConnector Started
>> ServerConnector@2066f0d3{SSL,[ssl, http/1.1]}{0.0.0.0:8443}
>>
>>
>> which seems to indicate Jetty is able to listen for https connections on
>> port 8443 using certificates described in SslContextFactory. No ?
>>
>> Le 13/08/2019 à 15:40, Nicolas Delsaux a écrit :
>> > I'm currently trying to implement ldap user group authorization in nifi.
>> >
>> > For that, I've deployed nifi docker image with configuration files
>> > containing required config elements (a ldap identity provider, a ldap
>> > user group provider).
>> >
>> > I've also configured https with a keystore/truststore that are injected
>> > into docker container through volumes.
>> >
>> > Once all is configured, i've taken the time to do some debug session to
>> > make sure tue FileAccessPolicyProvider correctly loads my user from
>> > ldap, and it works ok.
>> >
>> > Unfortunatly, now, when i try to load Nifi admin interface, I get a
>> > strange http response containing only the string "   �  P".
>> >
>> > In other words,
>> >
>> >
>> > nicolas-delsaux@NICOLASDELSAUX C:\Users\nicolas-delsaux
>> > $ curl -v -H "Host: nifi-psh.adeo.com" http://localhost:38080/
>> --output -
>> > *   Trying ::1...
>> > 

Re: My nifi no more serve admin interface

[Nicolas Delsaux]

Oh damn

It appeared (after a long search) that my keystore was incorrectly built.

Indeed, it contained the server certificate as a trusted certificate,
where it should had been a key pair (with both private and public keys
in) as is explained in Jetty documentation
(https://www.eclipse.org/jetty/documentation/9.4.19.v20190610/configuring-ssl.html#understanding-certificates-and-keys
- see part Layout of keystore and truststore). And this happened because
I'm really bad at certificates.

Sorry to have consumed some of your time, you all.

Le 13/08/2019 à 16:21, Nicolas Delsaux a écrit :


oh, sorry, I forgot to mention i use the nifi docker image, with
configuration

services:
nifi-runner:
hostname: nifi-psh.adeo.com
image: apache/nifi:1.9.2
ports:
- "38080:8443"
- "5000:8000"
volumes:
-
${project.basedir}/target/docker-compose/includes/nifi/node/conf:/opt/nifi/nifi-current/conf
-
${project.basedir}/target/docker-compose/includes/nifi/node/cacerts.jks:/opt/certs/cacerts.jks
-
${project.basedir}/target/docker-compose/includes/nifi/node/https_certificates.pkcs:/opt/certs/https_certificates.pkcs

And port 8443 is standard http port, I guess (the port 8000 is the
standard debug one)


Le 13/08/2019 à 16:10, Pierre Villard a écrit :

Might be a dumb question but I'm wondering why you're trying with
port 38080? Did you change the configuration to use that specific
port with a secured instance?

Pierre

Le mar. 13 août 2019 à 16:00, Nicolas Delsaux mailto:nicolas.dels...@gmx.fr>> a écrit :

To go a little further, a test with openssl s_client gives the
following

nicolas-delsaux@NICOLASDELSAUX C:\Users\nicolas-delsaux
$ openssl s_client -host localhost -port 38080
CONNECTED(0164)
416:error:14094410:SSL routines:ssl3_read_bytes:sslv3 alert handshake
failure:ssl\record\rec_layer_s3.c:1399:SSL alert number 40
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 7 bytes and written 176 bytes
Verification: OK
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
 Protocol  : TLSv1.2
 Cipher    : 
 Session-ID:
 Session-ID-ctx:
 Master-Key:
 PSK identity: None
 PSK identity hint: None
 SRP username: None
 Start Time: 1565704262
 Timeout   : 7200 (sec)
 Verify return code: 0 (ok)
 Extended master secret: no
---


Which i weird considering nifi outputs in its startup log the lines

nifi-runner_1  | 2019-08-13 13:37:52,315 INFO [main]
o.e.jetty.server.handler.ContextHandler Started

o.e.j.w.WebAppContext@7cb81ae{nifi-error,/,file:///opt/nifi/nifi-current/work/jetty/nifi-web-error-1.9.2.war/webapp/,AVAILABLE}{./work/nar/framework/nifi-framework-nar-1.9.2.nar-unpacked/NAR-INF/bundled-dependencies/nifi-web-error-1.9.2.war}
nifi-runner_1  | 2019-08-13 13:37:52,490 INFO [main]
o.e.jetty.util.ssl.SslContextFactory
x509=X509@3d94d7f3(nifi-psh.adeo.com  (adeo
ca),h=[nifi-psh.adeo.com ],w=[]) for

SslContextFactory@da1abd6[provider=null,keyStore=file:///opt/certs/https_certificates.pkcs,trustStore=file:///opt/certs/cacerts.jks]
nifi-runner_1  | 2019-08-13 13:37:52,510 INFO [main]
o.eclipse.jetty.server.AbstractConnector Started
ServerConnector@2066f0d3{SSL,[ssl, http/1.1]}{0.0.0.0:8443
}


which seems to indicate Jetty is able to listen for https
connections on
port 8443 using certificates described in SslContextFactory. No ?

Le 13/08/2019 à 15:40, Nicolas Delsaux a écrit :
> I'm currently trying to implement ldap user group authorization
in nifi.
>
> For that, I've deployed nifi docker image with configuration files
> containing required config elements (a ldap identity provider,
a ldap
> user group provider).
>
> I've also configured https with a keystore/truststore that are
injected
> into docker container through volumes.
>
> Once all is configured, i've taken the time to do some debug
session to
> make sure tue FileAccessPolicyProvider correctly loads my user from
> ldap, and it works ok.
>
> Unfortunatly, now, when i try to load Nifi admin interface, I get a
> strange http response containing only the string "   � P".
>
> In other words,
>
>
> nicolas-delsaux@NICOLASDELSAUX C:\Users\nicolas-delsaux
> $ curl -v -H "Host: nifi-psh.adeo.com
" http://localhost:38080/ --output -
> *   Trying ::1...
> * TCP_NODELAY set
> * Connected to localhost (::1) port 38080 (#0)
> > GET / HTTP/1.1
> > Host: nifi-psh.adeo.com 
> > User-Agent: curl/7.55.1
> > Accept: */*
> >
> §♥♥ ☻☻P* Connection 

Re: Custom Processor Upgrade

[Bryan Bende]

Without access to the code for your NAR I can only really guess, but it
sounds like an exception is happening  when trying to call the constructor
of your processor and then it bounces into creating a ghost processor.

What is in the logs at the time you get the ghost processor?

On Tue, Aug 13, 2019 at 10:54 PM Bimal Mehta  wrote:

> Does that mean I need to recreate the processor? Or there is some
> workaround?
>
> The processor gets unpacked and its bundled dependencies go in NAR_INF.
> However when I drag the processor on the canvas, it comes with a yellow
> triangle (and gives the error message I stated above) and properties are
> missing as well.
>
>
> On Tue, Aug 13, 2019 at 10:47 PM Bryan Bende  wrote:
>
>> I don’t remember all the reasoning behind the change, but it had to do
>> with an issue when we upgraded Jetty...
>>
>> https://issues.apache.org/jira/browse/NIFI-5479
>>
>> On Tue, Aug 13, 2019 at 9:47 PM Bimal Mehta  wrote:
>>
>>> Yes it does show as an option.
>>> One thing I noticed is that the when the nar is unpacked, the bundled
>>> dependencies are inside META_INF in the work folder in NiFi 1.6.0, however
>>> in NiFI 1.9.0 they go inside NAR_INF.
>>> Why does this happen?
>>> It seems the custom processor that we have uses Springboot, and
>>> references applicationcontext file which was inside META_INF when it was
>>> built. However I cant see that file anymore in the unpacked nar.
>>>
>>> On Tue, Aug 13, 2019 at 8:57 PM Bryan Bende  wrote:
>>>
 Does that custom processor type show as an option if you try to add a
 new processor to the canvas?

 On Tue, Aug 13, 2019 at 4:54 PM Bimal Mehta  wrote:

> Hi Mike and Bryan,
>
> One of my custom processors appears as inactive in NiFi with a yellow
> triangle error.
> When I hover over it I see a message saying 'Missing Processor'
> validated against 'Any Property' is invalid. This is not a valid 
> processor.
> In the log it seems to invoke GhostProcessor.java which is giving the
> above error when restarting nifi.
> This custom processor sits (with my other processors) in my
> custom_lib  folder and I have provided that path in the nifi properties
> file as
>
> *nifi.nar.library.directory.custom=/opt/nifi/custom_lib*
>
>
> Not sure what I missed?
>
> Do I need to make entry of this custom processor somewhere?
>
>
> On Thu, Aug 8, 2019 at 9:14 AM Bimal Mehta  wrote:
>
>> Thanks Mike and Bryan.
>> Yes it seems my template was still referring the old version.
>> I will have it updated now and will reimport.
>> Also the version of NiFi we are using is the one that comes with CDF.
>> I am not sure if CDF supports 1.9.2 yet or not. I will reach out to
>> Cloudera and see if we can get it upgraded.
>>
>>
>>
>> On Thu, Aug 8, 2019, 8:51 AM Bryan Bende  wrote:
>>
>>> What is in the template for the bundle coordinates of your processor?
>>> and does that match the coordinates of the NAR that is deployed?
>>>
>>> Example:
>>>
>>>
>>>   org.apache.nifi
>>>   nifi-update-attribute-nar
>>>   1.10.0-SNAPSHOT
>>> 
>>>
>>> If you made a new version of your NAR, say 2.0.0 and your template
>>> references 1.0.0, then you'll need to update your template.
>>>
>>> On Wed, Aug 7, 2019 at 10:05 PM Mike Thomsen 
>>> wrote:
>>> >
>>> > If it's happening immediately upon trying to import the template,
>>> I believe that's the error message saying that the 1.9 instance cannot 
>>> find
>>> the NAR file which provided the processor. Also, if you're referring to
>>> 1.9.0 and not 1.9.2 you're going to want to upgrade to the latter 
>>> because
>>> there are a few critical bugs fixed in 1.9.2.
>>> >
>>> > On Wed, Aug 7, 2019 at 9:19 PM Bimal Mehta 
>>> wrote:
>>> >>
>>> >> Thanks Bryan.
>>> >> My custom processors are part of a template. However when I try
>>> to import my template in NiFi 1.9, I get an error message saying
>>> >> PutFeedMetadata is not known to this NiFi instance. I did update
>>> all the dependencies to NiFi 1.9 and even the plugins. We are using a
>>> Cloudera distributed version of NiFi 1.9.
>>> >> Any idea why is this happening?
>>> >>
>>> >> Thanks
>>> >>
>>> >>
>>> >>
>>> >> On Wed, Aug 7, 2019 at 3:46 PM Bryan Bende 
>>> wrote:
>>> >>>
>>> >>> Hello,
>>> >>>
>>> >>> Most likely your processor built against 1.6 would run fine in
>>> 1.9,
>>> >>> but to make sure you just need to update any nifi dependencies
>>> in your
>>> >>> poms to 1.9.2.
>>> >>>
>>> >>> If you created your project from the archetype and didn't change
>>> >>> anything, then this should just be changing the parent in the
>>> root pom
>>> >>> to the new version of 

Re: My nifi no more serve admin interface

[Edward Armes]

Hi Nicolas,

This is another dump question. As I've only ever seen this before when I've
accidentally connect to a secured Nifi cluster over HTTP and not HTTPS.
>From I've seen Nifi won't ask your browser to do a connection upgrade (HTTP
-> HTTPS),

When you type in the address are you sure your browser is not cutting of
the *s* from the *https* in the URL you entering?

Edward

On Tue, Aug 13, 2019 at 3:22 PM Nicolas Delsaux 
wrote:

> oh, sorry, I forgot to mention i use the nifi docker image, with
> configuration
> services:
> nifi-runner:
> hostname: nifi-psh.adeo.com
> image: apache/nifi:1.9.2
> ports:
> - "38080:8443"
> - "5000:8000"
> volumes:
> -
> ${project.basedir}/target/docker-compose/includes/nifi/node/conf:/opt/nifi/nifi-current/conf
> -
> ${project.basedir}/target/docker-compose/includes/nifi/node/cacerts.jks:/opt/certs/cacerts.jks
> -
> ${project.basedir}/target/docker-compose/includes/nifi/node/https_certificates.pkcs:/opt/certs/https_certificates.pkcs
>
> And port 8443 is standard http port, I guess (the port 8000 is the
> standard debug one)
>
>
> Le 13/08/2019 à 16:10, Pierre Villard a écrit :
>
> Might be a dumb question but I'm wondering why you're trying with port
> 38080? Did you change the configuration to use that specific port with a
> secured instance?
>
> Pierre
>
> Le mar. 13 août 2019 à 16:00, Nicolas Delsaux  a
> écrit :
>
>> To go a little further, a test with openssl s_client gives the following
>>
>> nicolas-delsaux@NICOLASDELSAUX C:\Users\nicolas-delsaux
>> $ openssl s_client -host localhost -port 38080
>> CONNECTED(0164)
>> 416:error:14094410:SSL routines:ssl3_read_bytes:sslv3 alert handshake
>> failure:ssl\record\rec_layer_s3.c:1399:SSL alert number 40
>> ---
>> no peer certificate available
>> ---
>> No client certificate CA names sent
>> ---
>> SSL handshake has read 7 bytes and written 176 bytes
>> Verification: OK
>> ---
>> New, (NONE), Cipher is (NONE)
>> Secure Renegotiation IS NOT supported
>> Compression: NONE
>> Expansion: NONE
>> No ALPN negotiated
>> SSL-Session:
>>  Protocol  : TLSv1.2
>>  Cipher: 
>>  Session-ID:
>>  Session-ID-ctx:
>>  Master-Key:
>>  PSK identity: None
>>  PSK identity hint: None
>>  SRP username: None
>>  Start Time: 1565704262
>>  Timeout   : 7200 (sec)
>>  Verify return code: 0 (ok)
>>  Extended master secret: no
>> ---
>>
>>
>> Which i weird considering nifi outputs in its startup log the lines
>>
>> nifi-runner_1  | 2019-08-13 13:37:52,315 INFO [main]
>> o.e.jetty.server.handler.ContextHandler Started
>> o.e.j.w.WebAppContext@7cb81ae{nifi-error,/,
>> file:///opt/nifi/nifi-current/work/jetty/nifi-web-error-1.9.2.war/webapp/,AVAILABLE
>> }{./work/nar/framework/nifi-framework-nar-1.9.2.nar-unpacked/NAR-INF/bundled-dependencies/nifi-web-error-1.9.2.war}
>> nifi-runner_1  | 2019-08-13 13:37:52,490 INFO [main]
>> o.e.jetty.util.ssl.SslContextFactory
>> x509=X509@3d94d7f3(nifi-psh.adeo.com (adeo
>> ca),h=[nifi-psh.adeo.com],w=[]) for
>> SslContextFactory@da1abd6[provider=null,keyStore=
>> file:///opt/certs/https_certificates.pkcs,trustStore=file:///opt/certs/cacerts.jks
>> ]
>> nifi-runner_1  | 2019-08-13 13:37:52,510 INFO [main]
>> o.eclipse.jetty.server.AbstractConnector Started
>> ServerConnector@2066f0d3{SSL,[ssl, http/1.1]}{0.0.0.0:8443}
>>
>>
>> which seems to indicate Jetty is able to listen for https connections on
>> port 8443 using certificates described in SslContextFactory. No ?
>>
>> Le 13/08/2019 à 15:40, Nicolas Delsaux a écrit :
>> > I'm currently trying to implement ldap user group authorization in nifi.
>> >
>> > For that, I've deployed nifi docker image with configuration files
>> > containing required config elements (a ldap identity provider, a ldap
>> > user group provider).
>> >
>> > I've also configured https with a keystore/truststore that are injected
>> > into docker container through volumes.
>> >
>> > Once all is configured, i've taken the time to do some debug session to
>> > make sure tue FileAccessPolicyProvider correctly loads my user from
>> > ldap, and it works ok.
>> >
>> > Unfortunatly, now, when i try to load Nifi admin interface, I get a
>> > strange http response containing only the string "   �  P".
>> >
>> > In other words,
>> >
>> >
>> > nicolas-delsaux@NICOLASDELSAUX C:\Users\nicolas-delsaux
>> > $ curl -v -H "Host: nifi-psh.adeo.com" http://localhost:38080/
>> --output -
>> > *   Trying ::1...
>> > * TCP_NODELAY set
>> > * Connected to localhost (::1) port 38080 (#0)
>> > > GET / HTTP/1.1
>> > > Host: nifi-psh.adeo.com
>> > > User-Agent: curl/7.55.1
>> > > Accept: */*
>> > >
>> > §♥♥ ☻☻P* Connection #0 to host localhost left intact
>> >
>> >
>> > http does not work (which i expects, since I've configured
>> > authentication/authorization
>> >
>> > nicolas-delsaux@NICOLASDELSAUX C:\Users\nicolas-delsaux
>> > $ curl -v -H "Host: nifi-psh.adeo.com" https://localhost:38080/
>> > --output -
>> > *   Trying ::1...
>> > * TCP_NODELAY set
>> > 

Re: Anti-Virus Scanning

[Jason Csencsits]

Edward,
Thank you .  Appreciate the help and details.
Jason



Sent via the Samsung Galaxy Note8, an AT 4G LTE smartphone


 Original message 
From: Edward Armes 
Date: 8/14/19 5:28 AM (GMT-05:00)
To: users@nifi.apache.org
Subject: Re: Anti-Virus Scanning

Hi Jason,

This isn't explicitly documented anywhere, however the locations of all the key 
paths for Nifi can be found in the documentation in general. Hopefully a 
combination of this email thread and official documentation should be enough 
for your client to give the AV exemptions you need.

Getting information added to the documentation is simple in one aspect as 
ticket just has to be raised on the ASF Nifi JIRA. However, given that large 
elements of Nifi are configurable via plugins it would be difficult to give 
definitive advise, as different plugins will have different AV requirements.

In the meantime I have created 
NIFI-6553 to look at improving 
the documentation around this as we are missing other bits of information like 
SELinux configuration that I think would also be useful to have in the 
documentation

Edward

On Tue, Aug 13, 2019 at 7:31 PM Jason Csencsits 
mailto:jcsencs...@technicallycreative.com>> 
wrote:
Joe,
Thank you for the information.  Is this documented anywhere as I have a client 
looking for it from Apache?.

Thank you,
:::
Jason Csencsits
Manager of Technical Operations
Technically Creative Inc.
Simplifying IT Solutions

Office: 845.725.7883
jcsencs...@technicallycreative.com
www.TechnicallyCreative.com

::

From: Joe Witt mailto:joe.w...@gmail.com>>
Sent: Tuesday, August 13, 2019 2:27 PM
To: users@nifi.apache.org
Subject: Re: Anti-Virus Scanning

Jason

The work dir gets created at startup and possible as new nars are loaded.  I 
think you'd be ok to scan this.

The flowfile and content repository and provenance directories as configured 
should be skipped. The logs dir should be skipped.  The state directory should 
be skipped.  All else I believe would be fair game.

Thanks

On Tue, Aug 13, 2019 at 2:24 PM Jason Csencsits 
mailto:jcsencs...@technicallycreative.com>> 
wrote:
What is the recommended anti virus scanning exclusions from active scans. Can 
not find anything in the documents. Need to make sure my linux redhat scans do 
not compromise the flow files or anything else.

Thank you,
:::
Jason Csencsits
Manager of Technical Operations
Technically Creative Inc.
Simplifying IT Solutions

Office: 845.725.7883
jcsencs...@technicallycreative.com
www.TechnicallyCreative.com

::

Re: Anti-Virus Scanning

[Edward Armes]

Hi Jason,

This isn't explicitly documented anywhere, however the locations of all the
key paths for Nifi can be found in the documentation in general. Hopefully
a combination of this email thread and official documentation should be
enough for your client to give the AV exemptions you need.

Getting information added to the documentation is simple in one aspect as
ticket just has to be raised on the ASF Nifi JIRA. However, given that
large elements of Nifi are configurable via plugins it would be difficult
to give definitive advise, as different plugins will have different AV
requirements.

In the meantime I have created NIFI-6553
 to look at improving the
documentation around this as we are missing other bits of information like
SELinux configuration that I think would also be useful to have in the
documentation

Edward

On Tue, Aug 13, 2019 at 7:31 PM Jason Csencsits <
jcsencs...@technicallycreative.com> wrote:

> Joe,
>
> Thank you for the information.  Is this documented anywhere as I have a
> client looking for it from Apache?.
>
>
>
> Thank you,
>
> :::
>
> *Jason Csencsits *
>
> Manager of Technical Operations
>
> Technically Creative Inc.
>
> *Simplifying IT Solutions*
>
>
>
> Office: 845.725.7883
>
> jcsencs...@technicallycreative.com
>
> www.TechnicallyCreative.com
> 
>
>
>
> ::
>
>
>
> *From:* Joe Witt 
> *Sent:* Tuesday, August 13, 2019 2:27 PM
> *To:* users@nifi.apache.org
> *Subject:* Re: Anti-Virus Scanning
>
>
>
> Jason
>
>
>
> The work dir gets created at startup and possible as new nars are loaded.
> I think you'd be ok to scan this.
>
>
>
> The flowfile and content repository and provenance directories as
> configured should be skipped. The logs dir should be skipped.  The state
> directory should be skipped.  All else I believe would be fair game.
>
>
>
> Thanks
>
>
>
> On Tue, Aug 13, 2019 at 2:24 PM Jason Csencsits <
> jcsencs...@technicallycreative.com> wrote:
>
> What is the recommended anti virus scanning exclusions from active scans.
> Can not find anything in the documents. Need to make sure my linux redhat
> scans do not compromise the flow files or anything else.
>
>
>
> Thank you,
>
> :::
>
> *Jason Csencsits *
>
> Manager of Technical Operations
>
> Technically Creative Inc.
>
> *Simplifying IT Solutions*
>
>
>
> Office: 845.725.7883
>
> jcsencs...@technicallycreative.com
>
> www.TechnicallyCreative.com
> 
>
>
>
> ::
>
>
>
>