Re: NiFi codebase warnings
Hi Joe Understanding completely the situation, I see a lot of open PRs, and the challenge reviewing them may present specially. I also understand the extra care with PRs from non regular contributors. I'll keep only once open at a time from now on. If you'd like that I create a set of Jira issues and someone can assign them to me when there's bandwidth available to review, that is also ok for me. If there's any extra way I can facilitate the process, let me know. JL On Thu, Apr 1, 2021 at 6:13 PM Joe Witt wrote: > Jose > > Thanks for your contributions and note. Such contributions and intent > are welcome and appreciated. > > The challenge is our PR review bandwidth. We receive far more > contributions than we have review bandwidth for. Four of your PRs > touch a lot of files and the problem is if we don't get to them > quickly your PR will get out of date/have merge conflicts. The energy > to contribute is awesome and I dont want to deter you. I just want to > recommend an approach to make it more enjoyable. Try to focus on a > single PR at a time for instance. Once you build some momentum and > folks in the community and doing reviews get more familiar with your > work it can get easier too. > > Anyway just wanted to give a little perspective. What you have done > is perfectly fine. Just sharing the challenge so many substantive PRs > at once can present. > > Thanks > > On Wed, Mar 31, 2021 at 3:54 AM José Luis Pedrosa > wrote: > > > > Hi Devs/Maintainers > > > > I've recently started using NiFi and I thought it would be a good idea to > > give something back to the community while getting familiar with the > > different areas of NiFi. I am writing to explain the relation between few > > PRs that are submitted and check with you if it's a good idea and if it > > would require a different approach. > > > > I saw during compilation that are some warnings related to deprecated > APIS, > > that could be solved without a lot of issues, so I've submitted a PR for > > each of the different APIs being deprecated, some of them internal, some > of > > them external, [1],[2],[3],[4]. > > I've also submitted an small PR regarding code inspections [5] (Make > inner > > classes static where possible). I guess after some more PRs all warnings > > would be gone and it would be possible to enable warnings as error during > > compilation. > > > > So those are the questions that popped in my mind: > > > >- Are these kind of initiatives wellcome? > >- Should I handle it differently at code level? > >- Should I do something different at Jira rather than creating > >individual issues? (Maybe an epic level that comprises all of this?) > >- Any particular improvement that sounds more urgent or you'd like to > >get it done first? > > > > > > Kind Regards > > JL > > > > [1] https://github.com/apache/nifi/pull/4947 > > [2] https://github.com/apache/nifi/pull/4945 > > [3] https://github.com/apache/nifi/pull/4942 > > [4] https://github.com/apache/nifi/pull/4941 > > [5] https://github.com/apache/nifi/pull/4940 >
Re: NiFi codebase warnings
Jose Thanks for your contributions and note. Such contributions and intent are welcome and appreciated. The challenge is our PR review bandwidth. We receive far more contributions than we have review bandwidth for. Four of your PRs touch a lot of files and the problem is if we don't get to them quickly your PR will get out of date/have merge conflicts. The energy to contribute is awesome and I dont want to deter you. I just want to recommend an approach to make it more enjoyable. Try to focus on a single PR at a time for instance. Once you build some momentum and folks in the community and doing reviews get more familiar with your work it can get easier too. Anyway just wanted to give a little perspective. What you have done is perfectly fine. Just sharing the challenge so many substantive PRs at once can present. Thanks On Wed, Mar 31, 2021 at 3:54 AM José Luis Pedrosa wrote: > > Hi Devs/Maintainers > > I've recently started using NiFi and I thought it would be a good idea to > give something back to the community while getting familiar with the > different areas of NiFi. I am writing to explain the relation between few > PRs that are submitted and check with you if it's a good idea and if it > would require a different approach. > > I saw during compilation that are some warnings related to deprecated APIS, > that could be solved without a lot of issues, so I've submitted a PR for > each of the different APIs being deprecated, some of them internal, some of > them external, [1],[2],[3],[4]. > I've also submitted an small PR regarding code inspections [5] (Make inner > classes static where possible). I guess after some more PRs all warnings > would be gone and it would be possible to enable warnings as error during > compilation. > > So those are the questions that popped in my mind: > >- Are these kind of initiatives wellcome? >- Should I handle it differently at code level? >- Should I do something different at Jira rather than creating >individual issues? (Maybe an epic level that comprises all of this?) >- Any particular improvement that sounds more urgent or you'd like to >get it done first? > > > Kind Regards > JL > > [1] https://github.com/apache/nifi/pull/4947 > [2] https://github.com/apache/nifi/pull/4945 > [3] https://github.com/apache/nifi/pull/4942 > [4] https://github.com/apache/nifi/pull/4941 > [5] https://github.com/apache/nifi/pull/4940
Re: Stale PR automatic closure
Thanks for the comments. Regarding the stale PR management I'll proceed with what I suggested in the earlier email hopefully sometime in the next couple of days. Thanks On Fri, Mar 26, 2021 at 12:21 PM Chris Sampson wrote: > > Somewhat related... would there be benefit in labeling PRs [1] to group > them for people to more easily identify changes they may be more > comfortable in reviewing? > > For example, I've submitted several Elasticsearch related PRs that are > awaiting review (and would be relatively happy trying to review similar), > but I'd be less confident in reviewing something related to Hadoop, for > example. > > Such labels would allow people to quickly filter for things related to the > areas in which they're interested/confident to review. > > Of course, this would need someone to add labels to existing PRs and maybe > a request for people to add them to new PRs could be part of the PR > template? > > > [1] > https://docs.github.com/en/github/managing-your-work-on-github/managing-labels > > Cheers, > > Chris Sampson > > On Thu, 25 Mar 2021, 20:47 Joe Witt, wrote: > > > Got ya > > > > On Thu, Mar 25, 2021 at 1:36 PM Otto Fowler > > wrote: > > > > > > > > https://cwiki.apache.org/confluence/display/INFRA/Git+-+.asf.yaml+features > > > What I mean is, beyond the GitHub settings it has, if it offered PR > > pruning > > > ( if that were possible ). > > > Seems silly that everyone has to roll their own. > > > > > > On Thu, Mar 25, 2021 at 3:05 PM Joe Witt wrote: > > > > > > > Otto > > > > > > > > I'm not clear on what you mean. But here we're talking about our > > > > communities usage of Github in particular as a convenient vehicle to > > > > support contributions. The ASF/Git (gitbox) is a different animal. > > > > > > > > Thanks > > > > > > > > On Thu, Mar 25, 2021 at 12:02 PM Otto Fowler > > > > wrote: > > > > > > > > > > What would be nice is if this could be part of the ASF official .yml > > for > > > > git > > > > > > > > > > On Thu, Mar 25, 2021 at 2:42 PM Joe Witt wrote: > > > > > > > > > > > Team, > > > > > > > > > > > > We've discussed it in various forums but I want to formally move to > > > > > > taking action on our PR situation. As of a couple hours ago we had > > > > > > 280 or so open PRs. Many many many of these are quite old (two or > > > > > > more years). These aren't just abandoned PRs because they were not > > > > > > good enough to be clear. There are actually a lot of good and > > > > > > interesting things in here. I think though we simply cannot keep > > up > > > > > > with the PRs that come in. The most difficult appear to be less > > about > > > > > > fixing bugs and more on adding features. Anyway, we clearly get a > > lot > > > > > > of PRs and we clearly close a lot of PRs but many remain > > unaddressed. > > > > > > Other communities such as Apache Spark [1],[2] have provided an > > auto > > > > > > close mechanism. It is intended to keep the PR queue tidy and not > > > > > > just go endlessly without a response. > > > > > > > > > > > > I am planning to implement the same with a 180 day > > auto-stale/closure > > > > > > using the same github/ci workflow [3] based on github action [4]. > > You > > > > > > can see the results of this here [5]. > > > > > > > > > > > > If someone disagrees please share a workable alternative that > > doesn't > > > > > > leave the PRs or those that submitted them hanging indefinitely > > which > > > > > > you are willing to implement. > > > > > > > > > > > > [1] > > > > > > > > > > > > http://apache-spark-developers-list.1001551.n3.nabble.com/Handling-stale-PRs-td8015i20.html#a9684 > > > > > > [2] > > > > > > > > > > > > http://apache-spark-developers-list.1001551.n3.nabble.com/Closing-stale-PRs-with-a-GitHub-Action-td28477.html > > > > > > [3] > > > > > > > > > > > > https://github.com/apache/spark/blob/master/.github/workflows/stale.yml > > > > > > [4] https://github.com/marketplace/actions/close-stale-issues > > > > > > [5] > > > > > > > > > > > > https://github.com/apache/spark/pulls?q=is%3Apr+is%3Aclosed+label%3AStale > > > > > > > > > > > > Thanks > > > > > > Joe > > > > > > > > > > > >
Developing a custom processor which manages it state
Hi all, I'm developing a custom processor which requires state management. According to the NIFI developer guide, the entire state map should be less than 1MB if we are doing the state management in cluster scope. However, since the state map can be grown more than 1MB in my case, I'm planning to use an external db which should be connected via a JDBC connection. Here, can we configure NIFI to use an external db for state management instead of default zookeeper mechanism? Or do I need to implement the whole state management by myself? If it is required to implement the state management by myself, I'm wondering if you can point out the things which I should be careful of. Thank You. Best Regards, Vibhath.
Re: Nifi authentication through Kerberos issues
That was it! I pulled out the line "renew_lifetime = 7d" and it worked!
Thank you so much.
On Thu, Apr 1, 2021 at 7:40 AM Bryan Bende wrote:
> The important part is:
>
> Caused by: sun.security.krb5.internal.KrbApErrException: Message stream
> modified (41)
>
> The code that produces this exception looks like this:
>
> // Reply to a renewable request should be renewable, but if request does
> // not contain renewable, KDC is free to issue a renewable ticket (for
> // example, if ticket_lifetime is too big).
> if (req.reqBody.kdcOptions.get(KDCOptions.RENEWABLE) &&
> !rep.encKDCRepPart.flags.get(KDCOptions.RENEWABLE)) {
> throw new KrbApErrException(Krb5.KRB_AP_ERR_MODIFIED);
> }
>
> From googling, a possible solution here:
> https://bugs.centos.org/view.php?id=17000
>
> On Wed, Mar 31, 2021 at 6:57 PM Derek Richardson wrote:
> >
> > It doesn't look like anything to me, but here's the stacktrace for when
> > logback.xml has all of the user_file stuff in debug mode:
> >
> > 2021-03-31 22:54:13,670 INFO [NiFi Web Server-22]
> > o.a.n.w.a.c.IllegalArgumentExceptionMapper
> > java.lang.IllegalArgumentException: The supplied username and password
> are
> > not valid.. Returning Bad Request response.
> > 2021-03-31 22:54:13,672 DEBUG [NiFi Web Server-22]
> > o.a.n.w.a.c.IllegalArgumentExceptionMapper
> > java.lang.IllegalArgumentException: The supplied username and password
> are
> > not valid.
> > at
> >
> org.apache.nifi.web.api.AccessResource.createAccessToken(AccessResource.java:734)
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > at
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> > at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > at java.lang.reflect.Method.invoke(Method.java:498)
> > at
> >
> org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:76)
> > at
> >
> org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:148)
> > at
> >
> org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:191)
> > at
> >
> org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$ResponseOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:200)
> > at
> >
> org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:103)
> > at
> >
> org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:493)
> > at
> >
> org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:415)
> > at
> >
> org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:104)
> > at
> org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:277)
> > at org.glassfish.jersey.internal.Errors$1.call(Errors.java:272)
> > at org.glassfish.jersey.internal.Errors$1.call(Errors.java:268)
> > at org.glassfish.jersey.internal.Errors.process(Errors.java:316)
> > at org.glassfish.jersey.internal.Errors.process(Errors.java:298)
> > at org.glassfish.jersey.internal.Errors.process(Errors.java:268)
> > at
> >
> org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:289)
> > at
> org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:256)
> > at
> >
> org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:703)
> > at
> >
> org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:416)
> > at
> org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:370)
> > at
> >
> org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:389)
> > at
> >
> org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:342)
> > at
> >
> org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:229)
> > at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:865)
> > at
> >
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1655)
> > at
> org.apache.nifi.web.filter.RequestLogger.doFilter(RequestLogger.java:66)
> > at
> >
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
> > at
> >
> org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:208)
> > at
> >
> org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:177)
> > at
> >
> org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:347)
> > at
> >
> org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:263)
> > at
> >
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
> > at
Re: Nifi authentication through Kerberos issues
The important part is:
Caused by: sun.security.krb5.internal.KrbApErrException: Message stream
modified (41)
The code that produces this exception looks like this:
// Reply to a renewable request should be renewable, but if request does
// not contain renewable, KDC is free to issue a renewable ticket (for
// example, if ticket_lifetime is too big).
if (req.reqBody.kdcOptions.get(KDCOptions.RENEWABLE) &&
!rep.encKDCRepPart.flags.get(KDCOptions.RENEWABLE)) {
throw new KrbApErrException(Krb5.KRB_AP_ERR_MODIFIED);
}
>From googling, a possible solution here:
https://bugs.centos.org/view.php?id=17000
On Wed, Mar 31, 2021 at 6:57 PM Derek Richardson wrote:
>
> It doesn't look like anything to me, but here's the stacktrace for when
> logback.xml has all of the user_file stuff in debug mode:
>
> 2021-03-31 22:54:13,670 INFO [NiFi Web Server-22]
> o.a.n.w.a.c.IllegalArgumentExceptionMapper
> java.lang.IllegalArgumentException: The supplied username and password are
> not valid.. Returning Bad Request response.
> 2021-03-31 22:54:13,672 DEBUG [NiFi Web Server-22]
> o.a.n.w.a.c.IllegalArgumentExceptionMapper
> java.lang.IllegalArgumentException: The supplied username and password are
> not valid.
> at
> org.apache.nifi.web.api.AccessResource.createAccessToken(AccessResource.java:734)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:76)
> at
> org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:148)
> at
> org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:191)
> at
> org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$ResponseOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:200)
> at
> org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:103)
> at
> org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:493)
> at
> org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:415)
> at
> org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:104)
> at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:277)
> at org.glassfish.jersey.internal.Errors$1.call(Errors.java:272)
> at org.glassfish.jersey.internal.Errors$1.call(Errors.java:268)
> at org.glassfish.jersey.internal.Errors.process(Errors.java:316)
> at org.glassfish.jersey.internal.Errors.process(Errors.java:298)
> at org.glassfish.jersey.internal.Errors.process(Errors.java:268)
> at
> org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:289)
> at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:256)
> at
> org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:703)
> at
> org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:416)
> at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:370)
> at
> org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:389)
> at
> org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:342)
> at
> org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:229)
> at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:865)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1655)
> at org.apache.nifi.web.filter.RequestLogger.doFilter(RequestLogger.java:66)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
> at
> org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:208)
> at
> org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:177)
> at
> org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:347)
> at
> org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:263)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
> at org.apache.nifi.web.filter.TimerFilter.doFilter(TimerFilter.java:51)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
> at
> org.apache.nifi.web.filter.ExceptionFilter.doFilter(ExceptionFilter.java:46)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1634)
> at
>
