Chris, We describe the upgrade considerations here [1]. We don't have an upgrade guide yet but will soon [2].
Yes SplitText exposes a defect which will use too much memory (for no good reason). There is a pretty straightforward workaround for now and we'll fix it in an upcoming release [3]. The workaround is to do the splitting in two stages with back-pressure applied. So, chain together two SplitText processors where the first one does 5K lines at a time and the second one does single lines. Then between the two set backpressure to say 10 or 100 or whatever makes sense for you. This way regardless of the size of an input file in terms of lines NiFi will only ever have N/5000 flow files in the first processor and 5000 flow files in the second. [1] https://cwiki.apache.org/confluence/display/NIFI/Migration+Guidance [2] https://issues.apache.org/jira/browse/NIFI-590 [3] https://issues.apache.org/jira/browse/NIFI-1008 Thanks Joe On Mon, Nov 23, 2015 at 9:28 AM, Chris Teoh <[email protected]> wrote: > Yea this is NiFi 0.21. Is there an upgrade path? I have had it run out of > memory before because I was using SplitText on an unrelated log file > containing millions of lines. I guess that might be a limitation I'll have > to find a workaround for. I've managed to recover it but I only noticed this > trying to troubleshoot why the system stopped running the jobs properly. > > On Tue, 24 Nov 2015 at 1:06 AM, Joe Witt <[email protected]> wrote: >> >> Chris, >> >> Questions: >> Is this is NiFi 0.2.1? >> Does it always fail when trying to view content? >> >> I just built 021 and verified it worked as expected. In addition to >> the classloading issue i notice the JSP null found issues at the >> bottom of what you pasted. I wonder if the work directory was >> incomplete on load or if space issues on that partition have cropped >> up while running (large guesses admittedly) >> >> Thanks >> Joe >> >> On Mon, Nov 23, 2015 at 8:43 AM, Chris Teoh <[email protected]> wrote: >> > Hi folks, >> > >> > I tried to view the contents of flowfile from the provenance window and >> > I >> > get a 500 error. >> > >> > From Web page:- >> > HTTP ERROR 500 >> > >> > Problem accessing /nifi-content-viewer/. Reason: >> > >> > Server Error >> > >> > Caused by: >> > >> > java.lang.NoClassDefFoundError: >> > org/apache/nifi/web/ContentViewerController$2 >> > at >> > >> > org.apache.nifi.web.ContentViewerController.getContentRequest(ContentViewerController.java:277) >> > at >> > >> > org.apache.nifi.web.ContentViewerController.doGet(ContentViewerController.java:74) >> > at javax.servlet.http.HttpServlet.service(HttpServlet.java:687) >> > at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) >> > at >> > org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:808) >> > at >> > >> > org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1669) >> > at >> > >> > org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330) >> > at >> > >> > org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:118) >> > at >> > >> > org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:84) >> > at >> > >> > org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) >> > at >> > >> > org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:113) >> > at >> > >> > org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) >> > at >> > >> > org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:113) >> > at >> > >> > org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) >> > at >> > >> > org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:154) >> > at >> > >> > org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) >> > at >> > >> > org.springframework.security.web.authentication.preauth.AbstractPreAuthenticatedProcessingFilter.doFilter(AbstractPreAuthenticatedProcessingFilter.java:107) >> > at >> > >> > org.apache.nifi.web.security.x509.X509AuthenticationFilter.doFilter(X509AuthenticationFilter.java:111) >> > at >> > >> > org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) >> > at >> > >> > org.apache.nifi.web.security.authorization.NodeAuthorizedUserFilter.doFilter(NodeAuthorizedUserFilter.java:121) >> > at >> > >> > org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) >> > at >> > >> > org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:50) >> > at >> > >> > org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) >> > at >> > >> > org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) >> > at >> > >> > org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:87) >> > at >> > >> > org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) >> > at >> > >> > org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:192) >> > at >> > >> > org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:160) >> > at >> > >> > org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:344) >> > at >> > >> > org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:261) >> > at >> > >> > org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652) >> > at >> > >> > org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585) >> > at >> > >> > org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) >> > at >> > >> > org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:577) >> > at >> > >> > org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223) >> > at >> > >> > org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127) >> > at >> > >> > org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515) >> > at >> > >> > org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185) >> > at >> > >> > org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061) >> > at >> > >> > org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) >> > at >> > >> > org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:110) >> > at >> > >> > org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97) >> > at org.eclipse.jetty.server.Server.handle(Server.java:499) >> > at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310) >> > at >> > >> > org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257) >> > at >> > >> > org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540) >> > at >> > >> > org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635) >> > at >> > >> > org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555) >> > at java.lang.Thread.run(Thread.java:745) >> > >> > Powered by Jetty:// >> > >> > >> > From nifi-app.log: >> > 2015-11-23 00:40:18,756 WARN [NiFi Web Server-311490] >> > org.eclipse.jetty.servlet.ServletHandler Error for /nifi-content-viewer/ >> > java.lang.NoClassDefFoundError: >> > org/apache/nifi/web/ContentViewerController$2 >> > at >> > >> > org.apache.nifi.web.ContentViewerController.getContentRequest(ContentViewerController.java:277) >> > ~[classes/:0.2.1] >> > at >> > >> > org.apache.nifi.web.ContentViewerController.doGet(ContentViewerController.java:74) >> > ~[classes/:0.2.1] >> > at javax.servlet.http.HttpServlet.service(HttpServlet.java:687) >> > ~[javax.servlet-api-3.1.0.jar:3.1.0] >> > at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) >> > ~[javax.servlet-api-3.1.0.jar:3.1.0] >> > at >> > org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:808) >> > ~[jetty-servlet-9.2.11.v20150529.jar:9.2.11.v20150529] >> > at >> > >> > org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1669) >> > ~[jetty-servlet-9.2.11.v20150529.jar:9.2.11.v20150529] >> > at >> > >> > org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330) >> > ~[spring-security-web-3.2.7.RELEASE.jar:3.2.7.RELEASE] >> > at >> > >> > org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:118) >> > ~[spring-security-web-3.2.7.RELEASE.jar:3.2.7.RELEASE] >> > at >> > >> > org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:84) >> > ~[spring-security-web-3.2.7.RELEASE.jar:3.2.7.RELEASE] >> > at >> > >> > org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) >> > ~[spring-security-web-3.2.7.RELEASE.jar:3.2.7.RELEASE] >> > at >> > >> > org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:113) >> > ~[spring-security-web-3.2.7.RELEASE.jar:3.2.7.RELEASE] >> > at >> > >> > org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) >> > ~[spring-security-web-3.2.7.RELEASE.jar:3.2.7.RELEASE] >> > at >> > >> > org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:113) >> > ~[spring-security-web-3.2.7.RELEASE.jar:3.2.7.RELEASE] >> > at >> > >> > org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) >> > ~[spring-security-web-3.2.7.RELEASE.jar:3.2.7.RELEASE] >> > at >> > >> > org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:154) >> > ~[spring-security-web-3.2.7.RELEASE.jar:3.2.7.RELEASE] >> > at >> > >> > org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) >> > ~[spring-security-web-3.2.7.RELEASE.jar:3.2.7.RELEASE] >> > at >> > >> > org.springframework.security.web.authentication.preauth.AbstractPreAuthenticatedProcessingFilter.doFilter(AbstractPreAuthenticatedProcessingFilter.java:107) >> > ~[spring-security-web-3.2.7.RELEASE.jar:3.2.7.RELEASE] >> > at >> > >> > org.apache.nifi.web.security.x509.X509AuthenticationFilter.doFilter(X509AuthenticationFilter.java:111) >> > ~[nifi-web-security-0.2.1.jar:0.2.1] >> > at >> > >> > org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) >> > ~[spring-security-web-3.2.7.RELEASE.jar:3.2.7.RELEASE] >> > at >> > >> > org.apache.nifi.web.security.authorization.NodeAuthorizedUserFilter.doFilter(NodeAuthorizedUserFilter.java:121) >> > ~[nifi-web-security-0.2.1.jar:0.2.1] >> > at >> > >> > org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) >> > ~[spring-security-web-3.2.7.RELEASE.jar:3.2.7.RELEASE] >> > at >> > >> > org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:50) >> > ~[spring-security-web-3.2.7.RELEASE.jar:3.2.7.RELEASE] >> > at >> > >> > org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) >> > ~[spring-web-4.1.6.RELEASE.jar:4.1.6.RELEASE] >> > at >> > >> > org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) >> > ~[spring-security-web-3.2.7.RELEASE.jar:3.2.7.RELEASE] >> > at >> > >> > org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:87) >> > ~[spring-security-web-3.2.7.RELEASE.jar:3.2.7.RELEASE] >> > at >> > >> > org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) >> > ~[spring-security-web-3.2.7.RELEASE.jar:3.2.7.RELEASE] >> > at >> > >> > org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:192) >> > ~[spring-security-web-3.2.7.RELEASE.jar:3.2.7.RELEASE] >> > at >> > >> > org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:160) >> > ~[spring-security-web-3.2.7.RELEASE.jar:3.2.7.RELEASE] >> > at >> > >> > org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:192) >> > ~[spring-security-web-3.2.7.RELEASE.jar:3.2.7.RELEASE] >> > at >> > >> > org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:160) >> > ~[spring-security-web-3.2.7.RELEASE.jar:3.2.7.RELEASE] >> > at >> > >> > org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:344) >> > ~[spring-web-4.1.6.RELEASE.jar:4.1.6.RELEASE] >> > at >> > >> > org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:261) >> > ~[spring-web-4.1.6.RELEASE.jar:4.1.6.RELEASE] >> > at >> > >> > org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652) >> > ~[jetty-servlet-9.2.11.v20150529.jar:9.2.11.v20150529] >> > at >> > >> > org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585) >> > [jetty-servlet-9.2.11.v20150529.jar:9.2.11.v20150529] >> > at >> > >> > org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) >> > [jetty-server-9.2.11.v20150529.jar:9.2.11.v20150529] >> > at >> > >> > org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:577) >> > [jetty-security-9.2.11.v20150529.jar:9.2.11.v20150529] >> > at >> > >> > org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223) >> > [jetty-server-9.2.11.v20150529.jar:9.2.11.v20150529] >> > at >> > >> > org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127) >> > [jetty-server-9.2.11.v20150529.jar:9.2.11.v20150529] >> > at >> > >> > org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515) >> > [jetty-servlet-9.2.11.v20150529.jar:9.2.11.v20150529] >> > at >> > >> > org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185) >> > [jetty-server-9.2.11.v20150529.jar:9.2.11.v20150529] >> > at >> > >> > org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061) >> > [jetty-server-9.2.11.v20150529.jar:9.2.11.v20150529] >> > at >> > >> > org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) >> > [jetty-server-9.2.11.v20150529.jar:9.2.11.v20150529] >> > at >> > >> > org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:110) >> > [jetty-server-9.2.11.v20150529.jar:9.2.11.v20150529] >> > at >> > >> > org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97) >> > [jetty-server-9.2.11.v20150529.jar:9.2.11.v20150529] >> > at org.eclipse.jetty.server.Server.handle(Server.java:499) >> > [jetty-server-9.2.11.v20150529.jar:9.2.11.v20150529] >> > at >> > org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310) >> > [jetty-server-9.2.11.v20150529.jar:9.2.11.v20150529] >> > at >> > >> > org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257) >> > [jetty-server-9.2.11.v20150529.jar:9.2.11.v20150529] >> > at >> > >> > org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540) >> > [jetty-io-9.2.11.v20150529.jar:9.2.11.v20150529] >> > at >> > >> > org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635) >> > [jetty-util-9.2.11.v20150529.jar:9.2.11.v20150529] >> > at >> > >> > org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555) >> > [jetty-util-9.2.11.v20150529.jar:9.2.11.v20150529] >> > at java.lang.Thread.run(Thread.java:745) [na:1.7.0_67] >> > Caused by: java.lang.ClassNotFoundException: >> > org.apache.nifi.web.ContentViewerController$2 >> > at java.net.URLClassLoader$1.run(URLClassLoader.java:366) >> > ~[na:1.7.0_67] >> > at java.net.URLClassLoader$1.run(URLClassLoader.java:355) >> > ~[na:1.7.0_67] >> > at java.security.AccessController.doPrivileged(Native Method) >> > ~[na:1.7.0_67] >> > at java.net.URLClassLoader.findClass(URLClassLoader.java:354) >> > ~[na:1.7.0_67] >> > at java.lang.ClassLoader.loadClass(ClassLoader.java:425) >> > ~[na:1.7.0_67] >> > at java.lang.ClassLoader.loadClass(ClassLoader.java:358) >> > ~[na:1.7.0_67] >> > at >> > >> > org.eclipse.jetty.webapp.WebAppClassLoader.loadClass(WebAppClassLoader.java:450) >> > ~[jetty-webapp-9.2.11.v20150529.jar:9.2.11.v20150529] >> > at >> > >> > org.eclipse.jetty.webapp.WebAppClassLoader.loadClass(WebAppClassLoader.java:403) >> > ~[jetty-webapp-9.2.11.v20150529.jar:9.2.11.v20150529] >> > ... 49 common frames omitted >> > 2015-11-23 00:40:18,945 ERROR [NiFi Web Server-311493] >> > org.apache.jasper.servlet.JspServlet PWC6117: File "null" not found >> > 2015-11-23 00:40:18,946 ERROR [NiFi Web Server-311493] >> > org.apache.jasper.servlet.JspServlet PWC6117: File "null" not found
