A simple solution it to move the java tmp directory location out of /tmp. Other than that, find and disable the tmp sweeper.
cheers On 23 May 2017 at 08:06, Pritha Srivastava <[email protected]> wrote: > > ----- Original Message ----- > > From: "Pritha Srivastava" <[email protected]> > > To: "JETTY user mailing list" <[email protected]> > > Sent: Tuesday, May 23, 2017 10:33:24 AM > > Subject: Re: [jetty-users] Error with shibboleth-idp and jetty > > > > > > ----- Original Message ----- > > > From: "Joakim Erdfelt" <[email protected]> > > > To: "JETTY user mailing list" <[email protected]> > > > Sent: Tuesday, May 23, 2017 4:20:58 AM > > > Subject: Re: [jetty-users] Error with shibboleth-idp and jetty > > > > > > This is the source of your Apache Jasper warning ... > > > > > > 2017-04-20 05:50:06.059:WARN:oaj. EmbeddedServletOptions: > qtp987405879-11: > > > The scratchDir you specified: /tmp/jetty-0.0.0.0-443-idp. war-_idp-any- > > > 6580006879402156844.dir/jsp is unusable. > > > > > > https://github.com/apache/tomcat85/blob/trunk/java/org/apache/jasper/ > EmbeddedServletOptions.java#L688-L691 > > > > > > Your /tmp/jetty-0.0.0.0-443-idp. war-_idp-any- > 6580006879402156844.dir/jsp > > > path fails one of the following checks .... > > > > > > > > > * must exist > > > * is a directory > > > * jetty user should be able to read from that directory > > > * jetty user should be able to write to that directory > > > Does this happen immediately? or after some time? > > > > > > If it happens after some time, then you likely have a process on your > unix > > > server that is periodically cleaning up the /tmp/ directory (which > broke > > > jetty). > > > > > > > Thank you for your reply. > > > > The error happened after some days. Initially things were working fine. > > > > After seeing this error, shibboleth-idp was restarted which in turn > restarted > > the jetty server as well, the server comes up fine as can be seen in the > > logs below: > > > > 2017-04-21 09:29:14.347:INFO:oejs.Server:main: jetty-9.4.0.M0 > > 2017-04-21 09:29:14.347:WARN:oejs.Server:main: THIS IS NOT A STABLE > RELEASE! > > DO NOT USE IN PRODUCTION! > > 2017-04-21 09:29:14.347:WARN:oejs.Server:main: Download a stable > release from > > http://download.eclipse.org/jetty/ > > 2017-04-21 09:29:14.396:INFO:oejdp.ScanningAppProvider:main: Deployment > > monitor [file:///opt/shibboleth-idp/jetty-base/webapps/] at interval 1 > > 2017-04-21 09:29:14.641:INFO:root:main: Warning: No > > org.apache.tomcat.JarScanner set in ServletContext. Falling back to > default > > JarScanner implementation. > > 2017-04-21 09:29:14.837:INFO:oajs.TldScanner:main: At least one JAR was > > scanned for TLDs yet contained no TLDs. Enable debug logging for this > logger > > for a complete list of JARs that were scanned but no TLDs were found in > > them. Skipping unneeded JARs during scanning can improve startup time and > > JSP compilation time. > > 2017-04-21 09:29:14.856:WARN:oejs.session:main: No workerName > configured for > > DefaultSessionIdManager, using node0 > > 2017-04-21 09:29:14.856:WARN:oejs.session:main: No SessionScavenger set, > > using defaults > > 2017-04-21 09:29:14.900:INFO:oejsh.ContextHandler:main: Started > > o.e.j.w.WebAppContext@5383967b{/,file:///opt/shibboleth-idp/ > jetty-base/webapps/root/,AVAILABLE}{/root} > > 2017-04-21 09:29:22.723:INFO:/idp:main: No Spring > WebApplicationInitializer > > types detected on classpath > > 2017-04-21 09:29:22.734:INFO:/idp:main: Warning: No > > org.apache.tomcat.JarScanner set in ServletContext. Falling back to > default > > JarScanner implementation. > > 2017-04-21 09:29:22.974:INFO:oajs.TldScanner:main: At least one JAR was > > scanned for TLDs yet contained no TLDs. Enable debug logging for this > logger > > for a complete list of JARs that were scanned but no TLDs were found in > > them. Skipping unneeded JARs during scanning can improve startup time and > > JSP compilation time. > > 2017-04-21 09:29:23.026:WARN:oejs.SecurityHandler:main: > > [email protected]@5db45159{/idp,[ > file:///tmp/jetty-0.0.0.0-443-idp.war-_idp-any-4503664939735826572.dir/ > webinf/, > > jar:file:///opt/shibboleth-idp/war/idp.war!/],STARTING}{/ > opt/shibboleth-idp/war/idp.war} > > has uncovered http methods for path: /* > > 2017-04-21 09:29:23.291:INFO:/idp:main: Initializing Spring root > > WebApplicationContext > > 2017-04-21 09:29:31.291:INFO:/idp:main: Initializing Spring > FrameworkServlet > > 'idp' > > 2017-04-21 09:29:32.483:INFO:oejsh.ContextHandler:main: Started > > o.e.j.w.WebAppContext@5db45159{/idp,[file:///tmp/jetty-0.0. > 0.0-443-idp.war-_idp-any-4503664939735826572.dir/webinf/, > > jar:file:///opt/shibboleth-idp/war/idp.war!/],AVAILABLE}{ > /opt/shibboleth-idp/war/idp.war} > > 2017-04-21 09:29:32.495:INFO:oejs.AbstractNCSARequestLog:main: Opened > > /opt/shibboleth-idp/jetty-base/logs/2017_04_21.request.log > > 2017-04-21 09:29:32.549:INFO:oejus.SslContextFactory:main: > > x509=X509@214b342f(creek,h=[creek.eng.arb.redhat.com],w=[]) for > > SslContextFactory@5db0003d(file:///opt/shibboleth-idp/ > credentials/idp-backchannel.p12,null) > > 2017-04-21 09:29:32.572:INFO:oejs.AbstractConnector:main: Started > > ServerConnector@1c0c9c99{SSL,[ssl, http/1.1]}{0.0.0.0:443} > > 2017-04-21 09:29:32.573:INFO:oejs.Server:main: Started @19519ms > > > > > > But again, as soon as I issue a request, I start getting the following > > errors: > > java.lang.IllegalStateException: Not valid for write: > > id=node0wcbt9cl1f24p11zowsnp88kb0 not resident > > at > > org.eclipse.jetty.server.session.Session. > checkValidForWrite(Session.java:587) > > at > > org.eclipse.jetty.server.session.Session.setAttribute( > Session.java:722) > > at > > org.eclipse.jetty.server.session.SessionHandler. > newHttpSession(SessionHandler.java:796) > > at org.eclipse.jetty.server.Request.getSession(Request. > java:1520) > > at org.eclipse.jetty.server.Request.getSession(Request. > java:1493) > > at > > net.shibboleth.idp.log.SLF4JMDCServletFilter.doFilter( > SLF4JMDCServletFilter.java:66) > > > > Is it still a problem with the /tmp directory getting wiped off? > > > > Ok, i checked the logs and the contents of /tmp again, as soon as jetty > server is restarted the jetty-0.0.0.0-443-idp.war-_idp-any-4503664939735826572 > folder appears fine under /tmp, but after sometime it goes away. I'll check > what has to be done to solve this. > > Thanks again for your help, > Pritha > > > Thanks, > > Pritha > > > > > > > > Joakim Erdfelt / [email protected] > > > > > > On Fri, May 19, 2017 at 3:26 AM, Pritha Srivastava < > [email protected] > > > > wrote: > > > > > > > > > Hi, > > > > > > We have a setup that has a shibboleth-idp with a jetty server. Any > requests > > > to the shibboleth-idp is returning an error. I issued the following > > > command: > > > > > > curl -kvL https://<hostname>/idp and instead of seeing the expected > result, > > > I > > > see errors like: > > > > > > <html> > > > > > > http-equiv="Content-Type" content="text/html;charset=utf-8"/> > > > Error 500 Server Error > > > > > > HTTP ERROR 500 > > > Problem accessing /idp/. Reason: > > > <pre> Server Errorpre>Caused > > > by:<pre>java.lang.IllegalStateException: Not valid for write: > > > id=node0xzkusdg1igdt4hol48y607u66 not resident > > > at > > > org.eclipse.jetty.server.session.Session.checkValidForWrite(Session. > java:587) > > > at > > > org.eclipse.jetty.server.session.Session.setAttribute( > Session.java:722) > > > > > > > > > Looking at some older logs, I could see an error, after which similar > > > errors > > > started appearing: > > > > > > 2017-04-20 05:50:06.059:WARN:oaj.EmbeddedServletOptions: > qtp987405879-11: > > > The scratchDir you specified: > > > /tmp/jetty-0.0.0.0-443-idp.war-_idp-any-6580006879402156844.dir/jsp is > > > unusable. > > > 2017-04-20 05:50:09.289:WARN:/idp:qtp987405879-11: unavailable > > > java.lang.ClassNotFoundException: > > > org.apache.jsp.WEB_002dINF.jsp.metadata_jsp > > > at java.net.URLClassLoader.findClass(URLClassLoader.java:381) > > > at org.apache.jasper.servlet.JasperLoader.loadClass( > JasperLoader.java:131) > > > at org.apache.jasper.servlet.JasperLoader.loadClass( > JasperLoader.java:62) > > > > > > > > > I have checked permissions for the scratch directory, and they seem to > be > > > fine. I have also restarted shibboleth-idp several times, but that > doesn't > > > seem to solve the problem. > > > Everytime I issue a request to the idp, I get back the same error as > above > > > > > > Does anyone have any idea as to what has gone wrong and what can be > done to > > > solve this issue? (Maybe, rebuild the WAR file?) > > > > > > Thanks in advance, > > > Pritha > > > > > > _______________________________________________ > > > jetty-users mailing list > > > [email protected] > > > To change your delivery options, retrieve your password, or unsubscribe > > > from > > > this list, visit > > > https://dev.eclipse.org/mailman/listinfo/jetty-users > > > > > > > > > _______________________________________________ > > > jetty-users mailing list > > > [email protected] > > > To change your delivery options, retrieve your password, or unsubscribe > > > from > > > this list, visit > > > https://dev.eclipse.org/mailman/listinfo/jetty-users > > _______________________________________________ > > jetty-users mailing list > > [email protected] > > To change your delivery options, retrieve your password, or unsubscribe > from > > this list, visit > > https://dev.eclipse.org/mailman/listinfo/jetty-users > > > _______________________________________________ > jetty-users mailing list > [email protected] > To change your delivery options, retrieve your password, or unsubscribe > from this list, visit > https://dev.eclipse.org/mailman/listinfo/jetty-users > -- Greg Wilkins <[email protected]> CTO http://webtide.com
_______________________________________________ jetty-users mailing list [email protected] To change your delivery options, retrieve your password, or unsubscribe from this list, visit https://dev.eclipse.org/mailman/listinfo/jetty-users
