Hi, Tim and Konstantin! Thanks for the quick response. I’m testing on my local environment, it’s a sort of home load testing to understand this service is robust or not. I use a bunch of file with different mime types such as pdf, docx, txt, mp4, pptx, odt, xsls, rtf, html. Env: Mac OS 13, Java 15 To test restarts I just override a default value of maxFiles and it doesn’t matter how many file descriptor OS has. Tim, yeah you are right, I also override maxForkedStartupMillis. Now I returned to a default value and restarting works well. I’ll continue testing, if get some issues I’ll let you know.
However, there is an interesting fact. When the hit max files are coming I stop calls by JMeter and call manually way and carefully to get the max files number. So, I’ve noticed if I don’t call while restarting tika server process a value as 6 sec is enough. And in case I continue calling it is not enough. From: Tim Allison <[email protected]> Date: Wednesday, 8 March 2023, 23:30 To: [email protected] <[email protected]> Subject: [EXTERNAL] - Re: Tika server crashes CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe. If you feel that the email is suspicious, please report it using PhishAlarm. HIT_MAX_FILES is expected. We designed that in to periodically restart the server to avoid memory leaks in badly behaving parsers. You can configure a value for the max file threshold if necessary. The restart failed, and that's a problem. Let me look into the code, I thought we offered more grace than 6 seconds to restart the server. Can you share any server settings in your config.xml? Please remember that as of 2.x tika-server will shutdown on oom, timeouts and max_files, and clients should be able to handle waiting for tika-server restarts. On Wed, Mar 8, 2023 at 2:58 PM Konstantin Gribov <[email protected]> wrote: > > Hello, Artur. > > How many concurrent requests did you have and are you running Tika Server on > Windows? And what kind of files did you use? > > You may have hit number of open files limit due to lot of reasons starting > from known Windows issue (JVM process holds file descriptors for mmaped files > until process killed) through just too low nofile limit to some Tika bug with > handling for example stdin/stdout for forked processes. > > Could you provide jvm thread dump and lsof output (or Windows analog)? > > -- > Best regards, > Konstantin Gribov. > > > On Wed, Mar 8, 2023 at 4:26 PM Artur Auhatov via user <[email protected]> > wrote: >> >> Hello! >> >> I have a few questions related to Tika server. >> >> >> >> We’ve started using tika server in our environment. While testing >> reliability of the tika-server we found it crashes during fork and can’t >> fork anymore until restart the main process. Is this known problem? >> >> In order to start tika-server we use command: java -jar >> tika-server-2.7.0.jar -c myconfig.xml >> >> >> >> There is a log message: >> >> >> >> 14:50:30,272 [INFO] [Thread-9] - Shutting down forked process with status: >> HIT_MAX_FILES [org.apache.tika.server.core.ServerStatusWatcher] >> >> INFO [pool-2-thread-1] 14:50:30,816 >> org.apache.tika.server.core.TikaServerWatchDog forked process exited with >> exit value 2 >> >> INFO [pool-2-thread-1] 14:50:36,876 >> org.apache.tika.server.core.TikaServerWatchDog about to shutdown process >> >> ERROR [main] 14:50:36,878 org.apache.tika.server.core.TikaServerCli Can't >> start: >> >> java.util.concurrent.ExecutionException: java.lang.RuntimeException: Forked >> process failed to start after 6022 (ms) >> >> at >> java.util.concurrent.FutureTask.report(FutureTask.java:122) ~[?:?] >> >> at java.util.concurrent.FutureTask.get(FutureTask.java:191) >> ~[?:?] >> >> at >> org.apache.tika.server.core.TikaServerCli.mainLoop(TikaServerCli.java:121) >> ~[tika-server-standard-2.7.0.jar:2.7.0] >> >> at >> org.apache.tika.server.core.TikaServerCli.execute(TikaServerCli.java:93) >> ~[tika-server-standard-2.7.0.jar:2.7.0] >> >> at >> org.apache.tika.server.core.TikaServerCli.main(TikaServerCli.java:80) >> ~[tika-server-standard-2.7.0.jar:2.7.0] >> >> Caused by: java.lang.RuntimeException: Forked process failed to start after >> 6022 (ms) >> >> at >> org.apache.tika.server.core.TikaServerWatchDog$ForkedProcess.<init>(TikaServerWatchDog.java:316) >> ~[tika-server-standard-2.7.0.jar:2.7.0] >> >> at >> org.apache.tika.server.core.TikaServerWatchDog$ForkedProcess.<init>(TikaServerWatchDog.java:287) >> ~[tika-server-standard-2.7.0.jar:2.7.0] >> >> at >> org.apache.tika.server.core.TikaServerWatchDog.startForkedProcess(TikaServerWatchDog.java:224) >> ~[tika-server-standard-2.7.0.jar:2.7.0] >> >> at >> org.apache.tika.server.core.TikaServerWatchDog.call(TikaServerWatchDog.java:143) >> ~[tika-server-standard-2.7.0.jar:2.7.0] >> >> at >> org.apache.tika.server.core.TikaServerWatchDog.call(TikaServerWatchDog.java:53) >> ~[tika-server-standard-2.7.0.jar:2.7.0] >> >> at java.util.concurrent.FutureTask.run(FutureTask.java:264) >> ~[?:?] >> >> at >> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) >> ~[?:?] >> >> at java.util.concurrent.FutureTask.run(FutureTask.java:264) >> ~[?:?] >> >> at >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) >> ~[?:?] >> >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) >> ~[?:?] >> >> at java.lang.Thread.run(Thread.java:832) ~[?:?] >> >>
