It was because of this: https://stackoverflow.com/questions/72841549/container-fails-to-start-insufficient-memory-for-the-java-runtime-environment-t#:~:text=Possible%20reasons%3A%20The%20system%20is,physical%20memory%20or%20swap%20space
The customer had installed docker 18.xx, and we have not had to specify the version before and this new NiFi 1.19.X version was put out with minimal testing on our servers (i.e. didn't check with older docker versions. Upgrading to docker 20.10.13 resolved the issue. Thanks! On Mon, Jan 30, 2023 at 11:52 PM Chris Sampson <[email protected]> wrote: > Have you tried an earlier version of nifi in the same environment? > > Does the image for nifi 1.18.0 boot successfully, for example? > > The reason I'd check this is mainly that the apache/nifi convenience > docker image switched from java 8 to java 11 for nifi version 1.19.0+. I > think the way in which memory is allocated to the jvm within a docker > container changed between java 8 and 11, so it might be worth checking to > see whether an earlier version works. > > Is the VMWare environment 64 or 32bit as I think that can also affect how > the jvm allocates memory (based on a brief search for parts of your error > message - https://www.baeldung.com/jvm-compressed-oops)? > > > On Tue, 31 Jan 2023, 03:36 Cannon Palms, <[email protected]> wrote: > >> Check the resources on the host. >> >> By default, the docker container will have all of the available memory on >> the host machine available to it. If you'd like to ensure that this >> available memory is at least X, you can use a memory reservation in your >> docker compose file: >> >> ``` >> ... >> mem_reservation: 4Gi >> ``` >> >> etc. >> >> You are correct, though, that this is not a Nifi-specific question. You >> may have more success on a docker-specific forum or platform. >> >> Best, >> Cannon >> >> >> On Mon, Jan 30, 2023, 8:27 PM David Early via users < >> [email protected]> wrote: >> >>> Hi all, >>> >>> We are deploying several NiFi containers as part of a service for a >>> customer and we have run into an issue we have not seen before. >>> >>> I THINK the problem may be more of a problem with Java/Docker/VMWare >>> than NiFi per se, but I wanted to ask here in case someone has seen this >>> before. >>> >>> We recently upgraded a small system in Azure to 1.19.1 using the >>> "official" docker container. This worked just fine. The host was an Azure >>> VM running an older CentOS 7 image. There were no issues. >>> >>> For the current install, we used the same Docker image but it is on prem >>> with a customer who is using VMWare to create the hosts we are running on. >>> The hosts (3 of them) are 8 core/64G. >>> >>> The error we are getting is: >>> >>> Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: replacing target >>>> file /opt/nifi/nifi-current/conf/bootstrap.conf >>>> Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: replacing target >>>> file /opt/nifi/nifi-current/conf/bootstrap.conf >>>> Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: replacing target >>>> file /opt/nifi/nifi-current/conf/nifi.properties >>>> Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: replacing target >>>> file /opt/nifi/nifi-current/conf/nifi.properties >>>> Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: replacing target >>>> file /opt/nifi/nifi-current/conf/nifi.properties >>>> Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: replacing target >>>> file /opt/nifi/nifi-current/conf/nifi.properties >>>> Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: replacing target >>>> file /opt/nifi/nifi-current/conf/nifi.properties >>>> Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: replacing target >>>> file /opt/nifi/nifi-current/conf/nifi.properties >>>> Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: replacing target >>>> file /opt/nifi/nifi-current/conf/nifi.properties >>>> Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: replacing target >>>> file /home/nifi/.nifi-cli.nifi.properties >>>> Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: replacing target >>>> file /home/nifi/.nifi-cli.nifi.properties >>>> Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: replacing target >>>> file /home/nifi/.nifi-cli.nifi.properties >>>> Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: replacing target >>>> file /home/nifi/.nifi-cli.nifi.properties >>>> Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: replacing target >>>> file /home/nifi/.nifi-cli.nifi.properties >>>> Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: replacing target >>>> file /opt/nifi/nifi-current/conf/nifi.properties >>>> Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: replacing target >>>> file /opt/nifi/nifi-current/conf/nifi.properties >>>> Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: replacing target >>>> file /opt/nifi/nifi-current/conf/nifi.properties >>>> Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: replacing target >>>> file /opt/nifi/nifi-current/conf/nifi.properties >>>> Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: replacing target >>>> file /opt/nifi/nifi-current/conf/nifi.properties >>>> Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: replacing target >>>> file /opt/nifi/nifi-current/conf/nifi.properties >>>> Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: replacing target >>>> file /opt/nifi/nifi-current/conf/nifi.properties >>>> Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: replacing target >>>> file /opt/nifi/nifi-current/conf/nifi.properties >>>> Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: replacing target >>>> file /opt/nifi/nifi-current/conf/nifi.properties >>>> Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: replacing target >>>> file /opt/nifi/nifi-current/conf/nifi.properties >>>> Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: replacing target >>>> file /opt/nifi/nifi-current/conf/nifi.properties >>>> Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: replacing target >>>> file /opt/nifi/nifi-current/conf/nifi.properties >>>> Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: replacing target >>>> file /opt/nifi/nifi-current/conf/nifi.properties >>>> Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: replacing target >>>> file /opt/nifi/nifi-current/conf/nifi.properties >>>> Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: replacing target >>>> file /opt/nifi/nifi-current/conf/nifi.properties >>>> Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: replacing target >>>> file /opt/nifi/nifi-current/conf/nifi.properties >>>> Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: >>>> Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: Java home: >>>> /opt/java/openjdk >>>> Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: NiFi home: >>>> /opt/nifi/nifi-current >>>> Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: >>>> Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: Bootstrap Config >>>> File: /opt/nifi/nifi-current/conf/bootstrap.conf >>>> Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: >>>> Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: >>>> [0.003s][warning][os,thread] Failed to start thread "GC Thread#0" - >>>> pthread_create failed (EPERM) for attributes: stacksize: 1024k, guardsize: >>>> 4k, detached. >>>> Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: # >>>> Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: # There is >>>> insufficient memory for the Java Runtime Environment to continue. >>>> Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: # Cannot create >>>> worker GC thread. Out of system resources. >>>> Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: # Can not save log >>>> file, dump to screen.. >>>> Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: # >>>> Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: # There is >>>> insufficient memory for the Java Runtime Environment to continue. >>>> Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: # Cannot create >>>> worker GC thread. Out of system resources. >>>> Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: # Possible reasons: >>>> Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: # The system is >>>> out of physical RAM or swap space >>>> Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: # The process is >>>> running with CompressedOops enabled, and the Java Heap may be blocking the >>>> growth of the native heap >>>> Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: # Possible >>>> solutions: >>>> Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: # Reduce memory >>>> load on the system >>>> Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: # Increase >>>> physical memory or swap space >>>> Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: # Check if swap >>>> backing store is full >>>> Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: # Decrease Java >>>> heap size (-Xmx/-Xms) >>>> Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: # Decrease >>>> number of Java threads >>>> Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: # Decrease Java >>>> thread stack sizes (-Xss) >>>> Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: # Set larger >>>> code cache with -XX:ReservedCodeCacheSize= >>>> Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: # JVM is running >>>> with Unscaled Compressed Oops mode in which the Java heap is >>>> Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: # placed in >>>> the first 4GB address space. The Java Heap base address is the >>>> Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: # maximum >>>> limit for the native heap growth. Please use -XX:HeapBaseMinAddress >>>> Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: # to set the >>>> Java Heap base and to place the Java Heap above 4GB virtual address. >>>> Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: # This output file >>>> may be truncated or incomplete. >>>> Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: # >>>> Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: # Out of Memory >>>> Error (workerManager.hpp:70), pid=51, tid=51 >>>> Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: # >>>> Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: # JRE version: >>>> (11.0.17+8) (build ) >>>> Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: # Java VM: OpenJDK >>>> 64-Bit Server VM (11.0.17+8, mixed mode, sharing, tiered, compressed oops, >>>> g1 gc, linux-amd64) >>>> Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: # Core dump will >>>> be written. Default location: /opt/nifi/nifi-current/%e.51.core >>>> Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: # >>>> Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: >>>> Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: --------------- S >>>> U M M A R Y ------------ >>>> Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: >>>> Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: Command Line: >>>> -Dnifi.properties.file.path=/opt/nifi/nifi-current/conf/nifi.properties >>>> org.apache.nifi.authentication.single.user.command.SetSingleUserCredentials >>>> admin password >>>> >>> >>> This repeats one more time in one given startup sequence, with different >>> Command Line, with the same "insufficient memory" error. >>> >>> Command Line: -Xms48m -Xmx48m >>>> -Dorg.apache.nifi.bootstrap.config.log.dir=/opt/nifi/nifi-current/logs >>>> -Dorg.apache.nifi.bootstrap.config.pid.dir=/opt/nifi/nifi-current/run >>>> -Dorg.apache.nifi.bootstrap.config.file=/opt/nifi/nifi-current/conf/bootstrap.conf >>>> org.apache.nifi.bootstrap.RunNiFi run >>>> >>> >>> The full log for one cycle is attached, I removed the dynamic libraries >>> section to shorten it a bit. >>> >>> As noted, this looks a bit like an interaction between >>> docker/java/VMWare, but we are not able to see any reason why this may be >>> the case. We have done this kind of install dozens of times, but this is >>> the first time doing it on VMWare running the hosts. That is about the >>> only thing we can think is causing the issue. >>> >>> We tried setting the bootstrap memory Xmx/Xms to all kinds of levels, >>> from 2G to 32G, but it made no difference with the memory error above. The >>> error appears to be part of the bootstrap, so not even getting to the point >>> where the memory config would make any difference. >>> >>> I am in the process of gathering some additional version and config >>> data, I can provide that shortly. >>> >>> But the basics: >>> >>> NiFi 1.19.1 docker image >>> docker-compose to start >>> host is RedHat7 >>> nothing else is running on the box except base OS stuff, probably a few >>> stray packages, but it shows >48G memory available all the time. >>> >>> Any ideas? >>> >>> Dave >>> >> -- David Early, Ph.D. [email protected] 720-470-7460 Cell
