Check the resources on the host. By default, the docker container will have all of the available memory on the host machine available to it. If you'd like to ensure that this available memory is at least X, you can use a memory reservation in your docker compose file:
``` ... mem_reservation: 4Gi ``` etc. You are correct, though, that this is not a Nifi-specific question. You may have more success on a docker-specific forum or platform. Best, Cannon On Mon, Jan 30, 2023, 8:27 PM David Early via users <[email protected]> wrote: > Hi all, > > We are deploying several NiFi containers as part of a service for a > customer and we have run into an issue we have not seen before. > > I THINK the problem may be more of a problem with Java/Docker/VMWare than > NiFi per se, but I wanted to ask here in case someone has seen this before. > > We recently upgraded a small system in Azure to 1.19.1 using the > "official" docker container. This worked just fine. The host was an Azure > VM running an older CentOS 7 image. There were no issues. > > For the current install, we used the same Docker image but it is on prem > with a customer who is using VMWare to create the hosts we are running on. > The hosts (3 of them) are 8 core/64G. > > The error we are getting is: > > Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: replacing target file >> /opt/nifi/nifi-current/conf/bootstrap.conf >> Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: replacing target >> file /opt/nifi/nifi-current/conf/bootstrap.conf >> Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: replacing target >> file /opt/nifi/nifi-current/conf/nifi.properties >> Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: replacing target >> file /opt/nifi/nifi-current/conf/nifi.properties >> Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: replacing target >> file /opt/nifi/nifi-current/conf/nifi.properties >> Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: replacing target >> file /opt/nifi/nifi-current/conf/nifi.properties >> Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: replacing target >> file /opt/nifi/nifi-current/conf/nifi.properties >> Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: replacing target >> file /opt/nifi/nifi-current/conf/nifi.properties >> Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: replacing target >> file /opt/nifi/nifi-current/conf/nifi.properties >> Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: replacing target >> file /home/nifi/.nifi-cli.nifi.properties >> Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: replacing target >> file /home/nifi/.nifi-cli.nifi.properties >> Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: replacing target >> file /home/nifi/.nifi-cli.nifi.properties >> Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: replacing target >> file /home/nifi/.nifi-cli.nifi.properties >> Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: replacing target >> file /home/nifi/.nifi-cli.nifi.properties >> Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: replacing target >> file /opt/nifi/nifi-current/conf/nifi.properties >> Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: replacing target >> file /opt/nifi/nifi-current/conf/nifi.properties >> Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: replacing target >> file /opt/nifi/nifi-current/conf/nifi.properties >> Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: replacing target >> file /opt/nifi/nifi-current/conf/nifi.properties >> Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: replacing target >> file /opt/nifi/nifi-current/conf/nifi.properties >> Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: replacing target >> file /opt/nifi/nifi-current/conf/nifi.properties >> Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: replacing target >> file /opt/nifi/nifi-current/conf/nifi.properties >> Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: replacing target >> file /opt/nifi/nifi-current/conf/nifi.properties >> Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: replacing target >> file /opt/nifi/nifi-current/conf/nifi.properties >> Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: replacing target >> file /opt/nifi/nifi-current/conf/nifi.properties >> Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: replacing target >> file /opt/nifi/nifi-current/conf/nifi.properties >> Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: replacing target >> file /opt/nifi/nifi-current/conf/nifi.properties >> Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: replacing target >> file /opt/nifi/nifi-current/conf/nifi.properties >> Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: replacing target >> file /opt/nifi/nifi-current/conf/nifi.properties >> Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: replacing target >> file /opt/nifi/nifi-current/conf/nifi.properties >> Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: replacing target >> file /opt/nifi/nifi-current/conf/nifi.properties >> Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: >> Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: Java home: >> /opt/java/openjdk >> Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: NiFi home: >> /opt/nifi/nifi-current >> Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: >> Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: Bootstrap Config >> File: /opt/nifi/nifi-current/conf/bootstrap.conf >> Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: >> Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: >> [0.003s][warning][os,thread] Failed to start thread "GC Thread#0" - >> pthread_create failed (EPERM) for attributes: stacksize: 1024k, guardsize: >> 4k, detached. >> Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: # >> Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: # There is >> insufficient memory for the Java Runtime Environment to continue. >> Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: # Cannot create >> worker GC thread. Out of system resources. >> Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: # Can not save log >> file, dump to screen.. >> Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: # >> Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: # There is >> insufficient memory for the Java Runtime Environment to continue. >> Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: # Cannot create >> worker GC thread. Out of system resources. >> Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: # Possible reasons: >> Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: # The system is >> out of physical RAM or swap space >> Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: # The process is >> running with CompressedOops enabled, and the Java Heap may be blocking the >> growth of the native heap >> Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: # Possible solutions: >> Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: # Reduce memory >> load on the system >> Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: # Increase >> physical memory or swap space >> Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: # Check if swap >> backing store is full >> Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: # Decrease Java >> heap size (-Xmx/-Xms) >> Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: # Decrease number >> of Java threads >> Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: # Decrease Java >> thread stack sizes (-Xss) >> Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: # Set larger code >> cache with -XX:ReservedCodeCacheSize= >> Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: # JVM is running >> with Unscaled Compressed Oops mode in which the Java heap is >> Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: # placed in the >> first 4GB address space. The Java Heap base address is the >> Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: # maximum limit >> for the native heap growth. Please use -XX:HeapBaseMinAddress >> Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: # to set the >> Java Heap base and to place the Java Heap above 4GB virtual address. >> Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: # This output file >> may be truncated or incomplete. >> Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: # >> Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: # Out of Memory >> Error (workerManager.hpp:70), pid=51, tid=51 >> Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: # >> Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: # JRE version: >> (11.0.17+8) (build ) >> Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: # Java VM: OpenJDK >> 64-Bit Server VM (11.0.17+8, mixed mode, sharing, tiered, compressed oops, >> g1 gc, linux-amd64) >> Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: # Core dump will be >> written. Default location: /opt/nifi/nifi-current/%e.51.core >> Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: # >> Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: >> Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: --------------- S U >> M M A R Y ------------ >> Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: >> Jan 27 16:32:36 dstsc01 cust-sts|2181237bc92f[1788]: Command Line: >> -Dnifi.properties.file.path=/opt/nifi/nifi-current/conf/nifi.properties >> org.apache.nifi.authentication.single.user.command.SetSingleUserCredentials >> admin password >> > > This repeats one more time in one given startup sequence, with different > Command Line, with the same "insufficient memory" error. > > Command Line: -Xms48m -Xmx48m >> -Dorg.apache.nifi.bootstrap.config.log.dir=/opt/nifi/nifi-current/logs >> -Dorg.apache.nifi.bootstrap.config.pid.dir=/opt/nifi/nifi-current/run >> -Dorg.apache.nifi.bootstrap.config.file=/opt/nifi/nifi-current/conf/bootstrap.conf >> org.apache.nifi.bootstrap.RunNiFi run >> > > The full log for one cycle is attached, I removed the dynamic libraries > section to shorten it a bit. > > As noted, this looks a bit like an interaction between docker/java/VMWare, > but we are not able to see any reason why this may be the case. We have > done this kind of install dozens of times, but this is the first time doing > it on VMWare running the hosts. That is about the only thing we can think > is causing the issue. > > We tried setting the bootstrap memory Xmx/Xms to all kinds of levels, from > 2G to 32G, but it made no difference with the memory error above. The > error appears to be part of the bootstrap, so not even getting to the point > where the memory config would make any difference. > > I am in the process of gathering some additional version and config data, > I can provide that shortly. > > But the basics: > > NiFi 1.19.1 docker image > docker-compose to start > host is RedHat7 > nothing else is running on the box except base OS stuff, probably a few > stray packages, but it shows >48G memory available all the time. > > Any ideas? > > Dave >
