Re: [Neo4j] Neo4j logging GB of errors when Neo4j cannot get enough memory

[Ray Cheng]

> afaik it was a bug in an *older version* that it used too much memory
during recovery. (Please upgrade to 3.0.0)
We are using 3.0.3, but a Community version.

> How big were the transaction logs? I.e. how much data had you inserted?
I don't recall this one, but transaction logs were limited to 10G in our
neo4j.conf. We have inserted Tera bytes to Neo4j.

> What is your configuration for the page-cache?
> Neo4j assumes it runs alone on the machine and grabs a percentage of RAM
- heap as page-cache when not configured.
*> Please configure the page-cache explicitely in the config.*
This page-cache setting may be a major cause. We are using the default,
i.e. (192GB - 40GB) / 2
I will set the page-cache explicitly to 10GB.

> For these memory sizes you shouldn't run community but on Enterprise.
Will do so.

> The error is from the operating system trying to allocate memory for us,
there is not much else we can do about it.
I see

*> You should also configure tx-log retention e.g. to 10 G*
We have set dbms.tx_log.rotation.retention_policy=10G size before this
incident.

Thanks Michael.
Ray

On Mon, Nov 13, 2017 at 9:08 PM, 'Michael Hunger' via Neo4j <
neo4j@googlegroups.com> wrote:

> Hi,
>
> afaik it was a bug in an *older version* that it used too much memory
> during recovery. (Please upgrade to 3.0.0)
> How big were the transaction logs? I.e. how much data had you inserted?
>
> What is your configuration for the page-cache?
> Neo4j assumes it runs alone on the machine and grabs a percentage of RAM -
> heap as page-cache when not configured.
> *Please configure the page-cache explicitely in the config.*
>
> For these memory sizes you shouldn't run community but on Enterprise.
>
> The error is from the operating system trying to allocate memory for us,
> there is not much else we can do about it.
>
> *You should also configure tx-log retention e.g. to 10 G*
>
> HTH
>
> Michael
>
>
> On Mon, Nov 13, 2017 at 8:08 PM,  wrote:
>
>> Hi,
>>
>> One of our Linux test servers has 192 GB RAM and the following disk
>> partitions (of some time earlier when / was not full). We ran Neo4j and
>> Elasticsearch on this server. We set both heap initial_size and heap max
>> size to 40GB.
>>
>>
>>
>> Filesystem  Size  Used Avail Use% Mounted on
>>
>> /dev/sda1   546G  5.1G  513G   1% /
>>
>> tmpfs95G   16K   95G   1% /dev/shm
>>
>> /dev/sdb1   6.5T  2.1T  4.1T  34% /data
>>
>>
>>
>> We set neo4j home of the Neo4j Community Edition 3.0.3 to root partition
>> under  /opt/…
>>
>>
>>
>> The problem we had was neo4j.log caused the 546GB root partition full.
>> After we truncated neo4j.log, neo4j could not start until we manually
>> removed transaction log files.
>>
>>
>>
>> I have the following questions, can anyone help?
>>
>> 1)   Is there a way to set a maximum file size or file rotation like
>> debug.log on neo4j.log from neo4j.conf in Enterprise version?
>>
>> 2)   In more than one incidents Neo4j could not restart successfully
>> from transaction log files, and it could not restart until I manually
>> removed transaction log files such as neostore.transaction.db.348. Can
>> Enterprise version restart without human intervention if it cannot recover
>> from transaction log files?
>>
>> 3)Is there a way to slow down reporting of 'Cannot allocate
>> memory' (errno=12) errors? In our case, the errors were dumped to neo4j.log
>> almost like an infinite loop.
>>
>> Error messages are as follows: (Neo4j.log has many millions of these
>> lines:)
>>
>> # There is insufficient memory for the Java Runtime Environment to
>> continue.
>>
>> # Native memory allocation (mmap) failed to map 4194304 bytes for
>> committing reserved memory.
>>
>> # An error report file with more information is saved as:
>>
>> # /tmp/hs_err_pid74296.log
>>
>> OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x7f6b2c00,
>> 4194304, 0) failed; error='Cannot allocate memory' (errno=12)
>>
>> #
>>
>> # There is insufficient memory for the Java Runtime Environment to
>> continue.
>>
>> # Native memory allocation (mmap) failed to map 4194304 bytes for
>> committing reserved memory.
>>
>> # An error report file with more information is saved as:
>>
>> # /tmp/hs_err_pid75375.log
>>
>> OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x7f3fb400,
>> 4194304, 0) failed; error='Cannot allocate memory' (errno=12)
>>
>> #
>>
>> # There is insufficient memory for the Java Runtime Environment to
>> continue.
>>
>> # Native memory allocation (mmap) failed to map 4194304 bytes for
>> committing reserved memory.
>>
>> # An error report file with more information is saved as:
>>
>> # /tmp/hs_err_pid76380.log
>>
>> OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x7f8d1400,
>> 4194304, 0) failed; error='Cannot allocate memory' (errno=12)
>>
>>
>>
>> The above repeated errors were followed by repeated errors below:
>>
>>
>>
>> 2017-10-20 

Re: [Neo4j] Neo4j logging GB of errors when Neo4j cannot get enough memory

['Michael Hunger' via Neo4j]

Hi,

afaik it was a bug in an *older version* that it used too much memory
during recovery. (Please upgrade to 3.0.0)
How big were the transaction logs? I.e. how much data had you inserted?

What is your configuration for the page-cache?
Neo4j assumes it runs alone on the machine and grabs a percentage of RAM -
heap as page-cache when not configured.
*Please configure the page-cache explicitely in the config.*

For these memory sizes you shouldn't run community but on Enterprise.

The error is from the operating system trying to allocate memory for us,
there is not much else we can do about it.

*You should also configure tx-log retention e.g. to 10 G*

HTH

Michael


On Mon, Nov 13, 2017 at 8:08 PM,  wrote:

> Hi,
>
> One of our Linux test servers has 192 GB RAM and the following disk
> partitions (of some time earlier when / was not full). We ran Neo4j and
> Elasticsearch on this server. We set both heap initial_size and heap max
> size to 40GB.
>
>
>
> Filesystem  Size  Used Avail Use% Mounted on
>
> /dev/sda1   546G  5.1G  513G   1% /
>
> tmpfs95G   16K   95G   1% /dev/shm
>
> /dev/sdb1   6.5T  2.1T  4.1T  34% /data
>
>
>
> We set neo4j home of the Neo4j Community Edition 3.0.3 to root partition
> under  /opt/…
>
>
>
> The problem we had was neo4j.log caused the 546GB root partition full.
> After we truncated neo4j.log, neo4j could not start until we manually
> removed transaction log files.
>
>
>
> I have the following questions, can anyone help?
>
> 1)   Is there a way to set a maximum file size or file rotation like
> debug.log on neo4j.log from neo4j.conf in Enterprise version?
>
> 2)   In more than one incidents Neo4j could not restart successfully
> from transaction log files, and it could not restart until I manually
> removed transaction log files such as neostore.transaction.db.348. Can
> Enterprise version restart without human intervention if it cannot recover
> from transaction log files?
>
> 3)Is there a way to slow down reporting of 'Cannot allocate
> memory' (errno=12) errors? In our case, the errors were dumped to neo4j.log
> almost like an infinite loop.
>
> Error messages are as follows: (Neo4j.log has many millions of these
> lines:)
>
> # There is insufficient memory for the Java Runtime Environment to
> continue.
>
> # Native memory allocation (mmap) failed to map 4194304 bytes for
> committing reserved memory.
>
> # An error report file with more information is saved as:
>
> # /tmp/hs_err_pid74296.log
>
> OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x7f6b2c00,
> 4194304, 0) failed; error='Cannot allocate memory' (errno=12)
>
> #
>
> # There is insufficient memory for the Java Runtime Environment to
> continue.
>
> # Native memory allocation (mmap) failed to map 4194304 bytes for
> committing reserved memory.
>
> # An error report file with more information is saved as:
>
> # /tmp/hs_err_pid75375.log
>
> OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x7f3fb400,
> 4194304, 0) failed; error='Cannot allocate memory' (errno=12)
>
> #
>
> # There is insufficient memory for the Java Runtime Environment to
> continue.
>
> # Native memory allocation (mmap) failed to map 4194304 bytes for
> committing reserved memory.
>
> # An error report file with more information is saved as:
>
> # /tmp/hs_err_pid76380.log
>
> OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x7f8d1400,
> 4194304, 0) failed; error='Cannot allocate memory' (errno=12)
>
>
>
> The above repeated errors were followed by repeated errors below:
>
>
>
> 2017-10-20 13:31:49.509+ INFO  Starting...
>
> 2017-10-20 13:31:52.125+ INFO  Bolt enabled on localhost:7687.
>
> 2017-10-21 10:23:41.773+ ERROR Failed to start Neo4j: Starting Neo4j
> failed: Component 
> 'org.neo4j.server.database.LifecycleManagingDatabase@5ab0b0fa'
> was successfully initialized, but failed to start. Please see attached
> cause exception. Starting Neo4j failed: Component
> 'org.neo4j.server.database.LifecycleManagingDatabase@5ab0b0fa' was
> successfully initialized, but failed to start. Please see attached cause
> exception.
>
> org.neo4j.server.ServerStartupException: Starting Neo4j failed: Component
> 'org.neo4j.server.database.LifecycleManagingDatabase@5ab0b0fa' was
> successfully initialized, but failed to start. Please see attached cause
> exception.
>
> at org.neo4j.server.exception.ServerStartupErrors.
> translateToServerStartupError(ServerStartupErrors.java:68)
>
> at org.neo4j.server.AbstractNeoServer.start(
> AbstractNeoServer.java:217)
>
> at org.neo4j.server.ServerBootstrapper.start(
> ServerBootstrapper.java:87)
>
> at org.neo4j.server.ServerBootstrapper.start(
> ServerBootstrapper.java:66)
>
> at org.neo4j.server.CommunityEntryPoint.main(
> CommunityEntryPoint.java:28)
>
> Caused by: