I monitored the heap using jconsole and much to my surprise observed that the
heap stayed relatively stable while the overall memory occupancy of the process
grew steadily until it reached the ~500M . I'm now rather confused as to what
else can be consuming memory like that.... Any ideas folks?
Sent from my iPhone
On 1 Jun 2011, at 20:52, Michael Hunger <michael.hun...@neotechnology.com>
wrote:
> props passed in to the batchinserter
> look into messages log
> you see the different gc behaviour
>
> Michael
>
> Sent from my iBrick4
>
>
> Am 01.06.2011 um 20:44 schrieb Paul Bandler <pband...@cseuk.co.uk>:
>
>> Is that simply set as a system property or via the Map passed as the second
>> parameter to the BatchInserterImpl constructor? I've tried both and doesn't
>> seem to help. Is there some way I can verify that it's being used?
>>
>> I'm using 1.3
>>
>> On 1 Jun 2011, at 18:49, Michael Hunger wrote:
>>
>>> you could use cache_type=weak
>>> in the db properties
>>>
>>> you can easily introspect java programs (heap) using jmap jconsole or
>>> visualvm
>>>
>>> what version of neo4j are you using?
>>>
>>> index.flush just sets a flag for immediate index querying
>>>
>>> Sent from my iBrick4
>>>
>>>
>>> Am 01.06.2011 um 19:18 schrieb Paul Bandler <pband...@cseuk.co.uk>:
>>>
>>>> I have a simple program that uses the BatchInserter to load rows from a
>>>> SQL database and am running it on a modestly configured Windows machine
>>>> with 2GB of RAM and setting the max heap to 500M.
>>>>
>>>> Initially it was running out of memory quite soon so I introduced a flush
>>>> after every 5000 nodes and it appeared that all was well. But having got
>>>> further in the data load it appears to hop along nicely but the memory
>>>> allocated (simply visible using windows task manager) grows and grows
>>>> until I suspect it's reached its max heap size and it's written about 2M
>>>> nodes then abruptly stops making any further discernible progress. It
>>>> doesn't fail, just the logging I've put in to log every 5000 nodes has
>>>> stopped and the CPU is 100% used - garbage collecting I suspect.
>>>>
>>>> Is there something I should be doing periodically in addition to the
>>>> index flush to stop the heap exhaustion? My code is really simple, here's
>>>> the method for loading nodes from each table:-
>>>>
>>>> public long restoreCollection() {
>>>> resolveSql();
>>>> _log.debug("restore collection:" + getCollectionName() + " using: "
>>>> + _sql + " and:" + Arrays.deepToString(_columns));
>>>> final BatchInserterIndex _index = makeIndex();
>>>> final long collectionNode = _inserter.createNode(MapUtil.map("name",
>>>> getCollectionName()));
>>>>
>>>> _log.debug("Query db...");
>>>> getJdbcTemplate().query(_sql, new Object[] {},
>>>> new RowCallbackHandler() {
>>>> public void processRow(ResultSet row) throws SQLException {
>>>> final Map<String, Object> properties =
>>>> extractproperties(row);
>>>> long node = _inserter.createNode(properties);
>>>> _inserter.createRelationship(node, collectionNode,
>>>> RdmRelationship.MEMBER_OF, null);
>>>> if (_index != null)
>>>> for (DbColumn col : _columns) {
>>>> if (col.isIndexed())
>>>> _index.add(node, MapUtil.map(col.getName(),
>>>> properties.get(col.getName())));
>>>> }
>>>> _collectionSize++;
>>>> if ((_collectionSize % FLUSH_INTERVAL == 0)) {
>>>> if (_index != null)
>>>> _index.flush();
>>>> _log.debug("Added node:" + _collectionSize
>>>> + " to: " + getCollectionName());
>>>> }
>>>> }
>>>> });
>>>>
>>>> // long collectionNode = -1;
>>>> if (_index != null) {
>>>> _index.flush();
>>>> }
>>>> _log.debug("Completed restoring " + _collectionSize + " to: "
>>>> + getCollectionName());
>>>> return collectionNode;
>>>> }
>>>>
>>>>
>>>> and then around that a higher level function that handles all tables:-
>>>>
>>>> public void run() {
>>>> throwIfNull(_restorers, "Restorers missing");
>>>> throwIfNull(_inserter, "Batch inserter missing");
>>>> int totalNodes = 0;
>>>> int totalRelationships = 0;
>>>> try {
>>>> for (CollectionRestorer r : _restorers) {
>>>> long collection = r.restoreCollection();
>>>> totalNodes += r.getCollectionSize();
>>>> _inserter.createRelationship(_inserter.getGraphDbService()
>>>> .getReferenceNode().getId(), collection,
>>>> RdmRelationship.CLASS_EXTENT, null);
>>>> }
>>>> for (ParentChildRelationshipBuilder r : _relators) {
>>>> r.makeRelationships();
>>>> totalRelationships += r.getRelations();
>>>>
>>>> }
>>>> } finally {
>>>> _inserter.shutdown();
>>>> _log.info("Batch inserter shutdown. Created: " + totalNodes + "
>>>> nodes and "
>>>> + totalRelationships + " relationships");
>>>> }
>>>> }
>>>>
>>>> Any suggestions welcome.
>>>> _______________________________________________
>>>> Neo4j mailing list
>>>> User@lists.neo4j.org
>>>> https://lists.neo4j.org/mailman/listinfo/user
>>> _______________________________________________
>>> Neo4j mailing list
>>> User@lists.neo4j.org
>>> https://lists.neo4j.org/mailman/listinfo/user
>>
>> _______________________________________________
>> Neo4j mailing list
>> User@lists.neo4j.org
>> https://lists.neo4j.org/mailman/listinfo/user
> _______________________________________________
> Neo4j mailing list
> User@lists.neo4j.org
> https://lists.neo4j.org/mailman/listinfo/user
_______________________________________________
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user
