On 7/11/2019 9:04 AM, Joseph_Tucker wrote:
Looks like I've managed to get some semblance of this working. The indexes are much faster, but the RAM usage by SolrJ is quite high. Is it normal to see around 6GB of RAM usage? (My test is indexing 250,000 records with the 50 child entities)
Whatever max heap value you tell Java it can have, it will eventually use. That's how Java's memory model works. You can try lowering the max heap, to see whether it actually needs that much memory. If the program really does require all the heap it's allowed, reducing the max heap size will cause the program to throw errors and probably behave in an unpredictable manner.
Many JDBC drivers will load the entire result set from a database query into memory by default, which can explain very high memory use. You would need to research your specific JDBC driver to see if it does this, and if so, learn how to have it stream the results instead of storing them.
Thanks, Shawn
