Hello all, At some point we will need to re-build an index that totals about 2 terrabytes in size (split over 10 shards). At our current indexing speed we estimate that this will take about 3 weeks. We would like to reduce that time. It appears that our main bottleneck is disk I/O. We currently have ramBufferSizeMB set to 32 and our merge factor is 10. If we increase ramBufferSizeMB to 320, we avoid a merge and the 9 disk writes and reads to merge 9+1 32MB segments into a 320MB segment.
Assuming we allocate enough memory to the JVM, would it make sense to increase ramBufferSize to 3200MB? What are people's experiences with very large ramBufferSizeMB sizes? Tom Burton-West University of Michigan Library www.hathitrust.org
