On Oct 16, 6:45 pm, Shi Yu <[email protected]> wrote: > I have also tried the CacheLoader API, it pops a java GC error. The > thing I haven't tried is to separate 6 million records into several > objects and try CacheLoader. But I don't think it should be that > fragile and complicated. I have spent a whole day on this issue, now I > just rely the hybrid approach to finish the work. But I would be very > interested to hear any solution to solve this issue.
I cannot make any suggestions as to why you got an error without knowing what you did and what error you got. I would not expect the same that you posted to work without a lot of memory, tweaking, and a very fast network since you're just filling an output queue as fast as java will allow you. You didn't share any code using CacheLoader, so I can only guess as to how you may have used it to get an error. There are three different methods you can use -- did you try to create a map with six million values and then pass it to the CacheLoader API (that would very likely give you an out of memory error). You could also be taxing the GC considerably by converting integers to strings to compute modulus if your jvm doesn't do proper escape analysis. I can assure you there's no magic that will make it fail to load six million records through the API as long as you account for the realities of your network (which CacheLoader does for you) and your available memory.
