Hmm I don't see anything obviously wrong with the code, assuming
TableReference and TableSchema both implement a hashCode() consistent with
equals() [I'm not 100% sure - could you check that?]... I'm hoping saving
heap dumps will shed some light.
On Thu, May 17, 2018 at 12:25 AM Jose Ignacio
Thanks Eugene,
I will try "saveHeapDumpsToGcsPath" to see if there are any head dumps that
I am missing.
I am using DynamicDestinations, indeed. You can find the code here:
https://pastebin.com/yrmXFQLH
Note: DocumentWithSchema is a case class that contains the document JSON
along with its
Thanks. This definitely looks like a memory usage issue to me, I recommend
you try running with --saveHeapDumpsToGcsPath=gs://... and see if workers
leave any heap dumps over there - perhaps there were some OOMs you missed?
If there are heap dumps, you can use any regular Java memory profiler on
Thanks for your response Eugene.
I don't see any more error traces than the shown in the previous mail in
the"worker" category, but checking the "jvm-gc" logs category I can see a
lot of these traces:
I 5498.913: [GC (Allocation Failure) 2018-05-09T17:42:59.661+:
5498.913: [DefNew:
Do you see any other exceptions in Stackdriver, e.g. (guess) OOM errors?
"worker lost contact with the service" is commonly caused by OOM crashes,
and GC thrashing could also cause network issues of the sort that you're
seeing.
On Fri, May 11, 2018 at 2:54 AM Jose Ignacio Honrado Benítez <
