cbomgit commented on issue #11821: URL: https://github.com/apache/hudi/issues/11821#issuecomment-2308978687
Thanks. Switching to SIMPLE index doesn't seem to help. Job fails at the following job and stage:    The task logs show OOM error: ``` 2024-08-25T05:22:43.611+0000: [Full GC (Allocation Failure) 36580M->33919M(40960M), 46.7644028 secs] [Eden: 0.0B(1792.0M)->0.0B(2048.0M) Survivors: 256.0M->0.0B Heap: 36580.4M(40960.0M)->33919.2M(40960.0M)], [Metaspace: 117312K->117312K(135168K)] [Times: user=77.83 sys=0.20, real=46.76 secs] 2024-08-25T05:23:30.376+0000: [Full GC (Allocation Failure) 33919M->33904M(40960M), 43.9530911 secs] [Eden: 0.0B(2048.0M)->0.0B(2048.0M) Survivors: 0.0B->0.0B Heap: 33919.2M(40960.0M)->33904.1M(40960.0M)], [Metaspace: 117312K->113897K(135168K)] [Times: user=74.71 sys=0.03, real=43.96 secs] # # java.lang.OutOfMemoryError: Java heap space # -XX:OnOutOfMemoryError="kill -9 %p kill -9 %p" # Executing /bin/sh -c "kill -9 829 kill -9 829"... ``` For settings, I set simple index and upsert parallelism to 1500, but it doesn't seem to be using that setting here. Are there any HUDI config knobs you recommend tweaking to increase the throughput here? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
