xushiyan commented on issue #7565:
URL: https://github.com/apache/hudi/issues/7565#issuecomment-1374522126
```java
inputRecords
.mapToPair(record -> Pair.of(
new Tuple2<>(record.getPartitionPath(),
Option.ofNullable(record.getCurrentLocation())), record))
.countByKey();
```
you should refer to
`org.apache.hudi.table.action.commit.BaseSparkCommitActionExecutor#buildProfile`
which is used by spark.
I think this is more of a spark job tuning issue, where parallelism and
executor memory should be tuned.
> Reason: Remote RPC client disassociated. Likely due to containers
exceeding thresholds, or network issues. Check driver logs for WARN messages.
Any further info on this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
