parthchandra commented on code in PR #1240:
URL: https://github.com/apache/parquet-mr/pull/1240#discussion_r1430737627
##########
parquet-hadoop/src/main/java/org/apache/parquet/hadoop/MemoryManager.java:
##########
@@ -74,7 +75,7 @@ private void checkRatio(float ratio) {
* @param writer the new created writer
* @param allocation the requested buffer size
*/
- synchronized void addWriter(InternalParquetRecordWriter<?> writer, Long
allocation) {
+ void addWriter(InternalParquetRecordWriter<?> writer, Long allocation) {
Review Comment:
@wgtmac I'm not too familiar with this part of the code path either.
However, it does look that the `updateAllocation` call needs to be synchronized
and it requires the `writers` map to not be mutated during that time.
Spark uses the default behaviour of ParquetOutputFormat and so does end up
using the MemoryManager. The impact of this change is not clear to me though.
BTW, @ravwojdyla this is a great bit of investigation you did there! Would
it be possible to share the jfr file (in the jira, perhaps)? I'd like to see if
I can gain any additional insights from the information you have collected.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
