ravwojdyla commented on code in PR #1240:
URL: https://github.com/apache/parquet-mr/pull/1240#discussion_r1424446622
##########
parquet-hadoop/src/main/java/org/apache/parquet/hadoop/MemoryManager.java:
##########
@@ -74,7 +75,7 @@ private void checkRatio(float ratio) {
* @param writer the new created writer
* @param allocation the requested buffer size
*/
- synchronized void addWriter(InternalParquetRecordWriter<?> writer, Long
allocation) {
+ void addWriter(InternalParquetRecordWriter<?> writer, Long allocation) {
Review Comment:
@ConeyLiu thank you for a prompt review. Excited to get this fixed. Agree
it's not the same, and further `synchronized` is not locking a whole method BUT
a whole **instance** which in the case of MemoryManager is the **single/global
instance**
([src](https://github.com/apache/parquet-mr/blob/afd39dde8fd762bf696fea3dab16d45eae1093c3/parquet-hadoop/src/main/java/org/apache/parquet/hadoop/ParquetOutputFormat.java#L568)),
so `synchronized` can be very expensive here. The cost is manifested by the
lock stats above. I can see the tests are all green. Is such an "excessive"
synchronization pattern on the global instance necessary for the semantic of
MemoryManager? Which use-cases require this level of locking, and could that be
relaxed? What do you recommend as next steps? Thanks in advance!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
