dongjoon-hyun commented on pull request #34846:
URL: https://github.com/apache/spark/pull/34846#issuecomment-1002336872
The new proposed code focused only on fitting to JVM G1HeapRegionSize and it
claims that it will reduce the Spark's pageSize (humongous) to smaller ones. In
this case, there is no guarantee that new defaultPageSizeBytes is better than
the previous calculated ones (whatever they were in the production) in terms of
the executor time (if no GC occurs because there is enough memory in the
extreme cases), isn't it, @srowen ?
```
private lazy val defaultPageSizeBytes = Utils.maybeG1HeapRegionSize match {
case Some(heapRegionSize) if tungstenMemoryMode == MemoryMode.ON_HEAP =>
heapRegionSize - Platform.LONG_ARRAY_OFFSET
```
If the smaller page size always wins, we may want to have a default value,
`1MB` (the minimum value of G1HeapRegionSize) - Platform.LONG_ARRAY_OFFSET, for
`spark.buffer.pageSize` configuration, instead of this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
