[
https://issues.apache.org/jira/browse/SPARK-9411?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Josh Rosen updated SPARK-9411:
------------------------------
Description:
We need to make page sizes configurable so we can reduce them in unit tests and
increase them in real production workloads.
The following hardcoded page sizes need to be updated:
- Spark Core: UnsafeShuffleExternalSorter.PAGE_SIZE
- Spark SQL: UnsafeExternalSorter.PAGE_SIZE
- Unsafe: BytesToBytesMap.PAGE_SIZE_BYTES
While updating the page sizes, we should also update certain size calculations
which are based on the page size so that they do not assume that all pages are
the same size. This isn't strictly necessary in this patch but should be done
eventually as part of supporting overflow pages for large records.
A number of unit tests also need to be updated to account for the new page
sizes.
was:
So we can reduce them in unit tests, and go up in real production workloads.
> Make page size configurable
> ---------------------------
>
> Key: SPARK-9411
> URL: https://issues.apache.org/jira/browse/SPARK-9411
> Project: Spark
> Issue Type: Improvement
> Reporter: Reynold Xin
> Assignee: Josh Rosen
> Priority: Critical
>
> We need to make page sizes configurable so we can reduce them in unit tests
> and increase them in real production workloads.
> The following hardcoded page sizes need to be updated:
> - Spark Core: UnsafeShuffleExternalSorter.PAGE_SIZE
> - Spark SQL: UnsafeExternalSorter.PAGE_SIZE
> - Unsafe: BytesToBytesMap.PAGE_SIZE_BYTES
> While updating the page sizes, we should also update certain size
> calculations which are based on the page size so that they do not assume that
> all pages are the same size. This isn't strictly necessary in this patch but
> should be done eventually as part of supporting overflow pages for large
> records.
> A number of unit tests also need to be updated to account for the new page
> sizes.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
