Github user michaelkamprath commented on a diff in the pull request:
https://github.com/apache/spark/pull/16403#discussion_r93983319
--- Diff: common/unsafe/src/main/java/org/apache/spark/unsafe/Platform.java
---
@@ -244,6 +257,34 @@ public static void throwException(Throwable t) {
LONG_ARRAY_OFFSET = _UNSAFE.arrayBaseOffset(long[].class);
FLOAT_ARRAY_OFFSET = _UNSAFE.arrayBaseOffset(float[].class);
DOUBLE_ARRAY_OFFSET = _UNSAFE.arrayBaseOffset(double[].class);
+
+ // determine whether double access should be aligned.
+ String arch = System.getProperty("os.arch", "");
+ if (arch.matches("^(arm|arm32)")) {
+ logger.info(
+ "Host platform '{}' requires aligned double access. "+
+ "Creating an aligned buffer for unsafe double reads.",
+ arch);
+
+ // allocate a 2x memory block to ensure buffer used is 8-byte
aligned. Java
+ // objects are always aligned, so we just need to ensure the
offset is aligned
+ // to an 8-byte boundary
+ byte[] heapObj = new byte[16];
+ long offset = BYTE_ARRAY_OFFSET;
+ long bufferSize = 16;
+ for (long i = 0; i < 8; ++i ) {
+ if ((offset+i)%8 == 0) {
+ logger.debug("Found aligned buffer offset at {} + {}", offset,
i);
+ offset += i;
+ bufferSize -= i;
+ break;
+ }
--- End diff --
No, we can't, because we would still need the buffer to avoid doing the
direct double read from an unaligned memory location on [line
131](https://github.com/michaelkamprath/spark/blob/30c6c997ca8e864a31d2f4dfa55d47b5aa629596/common/unsafe/src/main/java/org/apache/spark/unsafe/Platform.java#L131).
The reason for finding an aligned offset here is because on ARM7, the
actual memory address for the `byte[0]` location is not the memory address of
the `byte[]` object. That starting point for the 0-index item is indicated by
`Platform.BYTE_ARRAY_OFFSET`, which is the Java overhead memory used for
managing the `byte[]` object. So if we read from `byte[0]`, that could be
unaligned because of the Java overhead, hence the reason to find the index in
the byte buffer where overhead+index is aligned. I am depending on the Java
behavior of aligning on objects here, the `byte[]` object) for needing to only
check that the offset is also aligned.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
