Github user maropu commented on a diff in the pull request:
https://github.com/apache/spark/pull/7676#discussion_r36260211
--- Diff:
core/src/main/java/org/apache/spark/util/collection/unsafe/sort/PrefixComparators.java
---
@@ -52,6 +59,38 @@ public int compare(long bPrefix, long aPrefix) {
}
}
+ public static final class BinaryPrefixComparator extends
PrefixComparator {
+ @Override
+ public int compare(long aPrefix, long bPrefix) {
+ return UnsignedLongs.compare(aPrefix, bPrefix);
+ }
+
+ public static long computePrefix(byte[] bytes) {
+ if (bytes == null) {
+ return 0L;
+ } else {
+ /**
+ * TODO: If a wrapper for BinaryType is created (SPARK-8786),
+ * these codes below will be in the wrapper class.
+ */
+ final int minLen = Math.min(bytes.length, 8);
+ long p = 0;
+ for (int i = 1; i <= minLen; ++i) {
+ p |= (128L + PlatformDependent.UNSAFE.getByte(bytes,
BYTE_ARRAY_OFFSET + i - 1))
--- End diff --
I think it is hard to do that.
This byte access is needed to map signed bytes to unsigned ones, and
this mapping can make `UnsignedLongs#compare` compare them in a
order-preserving way according to `TypeUtils#compareBinary`.
If we have direct word access here, `BinaryPrefixComparator#compare` needs
to inefficiently compare them by using `TypeUtils#compareBinary`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
