Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/7676#discussion_r36246817
--- Diff:
core/src/main/java/org/apache/spark/util/collection/unsafe/sort/PrefixComparators.java
---
@@ -52,6 +59,38 @@ public int compare(long bPrefix, long aPrefix) {
}
}
+ public static final class BinaryPrefixComparator extends
PrefixComparator {
+ @Override
+ public int compare(long aPrefix, long bPrefix) {
+ return UnsignedLongs.compare(aPrefix, bPrefix);
+ }
+
+ public static long computePrefix(byte[] bytes) {
+ if (bytes == null) {
+ return 0L;
+ } else {
+ /**
+ * TODO: If a wrapper for BinaryType is created (SPARK-8786),
+ * these codes below will be in the wrapper class.
+ */
+ final int minLen = Math.min(bytes.length, 8);
+ long p = 0;
+ for (int i = 1; i <= minLen; ++i) {
+ p |= (128L + PlatformDependent.UNSAFE.getByte(bytes,
BYTE_ARRAY_OFFSET + i - 1))
--- End diff --
can we get a word directly, similar to how we do it in utf8string?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
