Reynold Xin created SPARK-9517: ---------------------------------- Summary: BytesToBytesMap should encode data the same way as UnsafeExternalSorter Key: SPARK-9517 URL: https://issues.apache.org/jira/browse/SPARK-9517 Project: Spark Issue Type: Sub-task Components: SQL Reporter: Reynold Xin Assignee: Reynold Xin
BytesToBytesMap current encodes key/value data in the following format: {code} 8B key length, key data, 8B value length, value data {code} UnsafeExternalSorter, on the other hand, encodes data this way: {code} 4B record length, data {code} As a result, we cannot pass records encoded by BytesToBytesMap directly into UnsafeExternalSorter for sorting. However, if we rearrange data slightly, we can then pass the key/value records directly into UnsafeExternalSorter: {code} 4B key+value length, 4B key length, key data, value data {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org