[
https://issues.apache.org/jira/browse/PHOENIX-1737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14364273#comment-14364273
]
Tulasi P commented on PHOENIX-1737:
-----------------------------------
For reference, attached is the generic version of the bulk-loading code. By
dedup, are you referring to handling of duplicate rowkeys or duplicate "parts"
of the data across rows?
> Provide APIs for creating Phoenix encoded rowkeys
> -------------------------------------------------
>
> Key: PHOENIX-1737
> URL: https://issues.apache.org/jira/browse/PHOENIX-1737
> Project: Phoenix
> Issue Type: Improvement
> Reporter: Tulasi P
> Attachments: TestPhoenixBulkLoad.java
>
>
> Here is the code I used for direct Phoenix encoding of the composite rowkey.
> Bulk-loading data with direct encoding can give upto 4x better performance
> compared to JDBC path used in the default csv bulk-loader.
> Providing APIs for performing Phoenix encoding will be useful in such
> scenarios.
> {code}
> // rowkey is a 3 column (unsigned & fixed-size) composite key
> // 3 column qualifiers - q1, q2, q3
> ImmutableBytesWritable outputKey = new ImmutableBytesWritable();
> byte[] key1 = new byte[1];
> byte[] key2 = new byte[4];
> byte[] key3 = new byte[4];
> byte[] outKeyByteArr = new byte[1 + key1.length + key2.length + key3.length];
>
> byte[] saltedKeyByteArr = new byte[outKeyByteArr.length];
> System.arraycopy(key1, 0, outKeyByteArr, 1, key1.length);
> System.arraycopy(key2, 0, outKeyByteArr, 1+key1.length, key2.length);
> System.arraycopy(key3, 0, outKeyByteArr, 1+key1.length+key2.length,
> key3.length);
> saltedKeyByteArr = SaltingUtil.getSaltedKey(new
> ImmutableBytesWritable(outKeyByteArr), NUM_BUCKETS);
> outputKey.set(saltedKeyByteArr);
> kv = new KeyValue(outputKey.get(),"0".getBytes(), "q1".getBytes(),
> v1.getBytes());
> context.write(outputKey, kv);
> kv = new KeyValue(outputKey.get(),"0".getBytes(), "q2".getBytes(),
> v2.getBytes());
> context.write(outputKey, kv);
> kv = new KeyValue(outputKey.get(),"0".getBytes(), "q3".getBytes(),
> v3.getBytes());
> context.write(outputKey, kv);
> kv = new KeyValue(outputKey.get(),"0".getBytes(), "_0".getBytes());
> context.write(outputKey, kv);
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
