lidavidm commented on code in PR #215:
URL: https://github.com/apache/arrow-cookbook/pull/215#discussion_r880809920
##########
java/source/create.rst:
##########
@@ -70,6 +70,62 @@ Array of Varchar
[one, two, three]
+In some scenarios could be more appropriate use `Dictionary-encoded Layout`_
to encoded data which takes much less space.
Review Comment:
```suggestion
Dictionary-Encoded Array of Varchar
-----------------------------------
In some scenarios `dictionary-encoding`_ a column is useful to save memory.
```
##########
java/source/create.rst:
##########
@@ -70,6 +70,62 @@ Array of Varchar
[one, two, three]
+In some scenarios could be more appropriate use `Dictionary-encoded Layout`_
to encoded data which takes much less space.
+
+.. testcode::
+
+ import org.apache.arrow.memory.BufferAllocator;
+ import org.apache.arrow.memory.RootAllocator;
+ import org.apache.arrow.vector.FieldVector;
+ import org.apache.arrow.vector.VarCharVector;
+ import org.apache.arrow.vector.dictionary.Dictionary;
+ import org.apache.arrow.vector.dictionary.DictionaryEncoder;
+ import org.apache.arrow.vector.types.pojo.DictionaryEncoding;
+
+ import java.nio.charset.StandardCharsets;
+
+ try (BufferAllocator root = new RootAllocator();
+ VarCharVector countries = new VarCharVector("country-dict", root);
+ VarCharVector myAppUseCountryDictionary = new
VarCharVector("app-use-country-dict", root)
Review Comment:
Can we avoid the `my` prefix on variable names?
##########
java/source/io.rst:
##########
@@ -443,3 +443,108 @@ Reading Parquet File
********************
Please check :doc:`Dataset <./dataset>`
+
+Dictionary-encoded Layout
+=========================
+
+In some scenarios could be more appropriate use `Dictionary-encoded Layout`_
to encoded data which takes much less space.
+
+.. testcode::
+
+ import org.apache.arrow.memory.BufferAllocator;
+ import org.apache.arrow.memory.RootAllocator;
+ import org.apache.arrow.vector.FieldVector;
+ import org.apache.arrow.vector.ValueVector;
+ import org.apache.arrow.vector.VarCharVector;
+ import org.apache.arrow.vector.VectorSchemaRoot;
+ import org.apache.arrow.vector.dictionary.Dictionary;
+ import org.apache.arrow.vector.dictionary.DictionaryEncoder;
+ import org.apache.arrow.vector.dictionary.DictionaryProvider;
+ import org.apache.arrow.vector.ipc.ArrowFileReader;
+ import org.apache.arrow.vector.ipc.ArrowFileWriter;
+ import org.apache.arrow.vector.ipc.message.ArrowBlock;
+ import org.apache.arrow.vector.types.pojo.DictionaryEncoding;
+
+ import java.io.File;
+ import java.io.FileInputStream;
+ import java.io.FileNotFoundException;
+ import java.io.FileOutputStream;
+ import java.io.IOException;
+ import java.nio.charset.StandardCharsets;
+
+ try (BufferAllocator root = new RootAllocator();
+ VarCharVector countries = new VarCharVector("country-dict", root);
+ VarCharVector myAppUseCountryDictionary = new
VarCharVector("app-use-country-dict", root)
+ ) {
+ countries.allocateNew(10);
+ countries.set(0, "Andorra".getBytes(StandardCharsets.UTF_8));
+ countries.set(1, "Cuba".getBytes(StandardCharsets.UTF_8));
+ countries.set(2, "Grecia".getBytes(StandardCharsets.UTF_8));
+ countries.set(3, "Guinea".getBytes(StandardCharsets.UTF_8));
+ countries.set(4, "Islandia".getBytes(StandardCharsets.UTF_8));
+ countries.set(5, "Malta".getBytes(StandardCharsets.UTF_8));
+ countries.set(6, "Tailandia".getBytes(StandardCharsets.UTF_8));
+ countries.set(7, "Uganda".getBytes(StandardCharsets.UTF_8));
+ countries.set(8, "Yemen".getBytes(StandardCharsets.UTF_8));
+ countries.set(9, "Zambia".getBytes(StandardCharsets.UTF_8));
+ countries.setValueCount(10);
+
+ Dictionary myCountryDictionary = new Dictionary(countries,
+ new DictionaryEncoding(/*id=*/123L, /*ordered=*/false,
/*indexType=*/null));
+ System.out.println("Dictionary used: " + myCountryDictionary);
+
+ myAppUseCountryDictionary.allocateNew(5);
+ myAppUseCountryDictionary.set(0,
"Andorra".getBytes(StandardCharsets.UTF_8));
+ myAppUseCountryDictionary.set(1,
"Guinea".getBytes(StandardCharsets.UTF_8));
+ myAppUseCountryDictionary.set(2,
"Islandia".getBytes(StandardCharsets.UTF_8));
+ myAppUseCountryDictionary.set(3,
"Malta".getBytes(StandardCharsets.UTF_8));
+ myAppUseCountryDictionary.set(4,
"Uganda".getBytes(StandardCharsets.UTF_8));
+ myAppUseCountryDictionary.setValueCount(5);
+ System.out.println("Data to retain: " + myAppUseCountryDictionary);
+
+ File file = new File("randon_access_to_file_with_dictionary.arrow");
Review Comment:
```suggestion
File file = new File("random_access_file_with_dictionary.arrow");
```
##########
java/source/io.rst:
##########
@@ -443,3 +443,108 @@ Reading Parquet File
********************
Please check :doc:`Dataset <./dataset>`
+
+Dictionary-encoded Layout
+=========================
Review Comment:
```suggestion
Handling Data with Dictionaries
*******************************
```
##########
java/source/io.rst:
##########
@@ -443,3 +443,108 @@ Reading Parquet File
********************
Please check :doc:`Dataset <./dataset>`
+
+Dictionary-encoded Layout
+=========================
+
+In some scenarios could be more appropriate use `Dictionary-encoded Layout`_
to encoded data which takes much less space.
+
+.. testcode::
+
+ import org.apache.arrow.memory.BufferAllocator;
+ import org.apache.arrow.memory.RootAllocator;
+ import org.apache.arrow.vector.FieldVector;
+ import org.apache.arrow.vector.ValueVector;
+ import org.apache.arrow.vector.VarCharVector;
+ import org.apache.arrow.vector.VectorSchemaRoot;
+ import org.apache.arrow.vector.dictionary.Dictionary;
+ import org.apache.arrow.vector.dictionary.DictionaryEncoder;
+ import org.apache.arrow.vector.dictionary.DictionaryProvider;
+ import org.apache.arrow.vector.ipc.ArrowFileReader;
+ import org.apache.arrow.vector.ipc.ArrowFileWriter;
+ import org.apache.arrow.vector.ipc.message.ArrowBlock;
+ import org.apache.arrow.vector.types.pojo.DictionaryEncoding;
+
+ import java.io.File;
+ import java.io.FileInputStream;
+ import java.io.FileNotFoundException;
+ import java.io.FileOutputStream;
+ import java.io.IOException;
+ import java.nio.charset.StandardCharsets;
+
+ try (BufferAllocator root = new RootAllocator();
+ VarCharVector countries = new VarCharVector("country-dict", root);
+ VarCharVector myAppUseCountryDictionary = new
VarCharVector("app-use-country-dict", root)
+ ) {
+ countries.allocateNew(10);
+ countries.set(0, "Andorra".getBytes(StandardCharsets.UTF_8));
+ countries.set(1, "Cuba".getBytes(StandardCharsets.UTF_8));
+ countries.set(2, "Grecia".getBytes(StandardCharsets.UTF_8));
+ countries.set(3, "Guinea".getBytes(StandardCharsets.UTF_8));
+ countries.set(4, "Islandia".getBytes(StandardCharsets.UTF_8));
+ countries.set(5, "Malta".getBytes(StandardCharsets.UTF_8));
+ countries.set(6, "Tailandia".getBytes(StandardCharsets.UTF_8));
+ countries.set(7, "Uganda".getBytes(StandardCharsets.UTF_8));
+ countries.set(8, "Yemen".getBytes(StandardCharsets.UTF_8));
+ countries.set(9, "Zambia".getBytes(StandardCharsets.UTF_8));
+ countries.setValueCount(10);
+
+ Dictionary myCountryDictionary = new Dictionary(countries,
+ new DictionaryEncoding(/*id=*/123L, /*ordered=*/false,
/*indexType=*/null));
+ System.out.println("Dictionary used: " + myCountryDictionary);
+
+ myAppUseCountryDictionary.allocateNew(5);
+ myAppUseCountryDictionary.set(0,
"Andorra".getBytes(StandardCharsets.UTF_8));
+ myAppUseCountryDictionary.set(1,
"Guinea".getBytes(StandardCharsets.UTF_8));
+ myAppUseCountryDictionary.set(2,
"Islandia".getBytes(StandardCharsets.UTF_8));
+ myAppUseCountryDictionary.set(3,
"Malta".getBytes(StandardCharsets.UTF_8));
+ myAppUseCountryDictionary.set(4,
"Uganda".getBytes(StandardCharsets.UTF_8));
+ myAppUseCountryDictionary.setValueCount(5);
+ System.out.println("Data to retain: " + myAppUseCountryDictionary);
+
+ File file = new File("randon_access_to_file_with_dictionary.arrow");
+ DictionaryProvider.MapDictionaryProvider provider = new
DictionaryProvider.MapDictionaryProvider();
+ provider.put(myCountryDictionary);
+ try (FieldVector myAppUseCountryDictionaryEncoded = (FieldVector)
DictionaryEncoder
+ .encode(myAppUseCountryDictionary, myCountryDictionary);
+ VectorSchemaRoot vectorSchemaRoot =
VectorSchemaRoot.of(myAppUseCountryDictionaryEncoded);
+ FileOutputStream fileOutputStream = new FileOutputStream(file);
+ ArrowFileWriter writer = new ArrowFileWriter(vectorSchemaRoot,
provider, fileOutputStream.getChannel())
+ ) {
+ System.out.println("Data to retain through Dictionary: "
+myAppUseCountryDictionaryEncoded);
+ writer.start();
+ writer.writeBatch();
+ writer.end();
+ System.out.println("Record batches written: " +
writer.getRecordBlocks().size() + ". Number of rows written: " +
vectorSchemaRoot.getRowCount());
+ try(
+ BufferAllocator rootAllocator = new RootAllocator();
+ FileInputStream fileInputStream = new
FileInputStream(file);
+ ArrowFileReader reader = new
ArrowFileReader(fileInputStream.getChannel(), rootAllocator)
+ ){
+ for (ArrowBlock arrowBlock : reader.getRecordBlocks()) {
+ reader.loadRecordBatch(arrowBlock);
+ FieldVector myAppUseCountryDictionaryEncodedRead =
reader.getVectorSchemaRoot().getVector("app-use-country-dict");
+ Dictionary myAppUseCountryDictionaryRead =
reader.getDictionaryVectors().get(123L);
+ System.out.println("Data to retain recovered: " +
myAppUseCountryDictionaryEncodedRead);
+ System.out.println("Dictionary recovered: " +
myAppUseCountryDictionaryRead);
+ try (ValueVector readVector =
DictionaryEncoder.decode(myAppUseCountryDictionaryEncodedRead,
myAppUseCountryDictionaryRead)) {
+ System.out.println("Data to retain recovered decoded:
" + readVector);
+ }
+ }
+ }
+ } catch (FileNotFoundException e) {
+ e.printStackTrace();
+ } catch (IOException e) {
+ e.printStackTrace();
+ }
+ }
+
+.. testoutput::
+
+ Dictionary used: Dictionary
DictionaryEncoding[id=123,ordered=false,indexType=Int(32, true)] [Andorra,
Cuba, Grecia, Guinea, Islandia, Malta, Tailandia, Uganda, Yemen, Zambia]
+ Data to retain: [Andorra, Guinea, Islandia, Malta, Uganda]
Review Comment:
In all of these examples I would call this "Unencoded data" and
"Dictionary-encoded data"
##########
java/source/create.rst:
##########
@@ -70,6 +70,62 @@ Array of Varchar
[one, two, three]
+In some scenarios could be more appropriate use `Dictionary-encoded Layout`_
to encoded data which takes much less space.
+
+.. testcode::
+
+ import org.apache.arrow.memory.BufferAllocator;
+ import org.apache.arrow.memory.RootAllocator;
+ import org.apache.arrow.vector.FieldVector;
+ import org.apache.arrow.vector.VarCharVector;
+ import org.apache.arrow.vector.dictionary.Dictionary;
+ import org.apache.arrow.vector.dictionary.DictionaryEncoder;
+ import org.apache.arrow.vector.types.pojo.DictionaryEncoding;
+
+ import java.nio.charset.StandardCharsets;
+
+ try (BufferAllocator root = new RootAllocator();
+ VarCharVector countries = new VarCharVector("country-dict", root);
+ VarCharVector myAppUseCountryDictionary = new
VarCharVector("app-use-country-dict", root)
+ ) {
+ countries.allocateNew(10);
+ countries.set(0, "Andorra".getBytes(StandardCharsets.UTF_8));
+ countries.set(1, "Cuba".getBytes(StandardCharsets.UTF_8));
+ countries.set(2, "Grecia".getBytes(StandardCharsets.UTF_8));
+ countries.set(3, "Guinea".getBytes(StandardCharsets.UTF_8));
+ countries.set(4, "Islandia".getBytes(StandardCharsets.UTF_8));
+ countries.set(5, "Malta".getBytes(StandardCharsets.UTF_8));
+ countries.set(6, "Tailandia".getBytes(StandardCharsets.UTF_8));
+ countries.set(7, "Uganda".getBytes(StandardCharsets.UTF_8));
+ countries.set(8, "Yemen".getBytes(StandardCharsets.UTF_8));
+ countries.set(9, "Zambia".getBytes(StandardCharsets.UTF_8));
+ countries.setValueCount(10);
+
+ Dictionary myCountryDictionary = new Dictionary(countries,
Review Comment:
Just `countriesDictionary`?
##########
java/source/io.rst:
##########
@@ -443,3 +443,108 @@ Reading Parquet File
********************
Please check :doc:`Dataset <./dataset>`
+
+Dictionary-encoded Layout
+=========================
+
+In some scenarios could be more appropriate use `Dictionary-encoded Layout`_
to encoded data which takes much less space.
Review Comment:
```suggestion
Reading and writing dictionary-encoded data requires separately tracking the
dictionaries.
```
##########
java/source/create.rst:
##########
@@ -70,6 +70,62 @@ Array of Varchar
[one, two, three]
+In some scenarios could be more appropriate use `Dictionary-encoded Layout`_
to encoded data which takes much less space.
+
+.. testcode::
+
+ import org.apache.arrow.memory.BufferAllocator;
+ import org.apache.arrow.memory.RootAllocator;
+ import org.apache.arrow.vector.FieldVector;
+ import org.apache.arrow.vector.VarCharVector;
+ import org.apache.arrow.vector.dictionary.Dictionary;
+ import org.apache.arrow.vector.dictionary.DictionaryEncoder;
+ import org.apache.arrow.vector.types.pojo.DictionaryEncoding;
+
+ import java.nio.charset.StandardCharsets;
+
+ try (BufferAllocator root = new RootAllocator();
+ VarCharVector countries = new VarCharVector("country-dict", root);
+ VarCharVector myAppUseCountryDictionary = new
VarCharVector("app-use-country-dict", root)
+ ) {
+ countries.allocateNew(10);
+ countries.set(0, "Andorra".getBytes(StandardCharsets.UTF_8));
+ countries.set(1, "Cuba".getBytes(StandardCharsets.UTF_8));
+ countries.set(2, "Grecia".getBytes(StandardCharsets.UTF_8));
+ countries.set(3, "Guinea".getBytes(StandardCharsets.UTF_8));
+ countries.set(4, "Islandia".getBytes(StandardCharsets.UTF_8));
+ countries.set(5, "Malta".getBytes(StandardCharsets.UTF_8));
+ countries.set(6, "Tailandia".getBytes(StandardCharsets.UTF_8));
+ countries.set(7, "Uganda".getBytes(StandardCharsets.UTF_8));
+ countries.set(8, "Yemen".getBytes(StandardCharsets.UTF_8));
+ countries.set(9, "Zambia".getBytes(StandardCharsets.UTF_8));
+ countries.setValueCount(10);
+
+ Dictionary myCountryDictionary = new Dictionary(countries,
+ new DictionaryEncoding(/*id=*/1L, /*ordered=*/false,
/*indexType=*/null));
Review Comment:
Perhaps explicitly use `new ArrowType.Int(8, true)` or something to also
demonstrate a smaller index type
##########
java/source/create.rst:
##########
@@ -70,6 +70,62 @@ Array of Varchar
[one, two, three]
+In some scenarios could be more appropriate use `Dictionary-encoded Layout`_
to encoded data which takes much less space.
+
+.. testcode::
+
+ import org.apache.arrow.memory.BufferAllocator;
+ import org.apache.arrow.memory.RootAllocator;
+ import org.apache.arrow.vector.FieldVector;
+ import org.apache.arrow.vector.VarCharVector;
+ import org.apache.arrow.vector.dictionary.Dictionary;
+ import org.apache.arrow.vector.dictionary.DictionaryEncoder;
+ import org.apache.arrow.vector.types.pojo.DictionaryEncoding;
+
+ import java.nio.charset.StandardCharsets;
+
+ try (BufferAllocator root = new RootAllocator();
+ VarCharVector countries = new VarCharVector("country-dict", root);
+ VarCharVector myAppUseCountryDictionary = new
VarCharVector("app-use-country-dict", root)
+ ) {
+ countries.allocateNew(10);
+ countries.set(0, "Andorra".getBytes(StandardCharsets.UTF_8));
+ countries.set(1, "Cuba".getBytes(StandardCharsets.UTF_8));
+ countries.set(2, "Grecia".getBytes(StandardCharsets.UTF_8));
+ countries.set(3, "Guinea".getBytes(StandardCharsets.UTF_8));
+ countries.set(4, "Islandia".getBytes(StandardCharsets.UTF_8));
+ countries.set(5, "Malta".getBytes(StandardCharsets.UTF_8));
+ countries.set(6, "Tailandia".getBytes(StandardCharsets.UTF_8));
+ countries.set(7, "Uganda".getBytes(StandardCharsets.UTF_8));
+ countries.set(8, "Yemen".getBytes(StandardCharsets.UTF_8));
+ countries.set(9, "Zambia".getBytes(StandardCharsets.UTF_8));
+ countries.setValueCount(10);
+
+ Dictionary myCountryDictionary = new Dictionary(countries,
+ new DictionaryEncoding(/*id=*/1L, /*ordered=*/false,
/*indexType=*/null));
Review Comment:
Hmm, shouldn't `indexType` be non-null?
##########
java/source/create.rst:
##########
@@ -70,6 +70,62 @@ Array of Varchar
[one, two, three]
+In some scenarios could be more appropriate use `Dictionary-encoded Layout`_
to encoded data which takes much less space.
+
+.. testcode::
+
+ import org.apache.arrow.memory.BufferAllocator;
+ import org.apache.arrow.memory.RootAllocator;
+ import org.apache.arrow.vector.FieldVector;
+ import org.apache.arrow.vector.VarCharVector;
+ import org.apache.arrow.vector.dictionary.Dictionary;
+ import org.apache.arrow.vector.dictionary.DictionaryEncoder;
+ import org.apache.arrow.vector.types.pojo.DictionaryEncoding;
+
+ import java.nio.charset.StandardCharsets;
+
+ try (BufferAllocator root = new RootAllocator();
+ VarCharVector countries = new VarCharVector("country-dict", root);
+ VarCharVector myAppUseCountryDictionary = new
VarCharVector("app-use-country-dict", root)
Review Comment:
Also it's unclear why `dictionary` is in the name, maybe
`appUserCountriesUnencoded` or something
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
