GitHub user gatorsmile opened a pull request:
https://github.com/apache/spark/pull/10165
[SPARK-12164] [SQL] Display the binary/encoded values
When the dataset is encoded, the existing display looks strange. Decimal
format is not common when the type is binary.
```
implicit val kryoEncoder = Encoders.kryo[KryoClassData]
val ds = Seq(KryoClassData("a", 1), KryoClassData("b", 2),
KryoClassData("c", 3)).toDS()
ds.show(20, false);
```
The output is like
```
+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
|value
|
+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
|[1, 0, 111, 114, 103, 46, 97, 112, 97, 99, 104, 101, 46, 115, 112, 97,
114, 107, 46, 115, 113, 108, 46, 75, 114, 121, 111, 67, 108, 97, 115, 115, 68,
97, 116, -31, 1, 1, -126, 97, 2]|
|[1, 0, 111, 114, 103, 46, 97, 112, 97, 99, 104, 101, 46, 115, 112, 97,
114, 107, 46, 115, 113, 108, 46, 75, 114, 121, 111, 67, 108, 97, 115, 115, 68,
97, 116, -31, 1, 1, -126, 98, 4]|
|[1, 0, 111, 114, 103, 46, 97, 112, 97, 99, 104, 101, 46, 115, 112, 97,
114, 107, 46, 115, 113, 108, 46, 75, 114, 121, 111, 67, 108, 97, 115, 115, 68,
97, 116, -31, 1, 1, -126, 99, 6]|
+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
```
After the fix, it will be like the below
```
+----------------------------------------------------------------------------------------------------------------------------+
|value
|
+----------------------------------------------------------------------------------------------------------------------------+
|[01 00 6F 72 67 2E 61 70 61 63 68 65 2E 73 70 61 72 6B 2E 73 71 6C 2E 4B
72 79 6F 43 6C 61 73 73 44 61 74 E1 01 01 82 61 02]|
|[01 00 6F 72 67 2E 61 70 61 63 68 65 2E 73 70 61 72 6B 2E 73 71 6C 2E 4B
72 79 6F 43 6C 61 73 73 44 61 74 E1 01 01 82 62 04]|
|[01 00 6F 72 67 2E 61 70 61 63 68 65 2E 73 70 61 72 6B 2E 73 71 6C 2E 4B
72 79 6F 43 6C 61 73 73 44 61 74 E1 01 01 82 63 06]|
+----------------------------------------------------------------------------------------------------------------------------+
```
In addition, do we need to add a new method to decode and then display the
contents?
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/gatorsmile/spark binaryOutput
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/10165.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #10165
----
commit f63c43519b2e8eeab9428397c519de1032e1ae45
Author: gatorsmile <[email protected]>
Date: 2015-12-05T00:50:03Z
Merge remote-tracking branch 'upstream/master' into binaryOutput
commit 8754979da599743112f392250cee5606a3ce8864
Author: gatorsmile <[email protected]>
Date: 2015-12-06T17:44:04Z
Displays the encoded content of the Dataset
commit 5d0d64c76772d8d8d1a164be130d61e0abb50352
Author: gatorsmile <[email protected]>
Date: 2015-12-06T17:44:56Z
Merge remote-tracking branch 'upstream/master' into binaryOutput
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
