Vinayak Joshi created TOREE-460:
-----------------------------------
Summary: Difference in output between Spark 2.1.0 and Spark 2.1.2+
Key: TOREE-460
URL: https://issues.apache.org/jira/browse/TOREE-460
Project: TOREE
Issue Type: Bug
Components: Kernel
Affects Versions: 0.2.0
Reporter: Vinayak Joshi
Using the current version of Toree (0.2.0) & Scala 2.11 when Scala code such as
this is executed:
```
var v = "abc"
```
against Spark 2.1.0, there is no output generated on the notebook. However,
against Spark 2.1.2 and above, the notebook displays the following lines as
output for the same code:
```
v = abc
Out[2]: abc
```
When used via a kernel-gateway, I could trace the websocket messages and
observed that a couple of extra messages are received by the notebook when the
code is executed against Spark 2.1.2+ :
```
{
"parent_header": {
...
},
"msg_type": "display_data",
"msg_id": "7ee0fe40-3283-45dc-8632-a9f013ae7052",
"content": {
"source": "user",
"data": {
"text/plain": "v = abc"
},
"metadata": {}
},
"header": {
...
},
"channel": "iopub",
"buffers": [],
"metadata": {
"timestamp": "1513154854626"
}
}
```
and
```
{
"parent_header": {
...
},
"msg_type": "execute_result",
"msg_id": "98dae777-c5cf-4ae7-9903-306e1fdf7bc1",
"content": {
"execution_count": 2,
"data": {
"text/plain": "abc"
},
"metadata": {}
},
"header": {
...
},
"channel": "iopub",
"buffers": [],
"metadata": {
"timestamp": "1513154854754"
}
}
```
These messages appear to be getting rendered on the notebook ui.
This is breaking consistency/compatibility for expected outputs when Spark
version changes for notebooks. Also there are users that use some kind of code
cell hiding to prevent credentials etc being displayed in a notebook - this use
case now breaks since the cell output now displays any credential values that
are assigned to variables.
So looks like something's changed between Spark 2.1.0 and later versions that
causes Toree to produce different replies to the same code being executed? Is
it something Toree needs to adapt to?
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
