Re: SparkR + binary type + how to get value
from the second image it looks like there is protocol mismatch. I’d check if
the SparkR package running there on Livy machine matches the Spark java release.
But in any case this seems more an issue with Livy config. I’d suggest checking
with the community there:
From: Thijs Haarhuis
Sent: Tuesday, February 19, 2019 5:28 AM
To: Felix Cheung; user@spark.apache.org
Subject: Re: SparkR + binary type + how to get value
Hi Felix,
Thanks. I got it working now by using the unlist function.
I have another question, maybe you can help me with, since I did see your
naming popping up regarding the spark.lapply function.
I am using Apache Livy and am having troubles using this function, I even
reported a jira ticket for it at:
https://jira.apache.org/jira/browse/LIVY-558
When I call the spark.lapply function it reports that SparkR is not initialized.
I have looked into the spark.lapply function and it seems there is no spark
context.
Any idea how I can debug this?
I hope you can help.
Regards,
Thijs
From: Felix Cheung
Sent: Sunday, February 17, 2019 7:18 PM
To: Thijs Haarhuis; user@spark.apache.org
Subject: Re: SparkR + binary type + how to get value
A byte buffer in R is the raw vector type, so seems like it is working as
expected. What do you have in the raw byte? You could convert into other types
or access individual byte directly...
https://stat.ethz.ch/R-manual/R-devel/library/base/html/raw.html
From: Thijs Haarhuis
Sent: Thursday, February 14, 2019 4:01 AM
To: Felix Cheung; user@spark.apache.org
Subject: Re: SparkR + binary type + how to get value
Hi Felix,
Sure..
I have the following code:
printSchema(results)
cat("\n\n\n")
firstRow <- first(results)
value <- firstRow$value
cat(paste0("Value Type: '",typeof(value),"'\n\n\n"))
cat(paste0("Value: '",value,"'\n\n\n"))
results is a Spark Data Frame here.
When I run this code the following is printed to console:
[cid:04497e3e-7983-488a-8516-5d2349778f03]
You can there is only a single column in this sdf of type binary
when I collect this value and print the type it prints it is a list.
Any idea how to get the actual value, or how to process the individual bytes?
Thanks
Thijs
From: Felix Cheung
Sent: Thursday, February 14, 2019 5:31 AM
To: Thijs Haarhuis; user@spark.apache.org
Subject: Re: SparkR + binary type + how to get value
Please share your code
From: Thijs Haarhuis
Sent: Wednesday, February 13, 2019 6:09 AM
To: user@spark.apache.org
Subject: SparkR + binary type + how to get value
Hi all,
Does anybody have any experience in accessing the data from a column which has
a binary type in a Spark Data Frame in R?
I have a Spark Data Frame which has a column which is of a binary type. I want
to access this data and process it.
In my case I collect the spark data frame to a R data frame and access the
first row.
When I print this row to the console it does print all the hex values correctly.
However when I access the column it prints it is a list of 1 …when I print the
type of the child element..it again prints it is a list.
I expected this value to be of a raw type.
Anybody has some experience with this?
Thanks
Thijs
Re: SparkR + binary type + how to get value
Hi Felix,
Thanks. I got it working now by using the unlist function.
I have another question, maybe you can help me with, since I did see your
naming popping up regarding the spark.lapply function.
I am using Apache Livy and am having troubles using this function, I even
reported a jira ticket for it at:
https://jira.apache.org/jira/browse/LIVY-558
When I call the spark.lapply function it reports that SparkR is not initialized.
I have looked into the spark.lapply function and it seems there is no spark
context.
Any idea how I can debug this?
I hope you can help.
Regards,
Thijs
From: Felix Cheung
Sent: Sunday, February 17, 2019 7:18 PM
To: Thijs Haarhuis; user@spark.apache.org
Subject: Re: SparkR + binary type + how to get value
A byte buffer in R is the raw vector type, so seems like it is working as
expected. What do you have in the raw byte? You could convert into other types
or access individual byte directly...
https://stat.ethz.ch/R-manual/R-devel/library/base/html/raw.html
From: Thijs Haarhuis
Sent: Thursday, February 14, 2019 4:01 AM
To: Felix Cheung; user@spark.apache.org
Subject: Re: SparkR + binary type + how to get value
Hi Felix,
Sure..
I have the following code:
printSchema(results)
cat("\n\n\n")
firstRow <- first(results)
value <- firstRow$value
cat(paste0("Value Type: '",typeof(value),"'\n\n\n"))
cat(paste0("Value: '",value,"'\n\n\n"))
results is a Spark Data Frame here.
When I run this code the following is printed to console:
[cid:04497e3e-7983-488a-8516-5d2349778f03]
You can there is only a single column in this sdf of type binary
when I collect this value and print the type it prints it is a list.
Any idea how to get the actual value, or how to process the individual bytes?
Thanks
Thijs
From: Felix Cheung
Sent: Thursday, February 14, 2019 5:31 AM
To: Thijs Haarhuis; user@spark.apache.org
Subject: Re: SparkR + binary type + how to get value
Please share your code
From: Thijs Haarhuis
Sent: Wednesday, February 13, 2019 6:09 AM
To: user@spark.apache.org
Subject: SparkR + binary type + how to get value
Hi all,
Does anybody have any experience in accessing the data from a column which has
a binary type in a Spark Data Frame in R?
I have a Spark Data Frame which has a column which is of a binary type. I want
to access this data and process it.
In my case I collect the spark data frame to a R data frame and access the
first row.
When I print this row to the console it does print all the hex values correctly.
However when I access the column it prints it is a list of 1 …when I print the
type of the child element..it again prints it is a list.
I expected this value to be of a raw type.
Anybody has some experience with this?
Thanks
Thijs
Re: SparkR + binary type + how to get value
A byte buffer in R is the raw vector type, so seems like it is working as
expected. What do you have in the raw byte? You could convert into other types
or access individual byte directly...
https://stat.ethz.ch/R-manual/R-devel/library/base/html/raw.html
From: Thijs Haarhuis
Sent: Thursday, February 14, 2019 4:01 AM
To: Felix Cheung; user@spark.apache.org
Subject: Re: SparkR + binary type + how to get value
Hi Felix,
Sure..
I have the following code:
printSchema(results)
cat("\n\n\n")
firstRow <- first(results)
value <- firstRow$value
cat(paste0("Value Type: '",typeof(value),"'\n\n\n"))
cat(paste0("Value: '",value,"'\n\n\n"))
results is a Spark Data Frame here.
When I run this code the following is printed to console:
[cid:04497e3e-7983-488a-8516-5d2349778f03]
You can there is only a single column in this sdf of type binary
when I collect this value and print the type it prints it is a list.
Any idea how to get the actual value, or how to process the individual bytes?
Thanks
Thijs
From: Felix Cheung
Sent: Thursday, February 14, 2019 5:31 AM
To: Thijs Haarhuis; user@spark.apache.org
Subject: Re: SparkR + binary type + how to get value
Please share your code
From: Thijs Haarhuis
Sent: Wednesday, February 13, 2019 6:09 AM
To: user@spark.apache.org
Subject: SparkR + binary type + how to get value
Hi all,
Does anybody have any experience in accessing the data from a column which has
a binary type in a Spark Data Frame in R?
I have a Spark Data Frame which has a column which is of a binary type. I want
to access this data and process it.
In my case I collect the spark data frame to a R data frame and access the
first row.
When I print this row to the console it does print all the hex values correctly.
However when I access the column it prints it is a list of 1 …when I print the
type of the child element..it again prints it is a list.
I expected this value to be of a raw type.
Anybody has some experience with this?
Thanks
Thijs
Re: SparkR + binary type + how to get value
Hi Felix,
Sure..
I have the following code:
printSchema(results)
cat("\n\n\n")
firstRow <- first(results)
value <- firstRow$value
cat(paste0("Value Type: '",typeof(value),"'\n\n\n"))
cat(paste0("Value: '",value,"'\n\n\n"))
results is a Spark Data Frame here.
When I run this code the following is printed to console:
[cid:04497e3e-7983-488a-8516-5d2349778f03]
You can there is only a single column in this sdf of type binary
when I collect this value and print the type it prints it is a list.
Any idea how to get the actual value, or how to process the individual bytes?
Thanks
Thijs
From: Felix Cheung
Sent: Thursday, February 14, 2019 5:31 AM
To: Thijs Haarhuis; user@spark.apache.org
Subject: Re: SparkR + binary type + how to get value
Please share your code
From: Thijs Haarhuis
Sent: Wednesday, February 13, 2019 6:09 AM
To: user@spark.apache.org
Subject: SparkR + binary type + how to get value
Hi all,
Does anybody have any experience in accessing the data from a column which has
a binary type in a Spark Data Frame in R?
I have a Spark Data Frame which has a column which is of a binary type. I want
to access this data and process it.
In my case I collect the spark data frame to a R data frame and access the
first row.
When I print this row to the console it does print all the hex values correctly.
However when I access the column it prints it is a list of 1 …when I print the
type of the child element..it again prints it is a list.
I expected this value to be of a raw type.
Anybody has some experience with this?
Thanks
Thijs
Re: SparkR + binary type + how to get value
Please share your code From: Thijs Haarhuis Sent: Wednesday, February 13, 2019 6:09 AM To: user@spark.apache.org Subject: SparkR + binary type + how to get value Hi all, Does anybody have any experience in accessing the data from a column which has a binary type in a Spark Data Frame in R? I have a Spark Data Frame which has a column which is of a binary type. I want to access this data and process it. In my case I collect the spark data frame to a R data frame and access the first row. When I print this row to the console it does print all the hex values correctly. However when I access the column it prints it is a list of 1 …when I print the type of the child element..it again prints it is a list. I expected this value to be of a raw type. Anybody has some experience with this? Thanks Thijs
SparkR + binary type + how to get value
Hi all, Does anybody have any experience in accessing the data from a column which has a binary type in a Spark Data Frame in R? I have a Spark Data Frame which has a column which is of a binary type. I want to access this data and process it. In my case I collect the spark data frame to a R data frame and access the first row. When I print this row to the console it does print all the hex values correctly. However when I access the column it prints it is a list of 1 ...when I print the type of the child element..it again prints it is a list. I expected this value to be of a raw type. Anybody has some experience with this? Thanks Thijs
