Re: How to convert a Dataset to a Dataset?

2022-06-06 Thread Stelios Philippou
", 4, 4L), Data("e", 5, 5L)).toDF > val ds = df.map(_.mkString(",")).as[String] > ds.show > > +-+ > |value| > +-+ > |a,1,1| > |b,2,2| > |c,3,3| > |d,4,4| > |e,5,5| > +-+ > > Regards, > Christophe. > > On 6/4/22 14

Re: How to convert a Dataset to a Dataset?

2022-06-06 Thread Christophe Préaud
s[String] ds.show +-+ |value| +-+ |a,1,1| |b,2,2| |c,3,3| |d,4,4| |e,5,5| +-+ Regards, Christophe. On 6/4/22 14:38, marc nicole wrote: > Hi, > How to convert a Dataset to a Dataset? > What i have tried is: > > List list = dataset.as > <http://dataset.as>(Encode

Re: How to convert a Dataset to a Dataset?

2022-06-04 Thread marc nicole
d DoubleType for example or something else. Basically I want to >>>> remove "+" from all dataset rows and then inferschema. >>>> Here my idea is to filter the rows not equal to "+" for the target >>>> columns (potentially all of them) and then

Re: How to convert a Dataset to a Dataset?

2022-06-04 Thread Enrico Minack
it doesn't work directly this way. You can use a UDF to call .toString on the Row of course, but, again what are you really trying to do? On Sat, Jun 4, 2022 at 7:35 AM marc nicole wrote: Hi,

Re: How to convert a Dataset to a Dataset?

2022-06-04 Thread marc nicole
correct column types. >>> What do you think? >>> >>> Le sam. 4 juin 2022 à 15:56, Sean Owen a écrit : >>> >>>> I don't think you want to do that. You get a string representation of >>>> structured data without the structure, at bes

Re: How to convert a Dataset to a Dataset?

2022-06-04 Thread Enrico Minack
7:35 AM marc nicole wrote: Hi, How to convert a Dataset to a Dataset? What i have tried is: List list = dataset.as <http://dataset.as>(Encoders.STRING()).collectAsList(); Dataset datasetSt = spark.c

Re: How to convert a Dataset to a Dataset?

2022-06-04 Thread marc nicole
tly this way. >>> You can use a UDF to call .toString on the Row of course, but, again >>> what are you really trying to do? >>> >>> On Sat, Jun 4, 2022 at 7:35 AM marc nicole wrote: >>> >>>> Hi, >>>> How to convert a Dataset to

Re: How to convert a Dataset to a Dataset?

2022-06-04 Thread Sean Owen
he reason >> it doesn't work directly this way. >> You can use a UDF to call .toString on the Row of course, but, again >> what are you really trying to do? >> >> On Sat, Jun 4, 2022 at 7:35 AM marc nicole wrote: >> >>> Hi, >>> How to co

Re: How to convert a Dataset to a Dataset?

2022-06-04 Thread marc nicole
ctly this way. > You can use a UDF to call .toString on the Row of course, but, again > what are you really trying to do? > > On Sat, Jun 4, 2022 at 7:35 AM marc nicole wrote: > >> Hi, >> How to convert a Dataset to a Dataset? >> What i have tried is: >> >&g

Re: How to convert a Dataset to a Dataset?

2022-06-04 Thread Sean Owen
Sat, Jun 4, 2022 at 7:35 AM marc nicole wrote: > Hi, > How to convert a Dataset to a Dataset? > What i have tried is: > > List list = dataset.as(Encoders.STRING()).collectAsList(); > Dataset datasetSt = spark.createDataset(list, Encoders.STRING());

How to convert a Dataset to a Dataset?

2022-06-04 Thread marc nicole
Hi, How to convert a Dataset to a Dataset? What i have tried is: List list = dataset.as(Encoders.STRING()).collectAsList(); Dataset datasetSt = spark.createDataset(list, Encoders.STRING()); // But this line raises a org.apache.spark.sql.AnalysisException: Try to map struct... to Tuple1, but