Re: Boto3 library send to pyspark

[Sebastian Schere]

Unsubscribe

On Wed, 17 Apr 2019 at 07:43 Gorka Bravo Martinez <
gorka.bravo.marti...@cern.ch> wrote:

> Hi Gourav,
>
> you mean by seting a different python environment while running pyspark?
>
> Cheers, Gorka.
> 
> From: Gourav Sengupta [gourav.sengu...@gmail.com]
> Sent: 17 April 2019 10:06
> To: Gorka Bravo Martinez
> Cc: user@spark.apache.org
> Subject: Re: Boto3 library send to pyspark
>
> Hi,
>
> there is addPyFile, and then there is python environment, try to search
> for using python package managers like canopy and conda.
>
> Regards,
> Gourav
>
> On Wed, Apr 17, 2019 at 8:50 AM Gorka Bravo Martinez <
> gorka.bravo.marti...@cern.ch> wrote:
> Hi all,
>
> I would like to send a boto/boto3 library while running pyspark with yarn
> client mode, how is it possible?
> I am aware sc.addFile() can add a .py file, is it the same for a
> library?
>
> Cheers, Gorka.
>
> -
> To unsubscribe e-mail: user-unsubscr...@spark.apache.org
>
> --
Saludos,
Sebastian

How to have map_from_arrays() in Spark 2.3

[Sebastian Schere]

I have the following Dataframe in Spark 2.3, coming from a JSON file:

root
 |-- ext_attr: array (nullable = true)
 ||-- element: struct (containsNull = true)
 |||-- key: string (nullable = true)
 |||-- value: string (nullable = true)

I would need to convert it to below DataFrame:

root
 |-- ext_attr_map: map (nullable = false)
 ||-- key: string
 ||-- value: string

I know that in Spark 2.4 I'd use map_from_arrays() SQL function.

How can I achieve this in Spark 2.3 ? Please provide a UDF or SQL code.


Thanks,

Sebastian

[UNSUBSCRIBE]

[Sebastian Schere]

UNSUBSCRIBE

unsubscribe

[Sebastian Schere]