Re: Save data to different S3

[William Li]

I see. Thanks!

From: Steve Loughran <ste...@hortonworks.com<mailto:ste...@hortonworks.com>>
Date: Friday, October 30, 2015 at 12:03 PM
To: William Li <a-...@expedia.com<mailto:a-...@expedia.com>>
Cc: "Zhang, Jingyu" 
<jingyu.zh...@news.com.au<mailto:jingyu.zh...@news.com.au>>, user 
<user@spark.apache.org<mailto:user@spark.apache.org>>
Subject: Re: Save data to different S3


On 30 Oct 2015, at 18:05, William Li 
<a-...@expedia.com<mailto:a-...@expedia.com>> wrote:

Thanks for your response. My secret has a back splash (/) so it didn't work...

that's a recurrent problem with the hadoop/java s3 clients. Keep trying to 
regenerate a secret until you get one that works

Re: Save data to different S3

[Steve Loughran]

On 30 Oct 2015, at 18:05, William Li 
> wrote:

Thanks for your response. My secret has a back splash (/) so it didn’t work…

that's a recurrent problem with the hadoop/java s3 clients. Keep trying to 
regenerate a secret until you get one that works

Re: Save data to different S3

[William Li]

Thanks for your response. My secret has a back splash (/) so it didn't work...

From: "Zhang, Jingyu" 
<jingyu.zh...@news.com.au<mailto:jingyu.zh...@news.com.au>>
Date: Thursday, October 29, 2015 at 5:16 PM
To: William Li <a-...@expedia.com<mailto:a-...@expedia.com>>
Cc: user <user@spark.apache.org<mailto:user@spark.apache.org>>
Subject: Re: Save data to different S3

Try s3://aws_key:aws_secret@bucketName/folderName with your access key and 
secret to save the data.

On 30 October 2015 at 10:55, William Li 
<a-...@expedia.com<mailto:a-...@expedia.com>> wrote:
Hi - I have a simple app running fine with Spark, it reads data from S3 and 
performs calculation.

When reading data from S3, I use hadoopConfiguration.set for both 
fs.s3n.awsAccessKeyId, and the fs.s3n.awsSecretAccessKey to it has permissions 
to load the data from customer sources.

However, after I complete the analysis, how do I save the results (it's a 
org.apache.spark.rdd.RDD[String]) into my own s3 bucket which requires 
different access key and secret? It seems one option is that I could save the 
results as local file to the spark cluster, then create a new SQLContext with 
the different access, then load the data from the local file.

Is there any other options without requiring save and re-load files?


Thanks,

William.


This message and its attachments may contain legally privileged or confidential 
information. It is intended solely for the named addressee. If you are not the 
addressee indicated in this message or responsible for delivery of the message 
to the addressee, you may not copy or deliver this message or its attachments 
to anyone. Rather, you should permanently delete this message and its 
attachments and kindly notify the sender by reply e-mail. Any content of this 
message and its attachments which does not relate to the official business of 
the sending company must be taken not to have been sent or endorsed by that 
company or any of its related entities. No warranty is made that the e-mail or 
attachments are free from computer virus or other defect.

Save data to different S3

[William Li]

Hi - I have a simple app running fine with Spark, it reads data from S3 and 
performs calculation.

When reading data from S3, I use hadoopConfiguration.set for both 
fs.s3n.awsAccessKeyId, and the fs.s3n.awsSecretAccessKey to it has permissions 
to load the data from customer sources.

However, after I complete the analysis, how do I save the results (it's a 
org.apache.spark.rdd.RDD[String]) into my own s3 bucket which requires 
different access key and secret? It seems one option is that I could save the 
results as local file to the spark cluster, then create a new SQLContext with 
the different access, then load the data from the local file.

Is there any other options without requiring save and re-load files?


Thanks,

William.

Re: Save data to different S3

[Zhang, Jingyu]

Try s3://aws_key:aws_secret@bucketName/folderName with your access key and
secret to save the data.

On 30 October 2015 at 10:55, William Li  wrote:

> Hi – I have a simple app running fine with Spark, it reads data from S3
> and performs calculation.
>
> When reading data from S3, I use hadoopConfiguration.set for both
> fs.s3n.awsAccessKeyId, and the fs.s3n.awsSecretAccessKey to it has
> permissions to load the data from customer sources.
>
> However, after I complete the analysis, how do I save the results (it’s a
> org.apache.spark.rdd.RDD[String]) into my own s3 bucket which requires
> different access key and secret? It seems one option is that I could save
> the results as local file to the spark cluster, then create a new
> SQLContext with the different access, then load the data from the local
> file.
>
> Is there any other options without requiring save and re-load files?
>
>
> Thanks,
>
> William.
>

-- 
This message and its attachments may contain legally privileged or 
confidential information. It is intended solely for the named addressee. If 
you are not the addressee indicated in this message or responsible for 
delivery of the message to the addressee, you may not copy or deliver this 
message or its attachments to anyone. Rather, you should permanently delete 
this message and its attachments and kindly notify the sender by reply 
e-mail. Any content of this message and its attachments which does not 
relate to the official business of the sending company must be taken not to 
have been sent or endorsed by that company or any of its related entities. 
No warranty is made that the e-mail or attachments are free from computer 
virus or other defect.