Re: Spark Read from Google store and save in AWS s3

2017-01-10 Thread A Shaikh
This should help https://cloud.google.com/hadoop/examples/bigquery-connector-spark-example On 8 January 2017 at 03:49, neil90 wrote: > Here is how you would read from Google Cloud Storage(note you need to > create > a service account key) -> > >

Re: Spark Read from Google store and save in AWS s3

2017-01-07 Thread neil90
Here is how you would read from Google Cloud Storage(note you need to create a service account key) -> os.environ['PYSPARK_SUBMIT_ARGS'] = """--jars /home/neil/Downloads/gcs-connector-latest-hadoop2.jar pyspark-shell""" from pyspark import SparkContext, SparkConf from pyspark.sql import

Re: Spark Read from Google store and save in AWS s3

2017-01-06 Thread Steve Loughran
On 5 Jan 2017, at 20:07, Manohar Reddy > wrote: Hi Steve, Thanks for the reply and below is follow-up help needed from you. Do you mean we can set up two native file system to single sparkcontext ,so then based on urls

RE: Spark Read from Google store and save in AWS s3

2017-01-05 Thread Manohar Reddy
understanding right? Manohar From: Steve Loughran [mailto:ste...@hortonworks.com] Sent: Thursday, January 5, 2017 11:05 PM To: Manohar Reddy Cc: user@spark.apache.org Subject: Re: Spark Read from Google store and save in AWS s3 On 5 Jan 2017, at 09:58, Manohar753 <manohar.re...@happiestminds.

Re: Spark Read from Google store and save in AWS s3

2017-01-05 Thread Steve Loughran
On 5 Jan 2017, at 09:58, Manohar753 > wrote: Hi All, Using spark is interoperability communication between two clouds(Google,AWS) possible. in my use case i need to take Google store as input to spark and do some