I think along with the storage integration name you also have to specify
the snowpipe to use.
SnowflakeIO.<>write()
.withDataSourceConfiguration(dc)
.to(TABLE_NAME)
.withStagingBucketName(stagingBucketName)
.withStorageIntegrationName(storageIntegrationName)
.withDebugMode(StreamingLogLevel.INFO) // change this to
get more information
.withSnowPipe("pipe")
However looking at your error I think there may be some other issue. It is
mainly related to how you are authenticating for S3 and looks like what
authenticator beam provides does not match with what is required by your
aws account.
What I can suggest is to just write to S3 using beam i/o.
Then configure the snowpipe to read from that S3 bucket into a snowflake
which will work independently from your beam application.
In my case I have almost always just written to S3 and then used snowpipe
to move data from S3 to snowflake and there is a standard way of doing that.
Thanks
Sachin
On Sun, Dec 3, 2023 at 11:33 AM Xinmin <mybeam...@gmail.com> wrote:
> Hello Sachin,
>
> I am trying to write data to Snowflake using SnowflakeIO Connector. As per
> https://docs.snowflake.com/en/user-guide/data-load-s3-config, there are
> three options to configure the access to S3 buckets in Snowflake. 1)
> Storage integration which avoids the need to provide access key and secret;
> 2) IAM role; 3) IAM user which needs us to provide the access key and
> secret. The option of IAM role is deprecated by Snowflake.
>
> As per https://beam.apache.org/documentation/io/built-in/snowflake/, it
> mentions the usage of storage integration.
>
> [image: image.png]
>
> In the meantime, it also mentions that we need to provide both keys, which
> should be related to the option of IAM user.
>
> [image: image.png]
>
> I googled the usage of SnowflakeIO connector and S3 bucket and then found
> examples using Python with both keys (access key and secret key).
>
> https://github.com/dennisbachmann/beam-s3-reader/tree/master
>
> https://python.plainenglish.io/using-apache-beam-and-aws-s3-storage-i-o-transforms-in-python-6cabe2a8d592
>
> https://medium.com/@varunkrishna97/apache-beam-reading-from-writing-to-s3-in-python-sdk-6f2ff6419a14
>
> Since we are not allowed to create an IAM user in AWS due to security
> policy, we would like to go with the option of storage integration, I did
> the test with that and got the error below. Not sure if it was caused by
> the environment or something else.
>
> org.apache.beam.sdk.Pipeline$PipelineExecutionException:
> java.io.IOException: com.amazonaws.services.s3.model.AmazonS3Exception: The
> provided token has expired. (Service: Amazon S3; Status Code: 400; Error
> Code: ExpiredToken; Request ID: 1RCH191011YK9NEJ; S3 Extended Request ID:
> foh3wogVMGT7+kQ+5sbnjPf7WA+9Vbz6zfwmnz8GDChNgkPhf0Wzk9MRBBNl9KqsjHDDB/Jt1r4=;
> Proxy: null), S3 Extended Request ID:
> foh3wogVMGT7+kQ+5sbnjPf7WA+9Vbz6zfwmnz8GDChNgkPhf0Wzk9MRBBNl9KqsjHDDB/Jt1r4=
>
> I would appreciate it if you could share with me the steps that you
> followed to do the test using the SnowflakeIO connector and S3 bucket.
> Thanks.
>
> Regards,
> Xinmin
>
>
>
>
> On Sat, Dec 2, 2023 at 11:46 AM Sachin Mittal <sjmit...@gmail.com> wrote:
>
>> Hi,
>> What are you trying to do ?
>> Write to snowflake or read from it ?
>>
>> Thanks
>> Sachin
>>
>>
>> On Fri, Dec 1, 2023 at 10:43 AM Xinmin <mybeam...@gmail.com> wrote:
>>
>>> Hello Sachin,
>>>
>>> You said that you were able to configure SnowflakeIO with S3 bucket. Can
>>> you please share with me the steps to configure and test it? I would really
>>> appreciate it. Thanks.
>>>
>>>
>>> Regards,
>>> Xinmin
>>>
>>>
>>> On Thu, Oct 26, 2023 at 9:42 AM mybeam <mybeam...@gmail.com> wrote:
>>>
>>>> Hi Sachin,
>>>>
>>>> Thanks for your information.
>>>>
>>>> On Tue, Oct 24, 2023 at 11:02 PM Sachin Mittal <sjmit...@gmail.com>
>>>> wrote:
>>>>
>>>>> I think AWS is supported and I was able to configure snowflake io with
>>>>> S3 buckets.
>>>>>
>>>>>
>>>>>
>>>>> On Wed, 25 Oct 2023 at 9:05 AM, mybeam <mybeam...@gmail.com> wrote:
>>>>>
>>>>>> Hello,
>>>>>>
>>>>>> As per the javadoc below, only the GCP bucket is supported currently
>>>>>> by SnowflakeIO connector? Can you please confirm that AWS and Azure are
>>>>>> supported now? Thanks.
>>>>>>
>>>>>>
>>>>>> https://beam.apache.org/releases/javadoc/2.50.0/index.html?org/apache/beam/sdk/io/snowflake/SnowflakeIO.html
>>>>>> ,
>>>>>>
>>>>>
