adisunw opened a new issue, #23303: URL: https://github.com/apache/beam/issues/23303
### What happened? I'm using the Snowflake IO Write and I'm noticing that the fileNameTemplate attribute is not actually being used anywhere. To give more context, I'm using AWS for the external stage here and I have a streaming pipeline that reads from Kafka. Say my external stage bucket is s3://test-bucket: Within `s3://test-bucket` i'm seeing CSV's written out like UUID1.csv.gz, UUID2.csv.gz, etc. However I wanted to create a file name template so that I could partition these files differently within s3://test-bucket. Ideally these files would have a date prefix to them so that they could be stored at `s3://test-bucket/year=YYYY/month=MM/day=DD/` This line of code https://github.com/apache/beam/blob/master/sdks/java/io/snowflake/src/main/java/org/apache/beam/sdk/io/snowflake/SnowflakeIO.java#L1090 hard codes the file name prefix while files are being written to the external stage. The fileNameTemplate attribute (https://github.com/apache/beam/blob/master/sdks/java/io/snowflake/src/main/java/org/apache/beam/sdk/io/snowflake/SnowflakeIO.java#L245) sounded promising at first but it doesn't look like this attribute is being used. ```java SnowflakeIO.<Map<String, String>>write() .withStagingBucketName(options.getStagingBucketName()) .withStorageIntegrationName(options.getStorageIntegrationName()) .withDataSourceConfiguration(dataSourceConfiguration) .withUserDataMapper(getStreamingCsvMapper()) .withSnowPipe(options.getSnowPipe()) .withFileNameTemplate("FOO") // --> this doesn't do anything? .withFlushTimeLimit(Duration.millis(60000 * 10)) .withDebugMode(StreamingLogLevel.INFO) .withFlushRowLimit(1000000) .withShardsNumber(1) .to("output_table")); ``` ### Issue Priority Priority: 2 ### Issue Component Component: io-java-snowflake -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
