[jira] [Commented] (FLINK-15098) CsvTableSink has no properties of a OVERWRITE mode to write, but the exception hint the user use it when a file or directory already exists
[ https://issues.apache.org/jira/browse/FLINK-15098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16989757#comment-16989757 ] Kurt Young commented on FLINK-15098: I think this is the same issue which I was encountered: FLINK-15066 > CsvTableSink has no properties of a OVERWRITE mode to write, but the > exception hint the user use it when a file or directory already exists > --- > > Key: FLINK-15098 > URL: https://issues.apache.org/jira/browse/FLINK-15098 > Project: Flink > Issue Type: Improvement > Components: Table SQL / Client >Affects Versions: 1.10.0 >Reporter: xiaojin.wy >Priority: Major > > When using sql-client to execute a sql , there will be a IOException just > like the content below if the file or directory already exists. But there is > no place for the user to set the overwrite mode. In another word, the hint > tell users to use overwrite mode, but flink has not support it now. > > > The exception: > Caused by: org.apache.flink.runtime.client.JobExecutionException: Cannot > initialize task 'HashAggregate(isMerge=[true], select=[Final_AVG(sum$0, > count$1) AS avg_1]) -> SinkConversionToRow -> Map -> Sink: > CsvTableSink(avg_1)': File or directory already exists. Existing files and > directories are not overwritten in NO_OVERWRITE mode. Use OVERWRITE mode to > overwrite existing files and directories.Caused by: > org.apache.flink.runtime.client.JobExecutionException: Cannot initialize task > 'HashAggregate(isMerge=[true], select=[Final_AVG(sum$0, count$1) AS avg_1]) > -> SinkConversionToRow -> Map -> Sink: CsvTableSink(avg_1)': File or > directory already exists. Existing files and directories are not overwritten > in NO_OVERWRITE mode. Use OVERWRITE mode to overwrite existing files and > directories. at > org.apache.flink.runtime.executiongraph.ExecutionGraphBuilder.buildGraph(ExecutionGraphBuilder.java:216) > at > org.apache.flink.runtime.scheduler.SchedulerBase.createExecutionGraph(SchedulerBase.java:243) > at > org.apache.flink.runtime.scheduler.SchedulerBase.createAndRestoreExecutionGraph(SchedulerBase.java:215) > at > org.apache.flink.runtime.scheduler.SchedulerBase.(SchedulerBase.java:203) > at > org.apache.flink.runtime.scheduler.DefaultScheduler.(DefaultScheduler.java:119) > at > org.apache.flink.runtime.scheduler.DefaultSchedulerFactory.createInstance(DefaultSchedulerFactory.java:104) > at > org.apache.flink.runtime.jobmaster.JobMaster.createScheduler(JobMaster.java:278) > at org.apache.flink.runtime.jobmaster.JobMaster.(JobMaster.java:266) > at > org.apache.flink.runtime.jobmaster.factories.DefaultJobMasterServiceFactory.createJobMasterService(DefaultJobMasterServiceFactory.java:98) > at > org.apache.flink.runtime.jobmaster.factories.DefaultJobMasterServiceFactory.createJobMasterService(DefaultJobMasterServiceFactory.java:40) > at > org.apache.flink.runtime.jobmaster.JobManagerRunnerImpl.(JobManagerRunnerImpl.java:146) > ... 10 moreCaused by: java.io.IOException: File or directory already exists. > Existing files and directories are not overwritten in NO_OVERWRITE mode. Use > OVERWRITE mode to overwrite existing files and directories. at > org.apache.flink.core.fs.FileSystem.initOutPathDistFS(FileSystem.java:937) at > org.apache.flink.api.common.io.FileOutputFormat.initializeGlobal(FileOutputFormat.java:286) > at > org.apache.flink.runtime.jobgraph.InputOutputFormatVertex.initializeOnMaster(InputOutputFormatVertex.java:100) > at > org.apache.flink.runtime.executiongraph.ExecutionGraphBuilder.buildGraph(ExecutionGraphBuilder.java:212) > ... 20 more > > > > The properties that a csvTableSink support now: > connector.type, connector.property-version, connector.path, format.type, > format.property-version, format.field-delimiter -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-15098) CsvTableSink has no properties of a OVERWRITE mode to write, but the exception hint the user use it when a file or directory already exists
[ https://issues.apache.org/jira/browse/FLINK-15098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16989680#comment-16989680 ] xiaojin.wy commented on FLINK-15098: [~ykt836] > CsvTableSink has no properties of a OVERWRITE mode to write, but the > exception hint the user use it when a file or directory already exists > --- > > Key: FLINK-15098 > URL: https://issues.apache.org/jira/browse/FLINK-15098 > Project: Flink > Issue Type: Improvement > Components: Table SQL / Client >Affects Versions: 1.10.0 >Reporter: xiaojin.wy >Priority: Major > > When using sql-client to execute a sql , there will be a IOException just > like the content below if the file or directory already exists. But there is > no place for the user to set the overwrite mode. In another word, the hint > tell users to use overwrite mode, but flink has not support it now. > > > The exception: > Caused by: org.apache.flink.runtime.client.JobExecutionException: Cannot > initialize task 'HashAggregate(isMerge=[true], select=[Final_AVG(sum$0, > count$1) AS avg_1]) -> SinkConversionToRow -> Map -> Sink: > CsvTableSink(avg_1)': File or directory already exists. Existing files and > directories are not overwritten in NO_OVERWRITE mode. Use OVERWRITE mode to > overwrite existing files and directories.Caused by: > org.apache.flink.runtime.client.JobExecutionException: Cannot initialize task > 'HashAggregate(isMerge=[true], select=[Final_AVG(sum$0, count$1) AS avg_1]) > -> SinkConversionToRow -> Map -> Sink: CsvTableSink(avg_1)': File or > directory already exists. Existing files and directories are not overwritten > in NO_OVERWRITE mode. Use OVERWRITE mode to overwrite existing files and > directories. at > org.apache.flink.runtime.executiongraph.ExecutionGraphBuilder.buildGraph(ExecutionGraphBuilder.java:216) > at > org.apache.flink.runtime.scheduler.SchedulerBase.createExecutionGraph(SchedulerBase.java:243) > at > org.apache.flink.runtime.scheduler.SchedulerBase.createAndRestoreExecutionGraph(SchedulerBase.java:215) > at > org.apache.flink.runtime.scheduler.SchedulerBase.(SchedulerBase.java:203) > at > org.apache.flink.runtime.scheduler.DefaultScheduler.(DefaultScheduler.java:119) > at > org.apache.flink.runtime.scheduler.DefaultSchedulerFactory.createInstance(DefaultSchedulerFactory.java:104) > at > org.apache.flink.runtime.jobmaster.JobMaster.createScheduler(JobMaster.java:278) > at org.apache.flink.runtime.jobmaster.JobMaster.(JobMaster.java:266) > at > org.apache.flink.runtime.jobmaster.factories.DefaultJobMasterServiceFactory.createJobMasterService(DefaultJobMasterServiceFactory.java:98) > at > org.apache.flink.runtime.jobmaster.factories.DefaultJobMasterServiceFactory.createJobMasterService(DefaultJobMasterServiceFactory.java:40) > at > org.apache.flink.runtime.jobmaster.JobManagerRunnerImpl.(JobManagerRunnerImpl.java:146) > ... 10 moreCaused by: java.io.IOException: File or directory already exists. > Existing files and directories are not overwritten in NO_OVERWRITE mode. Use > OVERWRITE mode to overwrite existing files and directories. at > org.apache.flink.core.fs.FileSystem.initOutPathDistFS(FileSystem.java:937) at > org.apache.flink.api.common.io.FileOutputFormat.initializeGlobal(FileOutputFormat.java:286) > at > org.apache.flink.runtime.jobgraph.InputOutputFormatVertex.initializeOnMaster(InputOutputFormatVertex.java:100) > at > org.apache.flink.runtime.executiongraph.ExecutionGraphBuilder.buildGraph(ExecutionGraphBuilder.java:212) > ... 20 more > > > > The properties that a csvTableSink support now: > connector.type, connector.property-version, connector.path, format.type, > format.property-version, format.field-delimiter -- This message was sent by Atlassian Jira (v8.3.4#803005)
