Hi, We are looking at enhancing the getindata connector to add support for a metadata column, that could surface errors from the connector to the SQL level, so the SQL could then act on this error information. See Getindata Flink HTTP connector issue [1].
For the HTTP Connector we are thinking of: create table api_lookup ( `customer_id` STRING NOT NULL, `customer_records_from_backend` STRING, `errorMessage` STRING METADATA FROM `error_string`, `responseCode` INT METADATA FROM `error_code`) WITH ( `connector` = 'rest_lookup', ... ) The subsequent lookup join getting nulls for the values in this error case, but the metadata columns would contain error information that would allow the stream to handle the errors. This sort of functionality allows an enterprise to handle errors more autonomously/ naturally in the flow – rather than jobs failing and logs needing to be analysed. I was thinking that this approach could be useful for JDBC – surfacing he JDBC errors. Also for the Kafka connector we have side outputs[2] for datastream. I wonder if the Kafka connector could surface badly formed events via metadata columns – allowing for the flow itself to manage errors in SQL I do wonder whether the errorMessage and responseCode could be baked into Flink as the way to pass this information into the stream. For now, we will implement in the HTTP connector. What do people think of this idea to enhance SQL to be “error aware”? Metadata columns seem a good way to do this, Flink SQL support for try catch could be another approach. Has SQL try catch in Flink ever been looked into? Kind regards, David. [1] https://github.com/getindata/flink-http-connector/issues/154 [2] https://nightlies.apache.org/flink/flink-docs-release-2.0/docs/dev/datastream/side_output/ Unless otherwise stated above: IBM United Kingdom Limited Registered in England and Wales with number 741598 Registered office: Building C, IBM Hursley Office, Hursley Park Road, Winchester, Hampshire SO21 2JN
