Ah yes, missed the kafka part and just saw the array part. FLINK-19771 definitely was solely in the postgres-specific code.
Dylan From: Jark Wu <imj...@gmail.com> Date: Thursday, November 19, 2020 at 9:12 AM To: Dylan Forciea <dy...@oseberg.io> Cc: Danny Chan <danny0...@apache.org>, Rex Fenley <r...@remind101.com>, Flink ML <user@flink.apache.org> Subject: Re: Filter Null in Array in SQL Connector Hi Dylan, I think Rex encountered another issue, because he is using Kafka with Debezium format. Hi Rex, If you can share the json data and the exception stack, that would be helpful! Besides, you can try to enable 'debezium-json.ignore-parse-errors' option [1] to skip the dirty data. Best, Jark [1]: https://ci.apache.org/projects/flink/flink-docs-release-1.11/dev/table/connectors/formats/debezium.html#debezium-json-ignore-parse-errors On Thu, 19 Nov 2020 at 21:13, Dylan Forciea <dy...@oseberg.io<mailto:dy...@oseberg.io>> wrote: Do you mean that the array contains values that are null, or that the entire array itself is null? If it’s the latter, I have an issue written, along with a PR to fix it that has been pending review [1]. Regards, Dylan Forciea [1] https://issues.apache.org/jira/browse/FLINK-19771 From: Danny Chan <danny0...@apache.org<mailto:danny0...@apache.org>> Date: Thursday, November 19, 2020 at 2:24 AM To: Rex Fenley <r...@remind101.com<mailto:r...@remind101.com>> Cc: Flink ML <user@flink.apache.org<mailto:user@flink.apache.org>> Subject: Re: Filter Null in Array in SQL Connector Hi, Fenley ~ You are right, parsing nulls of ARRAY field is not supported now, i have logged an issue [1] and would fix it soon ~ [1] https://issues.apache.org/jira/browse/FLINK-20234 Rex Fenley <r...@remind101.com<mailto:r...@remind101.com>> 于2020年11月19日周四 下午2:51写道： Hi, I recently discovered some of our data has NULL values arriving in an ARRAY<STRING> column. This column is being consumed by Flink via the Kafka connector Debezium format. We seem to be receiving NullPointerExceptions for when these NULL values in the arrays arrive which restarts the source operator in a loop. Is there any way to not throw or to possibly filter out NULLs in an Array of Strings in Flink? We're somewhat stuck on how to solve this problem, we'd like to be defensive about this on Flink's side. Thanks! (P.S. The exception was not that informative, there may be room for improvement in terms of a richer error message when this happens.) -- Rex Fenley | Software Engineer - Mobile and Backend Remind.com<https://www.remind.com/> | BLOG<http://blog.remind.com/> | FOLLOW US<https://twitter.com/remindhq> | LIKE US<https://www.facebook.com/remindhq>
