[
https://issues.apache.org/jira/browse/FLINK-24924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17444963#comment-17444963
]
lincoln lee commented on FLINK-24924:
-------------------------------------
+1 to keep the current behavior and add a new option.
I'd like to explain more, the error handling is an interesting question in
streaming computing.
If we report errors under any circumstances like traditional databases(not all
of them), users may suffer from endless failover due to some abnormal data,
which is an undesirable behavior for many users. As mentioned by caizhi,
streaming computing is a resident program. Users do not want it to frequently
fails and cannot automatically recover. If it is only a small amount or even
one record of ‘dirty’ data, users can do is to add various filter conditions
(defensive programming) to SQL. If the condition can't simply expressed by SQL,
e.g., validate the date field, then users write their own UDF to tolerate
exceptions. If we let this continue to happen, those built-in functions (which
cannot tolerate exceptions) will be replaced by various versions of
user-defined implementations in real use. This is not our goal to build the
built-in function library.
>From my user observations, few users complained about this exception tolerance
>behavior, but on the contrary, we received more user complaints in the early
>blink's strictly error reporting implementation.
>From my understanding this is because there is a big difference between
>streaming computing and adhoc queries on traditional databases.
First of all, the data processed in stream computing is very different, a
considerable part of it is log data (the data does not follow a strict database
paradigm, and some abnormalities may also occur during transmission).
The second is the resident feature of stream computing, which needs to maintain
a long-term stable running. While the query running on the database is
one-time, result is success or error, the user can adjust the SQL in time
according to the error info to continue to try the query until the expected
result is obtained.
In stream computing, it is difficult for users to fully predict the anomalies
that the task will encounter. This leads the engine to choose a processing
strategy:
1. Always report an error and wait for users to process
2. Adopt a relatively reasonable processing strategy which is acceptable to
users
exception-to-null strategy is a choice, which is indeed not perfect, especially
now flink processes more and more relational data (the data has strict paradigm
guarantees). Some users hope to discover such anomalies in time and obtain
‘strictly correct’ results. This is indeed a scenario that we need to consider.
One solution is to add a global option such as 'sql.exec.strict-mode', default
is false, to keep the default exception-to-null behavior, true to ensure throw
out exception in time.
so it can keep the compatibility and also meet the requirements of strict mode.
> TO_TIMESTAMP and TO_DATE should fail
> ------------------------------------
>
> Key: FLINK-24924
> URL: https://issues.apache.org/jira/browse/FLINK-24924
> Project: Flink
> Issue Type: Sub-task
> Reporter: Francesco Guardiani
> Priority: Major
>
> In a similar fashion to what described
> https://issues.apache.org/jira/browse/FLINK-24385, TO_TIMESTAMP and TO_DATE
> should fail instead of returning {{null}}.
> In particular for these two functions, a failure in parsing could lead to
> very unexpected behavior, for example it could lead to records with null
> rowtime.
> We should change these functions to fail by default when parsing generates an
> error. We can let users handle errors by letting them use TRY_CAST for the
> same functionality:
> {code:sql}
> -- This fails when input is invalid
> TO_TIMESTAMP(input)
> -- Behaves the same as above
> CAST(input AS TIMESTAMP)
> -- This returns null when input is invalid
> TRY_CAST(input AS TIMESTAMP)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
