[
https://issues.apache.org/jira/browse/IMPALA-7741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16659759#comment-16659759
]
Paul Rogers edited comment on IMPALA-7741 at 10/23/18 2:25 AM:
---------------------------------------------------------------
{{gen_builtins_catalog.py}} generates {{ScalarBuiltins.java}} from information
in {{impala_functions.py}}. This presents a difficulty. {{impala_functions.py}}
lists functions implemented in the backend. However, the FE rewrites multiple
functions out of existence, so no BE implementation is necessary.
{{impala_functions.py}} divides functions into two categories:
{{visible_functions}} and {{invisible_functions}}. The {{visible_functions}}
category already includes some FE-only functions, so we can expand on this idea.
It turns out that {{nvl2(expr, ifNotNull, ifNull)}} is a bit nasty. The two
second arguments can be of any type, but must be the same type. The first
argument can be of any type, not necessarily the same as the other two. This
can lead to an n^2 number of entries in the table.
An easy answer is to leverage the {{NULL}} type for the first argument since we
only care about its nullness, not its type or value. This turns out to work.
That leaves {{decode()}} which also has the n^2 problem, but there seems to be
"cheat" for this one.
Turns out that the code does try to include {{decode()}}, there is code in
{{CaseExpr.initBuiltins()}} that tries to do so. However, it does so as an
operator, which makes the function invisible to the user.
was (Author: paul.rogers):
{{gen_builtins_catalog.py}} generates {{ScalarBuiltins.java}} from information
in {{impala_functions.py}}. This presents a difficulty. {{impala_functions.py}}
lists functions implemented in the backend. However, the FE rewrites multiple
functions out of existence, so no BE implementation is necessary.
{{impala_functions.py}} divides functions into two categories:
{{visible_functions}} and {{invisible_functions}}. The {{visible_functions}}
category already includes some FE-only functions, so we can expand on this idea.
It turns out that {{nvl2(expr, ifNotNull, ifNull)}} is a bit nasty. The two
second arguments can be of any type, but must be the same type. The first
argument can be of any type, not necessarily the same as the other two. This
can lead to an n^2 number of entries in the table.
An easy answer is to leverage the {{NULL}} type for the first argument since we
only care about its nullness, not its type or value. This turns out to work.
That leaves {{decode()}} which also has the n^2 problem, but there seems to be
"cheat" for this one.
> Functions nvl2(), decode(), nullif() not listed in _impala_builtins
> -------------------------------------------------------------------
>
> Key: IMPALA-7741
> URL: https://issues.apache.org/jira/browse/IMPALA-7741
> Project: IMPALA
> Issue Type: Improvement
> Affects Versions: Impala 3.0
> Reporter: Paul Rogers
> Priority: Major
>
> The
> [docs|https://impala.apache.org/docs/build3x/html/topics/impala_show.html]
> for {{SHOW FUNCTIONS}} says that we can use the following to list all
> built-in functions:
> {code:sql}
> show functions in _impala_builtins like '*week*';
> {code}
> However several Impala functions are removed early in the FE planning process
> and thus do not appear in the FE's function table in {{ScalarBuiltins}}:
> {{nvl2()}}, {{decode()}}, and {{nullif()}}. For example:
> {noformat}
> show functions in _impala_builtins like '*decode**'
> +-------------+------------------------------+-------------+---------------+
> | return type | signature | binary type | is persistent |
> +-------------+------------------------------+-------------+---------------+
> | STRING | base64decode(STRING) | BUILTIN | true |
> | STRING | madlib_decode_vector(STRING) | BUILTIN | true |
> +-------------+------------------------------+-------------+---------------+
> {noformat}
> However, since these three are perfectly valid functions, would have expected
> them to appear in the table. How they are processed internally is an
> implementation detail unimportant to the end user.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
