[
https://issues.apache.org/jira/browse/CALCITE-3582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16992807#comment-16992807
]
Rui Wang edited comment on CALCITE-3582 at 12/10/19 6:21 PM:
-------------------------------------------------------------
[[email protected]] I remember that it was more like a historic decision.
Quotes from the relevant thread "Flattening was introduced mainly because the
original engine used flat column-oriented storage".
[~zabetak] Indeed the use case is to maintain the final structure in the top
SELECT. In most cases struct does not matter in the middle of the plan. Sounds
like it maybe just because flattener fails to reconstruct the struct. I will
double check it.
was (Author: amaliujia):
[[email protected]] I remember that it was more like a historic decision.
Quotes from the relevant thread "Flattening was introduced mainly because the
original engine used flat column-oriented storage".
[~zabetak] Indeed the use case is to maintain the final structure in the top
SELECT. So sounds like it maybe just because flattener fails to reconstruct the
struct. I will double check it.
> Make RelStructuredTypeFlattener configurable in PlannerImpl
> -----------------------------------------------------------
>
> Key: CALCITE-3582
> URL: https://issues.apache.org/jira/browse/CALCITE-3582
> Project: Calcite
> Issue Type: Task
> Reporter: Rui Wang
> Assignee: Rui Wang
> Priority: Major
>
> There is a use case where users want to mix Beam programming model with Beam
> SQL together to process a dataset. The following is an example of the use
> case:
> dataset.apply(something user defined)
> .apply(SELECT row_a, row_b ...)
> .apply(something user defined)
> As you can see, after the SQL statement is applied, the data structure should
> be preserved for further processing(e.g. users expect to see there are two
> rows, than flattened fields of rows)
> So far as RelStructuredTypeFlattener is enabled by default in PlannerImpl,
> Row/Struct will be flattened. But users are very likely defined an output
> scheme that include row/struct, as that's what they processed in SQL
> statement.
> Make calling RelStructuredTypeFlattener configurable in PlannerImpl to allow
> disabling it when not needed can be a solution.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
