[
https://issues.apache.org/jira/browse/CALCITE-3080?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16847736#comment-16847736
]
Xu Mingmin commented on CALCITE-3080:
-------------------------------------
Move discussion in github here:
{code}
RelDataType rowDataType(String topicName);
{code}
h4. *[asereda-gs|https://github.com/asereda-gs]* [ 14 days ago
|https://github.com/apache/calcite/pull/1127/files#r283079680]
Member
>From configuration I see that you define one converter per topic. If so, why
>do you need this method ({{rowDataType(String)}}) ?
[!https://avatars3.githubusercontent.com/u/2017944?s=60&v=4|width=28,height=28!|https://github.com/XuMingmin]
h4. *[XuMingmin|https://github.com/XuMingmin]* [ 13 days ago
|https://github.com/apache/calcite/pull/1127/files#r283087334]
Author Member
I don't put row schema in constructor as {{KafkaRowConverter}} is defined as
interface. Usually we use an external metadata system to manage Kafka message
schema, calling {{rowDataType(String topic)}} would be more clear IMO.
[!https://avatars3.githubusercontent.com/u/25229979?s=60&v=4|width=28,height=28!|https://github.com/asereda-gs]
h4. *[asereda-gs|https://github.com/asereda-gs]* [ 10 days ago
|https://github.com/apache/calcite/pull/1127/files#r283913361] •
h4.
Member
I feel like current interface is both one-to-one (between topic and row) and
one-to-many (between topic and {{RelDataType}}). It seems more logical to have
the following:
interface KafkaRowConverter { RelDataType rowDataType(); Object[]
toRow(ConsumerRecord<K, V> message); }
Ie have separate KafkaRowConverter _per topic_.
Another option is to provide {{RelDataType}} as constructor argument to
{{KafkaTable}}.
Pls let me know if you agree with such API.
[!https://avatars3.githubusercontent.com/u/25229979?s=60&v=4|width=28,height=28!|https://github.com/asereda-gs]
h4. *[asereda-gs|https://github.com/asereda-gs]* [ 10 days ago
|https://github.com/apache/calcite/pull/1127/files#r283996688]
Member
Traditionally we have been using single {{Function1<Consumer<K,V>, Object[]>}}
for such transformations.
[!https://avatars3.githubusercontent.com/u/2017944?s=60&v=4|width=28,height=28!|https://github.com/XuMingmin]
h4. *[XuMingmin|https://github.com/XuMingmin]* [ 4 days ago
|https://github.com/apache/calcite/pull/1127/files#r285485710]
Author Member
[@asereda-gs|https://github.com/asereda-gs] sry that I was lost in the thread,
the proposed interface looks good to me. If it's not a block issue, I would
like to close this PR and update it in a new task(CALCITE-3080), to handle
user-specified columns together. In this way, there would be two provided
implementations: 1) to handle user-specified columns, 2) to support external
row metadata;
[!https://avatars3.githubusercontent.com/u/25229979?s=60&v=4|width=28,height=28!|https://github.com/asereda-gs]
h4. *[asereda-gs|https://github.com/asereda-gs]* [ a day ago
|https://github.com/apache/calcite/pull/1127/files#r286996689]
Member
OK if CALCITE-3080 addresses it separately.
> support user-specified row schema in KafkaAdapter
> -------------------------------------------------
>
> Key: CALCITE-3080
> URL: https://issues.apache.org/jira/browse/CALCITE-3080
> Project: Calcite
> Issue Type: Improvement
> Reporter: Xu Mingmin
> Priority: Major
>
> Currently rowSchema is defined by extending `KafkaRowConverter` only. Feature
> in this task would support user-defined row schema when creating a table.
>
> Here's one proposed interface in
> https://github.com/apache/calcite/pull/1127/files#r283913361
> {code}
> interface KafkaRowConverter {
> RelDataType rowDataType();
> Object[] toRow(ConsumerRecord<K, V> message);
> }
> {code}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
