soumilshah1995 opened a new issue, #10499:
URL: https://github.com/apache/hudi/issues/10499
Hello, I'm currently experimenting with the Hudi delta streamer and working
on creating part 12 of the delta streamer playlist. For the next video, my goal
is to cover the Hudi SQL-based transformer and the flattening transformer.
I've encountered a challenge with the flattening transformer. The data
schema in the schema registry is structured as follows:
```
{
"type": "record",
"name": "Order",
"fields": [
{
"name": "order_id",
"type": "string"
},
{
"name": "name",
"type": "string"
},
{
"name": "order_value",
"type": "string"
},
{
"name": "priority",
"type": "string"
},
{
"name": "order_date",
"type": "string"
},
{
"name": "ts",
"type": "string"
},
{
"name": "customer",
"type": {
"type": "record",
"name": "Customer",
"fields": [
{
"name": "customer_id",
"type": "string"
}
]
}
}
]
}
```
Without the flattening transformer, everything works fine using the
following configuration:
```
--transformer-class
org.apache.hudi.utilities.transform.SqlQueryBasedTransformer \
hoodie.streamer.transformer.sql=SELECT * FROM <SRC> a
```
Now, I'm attempting to implement the flattening transformer by following the
guide [here](https://github.com/apache/hudi/pull/4097) and [official
documentation](https://hudi.apache.org/docs/next/transforms/#flattening-transformer).
Here is the configuration I've tried:
```
--transformer-class
org.apache.hudi.utilities.transform.FlatteningTransformer,org.apache.hudi.utilities.transform.SqlQueryBasedTransformer
\
```
And the SQL query variations:
```
hoodie.streamer.transformer.sql=SELECT order_id, name, order_value,
priority, order_date, ts, customer.customer_id FROM <SRC> a
OR
hoodie.streamer.transformer.sql=SELECT a.order_id, a.name, a.order_value,
a.priority, a.order_date, a.ts, a.customer_id FROM <SRC> a
OR
hoodie.streamer.transformer.sql=SELECT a.order_id, a.name, a.order_value,
a.priority, a.order_date, a.ts, a.customer_id FROM <SRC> a
OR
hoodie.streamer.transformer.sql=SELECT order_id, name, order_value,
priority, order_date, ts, customer_id FROM <SRC> a
```
However, it seems like the flattening transformer is not working, or I might
be using it incorrectly. Any assistance or guidance on this matter would be
greatly appreciated. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
