[jira] [Commented] (FLINK-31828) List field in a POJO data stream results in table program compilation failure
[
https://issues.apache.org/jira/browse/FLINK-31828?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17725311#comment-17725311
]
Aitozi commented on FLINK-31828:
Any sql guys can help verify this fix ?
> List field in a POJO data stream results in table program compilation failure
> -
>
> Key: FLINK-31828
> URL: https://issues.apache.org/jira/browse/FLINK-31828
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Runtime
>Affects Versions: 1.16.1
> Environment: Java 11
> Flink 1.16.1
>Reporter: Vladimir Matveev
>Priority: Major
> Labels: pull-request-available
> Attachments: MainPojo.java, generated-code.txt, stacktrace.txt
>
>
> Suppose I have a POJO class like this:
> {code:java}
> public class Example {
> private String key;
> private List> values;
> // getters, setters, equals+hashCode omitted
> }
> {code}
> When a DataStream with this class is converted to a table, and some
> operations are performed on it, it results in an exception which explicitly
> says that I should file a ticket:
> {noformat}
> Caused by: org.apache.flink.api.common.InvalidProgramException: Table program
> cannot be compiled. This is a bug. Please file an issue.
> {noformat}
> Please find the example Java code and the full stack trace attached.
> From the exception and generated code it seems that Flink is upset with the
> list field being treated as an array - but I cannot have an array type there
> in the real code.
> Also note that if I _don't_ specify the schema explicitly, it then maps the
> {{values}} field to a `RAW('java.util.List', '...')` type, which also does
> not work correctly and fails the job in case of even simplest operations like
> printing.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-31828) List field in a POJO data stream results in table program compilation failure
[
https://issues.apache.org/jira/browse/FLINK-31828?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17717792#comment-17717792
]
Vladimir Matveev commented on FLINK-31828:
--
Cool, thank you very much!
> List field in a POJO data stream results in table program compilation failure
> -
>
> Key: FLINK-31828
> URL: https://issues.apache.org/jira/browse/FLINK-31828
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Runtime
>Affects Versions: 1.16.1
> Environment: Java 11
> Flink 1.16.1
>Reporter: Vladimir Matveev
>Priority: Major
> Labels: pull-request-available
> Attachments: MainPojo.java, generated-code.txt, stacktrace.txt
>
>
> Suppose I have a POJO class like this:
> {code:java}
> public class Example {
> private String key;
> private List> values;
> // getters, setters, equals+hashCode omitted
> }
> {code}
> When a DataStream with this class is converted to a table, and some
> operations are performed on it, it results in an exception which explicitly
> says that I should file a ticket:
> {noformat}
> Caused by: org.apache.flink.api.common.InvalidProgramException: Table program
> cannot be compiled. This is a bug. Please file an issue.
> {noformat}
> Please find the example Java code and the full stack trace attached.
> From the exception and generated code it seems that Flink is upset with the
> list field being treated as an array - but I cannot have an array type there
> in the real code.
> Also note that if I _don't_ specify the schema explicitly, it then maps the
> {{values}} field to a `RAW('java.util.List', '...')` type, which also does
> not work correctly and fails the job in case of even simplest operations like
> printing.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-31828) List field in a POJO data stream results in table program compilation failure
[
https://issues.apache.org/jira/browse/FLINK-31828?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17715451#comment-17715451
]
Aitozi commented on FLINK-31828:
Hi [~netvl] you could also work with this in your case.
{code:java}
final var schema =
Schema.newBuilder()
.column("key", DataTypes.STRING())
.column(
"values",
DataTypes.ARRAY(
DataTypes.MAP(
DataTypes.STRING(),
DataTypes.STRING()))
.bridgedTo(List.class))
.build(); {code}
> List field in a POJO data stream results in table program compilation failure
> -
>
> Key: FLINK-31828
> URL: https://issues.apache.org/jira/browse/FLINK-31828
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Runtime
>Affects Versions: 1.16.1
> Environment: Java 11
> Flink 1.16.1
>Reporter: Vladimir Matveev
>Priority: Major
> Labels: pull-request-available
> Attachments: MainPojo.java, generated-code.txt, stacktrace.txt
>
>
> Suppose I have a POJO class like this:
> {code:java}
> public class Example {
> private String key;
> private List> values;
> // getters, setters, equals+hashCode omitted
> }
> {code}
> When a DataStream with this class is converted to a table, and some
> operations are performed on it, it results in an exception which explicitly
> says that I should file a ticket:
> {noformat}
> Caused by: org.apache.flink.api.common.InvalidProgramException: Table program
> cannot be compiled. This is a bug. Please file an issue.
> {noformat}
> Please find the example Java code and the full stack trace attached.
> From the exception and generated code it seems that Flink is upset with the
> list field being treated as an array - but I cannot have an array type there
> in the real code.
> Also note that if I _don't_ specify the schema explicitly, it then maps the
> {{values}} field to a `RAW('java.util.List', '...')` type, which also does
> not work correctly and fails the job in case of even simplest operations like
> printing.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-31828) List field in a POJO data stream results in table program compilation failure
[
https://issues.apache.org/jira/browse/FLINK-31828?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17715403#comment-17715403
]
Aitozi commented on FLINK-31828:
[~twalthr] could help review this fix :) ? Since I see you help review the last
commit for this part
> List field in a POJO data stream results in table program compilation failure
> -
>
> Key: FLINK-31828
> URL: https://issues.apache.org/jira/browse/FLINK-31828
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Runtime
>Affects Versions: 1.16.1
> Environment: Java 11
> Flink 1.16.1
>Reporter: Vladimir Matveev
>Priority: Major
> Labels: pull-request-available
> Attachments: MainPojo.java, generated-code.txt, stacktrace.txt
>
>
> Suppose I have a POJO class like this:
> {code:java}
> public class Example {
> private String key;
> private List> values;
> // getters, setters, equals+hashCode omitted
> }
> {code}
> When a DataStream with this class is converted to a table, and some
> operations are performed on it, it results in an exception which explicitly
> says that I should file a ticket:
> {noformat}
> Caused by: org.apache.flink.api.common.InvalidProgramException: Table program
> cannot be compiled. This is a bug. Please file an issue.
> {noformat}
> Please find the example Java code and the full stack trace attached.
> From the exception and generated code it seems that Flink is upset with the
> list field being treated as an array - but I cannot have an array type there
> in the real code.
> Also note that if I _don't_ specify the schema explicitly, it then maps the
> {{values}} field to a `RAW('java.util.List', '...')` type, which also does
> not work correctly and fails the job in case of even simplest operations like
> printing.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-31828) List field in a POJO data stream results in table program compilation failure
[
https://issues.apache.org/jira/browse/FLINK-31828?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17715356#comment-17715356
]
Aitozi commented on FLINK-31828:
I have pushed a fix for it. Before this fix the demo will fails with
EOFException when print the raw type (as shown in the pull request)
After this, in this example, user can define the schema as below. Then
{code:java}
final var schema =
Schema.newBuilder()
.column("key", DataTypes.STRING())
.column("values", DataTypes.RAW(java.util.List.class))
.build(); {code}
> List field in a POJO data stream results in table program compilation failure
> -
>
> Key: FLINK-31828
> URL: https://issues.apache.org/jira/browse/FLINK-31828
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Runtime
>Affects Versions: 1.16.1
> Environment: Java 11
> Flink 1.16.1
>Reporter: Vladimir Matveev
>Priority: Major
> Labels: pull-request-available
> Attachments: MainPojo.java, generated-code.txt, stacktrace.txt
>
>
> Suppose I have a POJO class like this:
> {code:java}
> public class Example {
> private String key;
> private List> values;
> // getters, setters, equals+hashCode omitted
> }
> {code}
> When a DataStream with this class is converted to a table, and some
> operations are performed on it, it results in an exception which explicitly
> says that I should file a ticket:
> {noformat}
> Caused by: org.apache.flink.api.common.InvalidProgramException: Table program
> cannot be compiled. This is a bug. Please file an issue.
> {noformat}
> Please find the example Java code and the full stack trace attached.
> From the exception and generated code it seems that Flink is upset with the
> list field being treated as an array - but I cannot have an array type there
> in the real code.
> Also note that if I _don't_ specify the schema explicitly, it then maps the
> {{values}} field to a `RAW('java.util.List', '...')` type, which also does
> not work correctly and fails the job in case of even simplest operations like
> printing.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-31828) List field in a POJO data stream results in table program compilation failure
[
https://issues.apache.org/jira/browse/FLINK-31828?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17715160#comment-17715160
]
Vladimir Matveev commented on FLINK-31828:
--
Hi [~aitozi], thank you for the quick response! Yes, I had the EOF exception
when trying to print the table with RAW columns. Great to know that it will be
fixed :)
> List field in a POJO data stream results in table program compilation failure
> -
>
> Key: FLINK-31828
> URL: https://issues.apache.org/jira/browse/FLINK-31828
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Runtime
>Affects Versions: 1.16.1
> Environment: Java 11
> Flink 1.16.1
>Reporter: Vladimir Matveev
>Priority: Major
> Attachments: MainPojo.java, generated-code.txt, stacktrace.txt
>
>
> Suppose I have a POJO class like this:
> {code:java}
> public class Example {
> private String key;
> private List> values;
> // getters, setters, equals+hashCode omitted
> }
> {code}
> When a DataStream with this class is converted to a table, and some
> operations are performed on it, it results in an exception which explicitly
> says that I should file a ticket:
> {noformat}
> Caused by: org.apache.flink.api.common.InvalidProgramException: Table program
> cannot be compiled. This is a bug. Please file an issue.
> {noformat}
> Please find the example Java code and the full stack trace attached.
> From the exception and generated code it seems that Flink is upset with the
> list field being treated as an array - but I cannot have an array type there
> in the real code.
> Also note that if I _don't_ specify the schema explicitly, it then maps the
> {{values}} field to a `RAW('java.util.List', '...')` type, which also does
> not work correctly and fails the job in case of even simplest operations like
> printing.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-31828) List field in a POJO data stream results in table program compilation failure
[
https://issues.apache.org/jira/browse/FLINK-31828?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17715111#comment-17715111
]
Aitozi commented on FLINK-31828:
Hi [~netvl] Thanks for this detailed bug report.
I have reproduced your problem. And I also spend some time to dig the way to
use RAW type to declare the List type in your case. I found that there's a bug
in the cast rule (using the wrong serializer), so it will fail with EOF
exception as you mentioned (hope it's the same error with you).
I will prepare a PR to solve this bug
> List field in a POJO data stream results in table program compilation failure
> -
>
> Key: FLINK-31828
> URL: https://issues.apache.org/jira/browse/FLINK-31828
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Runtime
>Affects Versions: 1.16.1
> Environment: Java 11
> Flink 1.16.1
>Reporter: Vladimir Matveev
>Priority: Major
> Attachments: MainPojo.java, generated-code.txt, stacktrace.txt
>
>
> Suppose I have a POJO class like this:
> {code:java}
> public class Example {
> private String key;
> private List> values;
> // getters, setters, equals+hashCode omitted
> }
> {code}
> When a DataStream with this class is converted to a table, and some
> operations are performed on it, it results in an exception which explicitly
> says that I should file a ticket:
> {noformat}
> Caused by: org.apache.flink.api.common.InvalidProgramException: Table program
> cannot be compiled. This is a bug. Please file an issue.
> {noformat}
> Please find the example Java code and the full stack trace attached.
> From the exception and generated code it seems that Flink is upset with the
> list field being treated as an array - but I cannot have an array type there
> in the real code.
> Also note that if I _don't_ specify the schema explicitly, it then maps the
> {{values}} field to a `RAW('java.util.List', '...')` type, which also does
> not work correctly and fails the job in case of even simplest operations like
> printing.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
