[jira] [Commented] (FLINK-22281) flink sql consumer kakfa canal-json message then sum(amount)
[
https://issues.apache.org/jira/browse/FLINK-22281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17321917#comment-17321917
]
xx chai commented on FLINK-22281:
-
Yes, I have added this parameter correctly
> flink sql consumer kakfa canal-json message then sum(amount)
> --
>
> Key: FLINK-22281
> URL: https://issues.apache.org/jira/browse/FLINK-22281
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / API
>Affects Versions: 1.12.0
> Environment: flink 1.12 local
>Reporter: xx chai
>Priority: Major
> Attachments: screenshot-1.png
>
>
> I use flink sql to consumer kafka canal-json message the sql is
> CREATE TABLE kafka_mall_order_info (
> id int,
> amount double,
>PRIMARY KEY ( id) NOT ENFORCED
>) WITH (
> 'connector' = 'kafka',
> 'topic' = 'topic_yx-dc-3-102_3306',
> 'properties.bootstrap.servers' = 'localhost:9092',
> 'properties.group.id' = 'kafka_to_hive',
> 'scan.startup.mode' = 'latest-offset',
> 'format' = 'canal-json');
> create table t2 (amount double) with ('connector' = 'print');
>
>
> insert into t2 select sum(amount) from kafka_mall_order_info ;
> but the result is not i think
> the result in image
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-22281) flink sql consumer kakfa canal-json message then sum(amount)
[
https://issues.apache.org/jira/browse/FLINK-22281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17321878#comment-17321878
]
xx chai commented on FLINK-22281:
-
thanks [~jark] I solve the question .
> flink sql consumer kakfa canal-json message then sum(amount)
> --
>
> Key: FLINK-22281
> URL: https://issues.apache.org/jira/browse/FLINK-22281
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / API
>Affects Versions: 1.12.0
> Environment: flink 1.12 local
>Reporter: xx chai
>Priority: Major
> Attachments: screenshot-1.png
>
>
> I use flink sql to consumer kafka canal-json message the sql is
> CREATE TABLE kafka_mall_order_info (
> id int,
> amount double,
>PRIMARY KEY ( id) NOT ENFORCED
>) WITH (
> 'connector' = 'kafka',
> 'topic' = 'topic_yx-dc-3-102_3306',
> 'properties.bootstrap.servers' = 'localhost:9092',
> 'properties.group.id' = 'kafka_to_hive',
> 'scan.startup.mode' = 'latest-offset',
> 'format' = 'canal-json');
> create table t2 (amount double) with ('connector' = 'print');
>
>
> insert into t2 select sum(amount) from kafka_mall_order_info ;
> but the result is not i think
> the result in image
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-22281) flink sql consumer kakfa canal-json message then sum(amount)
[
https://issues.apache.org/jira/browse/FLINK-22281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17321856#comment-17321856
]
xx chai commented on FLINK-22281:
-
i add the Parameter
streamTableEnvironment.getConfig().addJobParameter("table.exec.source.cdc-events-duplicate","true");
but the result is not change
> flink sql consumer kakfa canal-json message then sum(amount)
> --
>
> Key: FLINK-22281
> URL: https://issues.apache.org/jira/browse/FLINK-22281
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / API
>Affects Versions: 1.12.0
> Environment: flink 1.12 local
>Reporter: xx chai
>Priority: Major
> Attachments: screenshot-1.png
>
>
> I use flink sql to consumer kafka canal-json message the sql is
> CREATE TABLE kafka_mall_order_info (
> id int,
> amount double,
>PRIMARY KEY ( id) NOT ENFORCED
>) WITH (
> 'connector' = 'kafka',
> 'topic' = 'topic_yx-dc-3-102_3306',
> 'properties.bootstrap.servers' = 'localhost:9092',
> 'properties.group.id' = 'kafka_to_hive',
> 'scan.startup.mode' = 'latest-offset',
> 'format' = 'canal-json');
> create table t2 (amount double) with ('connector' = 'print');
>
>
> insert into t2 select sum(amount) from kafka_mall_order_info ;
> but the result is not i think
> the result in image
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-22281) flink sql consumer kakfa canal-json message then sum(amount)
[
https://issues.apache.org/jira/browse/FLINK-22281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
xx chai updated FLINK-22281:
Description:
I use flink sql to consumer kafka canal-json message the sql is
CREATE TABLE kafka_mall_order_info (
id int,
amount double,
PRIMARY KEY ( id) NOT ENFORCED
) WITH (
'connector' = 'kafka',
'topic' = 'topic_yx-dc-3-102_3306',
'properties.bootstrap.servers' = 'localhost:9092',
'properties.group.id' = 'kafka_to_hive',
'scan.startup.mode' = 'latest-offset',
'format' = 'canal-json');
create table t2 (amount double) with ('connector' = 'print');
insert into t2 select sum(amount) from kafka_mall_order_info ;
but the result is not i think
the result in image
was:
I use flink sql to consumer kafka canal-json message the sql is
CREATE TABLE kafka_mall_order_info (
id int,
amount double,
PRIMARY KEY ( id) NOT ENFORCED
) WITH (
'connector' = 'kafka',
'topic' = 'topic_yx-dc-3-102_3306',
'properties.bootstrap.servers' =
'192.168.3.100:9092,192.168.3.101:9092,192.168.3.102:9092',
'properties.group.id' = 'kafka_to_hive',
'scan.startup.mode' = 'latest-offset',
'format' = 'canal-json');
create table t2 (amount double) with ('connector' = 'print');
insert into t2 select sum(amount) from kafka_mall_order_info ;
but the result is not i think
the result in image
> flink sql consumer kakfa canal-json message then sum(amount)
> --
>
> Key: FLINK-22281
> URL: https://issues.apache.org/jira/browse/FLINK-22281
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / API
>Affects Versions: 1.12.0
> Environment: flink 1.12 local
>Reporter: xx chai
>Priority: Major
> Attachments: screenshot-1.png
>
>
> I use flink sql to consumer kafka canal-json message the sql is
> CREATE TABLE kafka_mall_order_info (
> id int,
> amount double,
>PRIMARY KEY ( id) NOT ENFORCED
>) WITH (
> 'connector' = 'kafka',
> 'topic' = 'topic_yx-dc-3-102_3306',
> 'properties.bootstrap.servers' = 'localhost:9092',
> 'properties.group.id' = 'kafka_to_hive',
> 'scan.startup.mode' = 'latest-offset',
> 'format' = 'canal-json');
> create table t2 (amount double) with ('connector' = 'print');
>
>
> insert into t2 select sum(amount) from kafka_mall_order_info ;
> but the result is not i think
> the result in image
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-22281) flink sql consumer kakfa canal-json message then sum(amount)
[
https://issues.apache.org/jira/browse/FLINK-22281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
xx chai updated FLINK-22281:
Description:
I use flink sql to consumer kafka canal-json message the sql is
CREATE TABLE kafka_mall_order_info (
id int,
amount double,
PRIMARY KEY ( id) NOT ENFORCED
) WITH (
'connector' = 'kafka',
'topic' = 'topic_yx-dc-3-102_3306',
'properties.bootstrap.servers' =
'192.168.3.100:9092,192.168.3.101:9092,192.168.3.102:9092',
'properties.group.id' = 'kafka_to_hive',
'scan.startup.mode' = 'latest-offset',
'format' = 'canal-json');
create table t2 (amount double) with ('connector' = 'print');
insert into t2 select sum(amount) from kafka_mall_order_info ;
but the result is not i think
the result in image
was:
I use flink sql to consumer kafka canal-json message the sql is
CREATE TABLE kafka_mall_order_info (
id int,
amount double,
PRIMARY KEY ( id) NOT ENFORCED
) WITH (
'connector' = 'kafka',
'topic' = 'topic_yx-dc-3-102_3306',
'properties.bootstrap.servers' =
'192.168.3.100:9092,192.168.3.101:9092,192.168.3.102:9092',
'properties.group.id' = 'kafka_to_hive',
'scan.startup.mode' = 'latest-offset',
'format' = 'canal-json');
create table t2 (amount double) with ('connector' = 'print');
insert into t2 select sum(amount) from kafka_mall_order_info ;
but the result is not i think
> flink sql consumer kakfa canal-json message then sum(amount)
> --
>
> Key: FLINK-22281
> URL: https://issues.apache.org/jira/browse/FLINK-22281
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / API
>Affects Versions: 1.12.0
> Environment: flink 1.12 local
>Reporter: xx chai
>Priority: Major
> Attachments: screenshot-1.png
>
>
> I use flink sql to consumer kafka canal-json message the sql is
> CREATE TABLE kafka_mall_order_info (
> id int,
> amount double,
>PRIMARY KEY ( id) NOT ENFORCED
>) WITH (
> 'connector' = 'kafka',
> 'topic' = 'topic_yx-dc-3-102_3306',
> 'properties.bootstrap.servers' =
> '192.168.3.100:9092,192.168.3.101:9092,192.168.3.102:9092',
> 'properties.group.id' = 'kafka_to_hive',
> 'scan.startup.mode' = 'latest-offset',
> 'format' = 'canal-json');
> create table t2 (amount double) with ('connector' = 'print');
>
>
> insert into t2 select sum(amount) from kafka_mall_order_info ;
> but the result is not i think
> the result in image
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-22281) flink sql consumer kakfa canal-json message then sum(amount)
[
https://issues.apache.org/jira/browse/FLINK-22281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
xx chai updated FLINK-22281:
Attachment: screenshot-1.png
> flink sql consumer kakfa canal-json message then sum(amount)
> --
>
> Key: FLINK-22281
> URL: https://issues.apache.org/jira/browse/FLINK-22281
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / API
>Affects Versions: 1.12.0
> Environment: flink 1.12 local
>Reporter: xx chai
>Priority: Major
> Attachments: screenshot-1.png
>
>
> I use flink sql to consumer kafka canal-json message the sql is
> CREATE TABLE kafka_mall_order_info (
> id int,
> amount double,
>PRIMARY KEY ( id) NOT ENFORCED
>) WITH (
> 'connector' = 'kafka',
> 'topic' = 'topic_yx-dc-3-102_3306',
> 'properties.bootstrap.servers' =
> '192.168.3.100:9092,192.168.3.101:9092,192.168.3.102:9092',
> 'properties.group.id' = 'kafka_to_hive',
> 'scan.startup.mode' = 'latest-offset',
> 'format' = 'canal-json');
> create table t2 (amount double) with ('connector' = 'print');
>
>
> insert into t2 select sum(amount) from kafka_mall_order_info ;
> but the result is not i think
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-22281) flink sql consumer kakfa canal-json message then sum(amount)
[
https://issues.apache.org/jira/browse/FLINK-22281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
xx chai updated FLINK-22281:
Description:
I use flink sql to consumer kafka canal-json message the sql is
CREATE TABLE kafka_mall_order_info (
id int,
amount double,
PRIMARY KEY ( id) NOT ENFORCED
) WITH (
'connector' = 'kafka',
'topic' = 'topic_yx-dc-3-102_3306',
'properties.bootstrap.servers' =
'192.168.3.100:9092,192.168.3.101:9092,192.168.3.102:9092',
'properties.group.id' = 'kafka_to_hive',
'scan.startup.mode' = 'latest-offset',
'format' = 'canal-json');
create table t2 (amount double) with ('connector' = 'print');
insert into t2 select sum(amount) from kafka_mall_order_info ;
but the result is not i think
was:I use flink sql to consumer kafka canal-json message
Environment: flink 1.12 local
> flink sql consumer kakfa canal-json message then sum(amount)
> --
>
> Key: FLINK-22281
> URL: https://issues.apache.org/jira/browse/FLINK-22281
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / API
>Affects Versions: 1.12.0
> Environment: flink 1.12 local
>Reporter: xx chai
>Priority: Major
>
> I use flink sql to consumer kafka canal-json message the sql is
> CREATE TABLE kafka_mall_order_info (
> id int,
> amount double,
>PRIMARY KEY ( id) NOT ENFORCED
>) WITH (
> 'connector' = 'kafka',
> 'topic' = 'topic_yx-dc-3-102_3306',
> 'properties.bootstrap.servers' =
> '192.168.3.100:9092,192.168.3.101:9092,192.168.3.102:9092',
> 'properties.group.id' = 'kafka_to_hive',
> 'scan.startup.mode' = 'latest-offset',
> 'format' = 'canal-json');
> create table t2 (amount double) with ('connector' = 'print');
>
>
> insert into t2 select sum(amount) from kafka_mall_order_info ;
> but the result is not i think
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-22281) flink sql consumer kakfa canal-json message then sum(amount)
[ https://issues.apache.org/jira/browse/FLINK-22281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] xx chai updated FLINK-22281: Component/s: Table SQL / API Affects Version/s: 1.12.0 Description: I use flink sql to consumer kafka canal-json message Issue Type: Bug (was: Improvement) Summary: flink sql consumer kakfa canal-json message then sum(amount)(was: flink sql) > flink sql consumer kakfa canal-json message then sum(amount) > -- > > Key: FLINK-22281 > URL: https://issues.apache.org/jira/browse/FLINK-22281 > Project: Flink > Issue Type: Bug > Components: Table SQL / API >Affects Versions: 1.12.0 >Reporter: xx chai >Priority: Major > > I use flink sql to consumer kafka canal-json message -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-22281) flink sql
xx chai created FLINK-22281: --- Summary: flink sql Key: FLINK-22281 URL: https://issues.apache.org/jira/browse/FLINK-22281 Project: Flink Issue Type: Improvement Reporter: xx chai -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (FLINK-20870) FlinkKafkaSink
[
https://issues.apache.org/jira/browse/FLINK-20870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17260938#comment-17260938
]
xx chai edited comment on FLINK-20870 at 1/8/21, 6:59 AM:
--
!1610089153(1).png!
was (Author: chaixiaoxue):
!1610072517(1).jpg!
> FlinkKafkaSink
> --
>
> Key: FLINK-20870
> URL: https://issues.apache.org/jira/browse/FLINK-20870
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kafka
>Affects Versions: 1.12.0
> Environment: flink :1.12.0
> kafka 2.2.1
>Reporter: xx chai
>Priority: Major
> Attachments: 1610089153(1).png
>
>
> I consum from kafka sink to kafka,Then I split the message into then pieces.I
> guess the ten message should in one transaction. When the fifth message is
> sink kafka ,I throw a exception.but the first four are already in kafka.
> I set some parameters :
>properties.setProperty("transactional.id", "cxx");
> properties.setProperty("ack", "all");
> properties.put("enable.idempotence",true);
> properties.put("max.in.flight.requests.per.connection",5);
> properties.put("retries", 2);
> properties.setProperty("client.id", "producer-syn-2");
> properties.put("isolation.level","read_committed");
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-20870) FlinkKafkaSink
[
https://issues.apache.org/jira/browse/FLINK-20870?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
xx chai updated FLINK-20870:
Attachment: 1610089153(1).png
> FlinkKafkaSink
> --
>
> Key: FLINK-20870
> URL: https://issues.apache.org/jira/browse/FLINK-20870
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kafka
>Affects Versions: 1.12.0
> Environment: flink :1.12.0
> kafka 2.2.1
>Reporter: xx chai
>Priority: Major
> Attachments: 1610089153(1).png
>
>
> I consum from kafka sink to kafka,Then I split the message into then pieces.I
> guess the ten message should in one transaction. When the fifth message is
> sink kafka ,I throw a exception.but the first four are already in kafka.
> I set some parameters :
>properties.setProperty("transactional.id", "cxx");
> properties.setProperty("ack", "all");
> properties.put("enable.idempotence",true);
> properties.put("max.in.flight.requests.per.connection",5);
> properties.put("retries", 2);
> properties.setProperty("client.id", "producer-syn-2");
> properties.put("isolation.level","read_committed");
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-20870) FlinkKafkaSink
[
https://issues.apache.org/jira/browse/FLINK-20870?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
xx chai updated FLINK-20870:
Attachment: (was: 1610072517(1).jpg)
> FlinkKafkaSink
> --
>
> Key: FLINK-20870
> URL: https://issues.apache.org/jira/browse/FLINK-20870
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kafka
>Affects Versions: 1.12.0
> Environment: flink :1.12.0
> kafka 2.2.1
>Reporter: xx chai
>Priority: Major
>
> I consum from kafka sink to kafka,Then I split the message into then pieces.I
> guess the ten message should in one transaction. When the fifth message is
> sink kafka ,I throw a exception.but the first four are already in kafka.
> I set some parameters :
>properties.setProperty("transactional.id", "cxx");
> properties.setProperty("ack", "all");
> properties.put("enable.idempotence",true);
> properties.put("max.in.flight.requests.per.connection",5);
> properties.put("retries", 2);
> properties.setProperty("client.id", "producer-syn-2");
> properties.put("isolation.level","read_committed");
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-20870) FlinkKafkaSink
[
https://issues.apache.org/jira/browse/FLINK-20870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17260939#comment-17260939
]
xx chai commented on FLINK-20870:
-
this is my program
> FlinkKafkaSink
> --
>
> Key: FLINK-20870
> URL: https://issues.apache.org/jira/browse/FLINK-20870
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kafka
>Affects Versions: 1.12.0
> Environment: flink :1.12.0
> kafka 2.2.1
>Reporter: xx chai
>Priority: Major
> Attachments: 1610072517(1).jpg
>
>
> I consum from kafka sink to kafka,Then I split the message into then pieces.I
> guess the ten message should in one transaction. When the fifth message is
> sink kafka ,I throw a exception.but the first four are already in kafka.
> I set some parameters :
>properties.setProperty("transactional.id", "cxx");
> properties.setProperty("ack", "all");
> properties.put("enable.idempotence",true);
> properties.put("max.in.flight.requests.per.connection",5);
> properties.put("retries", 2);
> properties.setProperty("client.id", "producer-syn-2");
> properties.put("isolation.level","read_committed");
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-20870) FlinkKafkaSink
[
https://issues.apache.org/jira/browse/FLINK-20870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17260938#comment-17260938
]
xx chai commented on FLINK-20870:
-
!1610072517(1).jpg!
> FlinkKafkaSink
> --
>
> Key: FLINK-20870
> URL: https://issues.apache.org/jira/browse/FLINK-20870
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kafka
>Affects Versions: 1.12.0
> Environment: flink :1.12.0
> kafka 2.2.1
>Reporter: xx chai
>Priority: Major
> Attachments: 1610072517(1).jpg
>
>
> I consum from kafka sink to kafka,Then I split the message into then pieces.I
> guess the ten message should in one transaction. When the fifth message is
> sink kafka ,I throw a exception.but the first four are already in kafka.
> I set some parameters :
>properties.setProperty("transactional.id", "cxx");
> properties.setProperty("ack", "all");
> properties.put("enable.idempotence",true);
> properties.put("max.in.flight.requests.per.connection",5);
> properties.put("retries", 2);
> properties.setProperty("client.id", "producer-syn-2");
> properties.put("isolation.level","read_committed");
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-20870) FlinkKafkaSink
[
https://issues.apache.org/jira/browse/FLINK-20870?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
xx chai updated FLINK-20870:
Attachment: 1610072517(1).jpg
> FlinkKafkaSink
> --
>
> Key: FLINK-20870
> URL: https://issues.apache.org/jira/browse/FLINK-20870
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kafka
>Affects Versions: 1.12.0
> Environment: flink :1.12.0
> kafka 2.2.1
>Reporter: xx chai
>Priority: Major
> Attachments: 1610072517(1).jpg
>
>
> I consum from kafka sink to kafka,Then I split the message into then pieces.I
> guess the ten message should in one transaction. When the fifth message is
> sink kafka ,I throw a exception.but the first four are already in kafka.
> I set some parameters :
>properties.setProperty("transactional.id", "cxx");
> properties.setProperty("ack", "all");
> properties.put("enable.idempotence",true);
> properties.put("max.in.flight.requests.per.connection",5);
> properties.put("retries", 2);
> properties.setProperty("client.id", "producer-syn-2");
> properties.put("isolation.level","read_committed");
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-20870) FlinkKafkaSink
[
https://issues.apache.org/jira/browse/FLINK-20870?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
xx chai updated FLINK-20870:
Issue Type: Bug (was: Improvement)
> FlinkKafkaSink
> --
>
> Key: FLINK-20870
> URL: https://issues.apache.org/jira/browse/FLINK-20870
> Project: Flink
> Issue Type: Bug
> Components: API / DataStream
>Affects Versions: 1.12.0
> Environment: flink :1.12.0
> kafka 2.2.1
>Reporter: xx chai
>Priority: Major
>
> I consum from kafka sink to kafka,Then I split the message into then pieces.I
> guess the ten message should in one transaction. When the fifth message is
> sink kafka ,I throw a exception.but the first four are already in kafka.
> I set some parameters :
>properties.setProperty("transactional.id", "cxx");
> properties.setProperty("ack", "all");
> properties.put("enable.idempotence",true);
> properties.put("max.in.flight.requests.per.connection",5);
> properties.put("retries", 2);
> properties.setProperty("client.id", "producer-syn-2");
> properties.put("isolation.level","read_committed");
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-20870) FlinkKafkaSink
xx chai created FLINK-20870:
---
Summary: FlinkKafkaSink
Key: FLINK-20870
URL: https://issues.apache.org/jira/browse/FLINK-20870
Project: Flink
Issue Type: Improvement
Components: API / DataStream
Affects Versions: 1.12.0
Environment: flink :1.12.0
kafka 2.2.1
Reporter: xx chai
I consum from kafka sink to kafka,Then I split the message into then pieces.I
guess the ten message should in one transaction. When the fifth message is sink
kafka ,I throw a exception.but the first four are already in kafka.
I set some parameters :
properties.setProperty("transactional.id", "cxx");
properties.setProperty("ack", "all");
properties.put("enable.idempotence",true);
properties.put("max.in.flight.requests.per.connection",5);
properties.put("retries", 2);
properties.setProperty("client.id", "producer-syn-2");
properties.put("isolation.level","read_committed");
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-20780) Flink-sql-client query hive
[ https://issues.apache.org/jira/browse/FLINK-20780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17255497#comment-17255497 ] xx chai commented on FLINK-20780: - I may have found the reason，sql-client is not shut-down when ctrl+z or ./stop-cluster. I use the JPS command and kill sql-clients process .then resatrt and query hive is succeed > Flink-sql-client query hive > > > Key: FLINK-20780 > URL: https://issues.apache.org/jira/browse/FLINK-20780 > Project: Flink > Issue Type: Bug > Components: Connectors / Hive, Table SQL / Client > Environment: standalone >Reporter: xx chai >Priority: Major > Attachments: 1609127183(1).png, ps-flink-process.txt > > > flink-sql-client query hive is fail. > I've already configured hadoop classpath > and my hadoop is cdh > error : > Caused by: java.lang.ClassNotFoundException: > org.apache.hadoop.mapred.JobConfCaused by: java.lang.ClassNotFoundException: > org.apache.hadoop.mapred.JobConf at > java.net.URLClassLoader.findClass(URLClassLoader.java:382) at > java.lang.ClassLoader.loadClass(ClassLoader.java:418) at > sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:355) at > java.lang.ClassLoader.loadClass(ClassLoader.java:351) ... 56 more -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-20780) Flink-sql-client query hive
[ https://issues.apache.org/jira/browse/FLINK-20780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17255442#comment-17255442 ] xx chai commented on FLINK-20780: - [^ps-flink-process.txt] the flie is I execute aux | grep flink-process-id You can see from the file that hadoop classpath is loaded > Flink-sql-client query hive > > > Key: FLINK-20780 > URL: https://issues.apache.org/jira/browse/FLINK-20780 > Project: Flink > Issue Type: Bug > Components: Connectors / Hive, Table SQL / Client > Environment: standalone >Reporter: xx chai >Priority: Major > Attachments: 1609127183(1).png, ps-flink-process.txt > > > flink-sql-client query hive is fail. > I've already configured hadoop classpath > and my hadoop is cdh > error : > Caused by: java.lang.ClassNotFoundException: > org.apache.hadoop.mapred.JobConfCaused by: java.lang.ClassNotFoundException: > org.apache.hadoop.mapred.JobConf at > java.net.URLClassLoader.findClass(URLClassLoader.java:382) at > java.lang.ClassLoader.loadClass(ClassLoader.java:418) at > sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:355) at > java.lang.ClassLoader.loadClass(ClassLoader.java:351) ... 56 more -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-20780) Flink-sql-client query hive
[ https://issues.apache.org/jira/browse/FLINK-20780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] xx chai updated FLINK-20780: Attachment: ps-flink-process.txt > Flink-sql-client query hive > > > Key: FLINK-20780 > URL: https://issues.apache.org/jira/browse/FLINK-20780 > Project: Flink > Issue Type: Bug > Components: Connectors / Hive, Table SQL / Client > Environment: standalone >Reporter: xx chai >Priority: Major > Attachments: 1609127183(1).png, ps-flink-process.txt > > > flink-sql-client query hive is fail. > I've already configured hadoop classpath > and my hadoop is cdh > error : > Caused by: java.lang.ClassNotFoundException: > org.apache.hadoop.mapred.JobConfCaused by: java.lang.ClassNotFoundException: > org.apache.hadoop.mapred.JobConf at > java.net.URLClassLoader.findClass(URLClassLoader.java:382) at > java.lang.ClassLoader.loadClass(ClassLoader.java:418) at > sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:355) at > java.lang.ClassLoader.loadClass(ClassLoader.java:351) ... 56 more -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-20780) Flink-sql-client query hive
[ https://issues.apache.org/jira/browse/FLINK-20780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17255407#comment-17255407 ] xx chai commented on FLINK-20780: - There is only one node in my cluster,the cluster is my test cluster > Flink-sql-client query hive > > > Key: FLINK-20780 > URL: https://issues.apache.org/jira/browse/FLINK-20780 > Project: Flink > Issue Type: Bug > Components: Connectors / Hive, Table SQL / Client > Environment: standalone >Reporter: xx chai >Priority: Major > Attachments: 1609127183(1).png > > > flink-sql-client query hive is fail. > I've already configured hadoop classpath > and my hadoop is cdh > error : > Caused by: java.lang.ClassNotFoundException: > org.apache.hadoop.mapred.JobConfCaused by: java.lang.ClassNotFoundException: > org.apache.hadoop.mapred.JobConf at > java.net.URLClassLoader.findClass(URLClassLoader.java:382) at > java.lang.ClassLoader.loadClass(ClassLoader.java:418) at > sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:355) at > java.lang.ClassLoader.loadClass(ClassLoader.java:351) ... 56 more -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-20780) Flink-sql-client query hive
[
https://issues.apache.org/jira/browse/FLINK-20780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17255393#comment-17255393
]
xx chai commented on FLINK-20780:
-
export HADOOP_HOME=/opt/cloudera/parcels/CDH/lib/hadoop
export PATH=${HADOOP_HOME}/bin:$PATH
export HADOOP_CLASSPATH=`$HADOOP_HOME/bin/hadoop classpath`
I execute hadoop classpath,It prints the correct result
> Flink-sql-client query hive
>
>
> Key: FLINK-20780
> URL: https://issues.apache.org/jira/browse/FLINK-20780
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Hive, Table SQL / Client
> Environment: standalone
>Reporter: xx chai
>Priority: Major
> Attachments: 1609127183(1).png
>
>
> flink-sql-client query hive is fail.
> I've already configured hadoop classpath
> and my hadoop is cdh
> error :
> Caused by: java.lang.ClassNotFoundException:
> org.apache.hadoop.mapred.JobConfCaused by: java.lang.ClassNotFoundException:
> org.apache.hadoop.mapred.JobConf at
> java.net.URLClassLoader.findClass(URLClassLoader.java:382) at
> java.lang.ClassLoader.loadClass(ClassLoader.java:418) at
> sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:355) at
> java.lang.ClassLoader.loadClass(ClassLoader.java:351) ... 56 more
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-20780) Flink-sql-client query hive
[ https://issues.apache.org/jira/browse/FLINK-20780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17255387#comment-17255387 ] xx chai commented on FLINK-20780: - Every time i restart the cluster after you change jars in the lib folder > Flink-sql-client query hive > > > Key: FLINK-20780 > URL: https://issues.apache.org/jira/browse/FLINK-20780 > Project: Flink > Issue Type: Bug > Components: Connectors / Hive, Table SQL / Client > Environment: standalone >Reporter: xx chai >Priority: Major > Attachments: 1609127183(1).png > > > flink-sql-client query hive is fail. > I've already configured hadoop classpath > and my hadoop is cdh > error : > Caused by: java.lang.ClassNotFoundException: > org.apache.hadoop.mapred.JobConfCaused by: java.lang.ClassNotFoundException: > org.apache.hadoop.mapred.JobConf at > java.net.URLClassLoader.findClass(URLClassLoader.java:382) at > java.lang.ClassLoader.loadClass(ClassLoader.java:418) at > sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:355) at > java.lang.ClassLoader.loadClass(ClassLoader.java:351) ... 56 more -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-20780) Flink-sql-client query hive
[
https://issues.apache.org/jira/browse/FLINK-20780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17255375#comment-17255375
]
xx chai commented on FLINK-20780:
-
I have deleted {{flink-connector-hive_2.11-1.12.0.jar}},
{{hive-exec-2.1.1-cdh6.3.2.jar}}, {{hive-exec-2.2.0.jar.bak, but when I query
hive same mistake.}}
now lib:
flink-connector-jdbc_2.11-1.12.0.jar
flink-csv-1.12.0.jar
flink-dist_2.11-1.12.0.jar
flink-json-1.12.0.jar
flink-shaded-hadoop-3-uber-3.1.1.7.1.1.0-565-9.0.jar
flink-shaded-zookeeper-3.4.14.jar
flink-sql-connector-hive-2.2.0_2.11-1.12.0.jar
flink-sql-connector-kafka_2.11-1.12.0.jar
flink-table_2.11-1.12.0.jar
flink-table-api-java-bridge_2.11-1.12.0.jar
flink-table-blink_2.11-1.12.0.jar
log4j-1.2-api-2.12.1.jar
log4j-api-2.12.1.jar
log4j-core-2.12.1.jar
log4j-slf4j-impl-2.12.1.jar
mysql-connector-java-8.0.22.jar
> Flink-sql-client query hive
>
>
> Key: FLINK-20780
> URL: https://issues.apache.org/jira/browse/FLINK-20780
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Hive, Table SQL / Client
> Environment: standalone
>Reporter: xx chai
>Priority: Major
> Attachments: 1609127183(1).png
>
>
> flink-sql-client query hive is fail.
> I've already configured hadoop classpath
> and my hadoop is cdh
> error :
> Caused by: java.lang.ClassNotFoundException:
> org.apache.hadoop.mapred.JobConfCaused by: java.lang.ClassNotFoundException:
> org.apache.hadoop.mapred.JobConf at
> java.net.URLClassLoader.findClass(URLClassLoader.java:382) at
> java.lang.ClassLoader.loadClass(ClassLoader.java:418) at
> sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:355) at
> java.lang.ClassLoader.loadClass(ClassLoader.java:351) ... 56 more
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (FLINK-20780) Flink-sql-client query hive
[ https://issues.apache.org/jira/browse/FLINK-20780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17255369#comment-17255369 ] xx chai edited comment on FLINK-20780 at 12/28/20, 3:59 AM: Flink version is 1.12 chd version is 6.3.2 hadoop version is 3.0.0+cdh6.3.2 flink lib : flink-connector-hive_2.11-1.12.0.jar flink-connector-jdbc_2.11-1.12.0.jar flink-csv-1.12.0.jar flink-dist_2.11-1.12.0.jar flink-json-1.12.0.jar flink-shaded-hadoop-3-uber-3.1.1.7.1.1.0-565-9.0.jar flink-shaded-zookeeper-3.4.14.jar flink-sql-connector-hive-2.2.0_2.11-1.12.0.jar flink-sql-connector-kafka_2.11-1.12.0.jar flink-table_2.11-1.12.0.jar flink-table-api-java-bridge_2.11-1.12.0.jar flink-table-blink_2.11-1.12.0.jar hive-exec-2.1.1-cdh6.3.2.jar hive-exec-2.2.0.jar.bak log4j-1.2-api-2.12.1.jar log4j-api-2.12.1.jar log4j-core-2.12.1.jar log4j-slf4j-impl-2.12.1.jar mysql-connector-java-8.0.22.jar was (Author: chaixiaoxue): Flink version is 1.12 > Flink-sql-client query hive > > > Key: FLINK-20780 > URL: https://issues.apache.org/jira/browse/FLINK-20780 > Project: Flink > Issue Type: Bug > Components: Connectors / Hive, Table SQL / Client > Environment: standalone >Reporter: xx chai >Priority: Major > Attachments: 1609127183(1).png > > > flink-sql-client query hive is fail. > I've already configured hadoop classpath > and my hadoop is cdh > error : > Caused by: java.lang.ClassNotFoundException: > org.apache.hadoop.mapred.JobConfCaused by: java.lang.ClassNotFoundException: > org.apache.hadoop.mapred.JobConf at > java.net.URLClassLoader.findClass(URLClassLoader.java:382) at > java.lang.ClassLoader.loadClass(ClassLoader.java:418) at > sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:355) at > java.lang.ClassLoader.loadClass(ClassLoader.java:351) ... 56 more -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-20780) Flink-sql-client query hive
[ https://issues.apache.org/jira/browse/FLINK-20780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17255369#comment-17255369 ] xx chai commented on FLINK-20780: - Flink version is 1.12 > Flink-sql-client query hive > > > Key: FLINK-20780 > URL: https://issues.apache.org/jira/browse/FLINK-20780 > Project: Flink > Issue Type: Bug > Components: Connectors / Hive, Table SQL / Client > Environment: standalone >Reporter: xx chai >Priority: Major > Attachments: 1609127183(1).png > > > flink-sql-client query hive is fail. > I've already configured hadoop classpath > and my hadoop is cdh > error : > Caused by: java.lang.ClassNotFoundException: > org.apache.hadoop.mapred.JobConfCaused by: java.lang.ClassNotFoundException: > org.apache.hadoop.mapred.JobConf at > java.net.URLClassLoader.findClass(URLClassLoader.java:382) at > java.lang.ClassLoader.loadClass(ClassLoader.java:418) at > sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:355) at > java.lang.ClassLoader.loadClass(ClassLoader.java:351) ... 56 more -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-20780) Flink-sql-client query hive
[ https://issues.apache.org/jira/browse/FLINK-20780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] xx chai updated FLINK-20780: Attachment: 1609127183(1).png > Flink-sql-client query hive > > > Key: FLINK-20780 > URL: https://issues.apache.org/jira/browse/FLINK-20780 > Project: Flink > Issue Type: Bug > Components: Table SQL / Client > Environment: standalone >Reporter: xx chai >Priority: Major > Attachments: 1609127183(1).png > > > flink-sql-client query hive is fail. > I've already configured hadoop classpath > and my hadoop is cdh > error : > Caused by: java.lang.ClassNotFoundException: > org.apache.hadoop.mapred.JobConfCaused by: java.lang.ClassNotFoundException: > org.apache.hadoop.mapred.JobConf at > java.net.URLClassLoader.findClass(URLClassLoader.java:382) at > java.lang.ClassLoader.loadClass(ClassLoader.java:418) at > sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:355) at > java.lang.ClassLoader.loadClass(ClassLoader.java:351) ... 56 more -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-20780) Flink-sql-client query hive
xx chai created FLINK-20780: --- Summary: Flink-sql-client query hive Key: FLINK-20780 URL: https://issues.apache.org/jira/browse/FLINK-20780 Project: Flink Issue Type: Bug Components: Table SQL / Client Environment: standalone Reporter: xx chai flink-sql-client query hive is fail. I've already configured hadoop classpath and my hadoop is cdh error : Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.mapred.JobConfCaused by: java.lang.ClassNotFoundException: org.apache.hadoop.mapred.JobConf at java.net.URLClassLoader.findClass(URLClassLoader.java:382) at java.lang.ClassLoader.loadClass(ClassLoader.java:418) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:355) at java.lang.ClassLoader.loadClass(ClassLoader.java:351) ... 56 more -- This message was sent by Atlassian Jira (v8.3.4#803005)
