Re: [PR] [FLINK-34908][pipeline-connector][doris] Fix Mysql pipeline to doris will lost precision for timestamp [flink-cdc]

2024-06-10 Thread via GitHub 


gong commented on PR #3207:
URL: https://github.com/apache/flink-cdc/pull/3207#issuecomment-2159656572

   > @gong would you like to open a PR for release-3.1 branch?
   
   ok, I will open the PR.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] [FLINK-34908][pipeline-connector][doris] Fix Mysql pipeline to doris will lost precision for timestamp [flink-cdc]

2024-06-06 Thread via GitHub 


leonardBang commented on PR #3207:
URL: https://github.com/apache/flink-cdc/pull/3207#issuecomment-2152195587

   @gong would you like to open a PR for release-3.1 branch?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] [FLINK-34908][pipeline-connector][doris] Fix Mysql pipeline to doris will lost precision for timestamp [flink-cdc]

2024-06-06 Thread via GitHub 


leonardBang merged PR #3207:
URL: https://github.com/apache/flink-cdc/pull/3207


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] [FLINK-34908][pipeline-connector][doris] Fix Mysql pipeline to doris will lost precision for timestamp [flink-cdc]

2024-05-21 Thread via GitHub 


gong commented on PR #3207:
URL: https://github.com/apache/flink-cdc/pull/3207#issuecomment-2121831876

   @PatrickRen Hello, I rebase master.  CI fail is not relation with the PR. 
Please help to rerun CI.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] [FLINK-34908][pipeline-connector][doris] Fix Mysql pipeline to doris will lost precision for timestamp [flink-cdc]

2024-04-17 Thread via GitHub 


gong commented on PR #3207:
URL: https://github.com/apache/flink-cdc/pull/3207#issuecomment-2061042815

   @PatrickRen Hello, Please help to rerun CI. CI fail is not relation with the 
PR.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] [FLINK-34908][pipeline-connector][doris] Fix Mysql pipeline to doris will lost precision for timestamp [flink-cdc]

2024-04-17 Thread via GitHub 


gong commented on code in PR #3207:
URL: https://github.com/apache/flink-cdc/pull/3207#discussion_r1568504940


##
flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-doris/src/test/java/org/apache/flink/cdc/connectors/doris/sink/DorisRowConverterTest.java:
##
@@ -86,7 +113,9 @@ public void testExternalConvert() {
 row.add(converter.serialize(i, recordData));
 }
 Assert.assertEquals(
-"[true, 1.2, 1.2345, 1, 32, 64, 128, 2021-01-01 08:00:00, 
2021-01-01, a, doris]",
+"[true, 1.2, 1.2345, 1, 32, 64, 128, 2021-01-01 
08:00:00.00, 2021-01-01, a, doris, 2021-01-01 "
++ "08:01:11.00, 2021-01-01 08:01:11.123000, 
2021-01-01 08:01:11.123456, 2021-01-01 "
++ "16:01:11.00, 2021-01-01 16:01:11.123000, 
2021-01-01 16:01:11.123456]",

Review Comment:
   I set pipelineZoneId GMT+08:00



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] [FLINK-34908][pipeline-connector][doris] Fix Mysql pipeline to doris will lost precision for timestamp [flink-cdc]

2024-04-17 Thread via GitHub 


yuxiqian commented on code in PR #3207:
URL: https://github.com/apache/flink-cdc/pull/3207#discussion_r1568485784


##
flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-doris/src/test/java/org/apache/flink/cdc/connectors/doris/sink/DorisRowConverterTest.java:
##
@@ -86,7 +113,9 @@ public void testExternalConvert() {
 row.add(converter.serialize(i, recordData));
 }
 Assert.assertEquals(
-"[true, 1.2, 1.2345, 1, 32, 64, 128, 2021-01-01 08:00:00, 
2021-01-01, a, doris]",
+"[true, 1.2, 1.2345, 1, 32, 64, 128, 2021-01-01 
08:00:00.00, 2021-01-01, a, doris, 2021-01-01 "
++ "08:01:11.00, 2021-01-01 08:01:11.123000, 
2021-01-01 08:01:11.123456, 2021-01-01 "
++ "16:01:11.00, 2021-01-01 16:01:11.123000, 
2021-01-01 16:01:11.123456]",

Review Comment:
   This assertion will fail when running on any machine which doesn't use UTC+8 
timezone. Maybe dynamically generating expected data based on local timezone?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] [FLINK-34908][pipeline-connector][doris] Fix Mysql pipeline to doris will lost precision for timestamp [flink-cdc]

2024-04-17 Thread via GitHub 


gong commented on code in PR #3207:
URL: https://github.com/apache/flink-cdc/pull/3207#discussion_r1568450309


##
flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-doris/src/main/java/org/apache/flink/cdc/connectors/doris/sink/DorisRowConverter.java:
##
@@ -103,7 +103,8 @@ static SerializationConverter 
createExternalConverter(DataType type, ZoneId pipe
 return (index, val) ->
 val.getTimestamp(index, 
DataTypeChecks.getPrecision(type))
 .toLocalDateTime()
-
.format(DorisEventSerializer.DATE_TIME_FORMATTER);
+.toString()
+.replace('T', ' ');

Review Comment:
   @yuxiqian ok



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] [FLINK-34908][pipeline-connector][doris] Fix Mysql pipeline to doris will lost precision for timestamp [flink-cdc]

2024-04-16 Thread via GitHub 


yuxiqian commented on code in PR #3207:
URL: https://github.com/apache/flink-cdc/pull/3207#discussion_r1568134905


##
flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-doris/src/main/java/org/apache/flink/cdc/connectors/doris/sink/DorisRowConverter.java:
##
@@ -103,7 +103,8 @@ static SerializationConverter 
createExternalConverter(DataType type, ZoneId pipe
 return (index, val) ->
 val.getTimestamp(index, 
DataTypeChecks.getPrecision(type))
 .toLocalDateTime()
-
.format(DorisEventSerializer.DATE_TIME_FORMATTER);
+.toString()
+.replace('T', ' ');

Review Comment:
   What about fixing `DorisEventSerializer.DATE_TIME_FORMATTER` instead of 
doing string-level operations? It should make code clearer and more efficient.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] [FLINK-34908][pipeline-connector][doris] Fix Mysql pipeline to doris will lost precision for timestamp [flink-cdc]

2024-04-07 Thread via GitHub 


gong commented on PR #3207:
URL: https://github.com/apache/flink-cdc/pull/3207#issuecomment-2041406982

   > I wonder if changing `DorisEventSerializer.DATE_TIME_FORMATTER` to 
`-MM-dd HH:mm:ss.SS` fixes this problem, too?
   
   @yuxiqian I think that it can fix this problem, too. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] [FLINK-34908][pipeline-connector][doris] Fix Mysql pipeline to doris will lost precision for timestamp [flink-cdc]

2024-04-07 Thread via GitHub 


yuxiqian commented on PR #3207:
URL: https://github.com/apache/flink-cdc/pull/3207#issuecomment-2041372727

   I wonder if changing `DorisEventSerializer.DATE_TIME_FORMATTER` to 
`-MM-dd HH:mm:ss.SS` would fix this problem, too?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] [FLINK-34908][pipeline-connector][doris] Fix Mysql pipeline to doris will lost precision for timestamp [flink-cdc]

2024-04-06 Thread via GitHub 


gong commented on PR #3207:
URL: https://github.com/apache/flink-cdc/pull/3207#issuecomment-2041325325

   @leonardBang PTAL


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org