[jira] [Commented] (FLINK-11010) Flink SQL timestamp is inconsistent with currentProcessingTime()
[ https://issues.apache.org/jira/browse/FLINK-11010?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16718457#comment-16718457 ] ASF GitHub Bot commented on FLINK-11010: walterddr commented on issue #7180: [FLINK-11010] [TABLE] Flink SQL timestamp is inconsistent with currentProcessingTime() URL: https://github.com/apache/flink/pull/7180#issuecomment-446457485 > @walterddr could you please check the timezone problem when dealing with eventtime? > it seems that lamber-ken's commit only fixed proc time. > Would you please refer to my previous comment to take a look at my eventtime usecase. thanks a lot! > --update > i found the correct timestamp was changed in OutputRowtimeProcessFunction, using SqlFunctions.internalToTimestamp(), i'm not sure whether its redundant or not. > please fix this problem. thx FLINK-11010 only reports bugs for utilizing `currentProcessingTime()`. maybe we can mark the JIRA as duplicate of FLINK-8353 and focus on addressing the bigger overall timezone problem instead. what do you think? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Flink SQL timestamp is inconsistent with currentProcessingTime() > > > Key: FLINK-11010 > URL: https://issues.apache.org/jira/browse/FLINK-11010 > Project: Flink > Issue Type: Bug > Components: Table API SQL >Affects Versions: 1.6.2, 1.7.0, 1.8.0, 1.7.1 >Reporter: lamber-ken >Assignee: lamber-ken >Priority: Major > Labels: pull-request-available > > Flink SQL timestamp is inconsistent with currentProcessingTime(). > > the ProcessingTime is just implemented by invoking System.currentTimeMillis() > but the long value will be automatically wrapped to a Timestamp with the > following statement: > `new java.sql.Timestamp(time - TimeZone.getDefault().getOffset(time));` -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-11010) Flink SQL timestamp is inconsistent with currentProcessingTime()
[ https://issues.apache.org/jira/browse/FLINK-11010?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16718434#comment-16718434 ] ASF GitHub Bot commented on FLINK-11010: samsai edited a comment on issue #7180: [FLINK-11010] [TABLE] Flink SQL timestamp is inconsistent with currentProcessingTime() URL: https://github.com/apache/flink/pull/7180#issuecomment-446445016 @walterddr could you please check the timezone problem when dealing with eventtime? it seems that lamber-ken's commit only fixed proc time. Would you please refer to my previous comment to take a look at my eventtime usecase. thanks a lot! --update i found the correct timestamp was changed in OutputRowtimeProcessFunction, using SqlFunctions.internalToTimestamp(), i'm not sure whether its redundant or not. please fix this problem. thx This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Flink SQL timestamp is inconsistent with currentProcessingTime() > > > Key: FLINK-11010 > URL: https://issues.apache.org/jira/browse/FLINK-11010 > Project: Flink > Issue Type: Bug > Components: Table API SQL >Affects Versions: 1.6.2, 1.7.0, 1.8.0, 1.7.1 >Reporter: lamber-ken >Assignee: lamber-ken >Priority: Major > Labels: pull-request-available > > Flink SQL timestamp is inconsistent with currentProcessingTime(). > > the ProcessingTime is just implemented by invoking System.currentTimeMillis() > but the long value will be automatically wrapped to a Timestamp with the > following statement: > `new java.sql.Timestamp(time - TimeZone.getDefault().getOffset(time));` -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-11010) Flink SQL timestamp is inconsistent with currentProcessingTime()
[ https://issues.apache.org/jira/browse/FLINK-11010?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16718417#comment-16718417 ] ASF GitHub Bot commented on FLINK-11010: samsai commented on issue #7180: [FLINK-11010] [TABLE] Flink SQL timestamp is inconsistent with currentProcessingTime() URL: https://github.com/apache/flink/pull/7180#issuecomment-446445016 @walterddr could you please check the timezone problem when dealing with eventtime? it seems that lamber-ken's commit only fixed proc time. Would you please refer to my previous comment to take a look at my eventtime usecase. thanks a lot! This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Flink SQL timestamp is inconsistent with currentProcessingTime() > > > Key: FLINK-11010 > URL: https://issues.apache.org/jira/browse/FLINK-11010 > Project: Flink > Issue Type: Bug > Components: Table API SQL >Affects Versions: 1.6.2, 1.7.0, 1.8.0, 1.7.1 >Reporter: lamber-ken >Assignee: lamber-ken >Priority: Major > Labels: pull-request-available > > Flink SQL timestamp is inconsistent with currentProcessingTime(). > > the ProcessingTime is just implemented by invoking System.currentTimeMillis() > but the long value will be automatically wrapped to a Timestamp with the > following statement: > `new java.sql.Timestamp(time - TimeZone.getDefault().getOffset(time));` -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-11010) Flink SQL timestamp is inconsistent with currentProcessingTime()
[ https://issues.apache.org/jira/browse/FLINK-11010?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16717475#comment-16717475 ] ASF GitHub Bot commented on FLINK-11010: lamber-ken edited a comment on issue #7180: [FLINK-11010] [TABLE] Flink SQL timestamp is inconsistent with currentProcessingTime() URL: https://github.com/apache/flink/pull/7180#issuecomment-446264958 @walterddr, as the [https://issues.apache.org/jira/browse/FLINK-8353](https://issues.apache.org/jira/browse/FLINK-8353) desc. - need to modify each function which works with sql proctiome. - it's not user-friendly, because user need to use udf function to convert proctime. - flink sql shoud consistent with flink-jar program which user do not need to care the default timezone. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Flink SQL timestamp is inconsistent with currentProcessingTime() > > > Key: FLINK-11010 > URL: https://issues.apache.org/jira/browse/FLINK-11010 > Project: Flink > Issue Type: Bug > Components: Table API SQL >Affects Versions: 1.6.2, 1.7.0, 1.8.0, 1.7.1 >Reporter: lamber-ken >Assignee: lamber-ken >Priority: Major > Labels: pull-request-available > > Flink SQL timestamp is inconsistent with currentProcessingTime(). > > the ProcessingTime is just implemented by invoking System.currentTimeMillis() > but the long value will be automatically wrapped to a Timestamp with the > following statement: > `new java.sql.Timestamp(time - TimeZone.getDefault().getOffset(time));` -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-11010) Flink SQL timestamp is inconsistent with currentProcessingTime()
[ https://issues.apache.org/jira/browse/FLINK-11010?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16717473#comment-16717473 ] ASF GitHub Bot commented on FLINK-11010: lamber-ken commented on issue #7180: [FLINK-11010] [TABLE] Flink SQL timestamp is inconsistent with currentProcessingTime() URL: https://github.com/apache/flink/pull/7180#issuecomment-446264958 @walterddr, as the [https://issues.apache.org/jira/browse/FLINK-8353](https://issues.apache.org/jira/browse/FLINK-8353) desc. first, need to modify each function which works with sql proctiome. second, it's not user-friendly, because user need to use udf function to convert proctime third, flink sql shoud consistent with flink-jar program which user do not need to care the default timezone This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Flink SQL timestamp is inconsistent with currentProcessingTime() > > > Key: FLINK-11010 > URL: https://issues.apache.org/jira/browse/FLINK-11010 > Project: Flink > Issue Type: Bug > Components: Table API SQL >Affects Versions: 1.6.2, 1.7.0, 1.8.0, 1.7.1 >Reporter: lamber-ken >Assignee: lamber-ken >Priority: Major > Labels: pull-request-available > > Flink SQL timestamp is inconsistent with currentProcessingTime(). > > the ProcessingTime is just implemented by invoking System.currentTimeMillis() > but the long value will be automatically wrapped to a Timestamp with the > following statement: > `new java.sql.Timestamp(time - TimeZone.getDefault().getOffset(time));` -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-11010) Flink SQL timestamp is inconsistent with currentProcessingTime()
[
https://issues.apache.org/jira/browse/FLINK-11010?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16716320#comment-16716320
]
ASF GitHub Bot commented on FLINK-11010:
lamber-ken commented on a change in pull request #7180: [FLINK-11010] [TABLE]
Flink SQL timestamp is inconsistent with currentProcessingTime()
URL: https://github.com/apache/flink/pull/7180#discussion_r240484482
##
File path:
flink-libraries/flink-table/src/test/java/org/apache/flink/table/runtime/stream/sql/JavaSqlITCase.java
##
@@ -164,4 +167,35 @@ public void testUnion() throws Exception {
StreamITCase.compareWithList(expected);
}
+
+ @Test
+ public void testProctime() throws Exception {
+ StreamExecutionEnvironment env =
StreamExecutionEnvironment.getExecutionEnvironment();
+ StreamTableEnvironment tableEnv =
TableEnvironment.getTableEnvironment(env);
+
+ DataStream> ds =
JavaStreamTestData.getSmall3TupleDataSet(env);
+ tableEnv.registerDataStream("MyTable", ds, "a, b, c,
proctime.proctime");
+
+ String sqlQuery = "select proctime from MyTable";
+
+ Table result = tableEnv.sqlQuery(sqlQuery);
+
+ tableEnv
+ .toAppendStream(result, TypeInformation.of(Row.class))
+ .addSink(new SinkFunction() {
+ @Override
+ public void invoke(Row value, Context context)
throws Exception {
+
+ Timestamp procTimestamp = (Timestamp)
value.getField(0);
+
+ // validate the second here
+ long procSecondTime =
procTimestamp.getTime() / 1000;
Review comment:
yeah, it's better to validate hour. I'll update.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Flink SQL timestamp is inconsistent with currentProcessingTime()
>
>
> Key: FLINK-11010
> URL: https://issues.apache.org/jira/browse/FLINK-11010
> Project: Flink
> Issue Type: Bug
> Components: Table API SQL
>Affects Versions: 1.6.2, 1.7.0, 1.8.0, 1.7.1
>Reporter: lamber-ken
>Assignee: lamber-ken
>Priority: Major
> Labels: pull-request-available
>
> Flink SQL timestamp is inconsistent with currentProcessingTime().
>
> the ProcessingTime is just implemented by invoking System.currentTimeMillis()
> but the long value will be automatically wrapped to a Timestamp with the
> following statement:
> `new java.sql.Timestamp(time - TimeZone.getDefault().getOffset(time));`
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-11010) Flink SQL timestamp is inconsistent with currentProcessingTime()
[
https://issues.apache.org/jira/browse/FLINK-11010?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16716164#comment-16716164
]
ASF GitHub Bot commented on FLINK-11010:
walterddr commented on a change in pull request #7180: [FLINK-11010] [TABLE]
Flink SQL timestamp is inconsistent with currentProcessingTime()
URL: https://github.com/apache/flink/pull/7180#discussion_r240470904
##
File path:
flink-libraries/flink-table/src/test/java/org/apache/flink/table/runtime/stream/sql/JavaSqlITCase.java
##
@@ -164,4 +167,35 @@ public void testUnion() throws Exception {
StreamITCase.compareWithList(expected);
}
+
+ @Test
+ public void testProctime() throws Exception {
+ StreamExecutionEnvironment env =
StreamExecutionEnvironment.getExecutionEnvironment();
+ StreamTableEnvironment tableEnv =
TableEnvironment.getTableEnvironment(env);
+
+ DataStream> ds =
JavaStreamTestData.getSmall3TupleDataSet(env);
+ tableEnv.registerDataStream("MyTable", ds, "a, b, c,
proctime.proctime");
+
+ String sqlQuery = "select proctime from MyTable";
+
+ Table result = tableEnv.sqlQuery(sqlQuery);
+
+ tableEnv
+ .toAppendStream(result, TypeInformation.of(Row.class))
+ .addSink(new SinkFunction() {
+ @Override
+ public void invoke(Row value, Context context)
throws Exception {
+
+ Timestamp procTimestamp = (Timestamp)
value.getField(0);
+
+ // validate the second here
+ long procSecondTime =
procTimestamp.getTime() / 1000;
Review comment:
This causes me some trouble because cases can happen when they cross the
second boundary. this is not a stable ITCase in my opinion.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Flink SQL timestamp is inconsistent with currentProcessingTime()
>
>
> Key: FLINK-11010
> URL: https://issues.apache.org/jira/browse/FLINK-11010
> Project: Flink
> Issue Type: Bug
> Components: Table API SQL
>Affects Versions: 1.6.2, 1.7.0, 1.8.0, 1.7.1
>Reporter: lamber-ken
>Assignee: lamber-ken
>Priority: Major
> Labels: pull-request-available
>
> Flink SQL timestamp is inconsistent with currentProcessingTime().
>
> the ProcessingTime is just implemented by invoking System.currentTimeMillis()
> but the long value will be automatically wrapped to a Timestamp with the
> following statement:
> `new java.sql.Timestamp(time - TimeZone.getDefault().getOffset(time));`
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-11010) Flink SQL timestamp is inconsistent with currentProcessingTime()
[ https://issues.apache.org/jira/browse/FLINK-11010?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16716020#comment-16716020 ] ASF GitHub Bot commented on FLINK-11010: lamber-ken commented on issue #7180: [FLINK-11010] [TABLE] Flink SQL timestamp is inconsistent with currentProcessingTime() URL: https://github.com/apache/flink/pull/7180#issuecomment-446050898 hi, @zentol cc This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Flink SQL timestamp is inconsistent with currentProcessingTime() > > > Key: FLINK-11010 > URL: https://issues.apache.org/jira/browse/FLINK-11010 > Project: Flink > Issue Type: Bug > Components: Table API SQL >Affects Versions: 1.6.2 >Reporter: lamber-ken >Assignee: lamber-ken >Priority: Major > Labels: pull-request-available > > Flink SQL timestamp is inconsistent with currentProcessingTime(). > > the ProcessingTime is just implemented by invoking System.currentTimeMillis() > but the long value will be automatically wrapped to a Timestamp with the > following statement: > `new java.sql.Timestamp(time - TimeZone.getDefault().getOffset(time));` -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-11010) Flink SQL timestamp is inconsistent with currentProcessingTime()
[
https://issues.apache.org/jira/browse/FLINK-11010?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16715949#comment-16715949
]
ASF GitHub Bot commented on FLINK-11010:
samsai commented on issue #7180: [FLINK-11010] [TABLE] Flink SQL timestamp is
inconsistent with currentProcessingTime()
URL: https://github.com/apache/flink/pull/7180#issuecomment-446041717
i got the same problem too:
val format = new SimpleDateFormat("-MM-dd HH:mm:ssZ")
val origin: DataStream[TransactionEvent] = env.fromCollection(List(
TransactionEvent("u1", format.parse("2018-01-02 01:13:30+0800"), 10)
))
val source2 = origin
.assignTimestampsAndWatermarks(new
BoundedOutOfOrdernessTimestampExtractor[TransactionEvent](Time.minutes(1)){
override def extractTimestamp(element: TransactionEvent): Long = {
val timestamp = element.time.getTime
println(s"extractTimestamp:$timestamp")
timestamp
}
})
tEnv.fromDataStream(source2, 'user,'eventTime.rowtime)
.toAppendStream[Row].print()
I got eventTime as 2018-01-01 17:13:30.0, which is 8hours delayed
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Flink SQL timestamp is inconsistent with currentProcessingTime()
>
>
> Key: FLINK-11010
> URL: https://issues.apache.org/jira/browse/FLINK-11010
> Project: Flink
> Issue Type: Bug
> Components: Table API SQL
>Affects Versions: 1.6.2
>Reporter: lamber-ken
>Assignee: lamber-ken
>Priority: Major
> Labels: pull-request-available
>
> Flink SQL timestamp is inconsistent with currentProcessingTime().
>
> the ProcessingTime is just implemented by invoking System.currentTimeMillis()
> but the long value will be automatically wrapped to a Timestamp with the
> following statement:
> `new java.sql.Timestamp(time - TimeZone.getDefault().getOffset(time));`
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-11010) Flink SQL timestamp is inconsistent with currentProcessingTime()
[ https://issues.apache.org/jira/browse/FLINK-11010?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16714820#comment-16714820 ] ASF GitHub Bot commented on FLINK-11010: lamber-ken edited a comment on issue #7180: [FLINK-11010] [TABLE] Flink SQL timestamp is inconsistent with currentProcessingTime() URL: https://github.com/apache/flink/pull/7180#issuecomment-445698387 hi, @twalthr, @dawidwys, cc or talk about it, thanks. :) This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Flink SQL timestamp is inconsistent with currentProcessingTime() > > > Key: FLINK-11010 > URL: https://issues.apache.org/jira/browse/FLINK-11010 > Project: Flink > Issue Type: Bug > Components: Table API SQL >Affects Versions: 1.6.2 >Reporter: lamber-ken >Assignee: lamber-ken >Priority: Major > Labels: pull-request-available > > Flink SQL timestamp is inconsistent with currentProcessingTime(). > > the ProcessingTime is just implemented by invoking System.currentTimeMillis() > but the long value will be automatically wrapped to a Timestamp with the > following statement: > `new java.sql.Timestamp(time - TimeZone.getDefault().getOffset(time));` -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-11010) Flink SQL timestamp is inconsistent with currentProcessingTime()
[ https://issues.apache.org/jira/browse/FLINK-11010?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16714322#comment-16714322 ] ASF GitHub Bot commented on FLINK-11010: lamber-ken commented on issue #7180: [FLINK-11010] [TABLE] Flink SQL timestamp is inconsistent with currentProcessingTime() URL: https://github.com/apache/flink/pull/7180#issuecomment-445698387 hi, @twalthr, please cc or we should talk about it. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Flink SQL timestamp is inconsistent with currentProcessingTime() > > > Key: FLINK-11010 > URL: https://issues.apache.org/jira/browse/FLINK-11010 > Project: Flink > Issue Type: Bug > Components: Table API SQL >Affects Versions: 1.6.2 >Reporter: lamber-ken >Assignee: lamber-ken >Priority: Major > Labels: pull-request-available > > Flink SQL timestamp is inconsistent with currentProcessingTime(). > > the ProcessingTime is just implemented by invoking System.currentTimeMillis() > but the long value will be automatically wrapped to a Timestamp with the > following statement: > `new java.sql.Timestamp(time - TimeZone.getDefault().getOffset(time));` -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-11010) Flink SQL timestamp is inconsistent with currentProcessingTime()
[
https://issues.apache.org/jira/browse/FLINK-11010?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16714281#comment-16714281
]
ASF GitHub Bot commented on FLINK-11010:
lzqdename edited a comment on issue #7180: [FLINK-11010] [TABLE] Flink SQL
timestamp is inconsistent with currentProcessingTime()
URL: https://github.com/apache/flink/pull/7180#issuecomment-445667532
In the Flink
to get process time ,the result is OK
then , In the Flink,the result is transfromed several times
for example , Long ->Timestamp ->Long->Timestamp
when Long ->Timestamp,use code as follows:**[In SqlFunctions]**
public static java.sql.Time internalToTime(int v) {
return new java.sql.Time(v - LOCAL_TZ.getOffset(v));
}
when Timestamp->Long,use code as follows:**[In SqlFunctions]**
// mainly intended for java.sql.Timestamp but works for other dates also
public static long toLong(java.util.Date v, TimeZone timeZone) {
final long time = v.getTime();
return time + timeZone.getOffset(time);
}
In the final write step,use code as follows:
protected long _timestamp(Date value) {
return value == null ? 0L : **value.getTime()**;
}
the position is
:org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.ser.std.DateSerializer._timestamp
(DateSerializer.java:41)
now ,let me think, we use two different class to handle the transformation
between Long and Timestamp,
and the first class SqlFunctions imports TimeZone factor,
but the second class ,TimeZone factor not imported
It is not consitent when handling time transformation!
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Flink SQL timestamp is inconsistent with currentProcessingTime()
>
>
> Key: FLINK-11010
> URL: https://issues.apache.org/jira/browse/FLINK-11010
> Project: Flink
> Issue Type: Bug
> Components: Table API SQL
>Affects Versions: 1.6.2
>Reporter: lamber-ken
>Assignee: lamber-ken
>Priority: Major
> Labels: pull-request-available
>
> Flink SQL timestamp is inconsistent with currentProcessingTime().
>
> the ProcessingTime is just implemented by invoking System.currentTimeMillis()
> but the long value will be automatically wrapped to a Timestamp with the
> following statement:
> `new java.sql.Timestamp(time - TimeZone.getDefault().getOffset(time));`
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-11010) Flink SQL timestamp is inconsistent with currentProcessingTime()
[
https://issues.apache.org/jira/browse/FLINK-11010?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16714282#comment-16714282
]
ASF GitHub Bot commented on FLINK-11010:
lzqdename edited a comment on issue #7180: [FLINK-11010] [TABLE] Flink SQL
timestamp is inconsistent with currentProcessingTime()
URL: https://github.com/apache/flink/pull/7180#issuecomment-445667532
In the Flink
to get process time ,the result is OK
then , In the Flink,the result is transfromed several times
for example , Long ->Timestamp ->Long->Timestamp
when Long ->Timestamp,use code as follows:**[In SqlFunctions]**
public static java.sql.Time internalToTime(int v) {
return new java.sql.Time(v - LOCAL_TZ.getOffset(v));
}
when Timestamp->Long,use code as follows:**[In SqlFunctions]**
// mainly intended for java.sql.Timestamp but works for other dates also
public static long toLong(java.util.Date v, TimeZone timeZone) {
final long time = v.getTime();
return time + timeZone.getOffset(time);
}
In the final write step,
Timestamp to Long
use code as follows:
protected long _timestamp(Date value) {
return value == null ? 0L : **value.getTime()**;
}
the position is
:org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.ser.std.DateSerializer._timestamp
(DateSerializer.java:41)
now ,let me think, we use two different class to handle the transformation
between Long and Timestamp,
and the first class SqlFunctions imports TimeZone factor,
but the second class ,TimeZone factor not imported
It is not consitent when handling time transformation!
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Flink SQL timestamp is inconsistent with currentProcessingTime()
>
>
> Key: FLINK-11010
> URL: https://issues.apache.org/jira/browse/FLINK-11010
> Project: Flink
> Issue Type: Bug
> Components: Table API SQL
>Affects Versions: 1.6.2
>Reporter: lamber-ken
>Assignee: lamber-ken
>Priority: Major
> Labels: pull-request-available
>
> Flink SQL timestamp is inconsistent with currentProcessingTime().
>
> the ProcessingTime is just implemented by invoking System.currentTimeMillis()
> but the long value will be automatically wrapped to a Timestamp with the
> following statement:
> `new java.sql.Timestamp(time - TimeZone.getDefault().getOffset(time));`
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-11010) Flink SQL timestamp is inconsistent with currentProcessingTime()
[
https://issues.apache.org/jira/browse/FLINK-11010?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16714280#comment-16714280
]
ASF GitHub Bot commented on FLINK-11010:
lzqdename commented on issue #7180: [FLINK-11010] [TABLE] Flink SQL timestamp
is inconsistent with currentProcessingTime()
URL: https://github.com/apache/flink/pull/7180#issuecomment-445667532
In the Flink
to get process time ,the result is OK
then , In the Flink,the result is transfromed several times
for example , Long ->Timestamp ->Long->Timestamp
when Long ->Timestamp,use code as follows:**[In SqlFunctions]**
public static java.sql.Time internalToTime(int v) {
return new java.sql.Time(v - LOCAL_TZ.getOffset(v));
}
when Timestamp->Long,use code as follows:**[In SqlFunctions]**
// mainly intended for java.sql.Timestamp but works for other dates also
public static long toLong(java.util.Date v, TimeZone timeZone) {
final long time = v.getTime();
return time + timeZone.getOffset(time);
}
In the final write step,use code as follows:
protected long _timestamp(Date value) {
return value == null ? 0L : **value.getTime()**;
}
the position is
:org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.ser.std.DateSerializer._timestamp
(DateSerializer.java:41)
now ,letme think, we use two different class to handle the transformation
between Long and Timestamp,
and the first class SqlFunctions imports TimeZone factor,
but the second class ,TimeZone factor not imported
It is not consitent when handling time transformation!
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Flink SQL timestamp is inconsistent with currentProcessingTime()
>
>
> Key: FLINK-11010
> URL: https://issues.apache.org/jira/browse/FLINK-11010
> Project: Flink
> Issue Type: Bug
> Components: Table API SQL
>Affects Versions: 1.6.2
>Reporter: lamber-ken
>Assignee: lamber-ken
>Priority: Major
> Labels: pull-request-available
>
> Flink SQL timestamp is inconsistent with currentProcessingTime().
>
> the ProcessingTime is just implemented by invoking System.currentTimeMillis()
> but the long value will be automatically wrapped to a Timestamp with the
> following statement:
> `new java.sql.Timestamp(time - TimeZone.getDefault().getOffset(time));`
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-11010) Flink SQL timestamp is inconsistent with currentProcessingTime()
[
https://issues.apache.org/jira/browse/FLINK-11010?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16712292#comment-16712292
]
ASF GitHub Bot commented on FLINK-11010:
lamber-ken commented on issue #7180: [FLINK-11010] [TABLE] Flink SQL timestamp
is inconsistent with currentProcessingTime()
URL: https://github.com/apache/flink/pull/7180#issuecomment-445109507
@walterddr, you are welcome.
flink sql use `CodeGenerator#generateProctimeTimestamp` method to process
`proctime` and its type is `Timestamp`, flink use calcite to convert long to
Timestamp
but calcite implementation of the method as below,
`SqlFunctions#internalToTime`
```
public static java.sql.Time internalToTime(int v) {
return new java.sql.Time(v - LOCAL_TZ.getOffset(v));
}
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Flink SQL timestamp is inconsistent with currentProcessingTime()
>
>
> Key: FLINK-11010
> URL: https://issues.apache.org/jira/browse/FLINK-11010
> Project: Flink
> Issue Type: Bug
> Components: Table API SQL
>Affects Versions: 1.6.2
>Reporter: lamber-ken
>Assignee: lamber-ken
>Priority: Major
> Labels: pull-request-available
>
> Flink SQL timestamp is inconsistent with currentProcessingTime().
>
> the ProcessingTime is just implemented by invoking System.currentTimeMillis()
> but the long value will be automatically wrapped to a Timestamp with the
> following statement:
> `new java.sql.Timestamp(time - TimeZone.getDefault().getOffset(time));`
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-11010) Flink SQL timestamp is inconsistent with currentProcessingTime()
[ https://issues.apache.org/jira/browse/FLINK-11010?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16712272#comment-16712272 ] ASF GitHub Bot commented on FLINK-11010: walterddr commented on issue #7180: [FLINK-11010] [TABLE] Flink SQL timestamp is inconsistent with currentProcessingTime() URL: https://github.com/apache/flink/pull/7180#issuecomment-445103065 Hi @lamber-ken sorry for the late response. I am a bit confuse with the JIRA ticket description and the PR comments. I am under the impression that the goal should make all internal times GMT-based regardless of where the code runs. Thus `currentProcessingTime()` should return a GMT-based time. if my understanding is correct, shouldn't the fix be in the "timerService" side instead? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Flink SQL timestamp is inconsistent with currentProcessingTime() > > > Key: FLINK-11010 > URL: https://issues.apache.org/jira/browse/FLINK-11010 > Project: Flink > Issue Type: Bug > Components: Table API SQL >Affects Versions: 1.6.2 >Reporter: lamber-ken >Assignee: lamber-ken >Priority: Major > Labels: pull-request-available > > Flink SQL timestamp is inconsistent with currentProcessingTime(). > > the ProcessingTime is just implemented by invoking System.currentTimeMillis() > but the long value will be automatically wrapped to a Timestamp with the > following statement: > `new java.sql.Timestamp(time - TimeZone.getDefault().getOffset(time));` -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-11010) Flink SQL timestamp is inconsistent with currentProcessingTime()
[ https://issues.apache.org/jira/browse/FLINK-11010?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16712232#comment-16712232 ] ASF GitHub Bot commented on FLINK-11010: Jennifer-sarah commented on issue #7180: [FLINK-11010] [TABLE] Flink SQL timestamp is inconsistent with currentProcessingTime() URL: https://github.com/apache/flink/pull/7180#issuecomment-445092699 +1, from my side. I had the [same problem](http://mail-archives.apache.org/mod_mbox/flink-user/201711.mbox/%3c351fd9ab-7a28-4ce0-bd9c-c2a15e537...@163.com%3E). thanks This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Flink SQL timestamp is inconsistent with currentProcessingTime() > > > Key: FLINK-11010 > URL: https://issues.apache.org/jira/browse/FLINK-11010 > Project: Flink > Issue Type: Bug > Components: Table API SQL >Affects Versions: 1.6.2 >Reporter: lamber-ken >Assignee: lamber-ken >Priority: Major > Labels: pull-request-available > > Flink SQL timestamp is inconsistent with currentProcessingTime(). > > the ProcessingTime is just implemented by invoking System.currentTimeMillis() > but the long value will be automatically wrapped to a Timestamp with the > following statement: > `new java.sql.Timestamp(time - TimeZone.getDefault().getOffset(time));` -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-11010) Flink SQL timestamp is inconsistent with currentProcessingTime()
[
https://issues.apache.org/jira/browse/FLINK-11010?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16711073#comment-16711073
]
ASF GitHub Bot commented on FLINK-11010:
lzqdename edited a comment on issue #7180: [FLINK-11010] [TABLE] Flink SQL
timestamp is inconsistent with currentProcessingTime()
URL: https://github.com/apache/flink/pull/7180#issuecomment-444762927
let me show how to generate the wrong result
---
**background**: processing time in tumbling window flink:1.5.0
the invoke stack is as follows:
[1] org.apache.calcite.runtime.SqlFunctions.internalToTimestamp
(SqlFunctions.java:1,747)
[2]
org.apache.flink.table.runtime.aggregate.TimeWindowPropertyCollector.collect
(TimeWindowPropertyCollector.scala:53)
[3]
org.apache.flink.table.runtime.aggregate.IncrementalAggregateWindowFunction.apply
(IncrementalAggregateWindowFunction.scala:74)
[4]
org.apache.flink.table.runtime.aggregate.IncrementalAggregateTimeWindowFunction.apply
(IncrementalAggregateTimeWindowFunction.scala:72)
[5]
org.apache.flink.table.runtime.aggregate.IncrementalAggregateTimeWindowFunction.apply
(IncrementalAggregateTimeWindowFunction.scala:39)
[6]
org.apache.flink.streaming.runtime.operators.windowing.functions.InternalSingleValueWindowFunction.process
(InternalSingleValueWindowFunction.java:46)
[7]
org.apache.flink.streaming.runtime.operators.windowing.WindowOperator.emitWindowContents
(WindowOperator.java:550)
[8]
org.apache.flink.streaming.runtime.operators.windowing.WindowOperator.onProcessingTime
(WindowOperator.java:505)
[9]
org.apache.flink.streaming.api.operators.HeapInternalTimerService.onProcessingTime
(HeapInternalTimerService.java:266)
[10]
org.apache.flink.streaming.runtime.tasks.SystemProcessingTimeService$TriggerTask.run
(SystemProcessingTimeService.java:281)
[11] java.util.concurrent.Executors$RunnableAdapter.call
(Executors.java:511)
[12] java.util.concurrent.FutureTask.run (FutureTask.java:266)
[13]
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201
(ScheduledThreadPoolExecutor.java:180)
[14]
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run
(ScheduledThreadPoolExecutor.java:293)
[15] java.util.concurrent.ThreadPoolExecutor.runWorker
(ThreadPoolExecutor.java:1,142)
[16] java.util.concurrent.ThreadPoolExecutor$Worker.run
(ThreadPoolExecutor.java:617)
[17] java.lang.Thread.run (Thread.java:748)
now ,we are at [1]
org.apache.calcite.runtime.SqlFunctions.internalToTimestamp
(SqlFunctions.java:1,747)
and the code is as follows:
` public static Timestamp internalToTimestamp(long v)
{
return new Timestamp(v - LOCAL_TZ.getOffset(v));
}
`
let us print the value of windowStart:v
print v
v = 154407483
let us print the value of windowEnd:v
print v
v = 1544074833000
after this, come back to
[1]
org.apache.flink.table.runtime.aggregate.TimeWindowPropertyCollector.collect
(TimeWindowPropertyCollector.scala:51)
then,we will execute
`
if (windowStartOffset.isDefined) {
output.setField(
lastFieldPos + windowStartOffset.get,
SqlFunctions.internalToTimestamp(windowStart))
}
if (windowEndOffset.isDefined) {
output.setField(
lastFieldPos + windowEndOffset.get,
SqlFunctions.internalToTimestamp(windowEnd))
}
`
before execute,the output is
output =
"pro0,throwable0,ERROR,ip0,1,ymm-appmetric-dev-self1_5_924367729,null,null,null"
after execute,the output is
output =
"pro0,throwable0,ERROR,ip0,1,ymm-appmetric-dev-self1_5_924367729,2018-12-06
05:40:30.0,2018-12-06 05:40:33.0,null"
so,do you think the
long value 154407483 translated to be 2018-12-06 05:40:30.0
long value 1544074833000 translated to be 2018-12-06 05:40:33.0
would be right?
I am in China, I think the timestamp should be 2018-12-06 13:40:30.0 and
2018-12-06 13:40:33.0
okay,let us continue
now ,the data will be written to kafka,before write ,the data will be
serialized
let us see what happened!
the call stack is as follows:
[1]
org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.ser.std.DateSerializer._timestamp
(DateSerializer.java:41)
[2]
org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.ser.std.DateSerializer.serialize
(DateSerializer.java:48)
[3]
org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.ser.std.DateSerializer.serialize
(DateSerializer.java:15)
[4]
org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.ser.DefaultSerializerProvider.serializeValue
(DefaultSerializerProvider.java:130)
[5]
[jira] [Commented] (FLINK-11010) Flink SQL timestamp is inconsistent with currentProcessingTime()
[
https://issues.apache.org/jira/browse/FLINK-11010?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16711071#comment-16711071
]
ASF GitHub Bot commented on FLINK-11010:
lzqdename edited a comment on issue #7180: [FLINK-11010] [TABLE] Flink SQL
timestamp is inconsistent with currentProcessingTime()
URL: https://github.com/apache/flink/pull/7180#issuecomment-444762927
let me show how to generate the wrong result
---
**background**: processing time in tumbling window flink:1.5.0
the invoke stack is as follows:
[1] org.apache.calcite.runtime.SqlFunctions.internalToTimestamp
(SqlFunctions.java:1,747)
[2]
org.apache.flink.table.runtime.aggregate.TimeWindowPropertyCollector.collect
(TimeWindowPropertyCollector.scala:53)
[3]
org.apache.flink.table.runtime.aggregate.IncrementalAggregateWindowFunction.apply
(IncrementalAggregateWindowFunction.scala:74)
[4]
org.apache.flink.table.runtime.aggregate.IncrementalAggregateTimeWindowFunction.apply
(IncrementalAggregateTimeWindowFunction.scala:72)
[5]
org.apache.flink.table.runtime.aggregate.IncrementalAggregateTimeWindowFunction.apply
(IncrementalAggregateTimeWindowFunction.scala:39)
[6]
org.apache.flink.streaming.runtime.operators.windowing.functions.InternalSingleValueWindowFunction.process
(InternalSingleValueWindowFunction.java:46)
[7]
org.apache.flink.streaming.runtime.operators.windowing.WindowOperator.emitWindowContents
(WindowOperator.java:550)
[8]
org.apache.flink.streaming.runtime.operators.windowing.WindowOperator.onProcessingTime
(WindowOperator.java:505)
[9]
org.apache.flink.streaming.api.operators.HeapInternalTimerService.onProcessingTime
(HeapInternalTimerService.java:266)
[10]
org.apache.flink.streaming.runtime.tasks.SystemProcessingTimeService$TriggerTask.run
(SystemProcessingTimeService.java:281)
[11] java.util.concurrent.Executors$RunnableAdapter.call
(Executors.java:511)
[12] java.util.concurrent.FutureTask.run (FutureTask.java:266)
[13]
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201
(ScheduledThreadPoolExecutor.java:180)
[14]
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run
(ScheduledThreadPoolExecutor.java:293)
[15] java.util.concurrent.ThreadPoolExecutor.runWorker
(ThreadPoolExecutor.java:1,142)
[16] java.util.concurrent.ThreadPoolExecutor$Worker.run
(ThreadPoolExecutor.java:617)
[17] java.lang.Thread.run (Thread.java:748)
now ,we are at [1]
org.apache.calcite.runtime.SqlFunctions.internalToTimestamp
(SqlFunctions.java:1,747)
and the code is as follows:
` public static Timestamp internalToTimestamp(long v)
{
return new Timestamp(v - LOCAL_TZ.getOffset(v));
}
`
let us print the value of windowStart:v
print v
v = 154407483
let us print the value of windowEnd:v
print v
v = 1544074833000
after this, come back to
[1]
org.apache.flink.table.runtime.aggregate.TimeWindowPropertyCollector.collect
(TimeWindowPropertyCollector.scala:51)
then,we will execute
`
if (windowStartOffset.isDefined) {
output.setField(
lastFieldPos + windowStartOffset.get,
SqlFunctions.internalToTimestamp(windowStart))
}
if (windowEndOffset.isDefined) {
output.setField(
lastFieldPos + windowEndOffset.get,
SqlFunctions.internalToTimestamp(windowEnd))
}
`
before execute,the output is
output =
"pro0,throwable0,ERROR,ip0,1,ymm-appmetric-dev-self1_5_924367729,null,null,null"
after execute,the output is
output =
"pro0,throwable0,ERROR,ip0,1,ymm-appmetric-dev-self1_5_924367729,2018-12-06
05:40:30.0,2018-12-06 05:40:33.0,null"
so,do you think the
long value 154407483 translated to be 2018-12-06 05:40:30.0
long value 1544074833000 translated to be 2018-12-06 05:40:33.0
would be right?
I am in China, I think the timestamp should be 2018-12-06 13:40:30.0 and
2018-12-06 13:40:33.0
okay,let us continue
now ,the data will be written to kafka,before write ,the data will be
serialized
let us see what happened!
the call stack is as follows:
` [1]
org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.ser.std.DateSerializer._timestamp
(DateSerializer.java:41)
[2]
org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.ser.std.DateSerializer.serialize
(DateSerializer.java:48)
[3]
org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.ser.std.DateSerializer.serialize
(DateSerializer.java:15)
[4]
org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.ser.DefaultSerializerProvider.serializeValue
(DefaultSerializerProvider.java:130)
[5]
[jira] [Commented] (FLINK-11010) Flink SQL timestamp is inconsistent with currentProcessingTime()
[ https://issues.apache.org/jira/browse/FLINK-11010?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16711063#comment-16711063 ] ASF GitHub Bot commented on FLINK-11010: lamber-ken commented on issue #7180: [FLINK-11010] [TABLE] Flink SQL timestamp is inconsistent with currentProcessingTime() URL: https://github.com/apache/flink/pull/7180#issuecomment-444773602 @lzqdename, thanks for your detail description. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Flink SQL timestamp is inconsistent with currentProcessingTime() > > > Key: FLINK-11010 > URL: https://issues.apache.org/jira/browse/FLINK-11010 > Project: Flink > Issue Type: Bug > Components: Table API SQL >Affects Versions: 1.6.2 >Reporter: lamber-ken >Assignee: lamber-ken >Priority: Major > Labels: pull-request-available > > Flink SQL timestamp is inconsistent with currentProcessingTime(). > > the ProcessingTime is just implemented by invoking System.currentTimeMillis() > but the long value will be automatically wrapped to a Timestamp with the > following statement: > `new java.sql.Timestamp(time - TimeZone.getDefault().getOffset(time));` -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-11010) Flink SQL timestamp is inconsistent with currentProcessingTime()
[
https://issues.apache.org/jira/browse/FLINK-11010?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16711033#comment-16711033
]
ASF GitHub Bot commented on FLINK-11010:
lzqdename edited a comment on issue #7180: [FLINK-11010] [TABLE] Flink SQL
timestamp is inconsistent with currentProcessingTime()
URL: https://github.com/apache/flink/pull/7180#issuecomment-444762927
let me show how to generate the wrong result
---
**background**: processing time in tumbling window flink:1.5.0
the invoke stack is as follows:
[1] org.apache.calcite.runtime.SqlFunctions.internalToTimestamp
(SqlFunctions.java:1,747)
[2]
org.apache.flink.table.runtime.aggregate.TimeWindowPropertyCollector.collect
(TimeWindowPropertyCollector.scala:53)
[3]
org.apache.flink.table.runtime.aggregate.IncrementalAggregateWindowFunction.apply
(IncrementalAggregateWindowFunction.scala:74)
[4]
org.apache.flink.table.runtime.aggregate.IncrementalAggregateTimeWindowFunction.apply
(IncrementalAggregateTimeWindowFunction.scala:72)
[5]
org.apache.flink.table.runtime.aggregate.IncrementalAggregateTimeWindowFunction.apply
(IncrementalAggregateTimeWindowFunction.scala:39)
[6]
org.apache.flink.streaming.runtime.operators.windowing.functions.InternalSingleValueWindowFunction.process
(InternalSingleValueWindowFunction.java:46)
[7]
org.apache.flink.streaming.runtime.operators.windowing.WindowOperator.emitWindowContents
(WindowOperator.java:550)
[8]
org.apache.flink.streaming.runtime.operators.windowing.WindowOperator.onProcessingTime
(WindowOperator.java:505)
[9]
org.apache.flink.streaming.api.operators.HeapInternalTimerService.onProcessingTime
(HeapInternalTimerService.java:266)
[10]
org.apache.flink.streaming.runtime.tasks.SystemProcessingTimeService$TriggerTask.run
(SystemProcessingTimeService.java:281)
[11] java.util.concurrent.Executors$RunnableAdapter.call
(Executors.java:511)
[12] java.util.concurrent.FutureTask.run (FutureTask.java:266)
[13]
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201
(ScheduledThreadPoolExecutor.java:180)
[14]
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run
(ScheduledThreadPoolExecutor.java:293)
[15] java.util.concurrent.ThreadPoolExecutor.runWorker
(ThreadPoolExecutor.java:1,142)
[16] java.util.concurrent.ThreadPoolExecutor$Worker.run
(ThreadPoolExecutor.java:617)
[17] java.lang.Thread.run (Thread.java:748)
now ,we are at [1]
org.apache.calcite.runtime.SqlFunctions.internalToTimestamp
(SqlFunctions.java:1,747)
and the code is as follows:
` public static Timestamp internalToTimestamp(long v)
{
return new Timestamp(v - LOCAL_TZ.getOffset(v));
}
`
let us print the value of windowStart:v
print v
v = 154407483
let us print the value of windowEnd:v
print v
v = 1544074833000
after this, come back to
[1]
org.apache.flink.table.runtime.aggregate.TimeWindowPropertyCollector.collect
(TimeWindowPropertyCollector.scala:51)
then,we will execute
`if (windowStartOffset.isDefined) {
output.setField(
lastFieldPos + windowStartOffset.get,
SqlFunctions.internalToTimestamp(windowStart))
}
if (windowEndOffset.isDefined) {
output.setField(
lastFieldPos + windowEndOffset.get,
SqlFunctions.internalToTimestamp(windowEnd))
}
`
before execute,the output is
output =
"pro0,throwable0,ERROR,ip0,1,ymm-appmetric-dev-self1_5_924367729,null,null,null"
after execute,the output is
output =
"pro0,throwable0,ERROR,ip0,1,ymm-appmetric-dev-self1_5_924367729,2018-12-06
05:40:30.0,2018-12-06 05:40:33.0,null"
so,do you think the
long value 154407483 translated to be 2018-12-06 05:40:30.0
long value 1544074833000 translated to be 2018-12-06 05:40:33.0
would be right?
I am in China, I think the timestamp should be 2018-12-06 13:40:30.0 and
2018-12-06 13:40:33.0
okay,let us continue
now ,the data will be write to kafka,before write ,the data will be
serialized
let us see what happened!
the call stack is as follows:
` [1]
org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.ser.std.DateSerializer._timestamp
(DateSerializer.java:41)
[2]
org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.ser.std.DateSerializer.serialize
(DateSerializer.java:48)
[3]
org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.ser.std.DateSerializer.serialize
(DateSerializer.java:15)
[4]
org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.ser.DefaultSerializerProvider.serializeValue
(DefaultSerializerProvider.java:130)
[5]
[jira] [Commented] (FLINK-11010) Flink SQL timestamp is inconsistent with currentProcessingTime()
[
https://issues.apache.org/jira/browse/FLINK-11010?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16711035#comment-16711035
]
ASF GitHub Bot commented on FLINK-11010:
lzqdename edited a comment on issue #7180: [FLINK-11010] [TABLE] Flink SQL
timestamp is inconsistent with currentProcessingTime()
URL: https://github.com/apache/flink/pull/7180#issuecomment-444762927
let me show how to generate the wrong result
---
**background**: processing time in tumbling window flink:1.5.0
the invoke stack is as follows:
[1] org.apache.calcite.runtime.SqlFunctions.internalToTimestamp
(SqlFunctions.java:1,747)
[2]
org.apache.flink.table.runtime.aggregate.TimeWindowPropertyCollector.collect
(TimeWindowPropertyCollector.scala:53)
[3]
org.apache.flink.table.runtime.aggregate.IncrementalAggregateWindowFunction.apply
(IncrementalAggregateWindowFunction.scala:74)
[4]
org.apache.flink.table.runtime.aggregate.IncrementalAggregateTimeWindowFunction.apply
(IncrementalAggregateTimeWindowFunction.scala:72)
[5]
org.apache.flink.table.runtime.aggregate.IncrementalAggregateTimeWindowFunction.apply
(IncrementalAggregateTimeWindowFunction.scala:39)
[6]
org.apache.flink.streaming.runtime.operators.windowing.functions.InternalSingleValueWindowFunction.process
(InternalSingleValueWindowFunction.java:46)
[7]
org.apache.flink.streaming.runtime.operators.windowing.WindowOperator.emitWindowContents
(WindowOperator.java:550)
[8]
org.apache.flink.streaming.runtime.operators.windowing.WindowOperator.onProcessingTime
(WindowOperator.java:505)
[9]
org.apache.flink.streaming.api.operators.HeapInternalTimerService.onProcessingTime
(HeapInternalTimerService.java:266)
[10]
org.apache.flink.streaming.runtime.tasks.SystemProcessingTimeService$TriggerTask.run
(SystemProcessingTimeService.java:281)
[11] java.util.concurrent.Executors$RunnableAdapter.call
(Executors.java:511)
[12] java.util.concurrent.FutureTask.run (FutureTask.java:266)
[13]
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201
(ScheduledThreadPoolExecutor.java:180)
[14]
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run
(ScheduledThreadPoolExecutor.java:293)
[15] java.util.concurrent.ThreadPoolExecutor.runWorker
(ThreadPoolExecutor.java:1,142)
[16] java.util.concurrent.ThreadPoolExecutor$Worker.run
(ThreadPoolExecutor.java:617)
[17] java.lang.Thread.run (Thread.java:748)
now ,we are at [1]
org.apache.calcite.runtime.SqlFunctions.internalToTimestamp
(SqlFunctions.java:1,747)
and the code is as follows:
` public static Timestamp internalToTimestamp(long v)
{
return new Timestamp(v - LOCAL_TZ.getOffset(v));
}
`
let us print the value of windowStart:v
print v
v = 154407483
let us print the value of windowEnd:v
print v
v = 1544074833000
after this, come back to
[1]
org.apache.flink.table.runtime.aggregate.TimeWindowPropertyCollector.collect
(TimeWindowPropertyCollector.scala:51)
then,we will execute
`
if (windowStartOffset.isDefined) {
output.setField(
lastFieldPos + windowStartOffset.get,
SqlFunctions.internalToTimestamp(windowStart))
}
if (windowEndOffset.isDefined) {
output.setField(
lastFieldPos + windowEndOffset.get,
SqlFunctions.internalToTimestamp(windowEnd))
}
`
before execute,the output is
output =
"pro0,throwable0,ERROR,ip0,1,ymm-appmetric-dev-self1_5_924367729,null,null,null"
after execute,the output is
output =
"pro0,throwable0,ERROR,ip0,1,ymm-appmetric-dev-self1_5_924367729,2018-12-06
05:40:30.0,2018-12-06 05:40:33.0,null"
so,do you think the
long value 154407483 translated to be 2018-12-06 05:40:30.0
long value 1544074833000 translated to be 2018-12-06 05:40:33.0
would be right?
I am in China, I think the timestamp should be 2018-12-06 13:40:30.0 and
2018-12-06 13:40:33.0
okay,let us continue
now ,the data will be write to kafka,before write ,the data will be
serialized
let us see what happened!
the call stack is as follows:
` [1]
org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.ser.std.DateSerializer._timestamp
(DateSerializer.java:41)
[2]
org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.ser.std.DateSerializer.serialize
(DateSerializer.java:48)
[3]
org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.ser.std.DateSerializer.serialize
(DateSerializer.java:15)
[4]
org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.ser.DefaultSerializerProvider.serializeValue
(DefaultSerializerProvider.java:130)
[5]
[jira] [Commented] (FLINK-11010) Flink SQL timestamp is inconsistent with currentProcessingTime()
[
https://issues.apache.org/jira/browse/FLINK-11010?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16711023#comment-16711023
]
ASF GitHub Bot commented on FLINK-11010:
lzqdename commented on issue #7180: [FLINK-11010] [TABLE] Flink SQL timestamp
is inconsistent with currentProcessingTime()
URL: https://github.com/apache/flink/pull/7180#issuecomment-444762927
let me show how to generate the wrong result
---
**background**: processing time in tumbling window flink:1.5.0
the invoke stack is as follows:
[1] org.apache.calcite.runtime.SqlFunctions.internalToTimestamp
(SqlFunctions.java:1,747)
[2]
org.apache.flink.table.runtime.aggregate.TimeWindowPropertyCollector.collect
(TimeWindowPropertyCollector.scala:53)
[3]
org.apache.flink.table.runtime.aggregate.IncrementalAggregateWindowFunction.apply
(IncrementalAggregateWindowFunction.scala:74)
[4]
org.apache.flink.table.runtime.aggregate.IncrementalAggregateTimeWindowFunction.apply
(IncrementalAggregateTimeWindowFunction.scala:72)
[5]
org.apache.flink.table.runtime.aggregate.IncrementalAggregateTimeWindowFunction.apply
(IncrementalAggregateTimeWindowFunction.scala:39)
[6]
org.apache.flink.streaming.runtime.operators.windowing.functions.InternalSingleValueWindowFunction.process
(InternalSingleValueWindowFunction.java:46)
[7]
org.apache.flink.streaming.runtime.operators.windowing.WindowOperator.emitWindowContents
(WindowOperator.java:550)
[8]
org.apache.flink.streaming.runtime.operators.windowing.WindowOperator.onProcessingTime
(WindowOperator.java:505)
[9]
org.apache.flink.streaming.api.operators.HeapInternalTimerService.onProcessingTime
(HeapInternalTimerService.java:266)
[10]
org.apache.flink.streaming.runtime.tasks.SystemProcessingTimeService$TriggerTask.run
(SystemProcessingTimeService.java:281)
[11] java.util.concurrent.Executors$RunnableAdapter.call
(Executors.java:511)
[12] java.util.concurrent.FutureTask.run (FutureTask.java:266)
[13]
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201
(ScheduledThreadPoolExecutor.java:180)
[14]
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run
(ScheduledThreadPoolExecutor.java:293)
[15] java.util.concurrent.ThreadPoolExecutor.runWorker
(ThreadPoolExecutor.java:1,142)
[16] java.util.concurrent.ThreadPoolExecutor$Worker.run
(ThreadPoolExecutor.java:617)
[17] java.lang.Thread.run (Thread.java:748)
now ,we are at [1]
org.apache.calcite.runtime.SqlFunctions.internalToTimestamp
(SqlFunctions.java:1,747)
and the code is as follows:
` public static Timestamp internalToTimestamp(long v)
{
return new Timestamp(v - LOCAL_TZ.getOffset(v));
}
`
let us print the value of windowStart:v
print v
v = 154407483
let us print the value of windowEnd:v
print v
v = 1544074833000
after this, come back to
[1]
org.apache.flink.table.runtime.aggregate.TimeWindowPropertyCollector.collect
(TimeWindowPropertyCollector.scala:51)
then,we will execute
`if (windowStartOffset.isDefined) {
output.setField(
lastFieldPos + windowStartOffset.get,
SqlFunctions.internalToTimestamp(windowStart))
}
`
before execute,the output is
output =
"pro0,throwable0,ERROR,ip0,1,ymm-appmetric-dev-self1_5_924367729,null,null,null"
after execute,the output is
output =
"pro0,throwable0,ERROR,ip0,1,ymm-appmetric-dev-self1_5_924367729,2018-12-06
05:40:30.0,2018-12-06 05:40:33.0,null"
so,do you think the
long value 154407483 translated to be 2018-12-06 05:40:30.0
long value 1544074833000 translated to be 2018-12-06 05:40:33.0
would be right?
I am in China, I think the timestamp should be 2018-12-06 13:40:30.0 and
2018-12-06 13:40:33.0
okay,let us continue
now ,the data will be write to kafka,before write ,the data will be
serialized
let us see what happened!
the call stack is as follows:
` [1]
org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.ser.std.DateSerializer._timestamp
(DateSerializer.java:41)
[2]
org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.ser.std.DateSerializer.serialize
(DateSerializer.java:48)
[3]
org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.ser.std.DateSerializer.serialize
(DateSerializer.java:15)
[4]
org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.ser.DefaultSerializerProvider.serializeValue
(DefaultSerializerProvider.java:130)
[5]
org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.ObjectMapper.writeValue
(ObjectMapper.java:2,444)
[6]
org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.ObjectMapper.valueToTree
(ObjectMapper.java:2,586)
[7]
[jira] [Commented] (FLINK-11010) Flink SQL timestamp is inconsistent with currentProcessingTime()
[ https://issues.apache.org/jira/browse/FLINK-11010?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16710996#comment-16710996 ] ASF GitHub Bot commented on FLINK-11010: lzqdename removed a comment on issue #7180: [FLINK-11010] [TABLE] Flink SQL timestamp is inconsistent with currentProcessingTime() URL: https://github.com/apache/flink/pull/7180#issuecomment-444723623 In flink, I think it is better to avoid timezone in calculation ,because the input is always UTC This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Flink SQL timestamp is inconsistent with currentProcessingTime() > > > Key: FLINK-11010 > URL: https://issues.apache.org/jira/browse/FLINK-11010 > Project: Flink > Issue Type: Bug > Components: Table API SQL >Affects Versions: 1.6.2 >Reporter: lamber-ken >Assignee: lamber-ken >Priority: Major > Labels: pull-request-available > > Flink SQL timestamp is inconsistent with currentProcessingTime(). > > the ProcessingTime is just implemented by invoking System.currentTimeMillis() > but the long value will be automatically wrapped to a Timestamp with the > following statement: > `new java.sql.Timestamp(time - TimeZone.getDefault().getOffset(time));` -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-11010) Flink SQL timestamp is inconsistent with currentProcessingTime()
[ https://issues.apache.org/jira/browse/FLINK-11010?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16710899#comment-16710899 ] ASF GitHub Bot commented on FLINK-11010: lzqdename commented on issue #7180: [FLINK-11010] [TABLE] Flink SQL timestamp is inconsistent with currentProcessingTime() URL: https://github.com/apache/flink/pull/7180#issuecomment-444723623 In flink, I think it is better to avoid timezone in calculation ,because the input is always UTC This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Flink SQL timestamp is inconsistent with currentProcessingTime() > > > Key: FLINK-11010 > URL: https://issues.apache.org/jira/browse/FLINK-11010 > Project: Flink > Issue Type: Bug > Components: Table API SQL >Affects Versions: 1.6.2 >Reporter: lamber-ken >Assignee: lamber-ken >Priority: Major > Labels: pull-request-available > > Flink SQL timestamp is inconsistent with currentProcessingTime(). > > the ProcessingTime is just implemented by invoking System.currentTimeMillis() > but the long value will be automatically wrapped to a Timestamp with the > following statement: > `new java.sql.Timestamp(time - TimeZone.getDefault().getOffset(time));` -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-11010) Flink SQL timestamp is inconsistent with currentProcessingTime()
[ https://issues.apache.org/jira/browse/FLINK-11010?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16709039#comment-16709039 ] ASF GitHub Bot commented on FLINK-11010: lamber-ken commented on issue #7180: [FLINK-11010] [TABLE] Flink SQL timestamp is inconsistent with currentProcessingTime() URL: https://github.com/apache/flink/pull/7180#issuecomment-444193677 hi, @tillrohrmann, the ci build failed, but I can't find something helpful for me from log, how to rebuild ? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Flink SQL timestamp is inconsistent with currentProcessingTime() > > > Key: FLINK-11010 > URL: https://issues.apache.org/jira/browse/FLINK-11010 > Project: Flink > Issue Type: Bug > Components: Table API SQL >Affects Versions: 1.6.2 >Reporter: lamber-ken >Assignee: lamber-ken >Priority: Major > Labels: pull-request-available > > Flink SQL timestamp is inconsistent with currentProcessingTime(). > > the ProcessingTime is just implemented by invoking System.currentTimeMillis() > but the long value will be automatically wrapped to a Timestamp with the > following statement: > `new java.sql.Timestamp(time - TimeZone.getDefault().getOffset(time));` -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-11010) Flink SQL timestamp is inconsistent with currentProcessingTime()
[ https://issues.apache.org/jira/browse/FLINK-11010?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16709044#comment-16709044 ] ASF GitHub Bot commented on FLINK-11010: lamber-ken edited a comment on issue #7180: [FLINK-11010] [TABLE] Flink SQL timestamp is inconsistent with currentProcessingTime() URL: https://github.com/apache/flink/pull/7180#issuecomment-444193677 hi, @tillrohrmann, the ci build failed, but I can't find something helpful for me from log, how to rebuild ? here is [ci log detail](https://travis-ci.org/apache/flink/jobs/463223773) This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Flink SQL timestamp is inconsistent with currentProcessingTime() > > > Key: FLINK-11010 > URL: https://issues.apache.org/jira/browse/FLINK-11010 > Project: Flink > Issue Type: Bug > Components: Table API SQL >Affects Versions: 1.6.2 >Reporter: lamber-ken >Assignee: lamber-ken >Priority: Major > Labels: pull-request-available > > Flink SQL timestamp is inconsistent with currentProcessingTime(). > > the ProcessingTime is just implemented by invoking System.currentTimeMillis() > but the long value will be automatically wrapped to a Timestamp with the > following statement: > `new java.sql.Timestamp(time - TimeZone.getDefault().getOffset(time));` -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-11010) Flink SQL timestamp is inconsistent with currentProcessingTime()
[ https://issues.apache.org/jira/browse/FLINK-11010?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16708438#comment-16708438 ] ASF GitHub Bot commented on FLINK-11010: lamber-ken commented on issue #7180: [FLINK-11010] [TABLE] Flink SQL timestamp is inconsistent with currentProcessingTime() URL: https://github.com/apache/flink/pull/7180#issuecomment-444025862 @walterddr , hi, I had add process time. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Flink SQL timestamp is inconsistent with currentProcessingTime() > > > Key: FLINK-11010 > URL: https://issues.apache.org/jira/browse/FLINK-11010 > Project: Flink > Issue Type: Bug > Components: Table API SQL >Affects Versions: 1.6.2 >Reporter: lamber-ken >Assignee: lamber-ken >Priority: Major > Labels: pull-request-available > > Flink SQL timestamp is inconsistent with currentProcessingTime(). > > the ProcessingTime is just implemented by invoking System.currentTimeMillis() > but the long value will be automatically wrapped to a Timestamp with the > following statement: > `new java.sql.Timestamp(time - TimeZone.getDefault().getOffset(time));` -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-11010) Flink SQL timestamp is inconsistent with currentProcessingTime()
[ https://issues.apache.org/jira/browse/FLINK-11010?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16708439#comment-16708439 ] ASF GitHub Bot commented on FLINK-11010: lamber-ken edited a comment on issue #7180: [FLINK-11010] [TABLE] Flink SQL timestamp is inconsistent with currentProcessingTime() URL: https://github.com/apache/flink/pull/7180#issuecomment-444025862 @walterddr , hi, I had add sql proctime test. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Flink SQL timestamp is inconsistent with currentProcessingTime() > > > Key: FLINK-11010 > URL: https://issues.apache.org/jira/browse/FLINK-11010 > Project: Flink > Issue Type: Bug > Components: Table API SQL >Affects Versions: 1.6.2 >Reporter: lamber-ken >Assignee: lamber-ken >Priority: Major > Labels: pull-request-available > > Flink SQL timestamp is inconsistent with currentProcessingTime(). > > the ProcessingTime is just implemented by invoking System.currentTimeMillis() > but the long value will be automatically wrapped to a Timestamp with the > following statement: > `new java.sql.Timestamp(time - TimeZone.getDefault().getOffset(time));` -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-11010) Flink SQL timestamp is inconsistent with currentProcessingTime()
[ https://issues.apache.org/jira/browse/FLINK-11010?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16708440#comment-16708440 ] ASF GitHub Bot commented on FLINK-11010: lamber-ken edited a comment on issue #7180: [FLINK-11010] [TABLE] Flink SQL timestamp is inconsistent with currentProcessingTime() URL: https://github.com/apache/flink/pull/7180#issuecomment-444025862 hi, @walterddr, I had add sql proctime test. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Flink SQL timestamp is inconsistent with currentProcessingTime() > > > Key: FLINK-11010 > URL: https://issues.apache.org/jira/browse/FLINK-11010 > Project: Flink > Issue Type: Bug > Components: Table API SQL >Affects Versions: 1.6.2 >Reporter: lamber-ken >Assignee: lamber-ken >Priority: Major > Labels: pull-request-available > > Flink SQL timestamp is inconsistent with currentProcessingTime(). > > the ProcessingTime is just implemented by invoking System.currentTimeMillis() > but the long value will be automatically wrapped to a Timestamp with the > following statement: > `new java.sql.Timestamp(time - TimeZone.getDefault().getOffset(time));` -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-11010) Flink SQL timestamp is inconsistent with currentProcessingTime()
[ https://issues.apache.org/jira/browse/FLINK-11010?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16706400#comment-16706400 ] ASF GitHub Bot commented on FLINK-11010: walterddr edited a comment on issue #7180: [FLINK-11010] [TABLE] Flink SQL timestamp is inconsistent with currentProcessingTime() URL: https://github.com/apache/flink/pull/7180#issuecomment-443522584 Hi @lamber-ken . the minimum reproduction example should be a good starting point for adding test/ITCase to justify your change. I also commented on the JIRA ticket as well. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Flink SQL timestamp is inconsistent with currentProcessingTime() > > > Key: FLINK-11010 > URL: https://issues.apache.org/jira/browse/FLINK-11010 > Project: Flink > Issue Type: Bug > Components: Table API SQL >Affects Versions: 1.6.2 >Reporter: lamber-ken >Assignee: lamber-ken >Priority: Major > Labels: pull-request-available > > Flink SQL timestamp is inconsistent with currentProcessingTime(). > > the ProcessingTime is just implemented by invoking System.currentTimeMillis() > but the long value will be automatically wrapped to a Timestamp with the > following statement: > `new java.sql.Timestamp(time - TimeZone.getDefault().getOffset(time));` -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-11010) Flink SQL timestamp is inconsistent with currentProcessingTime()
[
https://issues.apache.org/jira/browse/FLINK-11010?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16706398#comment-16706398
]
Rong Rong commented on FLINK-11010:
---
Based on the discussion in the mailing list:
http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Question-about-Timestamp-in-Flink-SQL-td16928.html,
I think Flink's internal time service using GMT as default would require a
more sophisticated solution.
Currently I think the best solution is to use DATE_FORMAT or other SQL
timezone-based operation to convert proctime. and always treat proctime as GMT
on queries like: {{select proctime from tbl}}.
I've also linked https://issues.apache.org/jira/browse/FLINK-8353 for the
reference.
> Flink SQL timestamp is inconsistent with currentProcessingTime()
>
>
> Key: FLINK-11010
> URL: https://issues.apache.org/jira/browse/FLINK-11010
> Project: Flink
> Issue Type: Bug
> Components: Table API SQL
>Affects Versions: 1.6.2
>Reporter: lamber-ken
>Assignee: lamber-ken
>Priority: Major
> Labels: pull-request-available
>
> Flink SQL timestamp is inconsistent with currentProcessingTime().
>
> the ProcessingTime is just implemented by invoking System.currentTimeMillis()
> but the long value will be automatically wrapped to a Timestamp with the
> following statement:
> `new java.sql.Timestamp(time - TimeZone.getDefault().getOffset(time));`
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-11010) Flink SQL timestamp is inconsistent with currentProcessingTime()
[ https://issues.apache.org/jira/browse/FLINK-11010?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16706399#comment-16706399 ] ASF GitHub Bot commented on FLINK-11010: walterddr commented on issue #7180: [FLINK-11010] [TABLE] Flink SQL timestamp is inconsistent with currentProcessingTime() URL: https://github.com/apache/flink/pull/7180#issuecomment-443522584 Hi @lamber-ken . the minimum reproduction result should be a good starting point for adding test/ITCase to justify your change. I also commented on the JIRA ticket as well. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Flink SQL timestamp is inconsistent with currentProcessingTime() > > > Key: FLINK-11010 > URL: https://issues.apache.org/jira/browse/FLINK-11010 > Project: Flink > Issue Type: Bug > Components: Table API SQL >Affects Versions: 1.6.2 >Reporter: lamber-ken >Assignee: lamber-ken >Priority: Major > Labels: pull-request-available > > Flink SQL timestamp is inconsistent with currentProcessingTime(). > > the ProcessingTime is just implemented by invoking System.currentTimeMillis() > but the long value will be automatically wrapped to a Timestamp with the > following statement: > `new java.sql.Timestamp(time - TimeZone.getDefault().getOffset(time));` -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-11010) Flink SQL timestamp is inconsistent with currentProcessingTime()
[ https://issues.apache.org/jira/browse/FLINK-11010?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16706058#comment-16706058 ] ASF GitHub Bot commented on FLINK-11010: lamber-ken commented on issue #7180: [FLINK-11010] [TABLE] Flink SQL timestamp is inconsistent with currentProcessingTime() URL: https://github.com/apache/flink/pull/7180#issuecomment-443472946 Thanks for review @walterdd. Yes, Ok, I'm very glad to. read more about [Calcite SqlFunctions#internalToTime](https://github.com/apache/calcite/blob/1d2929105f47b944c65529fe59b3072cce167116/core/src/main/java/org/apache/calcite/runtime/SqlFunctions.java#L1751). and if I give it an event time with unix timestamp 0, then I got the Timestamp(-2880). I am confused why `internalToTimestamp` need to subtract the offset? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Flink SQL timestamp is inconsistent with currentProcessingTime() > > > Key: FLINK-11010 > URL: https://issues.apache.org/jira/browse/FLINK-11010 > Project: Flink > Issue Type: Bug > Components: Table API SQL >Affects Versions: 1.6.2 >Reporter: lamber-ken >Assignee: lamber-ken >Priority: Major > Labels: pull-request-available > > Flink SQL timestamp is inconsistent with currentProcessingTime(). > > the ProcessingTime is just implemented by invoking System.currentTimeMillis() > but the long value will be automatically wrapped to a Timestamp with the > following statement: > `new java.sql.Timestamp(time - TimeZone.getDefault().getOffset(time));` -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-11010) Flink SQL timestamp is inconsistent with currentProcessingTime()
[
https://issues.apache.org/jira/browse/FLINK-11010?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16700116#comment-16700116
]
ASF GitHub Bot commented on FLINK-11010:
lamber-ken edited a comment on issue #7180: [FLINK-11010] [TABLE] Flink SQL
timestamp is inconsistent with currentProcessingTime()
URL: https://github.com/apache/flink/pull/7180#issuecomment-441977232
more discussion details, please read [Question about Timestamp in Flink SQL
](http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Question-about-Timestamp-in-Flink-SQL-td16928.html)
## minimal reproducible example
```
public static void main(String[] args) throws Exception {
StreamExecutionEnvironment env =
StreamExecutionEnvironment.getExecutionEnvironment();
StreamTableEnvironment tableEnv =
TableEnvironment.getTableEnvironment(env);
DataStreamSource socket = env.socketTextStream("localhost", );
tableEnv.registerDataStream("orders", socket, "user,proctime.proctime");
Table result = tableEnv.sqlQuery("select proctime from orders");
tableEnv
.toRetractStream(result, TypeInformation.of(Row.class))
.addSink(new SinkFunction>() {
@Override
public void invoke(Tuple2 value, Context context)
throws Exception {
String sysTime = new SimpleDateFormat("-MM-dd
HH:mm:ss.SSS").format(new Date());
String procTime = value.f1.toString();
System.out.println("sysTime: " + sysTime );
System.out.println("procTime: " + procTime );
System.out.println();
}
});
env.execute();
}
```
## result
```
sysTime: 2018-11-27 17:29:53.744
procTime: 2018-11-27 09:29:53.744
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Flink SQL timestamp is inconsistent with currentProcessingTime()
>
>
> Key: FLINK-11010
> URL: https://issues.apache.org/jira/browse/FLINK-11010
> Project: Flink
> Issue Type: Bug
> Components: Table API SQL
>Affects Versions: 1.6.2
>Reporter: lamber-ken
>Assignee: lamber-ken
>Priority: Major
> Labels: pull-request-available
>
> Flink SQL timestamp is inconsistent with currentProcessingTime().
>
> the ProcessingTime is just implemented by invoking System.currentTimeMillis()
> but the long value will be automatically wrapped to a Timestamp with the
> following statement:
> `new java.sql.Timestamp(time - TimeZone.getDefault().getOffset(time));`
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-11010) Flink SQL timestamp is inconsistent with currentProcessingTime()
[
https://issues.apache.org/jira/browse/FLINK-11010?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16700094#comment-16700094
]
ASF GitHub Bot commented on FLINK-11010:
lamber-ken edited a comment on issue #7180: [FLINK-11010] [TABLE] Flink SQL
timestamp is inconsistent with currentProcessingTime()
URL: https://github.com/apache/flink/pull/7180#issuecomment-441977232
more discussion details, please read [Question about Timestamp in Flink SQL
](http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Question-about-Timestamp-in-Flink-SQL-td16928.html)
## minimal reproducible example
```
public static void main(String[] args) throws Exception {
StreamExecutionEnvironment env =
StreamExecutionEnvironment.getExecutionEnvironment();
StreamTableEnvironment tableEnv =
TableEnvironment.getTableEnvironment(env);
DataStreamSource socket = env.socketTextStream("localhost",
);
tableEnv.registerDataStream("orders", socket,
"user,proctime.proctime");
Table result = tableEnv.sqlQuery("select proctime from orders");
tableEnv
.toRetractStream(result, TypeInformation.of(Row.class))
.print();
env.execute();
}
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Flink SQL timestamp is inconsistent with currentProcessingTime()
>
>
> Key: FLINK-11010
> URL: https://issues.apache.org/jira/browse/FLINK-11010
> Project: Flink
> Issue Type: Bug
> Components: Table API SQL
>Affects Versions: 1.6.2
>Reporter: lamber-ken
>Assignee: lamber-ken
>Priority: Major
> Labels: pull-request-available
>
> Flink SQL timestamp is inconsistent with currentProcessingTime().
>
> the ProcessingTime is just implemented by invoking System.currentTimeMillis()
> but the long value will be automatically wrapped to a Timestamp with the
> following statement:
> `new java.sql.Timestamp(time - TimeZone.getDefault().getOffset(time));`
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-11010) Flink SQL timestamp is inconsistent with currentProcessingTime()
[ https://issues.apache.org/jira/browse/FLINK-11010?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16700080#comment-16700080 ] ASF GitHub Bot commented on FLINK-11010: lamber-ken commented on issue #7180: [FLINK-11010] [TABLE] Flink SQL timestamp is inconsistent with currentProcessingTime() URL: https://github.com/apache/flink/pull/7180#issuecomment-441977232 more discussion details, please read [Question about Timestamp in Flink SQL ](http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Question-about-Timestamp-in-Flink-SQL-td16928.html) This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Flink SQL timestamp is inconsistent with currentProcessingTime() > > > Key: FLINK-11010 > URL: https://issues.apache.org/jira/browse/FLINK-11010 > Project: Flink > Issue Type: Bug > Components: Table API SQL >Affects Versions: 1.6.2 >Reporter: lamber-ken >Assignee: lamber-ken >Priority: Major > Labels: pull-request-available > > Flink SQL timestamp is inconsistent with currentProcessingTime(). > > the ProcessingTime is just implemented by invoking System.currentTimeMillis() > but the long value will be automatically wrapped to a Timestamp with the > following statement: > `new java.sql.Timestamp(time - TimeZone.getDefault().getOffset(time));` -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-11010) Flink SQL timestamp is inconsistent with currentProcessingTime()
[
https://issues.apache.org/jira/browse/FLINK-11010?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16700083#comment-16700083
]
ASF GitHub Bot commented on FLINK-11010:
lamber-ken edited a comment on issue #7180: [FLINK-11010] [TABLE] Flink SQL
timestamp is inconsistent with currentProcessingTime()
URL: https://github.com/apache/flink/pull/7180#issuecomment-441977232
more discussion details, please read [Question about Timestamp in Flink SQL
](http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Question-about-Timestamp-in-Flink-SQL-td16928.html)
## minimal reproducible example
```
public static void main(String[] args) throws Exception {
StreamExecutionEnvironment env =
StreamExecutionEnvironment.getExecutionEnvironment();
StreamTableEnvironment tableEnv =
TableEnvironment.getTableEnvironment(env);
DataStreamSource socket = env.socketTextStream("localhost",
);
tableEnv.registerDataStream("orders", socket,
"user,proctime.proctime");
Table result = tableEnv.sqlQuery("select proctime from orders");
tableEnv
.toRetractStream(result, TypeInformation.of(Row.class))
.print();
env.execute();
}
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Flink SQL timestamp is inconsistent with currentProcessingTime()
>
>
> Key: FLINK-11010
> URL: https://issues.apache.org/jira/browse/FLINK-11010
> Project: Flink
> Issue Type: Bug
> Components: Table API SQL
>Affects Versions: 1.6.2
>Reporter: lamber-ken
>Assignee: lamber-ken
>Priority: Major
> Labels: pull-request-available
>
> Flink SQL timestamp is inconsistent with currentProcessingTime().
>
> the ProcessingTime is just implemented by invoking System.currentTimeMillis()
> but the long value will be automatically wrapped to a Timestamp with the
> following statement:
> `new java.sql.Timestamp(time - TimeZone.getDefault().getOffset(time));`
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-11010) Flink SQL timestamp is inconsistent with currentProcessingTime()
[ https://issues.apache.org/jira/browse/FLINK-11010?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16700072#comment-16700072 ] ASF GitHub Bot commented on FLINK-11010: lamber-ken opened a new pull request #7180: [FLINK-11010] [TABLE] Flink SQL timestamp is inconsistent with currentProcessingTime() URL: https://github.com/apache/flink/pull/7180 ## What is the purpose of the change the ProcessingTime is just implemented by invoking System.currentTimeMillis() but the long value will be automatically wrapped to a Timestamp with the following statement: `new java.sql.Timestamp(time - TimeZone.getDefault().getOffset(time));` ## Brief change log `org.apache.flink.table.codegen.CodeGenerator#generateProctimeTimestamp` add Default TimeZone#offset ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (no) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (no) - The serializers: (no) - The runtime per-record code paths (performance sensitive): (no) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: (no) - The S3 file system connector: (no) ## Documentation - Does this pull request introduce a new feature? (no) - If yes, how is the feature documented? (no) This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Flink SQL timestamp is inconsistent with currentProcessingTime() > > > Key: FLINK-11010 > URL: https://issues.apache.org/jira/browse/FLINK-11010 > Project: Flink > Issue Type: Bug > Components: Table API SQL >Affects Versions: 1.6.2 >Reporter: lamber-ken >Assignee: lamber-ken >Priority: Major > Labels: pull-request-available > > Flink SQL timestamp is inconsistent with currentProcessingTime(). > > the ProcessingTime is just implemented by invoking System.currentTimeMillis() > but the long value will be automatically wrapped to a Timestamp with the > following statement: > `new java.sql.Timestamp(time - TimeZone.getDefault().getOffset(time));` -- This message was sent by Atlassian JIRA (v7.6.3#76005)
