[jira] [Commented] (FLINK-3856) Create types for java.sql.Date/Time/Timestamp
[
https://issues.apache.org/jira/browse/FLINK-3856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15282854#comment-15282854
]
Fabian Hueske commented on FLINK-3856:
--
Thanks for taking care of this Max!
> Create types for java.sql.Date/Time/Timestamp
> -
>
> Key: FLINK-3856
> URL: https://issues.apache.org/jira/browse/FLINK-3856
> Project: Flink
> Issue Type: New Feature
> Components: Core
>Reporter: Timo Walther
>Assignee: Timo Walther
> Fix For: 1.1.0
>
>
> At the moment there is only the {{Date}} type which is not sufficient for
> most use cases about time.
> The Table API would also benefit from having different types as output result.
> I would propose to add the three {{java.sql.}} types either as {{BasicTypes}}
> or in an additional class {{TimeTypes}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-3856) Create types for java.sql.Date/Time/Timestamp
[
https://issues.apache.org/jira/browse/FLINK-3856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15282843#comment-15282843
]
Maximilian Michels commented on FLINK-3856:
---
There is something wrong with our Maven configuration. You can see from the
test output that the test failed before but still Maven supported SUCCESS:
https://travis-ci.org/apache/flink/jobs/129943398#L5402
> Create types for java.sql.Date/Time/Timestamp
> -
>
> Key: FLINK-3856
> URL: https://issues.apache.org/jira/browse/FLINK-3856
> Project: Flink
> Issue Type: New Feature
> Components: Core
>Reporter: Timo Walther
>Assignee: Timo Walther
> Fix For: 1.1.0
>
>
> At the moment there is only the {{Date}} type which is not sufficient for
> most use cases about time.
> The Table API would also benefit from having different types as output result.
> I would propose to add the three {{java.sql.}} types either as {{BasicTypes}}
> or in an additional class {{TimeTypes}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-3856) Create types for java.sql.Date/Time/Timestamp
[
https://issues.apache.org/jira/browse/FLINK-3856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15282839#comment-15282839
]
Maximilian Michels commented on FLINK-3856:
---
Additional fix with 96b353d98f6b6d441ebedf69ec12cfa333a1d7c9.
> Create types for java.sql.Date/Time/Timestamp
> -
>
> Key: FLINK-3856
> URL: https://issues.apache.org/jira/browse/FLINK-3856
> Project: Flink
> Issue Type: New Feature
> Components: Core
>Reporter: Timo Walther
>Assignee: Timo Walther
> Fix For: 1.1.0
>
>
> At the moment there is only the {{Date}} type which is not sufficient for
> most use cases about time.
> The Table API would also benefit from having different types as output result.
> I would propose to add the three {{java.sql.}} types either as {{BasicTypes}}
> or in an additional class {{TimeTypes}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-3856) Create types for java.sql.Date/Time/Timestamp
[
https://issues.apache.org/jira/browse/FLINK-3856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15282704#comment-15282704
]
Fabian Hueske commented on FLINK-3856:
--
This is weird.
I ran Travis before committing and all five builds succeeded.
However, I can also reproduce the problem locally. :-/
> Create types for java.sql.Date/Time/Timestamp
> -
>
> Key: FLINK-3856
> URL: https://issues.apache.org/jira/browse/FLINK-3856
> Project: Flink
> Issue Type: New Feature
> Components: Core
>Reporter: Timo Walther
>Assignee: Timo Walther
> Fix For: 1.1.0
>
>
> At the moment there is only the {{Date}} type which is not sufficient for
> most use cases about time.
> The Table API would also benefit from having different types as output result.
> I would propose to add the three {{java.sql.}} types either as {{BasicTypes}}
> or in an additional class {{TimeTypes}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-3856) Create types for java.sql.Date/Time/Timestamp
[
https://issues.apache.org/jira/browse/FLINK-3856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15282696#comment-15282696
]
Maximilian Michels commented on FLINK-3856:
---
This breaks {{GroupReduceITCase.testGroupByGenericType}} because it checks for
{{
Assert.assertTrue(ec.getRegisteredKryoTypes().contains(java.sql.Date.class));}}.
Fixing this while merging FLINK-3701.
> Create types for java.sql.Date/Time/Timestamp
> -
>
> Key: FLINK-3856
> URL: https://issues.apache.org/jira/browse/FLINK-3856
> Project: Flink
> Issue Type: New Feature
> Components: Core
>Reporter: Timo Walther
>Assignee: Timo Walther
> Fix For: 1.1.0
>
>
> At the moment there is only the {{Date}} type which is not sufficient for
> most use cases about time.
> The Table API would also benefit from having different types as output result.
> I would propose to add the three {{java.sql.}} types either as {{BasicTypes}}
> or in an additional class {{TimeTypes}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-3856) Create types for java.sql.Date/Time/Timestamp
[
https://issues.apache.org/jira/browse/FLINK-3856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15279900#comment-15279900
]
ASF GitHub Bot commented on FLINK-3856:
---
Github user twalthr commented on a diff in the pull request:
https://github.com/apache/flink/pull/1959#discussion_r62823533
--- Diff:
flink-core/src/main/java/org/apache/flink/api/common/typeinfo/SqlTimeTypeInfo.java
---
@@ -0,0 +1,177 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.api.common.typeinfo;
+
+import java.lang.reflect.Constructor;
+import java.sql.Date;
+import java.sql.Time;
+import java.sql.Timestamp;
+import java.util.Objects;
+import org.apache.flink.annotation.PublicEvolving;
+import org.apache.flink.api.common.ExecutionConfig;
+import org.apache.flink.api.common.typeutils.TypeComparator;
+import org.apache.flink.api.common.typeutils.TypeSerializer;
+import org.apache.flink.api.common.typeutils.base.DateComparator;
+import org.apache.flink.api.common.typeutils.base.SqlDateSerializer;
+import org.apache.flink.api.common.typeutils.base.SqlTimeSerializer;
+import org.apache.flink.api.common.typeutils.base.SqlTimestampComparator;
+import org.apache.flink.api.common.typeutils.base.SqlTimestampSerializer;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+/**
+ * Type information for Java SQL Date/Time/Timestamp.
+ */
+@PublicEvolving
+public class SqlTimeTypeInfo extends TypeInformation implements
AtomicType {
--- End diff --
Ok, I will fix the unused import issues and merge once travis build passed.
> Create types for java.sql.Date/Time/Timestamp
> -
>
> Key: FLINK-3856
> URL: https://issues.apache.org/jira/browse/FLINK-3856
> Project: Flink
> Issue Type: New Feature
> Components: Core
>Reporter: Timo Walther
>Assignee: Timo Walther
>
> At the moment there is only the {{Date}} type which is not sufficient for
> most use cases about time.
> The Table API would also benefit from having different types as output result.
> I would propose to add the three {{java.sql.}} types either as {{BasicTypes}}
> or in an additional class {{TimeTypes}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-3856) Create types for java.sql.Date/Time/Timestamp
[
https://issues.apache.org/jira/browse/FLINK-3856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15279898#comment-15279898
]
ASF GitHub Bot commented on FLINK-3856:
---
Github user fhueske commented on a diff in the pull request:
https://github.com/apache/flink/pull/1959#discussion_r62823173
--- Diff:
flink-core/src/main/java/org/apache/flink/api/common/typeinfo/SqlTimeTypeInfo.java
---
@@ -0,0 +1,177 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.api.common.typeinfo;
+
+import java.lang.reflect.Constructor;
+import java.sql.Date;
+import java.sql.Time;
+import java.sql.Timestamp;
+import java.util.Objects;
+import org.apache.flink.annotation.PublicEvolving;
+import org.apache.flink.api.common.ExecutionConfig;
+import org.apache.flink.api.common.typeutils.TypeComparator;
+import org.apache.flink.api.common.typeutils.TypeSerializer;
+import org.apache.flink.api.common.typeutils.base.DateComparator;
+import org.apache.flink.api.common.typeutils.base.SqlDateSerializer;
+import org.apache.flink.api.common.typeutils.base.SqlTimeSerializer;
+import org.apache.flink.api.common.typeutils.base.SqlTimestampComparator;
+import org.apache.flink.api.common.typeutils.base.SqlTimestampSerializer;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+/**
+ * Type information for Java SQL Date/Time/Timestamp.
+ */
+@PublicEvolving
+public class SqlTimeTypeInfo extends TypeInformation implements
AtomicType {
--- End diff --
I agree. Let's stick to this solution then.
> Create types for java.sql.Date/Time/Timestamp
> -
>
> Key: FLINK-3856
> URL: https://issues.apache.org/jira/browse/FLINK-3856
> Project: Flink
> Issue Type: New Feature
> Components: Core
>Reporter: Timo Walther
>Assignee: Timo Walther
>
> At the moment there is only the {{Date}} type which is not sufficient for
> most use cases about time.
> The Table API would also benefit from having different types as output result.
> I would propose to add the three {{java.sql.}} types either as {{BasicTypes}}
> or in an additional class {{TimeTypes}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-3856) Create types for java.sql.Date/Time/Timestamp
[
https://issues.apache.org/jira/browse/FLINK-3856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15279891#comment-15279891
]
ASF GitHub Bot commented on FLINK-3856:
---
Github user twalthr commented on a diff in the pull request:
https://github.com/apache/flink/pull/1959#discussion_r62822604
--- Diff:
flink-core/src/main/java/org/apache/flink/api/common/typeinfo/SqlTimeTypeInfo.java
---
@@ -0,0 +1,177 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.api.common.typeinfo;
+
+import java.lang.reflect.Constructor;
+import java.sql.Date;
+import java.sql.Time;
+import java.sql.Timestamp;
+import java.util.Objects;
+import org.apache.flink.annotation.PublicEvolving;
+import org.apache.flink.api.common.ExecutionConfig;
+import org.apache.flink.api.common.typeutils.TypeComparator;
+import org.apache.flink.api.common.typeutils.TypeSerializer;
+import org.apache.flink.api.common.typeutils.base.DateComparator;
+import org.apache.flink.api.common.typeutils.base.SqlDateSerializer;
+import org.apache.flink.api.common.typeutils.base.SqlTimeSerializer;
+import org.apache.flink.api.common.typeutils.base.SqlTimestampComparator;
+import org.apache.flink.api.common.typeutils.base.SqlTimestampSerializer;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+/**
+ * Type information for Java SQL Date/Time/Timestamp.
+ */
+@PublicEvolving
+public class SqlTimeTypeInfo extends TypeInformation implements
AtomicType {
--- End diff --
I also thought about that. But this would also mean that `BasicTypeInfo`'s
`getInfoFor` needs to support types that are declared in an other class. So we
have to add the classes to the `TYPES` of `BasicTypeInfo`. I think complete
separation is a nicer design.
> Create types for java.sql.Date/Time/Timestamp
> -
>
> Key: FLINK-3856
> URL: https://issues.apache.org/jira/browse/FLINK-3856
> Project: Flink
> Issue Type: New Feature
> Components: Core
>Reporter: Timo Walther
>Assignee: Timo Walther
>
> At the moment there is only the {{Date}} type which is not sufficient for
> most use cases about time.
> The Table API would also benefit from having different types as output result.
> I would propose to add the three {{java.sql.}} types either as {{BasicTypes}}
> or in an additional class {{TimeTypes}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-3856) Create types for java.sql.Date/Time/Timestamp
[
https://issues.apache.org/jira/browse/FLINK-3856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15278371#comment-15278371
]
ASF GitHub Bot commented on FLINK-3856:
---
Github user fhueske commented on a diff in the pull request:
https://github.com/apache/flink/pull/1959#discussion_r62702515
--- Diff:
flink-core/src/main/java/org/apache/flink/api/common/typeinfo/SqlTimeTypeInfo.java
---
@@ -0,0 +1,177 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.api.common.typeinfo;
+
+import java.lang.reflect.Constructor;
+import java.sql.Date;
+import java.sql.Time;
+import java.sql.Timestamp;
+import java.util.Objects;
+import org.apache.flink.annotation.PublicEvolving;
+import org.apache.flink.api.common.ExecutionConfig;
+import org.apache.flink.api.common.typeutils.TypeComparator;
+import org.apache.flink.api.common.typeutils.TypeSerializer;
+import org.apache.flink.api.common.typeutils.base.DateComparator;
+import org.apache.flink.api.common.typeutils.base.SqlDateSerializer;
+import org.apache.flink.api.common.typeutils.base.SqlTimeSerializer;
+import org.apache.flink.api.common.typeutils.base.SqlTimestampComparator;
+import org.apache.flink.api.common.typeutils.base.SqlTimestampSerializer;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+/**
+ * Type information for Java SQL Date/Time/Timestamp.
+ */
+@PublicEvolving
+public class SqlTimeTypeInfo extends TypeInformation implements
AtomicType {
--- End diff --
Do we want to implement a new `TypeInformation` which is basically a copy
of `BasicTypeInfo`?
Alternatively we could make `SqlTimeTypeInfo` just a holder for three
`public static final BasicTypeInfo`s of the three time types. The `protected`
constructor of `BasicTypeInfo` is visible if `SqlTimeTypeInfo` is in the same
package as `BasicTypeInfo`.
> Create types for java.sql.Date/Time/Timestamp
> -
>
> Key: FLINK-3856
> URL: https://issues.apache.org/jira/browse/FLINK-3856
> Project: Flink
> Issue Type: New Feature
> Components: Core
>Reporter: Timo Walther
>Assignee: Timo Walther
>
> At the moment there is only the {{Date}} type which is not sufficient for
> most use cases about time.
> The Table API would also benefit from having different types as output result.
> I would propose to add the three {{java.sql.}} types either as {{BasicTypes}}
> or in an additional class {{TimeTypes}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-3856) Create types for java.sql.Date/Time/Timestamp
[
https://issues.apache.org/jira/browse/FLINK-3856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15278378#comment-15278378
]
ASF GitHub Bot commented on FLINK-3856:
---
Github user fhueske commented on the pull request:
https://github.com/apache/flink/pull/1959#issuecomment-218210159
@twalthr, should `SqlTimeTypeInfo` implement `TypeInformation`? What do you
think?
> Create types for java.sql.Date/Time/Timestamp
> -
>
> Key: FLINK-3856
> URL: https://issues.apache.org/jira/browse/FLINK-3856
> Project: Flink
> Issue Type: New Feature
> Components: Core
>Reporter: Timo Walther
>Assignee: Timo Walther
>
> At the moment there is only the {{Date}} type which is not sufficient for
> most use cases about time.
> The Table API would also benefit from having different types as output result.
> I would propose to add the three {{java.sql.}} types either as {{BasicTypes}}
> or in an additional class {{TimeTypes}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-3856) Create types for java.sql.Date/Time/Timestamp
[
https://issues.apache.org/jira/browse/FLINK-3856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15276340#comment-15276340
]
ASF GitHub Bot commented on FLINK-3856:
---
Github user twalthr commented on a diff in the pull request:
https://github.com/apache/flink/pull/1959#discussion_r62495413
--- Diff:
flink-core/src/main/java/org/apache/flink/api/common/typeutils/base/DateComparator.java
---
@@ -59,15 +55,37 @@ public boolean isNormalizedKeyPrefixOnly(int keyBytes) {
}
@Override
- public void putNormalizedKey(Date lValue, MemorySegment target, int
offset, int numBytes) {
- long value = lValue.getTime() - Long.MIN_VALUE;
-
+ public void putNormalizedKey(Date record, MemorySegment target, int
offset, int numBytes) {
+ putNormalizedKeyDate(record, target, offset, numBytes);
+ }
+
+ @Override
+ public DateComparator duplicate() {
+ return new DateComparator(ascendingComparison);
+ }
+
+ //
+ // Static Helpers for Date Comparison
+ //
+
+ public static int compareSerializedDate(DataInputView firstSource,
DataInputView secondSource,
+ boolean ascendingComparison) throws IOException {
+ final long l1 = firstSource.readLong();
+ final long l2 = secondSource.readLong();
+ final int comp = (l1 < l2 ? -1 : (l1 == l2 ? 0 : 1));
+ return ascendingComparison ? comp : -comp;
+ }
+
+ public static void putNormalizedKeyDate(Date record, MemorySegment
target, int offset, int numBytes) {
+ final long value = record.getTime() - Long.MIN_VALUE;
+
// see IntValue for an explanation of the logic
if (numBytes == 8) {
// default case, full normalized key
target.putLongBigEndian(offset, value);
}
else if (numBytes <= 0) {
--- End diff --
Thanks for the explanation. You are right. I have changed it.
> Create types for java.sql.Date/Time/Timestamp
> -
>
> Key: FLINK-3856
> URL: https://issues.apache.org/jira/browse/FLINK-3856
> Project: Flink
> Issue Type: New Feature
> Components: Core
>Reporter: Timo Walther
>Assignee: Timo Walther
>
> At the moment there is only the {{Date}} type which is not sufficient for
> most use cases about time.
> The Table API would also benefit from having different types as output result.
> I would propose to add the three {{java.sql.}} types either as {{BasicTypes}}
> or in an additional class {{TimeTypes}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-3856) Create types for java.sql.Date/Time/Timestamp
[
https://issues.apache.org/jira/browse/FLINK-3856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15276330#comment-15276330
]
ASF GitHub Bot commented on FLINK-3856:
---
Github user fhueske commented on a diff in the pull request:
https://github.com/apache/flink/pull/1959#discussion_r62493248
--- Diff:
flink-core/src/main/java/org/apache/flink/api/common/typeutils/base/DateComparator.java
---
@@ -59,15 +55,37 @@ public boolean isNormalizedKeyPrefixOnly(int keyBytes) {
}
@Override
- public void putNormalizedKey(Date lValue, MemorySegment target, int
offset, int numBytes) {
- long value = lValue.getTime() - Long.MIN_VALUE;
-
+ public void putNormalizedKey(Date record, MemorySegment target, int
offset, int numBytes) {
+ putNormalizedKeyDate(record, target, offset, numBytes);
+ }
+
+ @Override
+ public DateComparator duplicate() {
+ return new DateComparator(ascendingComparison);
+ }
+
+ //
+ // Static Helpers for Date Comparison
+ //
+
+ public static int compareSerializedDate(DataInputView firstSource,
DataInputView secondSource,
+ boolean ascendingComparison) throws IOException {
+ final long l1 = firstSource.readLong();
+ final long l2 = secondSource.readLong();
+ final int comp = (l1 < l2 ? -1 : (l1 == l2 ? 0 : 1));
+ return ascendingComparison ? comp : -comp;
+ }
+
+ public static void putNormalizedKeyDate(Date record, MemorySegment
target, int offset, int numBytes) {
+ final long value = record.getTime() - Long.MIN_VALUE;
+
// see IntValue for an explanation of the logic
if (numBytes == 8) {
// default case, full normalized key
target.putLongBigEndian(offset, value);
}
else if (numBytes <= 0) {
--- End diff --
for `numBytes == 0`, the `numBytes < 8` branch would match and the loop
would not be entered, because `numBytes > 0`. So the check is not necessary and
only adds overhead for some cases. Actually, `numBytes` should be larger than
`0` because otherwise calling this method would not make sense.
> Create types for java.sql.Date/Time/Timestamp
> -
>
> Key: FLINK-3856
> URL: https://issues.apache.org/jira/browse/FLINK-3856
> Project: Flink
> Issue Type: New Feature
> Components: Core
>Reporter: Timo Walther
>Assignee: Timo Walther
>
> At the moment there is only the {{Date}} type which is not sufficient for
> most use cases about time.
> The Table API would also benefit from having different types as output result.
> I would propose to add the three {{java.sql.}} types either as {{BasicTypes}}
> or in an additional class {{TimeTypes}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-3856) Create types for java.sql.Date/Time/Timestamp
[
https://issues.apache.org/jira/browse/FLINK-3856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15276300#comment-15276300
]
ASF GitHub Bot commented on FLINK-3856:
---
Github user twalthr commented on a diff in the pull request:
https://github.com/apache/flink/pull/1959#discussion_r62490034
--- Diff:
flink-core/src/main/java/org/apache/flink/api/common/typeutils/base/DateComparator.java
---
@@ -59,15 +55,37 @@ public boolean isNormalizedKeyPrefixOnly(int keyBytes) {
}
@Override
- public void putNormalizedKey(Date lValue, MemorySegment target, int
offset, int numBytes) {
- long value = lValue.getTime() - Long.MIN_VALUE;
-
+ public void putNormalizedKey(Date record, MemorySegment target, int
offset, int numBytes) {
+ putNormalizedKeyDate(record, target, offset, numBytes);
+ }
+
+ @Override
+ public DateComparator duplicate() {
+ return new DateComparator(ascendingComparison);
+ }
+
+ //
+ // Static Helpers for Date Comparison
+ //
+
+ public static int compareSerializedDate(DataInputView firstSource,
DataInputView secondSource,
+ boolean ascendingComparison) throws IOException {
+ final long l1 = firstSource.readLong();
+ final long l2 = secondSource.readLong();
+ final int comp = (l1 < l2 ? -1 : (l1 == l2 ? 0 : 1));
+ return ascendingComparison ? comp : -comp;
+ }
+
+ public static void putNormalizedKeyDate(Date record, MemorySegment
target, int offset, int numBytes) {
+ final long value = record.getTime() - Long.MIN_VALUE;
+
// see IntValue for an explanation of the logic
if (numBytes == 8) {
// default case, full normalized key
target.putLongBigEndian(offset, value);
}
else if (numBytes <= 0) {
--- End diff --
No, this case is for numBytes of 0.
> Create types for java.sql.Date/Time/Timestamp
> -
>
> Key: FLINK-3856
> URL: https://issues.apache.org/jira/browse/FLINK-3856
> Project: Flink
> Issue Type: New Feature
> Components: Core
>Reporter: Timo Walther
>Assignee: Timo Walther
>
> At the moment there is only the {{Date}} type which is not sufficient for
> most use cases about time.
> The Table API would also benefit from having different types as output result.
> I would propose to add the three {{java.sql.}} types either as {{BasicTypes}}
> or in an additional class {{TimeTypes}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-3856) Create types for java.sql.Date/Time/Timestamp
[
https://issues.apache.org/jira/browse/FLINK-3856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15276222#comment-15276222
]
ASF GitHub Bot commented on FLINK-3856:
---
Github user twalthr commented on the pull request:
https://github.com/apache/flink/pull/1959#issuecomment-217836281
I will move it to `SqlTimeTypeInfo`.
> Create types for java.sql.Date/Time/Timestamp
> -
>
> Key: FLINK-3856
> URL: https://issues.apache.org/jira/browse/FLINK-3856
> Project: Flink
> Issue Type: New Feature
> Components: Core
>Reporter: Timo Walther
>Assignee: Timo Walther
>
> At the moment there is only the {{Date}} type which is not sufficient for
> most use cases about time.
> The Table API would also benefit from having different types as output result.
> I would propose to add the three {{java.sql.}} types either as {{BasicTypes}}
> or in an additional class {{TimeTypes}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-3856) Create types for java.sql.Date/Time/Timestamp
[
https://issues.apache.org/jira/browse/FLINK-3856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15270778#comment-15270778
]
ASF GitHub Bot commented on FLINK-3856:
---
Github user fhueske commented on the pull request:
https://github.com/apache/flink/pull/1959#issuecomment-216893377
Just a few minor comments. What do you think about moving the types to a
`SqlTimeTypeInfo` class, @twalthr? @StephanEwen would that be OK for you?
> Create types for java.sql.Date/Time/Timestamp
> -
>
> Key: FLINK-3856
> URL: https://issues.apache.org/jira/browse/FLINK-3856
> Project: Flink
> Issue Type: New Feature
> Components: Core
>Reporter: Timo Walther
>Assignee: Timo Walther
>
> At the moment there is only the {{Date}} type which is not sufficient for
> most use cases about time.
> The Table API would also benefit from having different types as output result.
> I would propose to add the three {{java.sql.}} types either as {{BasicTypes}}
> or in an additional class {{TimeTypes}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-3856) Create types for java.sql.Date/Time/Timestamp
[
https://issues.apache.org/jira/browse/FLINK-3856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15270777#comment-15270777
]
ASF GitHub Bot commented on FLINK-3856:
---
Github user fhueske commented on a diff in the pull request:
https://github.com/apache/flink/pull/1959#discussion_r62054682
--- Diff:
flink-core/src/main/java/org/apache/flink/api/common/typeutils/base/DateComparator.java
---
@@ -59,15 +55,37 @@ public boolean isNormalizedKeyPrefixOnly(int keyBytes) {
}
@Override
- public void putNormalizedKey(Date lValue, MemorySegment target, int
offset, int numBytes) {
- long value = lValue.getTime() - Long.MIN_VALUE;
-
+ public void putNormalizedKey(Date record, MemorySegment target, int
offset, int numBytes) {
+ putNormalizedKeyDate(record, target, offset, numBytes);
+ }
+
+ @Override
+ public DateComparator duplicate() {
+ return new DateComparator(ascendingComparison);
+ }
+
+ //
+ // Static Helpers for Date Comparison
+ //
+
+ public static int compareSerializedDate(DataInputView firstSource,
DataInputView secondSource,
+ boolean ascendingComparison) throws IOException {
+ final long l1 = firstSource.readLong();
+ final long l2 = secondSource.readLong();
+ final int comp = (l1 < l2 ? -1 : (l1 == l2 ? 0 : 1));
+ return ascendingComparison ? comp : -comp;
+ }
+
+ public static void putNormalizedKeyDate(Date record, MemorySegment
target, int offset, int numBytes) {
+ final long value = record.getTime() - Long.MIN_VALUE;
+
// see IntValue for an explanation of the logic
if (numBytes == 8) {
// default case, full normalized key
target.putLongBigEndian(offset, value);
}
else if (numBytes <= 0) {
--- End diff --
This case can be removed, no?
> Create types for java.sql.Date/Time/Timestamp
> -
>
> Key: FLINK-3856
> URL: https://issues.apache.org/jira/browse/FLINK-3856
> Project: Flink
> Issue Type: New Feature
> Components: Core
>Reporter: Timo Walther
>Assignee: Timo Walther
>
> At the moment there is only the {{Date}} type which is not sufficient for
> most use cases about time.
> The Table API would also benefit from having different types as output result.
> I would propose to add the three {{java.sql.}} types either as {{BasicTypes}}
> or in an additional class {{TimeTypes}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-3856) Create types for java.sql.Date/Time/Timestamp
[
https://issues.apache.org/jira/browse/FLINK-3856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15270774#comment-15270774
]
ASF GitHub Bot commented on FLINK-3856:
---
Github user fhueske commented on a diff in the pull request:
https://github.com/apache/flink/pull/1959#discussion_r62054394
--- Diff:
flink-core/src/test/java/org/apache/flink/api/common/typeutils/base/SqlTimestampSerializerTest.java
---
@@ -0,0 +1,55 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.api.common.typeutils.base;
+
+import java.sql.Timestamp;
+import org.apache.flink.api.common.typeutils.SerializerTestBase;
+import org.apache.flink.api.common.typeutils.TypeSerializer;
+
+/**
+ * A test for the {@link SqlTimestampSerializer}.
+ */
+public class SqlTimestampSerializerTest extends
SerializerTestBase {
+
+ @Override
+ protected TypeSerializer createSerializer() {
+ return new SqlTimestampSerializer();
+ }
+
+ @Override
+ protected int getLength() {
+ return 12;
+ }
+
+ @Override
+ protected Class getTypeClass() {
+ return Timestamp.class;
+ }
+
+ @Override
+ protected Timestamp[] getTestData() {
+ return new Timestamp[] {
+ new Timestamp(0L),
+ Timestamp.valueOf("1970-01-01 00:00:00.000"),
--- End diff --
Add a Timestamp with nanos to check their serialization?
> Create types for java.sql.Date/Time/Timestamp
> -
>
> Key: FLINK-3856
> URL: https://issues.apache.org/jira/browse/FLINK-3856
> Project: Flink
> Issue Type: New Feature
> Components: Core
>Reporter: Timo Walther
>Assignee: Timo Walther
>
> At the moment there is only the {{Date}} type which is not sufficient for
> most use cases about time.
> The Table API would also benefit from having different types as output result.
> I would propose to add the three {{java.sql.}} types either as {{BasicTypes}}
> or in an additional class {{TimeTypes}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-3856) Create types for java.sql.Date/Time/Timestamp
[
https://issues.apache.org/jira/browse/FLINK-3856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15270772#comment-15270772
]
ASF GitHub Bot commented on FLINK-3856:
---
Github user fhueske commented on a diff in the pull request:
https://github.com/apache/flink/pull/1959#discussion_r62054304
--- Diff:
flink-core/src/test/java/org/apache/flink/api/common/typeutils/base/SqlTimestampComparatorTest.java
---
@@ -0,0 +1,48 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.api.common.typeutils.base;
+
+import java.sql.Timestamp;
+import org.apache.flink.api.common.typeutils.ComparatorTestBase;
+import org.apache.flink.api.common.typeutils.TypeComparator;
+import org.apache.flink.api.common.typeutils.TypeSerializer;
+
+public class SqlTimestampComparatorTest extends
ComparatorTestBase {
+
+ @SuppressWarnings("unchecked")
+ @Override
+ protected TypeComparator createComparator(boolean ascending)
{
+ return (TypeComparator) new SqlTimestampComparator(ascending);
+ }
+
+ @Override
+ protected TypeSerializer createSerializer() {
+ return new SqlTimestampSerializer();
+ }
+
+ @Override
+ protected Timestamp[] getSortedTestData() {
+ return new Timestamp[] {
+ Timestamp.valueOf("1970-01-01 00:00:00.000"),
--- End diff --
Add Timestamps that only differ in the nanos?
> Create types for java.sql.Date/Time/Timestamp
> -
>
> Key: FLINK-3856
> URL: https://issues.apache.org/jira/browse/FLINK-3856
> Project: Flink
> Issue Type: New Feature
> Components: Core
>Reporter: Timo Walther
>Assignee: Timo Walther
>
> At the moment there is only the {{Date}} type which is not sufficient for
> most use cases about time.
> The Table API would also benefit from having different types as output result.
> I would propose to add the three {{java.sql.}} types either as {{BasicTypes}}
> or in an additional class {{TimeTypes}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-3856) Create types for java.sql.Date/Time/Timestamp
[
https://issues.apache.org/jira/browse/FLINK-3856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15268625#comment-15268625
]
ASF GitHub Bot commented on FLINK-3856:
---
Github user fhueske commented on the pull request:
https://github.com/apache/flink/pull/1959#issuecomment-216513630
These types would also be useful for `flink-jdbc` and possibly other
modules. We can move them to a dedicated `TimeTypeInfo` or `SqlTimeTypeInfo`
class, but I think they should be part of `flink-core`.
> Create types for java.sql.Date/Time/Timestamp
> -
>
> Key: FLINK-3856
> URL: https://issues.apache.org/jira/browse/FLINK-3856
> Project: Flink
> Issue Type: New Feature
> Components: Core
>Reporter: Timo Walther
>Assignee: Timo Walther
>
> At the moment there is only the {{Date}} type which is not sufficient for
> most use cases about time.
> The Table API would also benefit from having different types as output result.
> I would propose to add the three {{java.sql.}} types either as {{BasicTypes}}
> or in an additional class {{TimeTypes}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-3856) Create types for java.sql.Date/Time/Timestamp
[
https://issues.apache.org/jira/browse/FLINK-3856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15268612#comment-15268612
]
ASF GitHub Bot commented on FLINK-3856:
---
Github user StephanEwen commented on the pull request:
https://github.com/apache/flink/pull/1959#issuecomment-216511830
Can these type infos exist independent of the BasicTypeInfo?
Either in some class like SQL type infos, or even only inside the Table API
/ SQL project?
> Create types for java.sql.Date/Time/Timestamp
> -
>
> Key: FLINK-3856
> URL: https://issues.apache.org/jira/browse/FLINK-3856
> Project: Flink
> Issue Type: New Feature
> Components: Core
>Reporter: Timo Walther
>Assignee: Timo Walther
>
> At the moment there is only the {{Date}} type which is not sufficient for
> most use cases about time.
> The Table API would also benefit from having different types as output result.
> I would propose to add the three {{java.sql.}} types either as {{BasicTypes}}
> or in an additional class {{TimeTypes}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-3856) Create types for java.sql.Date/Time/Timestamp
[
https://issues.apache.org/jira/browse/FLINK-3856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15268523#comment-15268523
]
ASF GitHub Bot commented on FLINK-3856:
---
GitHub user twalthr opened a pull request:
https://github.com/apache/flink/pull/1959
[FLINK-3856] [core] Create types for java.sql.Date/Time/Timestamp
Thanks for contributing to Apache Flink. Before you open your pull request,
please take the following check list into consideration.
If your changes take all of the items into account, feel free to open your
pull request. For more information and/or questions please refer to the [How To
Contribute guide](http://flink.apache.org/how-to-contribute.html).
In addition to going through the list, please provide a meaningful
description of your changes.
- [x] General
- The pull request references the related JIRA issue ("[FLINK-XXX] Jira
title text")
- The pull request addresses only one issue
- Each commit in the PR has a meaningful commit message (including the
JIRA id)
- [x] Documentation
- Documentation has been added for new functionality
- Old documentation affected by the pull request has been updated
- JavaDoc for public methods has been added
- [x] Tests & Build
- Functionality added by the pull request is covered by tests
- `mvn clean verify` has been executed successfully locally or a Travis
build has passed
This PR adds java.sql.Date/Time/Timestamp as basic types. I declared them
PublicEvolving, therefore I didn't add the types to the documentation. I
improved the Date serialization to use Long.MIN_VALUE instead of -1. But it
still does not solve FLINK-3858 completely.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/twalthr/flink DateTimeTimestamp
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/1959.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1959
commit b294fe3f1988582baf4b1d948d95e1efd5293d80
Author: twalthr
Date: 2016-05-02T14:31:45Z
[FLINK-3856] [core] Create types for java.sql.Date/Time/Timestamp
> Create types for java.sql.Date/Time/Timestamp
> -
>
> Key: FLINK-3856
> URL: https://issues.apache.org/jira/browse/FLINK-3856
> Project: Flink
> Issue Type: New Feature
> Components: Core
>Reporter: Timo Walther
>Assignee: Timo Walther
>
> At the moment there is only the {{Date}} type which is not sufficient for
> most use cases about time.
> The Table API would also benefit from having different types as output result.
> I would propose to add the three {{java.sql.}} types either as {{BasicTypes}}
> or in an additional class {{TimeTypes}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-3856) Create types for java.sql.Date/Time/Timestamp
[
https://issues.apache.org/jira/browse/FLINK-3856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15266907#comment-15266907
]
Fabian Hueske commented on FLINK-3856:
--
I think it makes sense to add these time types.
We can add them to BasicTypes, but we should mark them as {{@PublicEvolving}},
IMO.
> Create types for java.sql.Date/Time/Timestamp
> -
>
> Key: FLINK-3856
> URL: https://issues.apache.org/jira/browse/FLINK-3856
> Project: Flink
> Issue Type: New Feature
> Components: Core
>Reporter: Timo Walther
>Assignee: Timo Walther
>
> At the moment there is only the {{Date}} type which is not sufficient for
> most use cases about time.
> The Table API would also benefit from having different types as output result.
> I would propose to add the three {{java.sql.}} types either as {{BasicTypes}}
> or in an additional class {{TimeTypes}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-3856) Create types for java.sql.Date/Time/Timestamp
[
https://issues.apache.org/jira/browse/FLINK-3856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15266712#comment-15266712
]
Timo Walther commented on FLINK-3856:
-
I have created type information for all 3 types. You can find it here:
https://github.com/twalthr/flink/tree/DateTimeTimestamp
Actually, it does not look very nice due to the bad Java API design. Using
{{java.time}} would be better, but Java 8 is not mandatory yet. Supporting
these types is propaply the most industry-friendly solution, though.
What do you think about it [~fhueske]?
> Create types for java.sql.Date/Time/Timestamp
> -
>
> Key: FLINK-3856
> URL: https://issues.apache.org/jira/browse/FLINK-3856
> Project: Flink
> Issue Type: New Feature
> Components: Core
>Reporter: Timo Walther
>Assignee: Timo Walther
>
> At the moment there is only the {{Date}} type which is not sufficient for
> most use cases about time.
> The Table API would also benefit from having different types as output result.
> I would propose to add the three {{java.sql.}} types either as {{BasicTypes}}
> or in an additional class {{TimeTypes}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
