[jira] [Commented] (FLINK-4248) CsvTableSource does not support reading SqlTimeTypeInfo types
[
https://issues.apache.org/jira/browse/FLINK-4248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15512709#comment-15512709
]
ASF GitHub Bot commented on FLINK-4248:
---
Github user asfgit closed the pull request at:
https://github.com/apache/flink/pull/2303
> CsvTableSource does not support reading SqlTimeTypeInfo types
> -
>
> Key: FLINK-4248
> URL: https://issues.apache.org/jira/browse/FLINK-4248
> Project: Flink
> Issue Type: Improvement
> Components: Table API & SQL
>Affects Versions: 1.1.0
>Reporter: Till Rohrmann
>Assignee: Timo Walther
>
> The Table API's {{CsvTableSource}} does not support to read all Table API
> supported data types. For example, it is not possible to read
> {{SqlTimeTypeInfo}} types via the {{CsvTableSource}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-4248) CsvTableSource does not support reading SqlTimeTypeInfo types
[
https://issues.apache.org/jira/browse/FLINK-4248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15510703#comment-15510703
]
ASF GitHub Bot commented on FLINK-4248:
---
Github user fhueske commented on the issue:
https://github.com/apache/flink/pull/2303
I still think that methods should be named after what they do (and there is
nothing numeric about their behavior) and not in which context they are
supposed to be called. The error messages are numeric due to the original
context of the code which has been moved into separate methods. That context is
no longer present in these methods. Anyway, I don't want to start a
bikeshedding discussion about the names of internal utility methods and stop at
this point ;-)
> CsvTableSource does not support reading SqlTimeTypeInfo types
> -
>
> Key: FLINK-4248
> URL: https://issues.apache.org/jira/browse/FLINK-4248
> Project: Flink
> Issue Type: Improvement
> Components: Table API & SQL
>Affects Versions: 1.1.0
>Reporter: Till Rohrmann
>Assignee: Timo Walther
>
> The Table API's {{CsvTableSource}} does not support to read all Table API
> supported data types. For example, it is not possible to read
> {{SqlTimeTypeInfo}} types via the {{CsvTableSource}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-4248) CsvTableSource does not support reading SqlTimeTypeInfo types
[
https://issues.apache.org/jira/browse/FLINK-4248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15510349#comment-15510349
]
ASF GitHub Bot commented on FLINK-4248:
---
Github user twalthr commented on the issue:
https://github.com/apache/flink/pull/2303
@fhueske I think we should keep the methods as they currently are. I
renamed them to "nextNumericString"/"nextNumericStringEndPos" and added a
explanation what a numeric string is. You are right they don't do much numeric
at the moment but only the numeric classes will use it. Furthermore, the error
states and exceptions are numeric. E.g. if we don't do the whitespace checking
in `nextNumericString` we also need to return the position instead of the
string.
> CsvTableSource does not support reading SqlTimeTypeInfo types
> -
>
> Key: FLINK-4248
> URL: https://issues.apache.org/jira/browse/FLINK-4248
> Project: Flink
> Issue Type: Improvement
> Components: Table API & SQL
>Affects Versions: 1.1.0
>Reporter: Till Rohrmann
>Assignee: Timo Walther
>
> The Table API's {{CsvTableSource}} does not support to read all Table API
> supported data types. For example, it is not possible to read
> {{SqlTimeTypeInfo}} types via the {{CsvTableSource}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-4248) CsvTableSource does not support reading SqlTimeTypeInfo types
[
https://issues.apache.org/jira/browse/FLINK-4248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15509930#comment-15509930
]
ASF GitHub Bot commented on FLINK-4248:
---
Github user fhueske commented on a diff in the pull request:
https://github.com/apache/flink/pull/2303#discussion_r79833586
--- Diff:
flink-core/src/main/java/org/apache/flink/types/parser/FieldParser.java ---
@@ -174,6 +174,62 @@ protected void setErrorState(ParseErrorState error) {
public ParseErrorState getErrorState() {
return this.errorState;
}
+
+ /**
+* Returns the end position of a string with a numeric format (like
-XX-XX). Sets the error state if the
+* string contains leading/trailing whitespaces or if the column is
empty.
+*
+* @return the end position of the string or -1 if an error occurred
+*/
+ public final int formattedStringEndPos(byte[] bytes, int startPos, int
limit, byte[] delimiter) {
+ int len = startPos;
+
+ final int delimLimit = limit - delimiter.length + 1;
+
+ while (len < limit) {
+ if (len < delimLimit && delimiterNext(bytes, len,
delimiter)) {
+ if (len == startPos) {
+
setErrorState(ParseErrorState.EMPTY_COLUMN);
+ return -1;
+ }
+ break;
+ }
+ len++;
+ }
+
+ if (len > startPos &&
+ (Character.isWhitespace(bytes[startPos]) ||
Character.isWhitespace(bytes[(len - 1)]))) {
+
setErrorState(ParseErrorState.NUMERIC_VALUE_ILLEGAL_CHARACTER);
+ return -1;
+ }
+
+ return len;
+ }
+
+ /**
+* Returns a string with a numeric format (like -XX-XX). Throws an
exception if the
+* string contains leading/trailing whitespaces or if the column is
empty.
+*
+* @return the parsed string
+*/
+ public static final String formattedString(byte[] bytes, int startPos,
int length, char delimiter) {
+ if (length <= 0) {
+ throw new NumberFormatException("Invalid input: Empty
string");
--- End diff --
But this is not the `parseField` method. `parseField` could catch the
exception and pass it on as a `NumberFormatException`.
> CsvTableSource does not support reading SqlTimeTypeInfo types
> -
>
> Key: FLINK-4248
> URL: https://issues.apache.org/jira/browse/FLINK-4248
> Project: Flink
> Issue Type: Improvement
> Components: Table API & SQL
>Affects Versions: 1.1.0
>Reporter: Till Rohrmann
>Assignee: Timo Walther
>
> The Table API's {{CsvTableSource}} does not support to read all Table API
> supported data types. For example, it is not possible to read
> {{SqlTimeTypeInfo}} types via the {{CsvTableSource}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-4248) CsvTableSource does not support reading SqlTimeTypeInfo types
[
https://issues.apache.org/jira/browse/FLINK-4248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15509931#comment-15509931
]
ASF GitHub Bot commented on FLINK-4248:
---
Github user fhueske commented on a diff in the pull request:
https://github.com/apache/flink/pull/2303#discussion_r79833722
--- Diff:
flink-core/src/main/java/org/apache/flink/types/parser/FieldParser.java ---
@@ -174,6 +174,62 @@ protected void setErrorState(ParseErrorState error) {
public ParseErrorState getErrorState() {
return this.errorState;
}
+
+ /**
+* Returns the end position of a string with a numeric format (like
-XX-XX). Sets the error state if the
+* string contains leading/trailing whitespaces or if the column is
empty.
+*
+* @return the end position of the string or -1 if an error occurred
+*/
+ public final int formattedStringEndPos(byte[] bytes, int startPos, int
limit, byte[] delimiter) {
+ int len = startPos;
+
+ final int delimLimit = limit - delimiter.length + 1;
+
+ while (len < limit) {
+ if (len < delimLimit && delimiterNext(bytes, len,
delimiter)) {
+ if (len == startPos) {
+
setErrorState(ParseErrorState.EMPTY_COLUMN);
+ return -1;
+ }
+ break;
+ }
+ len++;
+ }
+
+ if (len > startPos &&
+ (Character.isWhitespace(bytes[startPos]) ||
Character.isWhitespace(bytes[(len - 1)]))) {
+
setErrorState(ParseErrorState.NUMERIC_VALUE_ILLEGAL_CHARACTER);
+ return -1;
+ }
+
+ return len;
+ }
+
+ /**
+* Returns a string with a numeric format (like -XX-XX). Throws an
exception if the
+* string contains leading/trailing whitespaces or if the column is
empty.
+*
+* @return the parsed string
+*/
+ public static final String formattedString(byte[] bytes, int startPos,
int length, char delimiter) {
+ if (length <= 0) {
+ throw new NumberFormatException("Invalid input: Empty
string");
+ }
+ int i = 0;
+ final byte delByte = (byte) delimiter;
+
+ while (i < length && bytes[startPos + i] != delByte) {
+ i++;
+ }
+
+ if (i > 0 &&
+ (Character.isWhitespace(bytes[startPos]) ||
Character.isWhitespace(bytes[startPos + i - 1]))) {
+ throw new NumberFormatException("There is leading or
trailing whitespace in the numeric field.");
--- End diff --
But it does not do anything related to numeric values or am I overlooking
something?
> CsvTableSource does not support reading SqlTimeTypeInfo types
> -
>
> Key: FLINK-4248
> URL: https://issues.apache.org/jira/browse/FLINK-4248
> Project: Flink
> Issue Type: Improvement
> Components: Table API & SQL
>Affects Versions: 1.1.0
>Reporter: Till Rohrmann
>Assignee: Timo Walther
>
> The Table API's {{CsvTableSource}} does not support to read all Table API
> supported data types. For example, it is not possible to read
> {{SqlTimeTypeInfo}} types via the {{CsvTableSource}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-4248) CsvTableSource does not support reading SqlTimeTypeInfo types
[
https://issues.apache.org/jira/browse/FLINK-4248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15509929#comment-15509929
]
ASF GitHub Bot commented on FLINK-4248:
---
Github user fhueske commented on a diff in the pull request:
https://github.com/apache/flink/pull/2303#discussion_r79833219
--- Diff:
flink-core/src/main/java/org/apache/flink/types/parser/FieldParser.java ---
@@ -174,6 +174,62 @@ protected void setErrorState(ParseErrorState error) {
public ParseErrorState getErrorState() {
return this.errorState;
}
+
+ /**
+* Returns the end position of a string with a numeric format (like
-XX-XX). Sets the error state if the
+* string contains leading/trailing whitespaces or if the column is
empty.
+*
+* @return the end position of the string or -1 if an error occurred
+*/
+ public final int formattedStringEndPos(byte[] bytes, int startPos, int
limit, byte[] delimiter) {
+ int len = startPos;
+
+ final int delimLimit = limit - delimiter.length + 1;
+
+ while (len < limit) {
+ if (len < delimLimit && delimiterNext(bytes, len,
delimiter)) {
+ if (len == startPos) {
+
setErrorState(ParseErrorState.EMPTY_COLUMN);
+ return -1;
+ }
+ break;
+ }
+ len++;
+ }
+
+ if (len > startPos &&
+ (Character.isWhitespace(bytes[startPos]) ||
Character.isWhitespace(bytes[(len - 1)]))) {
+
setErrorState(ParseErrorState.NUMERIC_VALUE_ILLEGAL_CHARACTER);
+ return -1;
+ }
+
+ return len;
+ }
+
+ /**
+* Returns a string with a numeric format (like -XX-XX). Throws an
exception if the
--- End diff --
It returns a plain string object and only checks if the string has a
leading or tailing whitespace. There are no checks for numeric characters or
similiar. I think the method should be named according to what it does and not
who's the intended caller.
> CsvTableSource does not support reading SqlTimeTypeInfo types
> -
>
> Key: FLINK-4248
> URL: https://issues.apache.org/jira/browse/FLINK-4248
> Project: Flink
> Issue Type: Improvement
> Components: Table API & SQL
>Affects Versions: 1.1.0
>Reporter: Till Rohrmann
>Assignee: Timo Walther
>
> The Table API's {{CsvTableSource}} does not support to read all Table API
> supported data types. For example, it is not possible to read
> {{SqlTimeTypeInfo}} types via the {{CsvTableSource}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-4248) CsvTableSource does not support reading SqlTimeTypeInfo types
[
https://issues.apache.org/jira/browse/FLINK-4248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15509532#comment-15509532
]
ASF GitHub Bot commented on FLINK-4248:
---
Github user twalthr commented on a diff in the pull request:
https://github.com/apache/flink/pull/2303#discussion_r79804868
--- Diff:
flink-core/src/main/java/org/apache/flink/types/parser/FieldParser.java ---
@@ -174,6 +174,62 @@ protected void setErrorState(ParseErrorState error) {
public ParseErrorState getErrorState() {
return this.errorState;
}
+
+ /**
+* Returns the end position of a string with a numeric format (like
-XX-XX). Sets the error state if the
+* string contains leading/trailing whitespaces or if the column is
empty.
+*
+* @return the end position of the string or -1 if an error occurred
+*/
+ public final int formattedStringEndPos(byte[] bytes, int startPos, int
limit, byte[] delimiter) {
+ int len = startPos;
+
+ final int delimLimit = limit - delimiter.length + 1;
+
+ while (len < limit) {
+ if (len < delimLimit && delimiterNext(bytes, len,
delimiter)) {
+ if (len == startPos) {
+
setErrorState(ParseErrorState.EMPTY_COLUMN);
+ return -1;
+ }
+ break;
+ }
+ len++;
+ }
+
+ if (len > startPos &&
+ (Character.isWhitespace(bytes[startPos]) ||
Character.isWhitespace(bytes[(len - 1)]))) {
+
setErrorState(ParseErrorState.NUMERIC_VALUE_ILLEGAL_CHARACTER);
+ return -1;
+ }
+
+ return len;
+ }
+
+ /**
+* Returns a string with a numeric format (like -XX-XX). Throws an
exception if the
+* string contains leading/trailing whitespaces or if the column is
empty.
+*
+* @return the parsed string
+*/
+ public static final String formattedString(byte[] bytes, int startPos,
int length, char delimiter) {
+ if (length <= 0) {
+ throw new NumberFormatException("Invalid input: Empty
string");
+ }
+ int i = 0;
+ final byte delByte = (byte) delimiter;
+
+ while (i < length && bytes[startPos + i] != delByte) {
+ i++;
+ }
+
+ if (i > 0 &&
+ (Character.isWhitespace(bytes[startPos]) ||
Character.isWhitespace(bytes[startPos + i - 1]))) {
+ throw new NumberFormatException("There is leading or
trailing whitespace in the numeric field.");
--- End diff --
I think we could just rename the method again. Maybe `numericString()`?
> CsvTableSource does not support reading SqlTimeTypeInfo types
> -
>
> Key: FLINK-4248
> URL: https://issues.apache.org/jira/browse/FLINK-4248
> Project: Flink
> Issue Type: Improvement
> Components: Table API & SQL
>Affects Versions: 1.1.0
>Reporter: Till Rohrmann
>Assignee: Timo Walther
>
> The Table API's {{CsvTableSource}} does not support to read all Table API
> supported data types. For example, it is not possible to read
> {{SqlTimeTypeInfo}} types via the {{CsvTableSource}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-4248) CsvTableSource does not support reading SqlTimeTypeInfo types
[
https://issues.apache.org/jira/browse/FLINK-4248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15509529#comment-15509529
]
ASF GitHub Bot commented on FLINK-4248:
---
Github user twalthr commented on a diff in the pull request:
https://github.com/apache/flink/pull/2303#discussion_r79804683
--- Diff:
flink-core/src/main/java/org/apache/flink/types/parser/FieldParser.java ---
@@ -174,6 +174,62 @@ protected void setErrorState(ParseErrorState error) {
public ParseErrorState getErrorState() {
return this.errorState;
}
+
+ /**
+* Returns the end position of a string with a numeric format (like
-XX-XX). Sets the error state if the
+* string contains leading/trailing whitespaces or if the column is
empty.
+*
+* @return the end position of the string or -1 if an error occurred
+*/
+ public final int formattedStringEndPos(byte[] bytes, int startPos, int
limit, byte[] delimiter) {
+ int len = startPos;
+
+ final int delimLimit = limit - delimiter.length + 1;
+
+ while (len < limit) {
+ if (len < delimLimit && delimiterNext(bytes, len,
delimiter)) {
+ if (len == startPos) {
+
setErrorState(ParseErrorState.EMPTY_COLUMN);
+ return -1;
+ }
+ break;
+ }
+ len++;
+ }
+
+ if (len > startPos &&
+ (Character.isWhitespace(bytes[startPos]) ||
Character.isWhitespace(bytes[(len - 1)]))) {
+
setErrorState(ParseErrorState.NUMERIC_VALUE_ILLEGAL_CHARACTER);
+ return -1;
+ }
+
+ return len;
+ }
+
+ /**
+* Returns a string with a numeric format (like -XX-XX). Throws an
exception if the
+* string contains leading/trailing whitespaces or if the column is
empty.
+*
+* @return the parsed string
+*/
+ public static final String formattedString(byte[] bytes, int startPos,
int length, char delimiter) {
+ if (length <= 0) {
+ throw new NumberFormatException("Invalid input: Empty
string");
--- End diff --
According to the Javadoc "parseField" should return a
`NumberFormatException`.
> CsvTableSource does not support reading SqlTimeTypeInfo types
> -
>
> Key: FLINK-4248
> URL: https://issues.apache.org/jira/browse/FLINK-4248
> Project: Flink
> Issue Type: Improvement
> Components: Table API & SQL
>Affects Versions: 1.1.0
>Reporter: Till Rohrmann
>Assignee: Timo Walther
>
> The Table API's {{CsvTableSource}} does not support to read all Table API
> supported data types. For example, it is not possible to read
> {{SqlTimeTypeInfo}} types via the {{CsvTableSource}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-4248) CsvTableSource does not support reading SqlTimeTypeInfo types
[
https://issues.apache.org/jira/browse/FLINK-4248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15509523#comment-15509523
]
ASF GitHub Bot commented on FLINK-4248:
---
Github user twalthr commented on a diff in the pull request:
https://github.com/apache/flink/pull/2303#discussion_r79804375
--- Diff:
flink-core/src/main/java/org/apache/flink/types/parser/FieldParser.java ---
@@ -174,6 +174,62 @@ protected void setErrorState(ParseErrorState error) {
public ParseErrorState getErrorState() {
return this.errorState;
}
+
+ /**
+* Returns the end position of a string with a numeric format (like
-XX-XX). Sets the error state if the
+* string contains leading/trailing whitespaces or if the column is
empty.
+*
+* @return the end position of the string or -1 if an error occurred
+*/
+ public final int formattedStringEndPos(byte[] bytes, int startPos, int
limit, byte[] delimiter) {
+ int len = startPos;
+
+ final int delimLimit = limit - delimiter.length + 1;
+
+ while (len < limit) {
+ if (len < delimLimit && delimiterNext(bytes, len,
delimiter)) {
+ if (len == startPos) {
+
setErrorState(ParseErrorState.EMPTY_COLUMN);
+ return -1;
+ }
+ break;
+ }
+ len++;
+ }
+
+ if (len > startPos &&
+ (Character.isWhitespace(bytes[startPos]) ||
Character.isWhitespace(bytes[(len - 1)]))) {
+
setErrorState(ParseErrorState.NUMERIC_VALUE_ILLEGAL_CHARACTER);
+ return -1;
+ }
+
+ return len;
+ }
+
+ /**
+* Returns a string with a numeric format (like -XX-XX). Throws an
exception if the
--- End diff --
Why not? Only parsers with a numeric format will use this method.
> CsvTableSource does not support reading SqlTimeTypeInfo types
> -
>
> Key: FLINK-4248
> URL: https://issues.apache.org/jira/browse/FLINK-4248
> Project: Flink
> Issue Type: Improvement
> Components: Table API & SQL
>Affects Versions: 1.1.0
>Reporter: Till Rohrmann
>Assignee: Timo Walther
>
> The Table API's {{CsvTableSource}} does not support to read all Table API
> supported data types. For example, it is not possible to read
> {{SqlTimeTypeInfo}} types via the {{CsvTableSource}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-4248) CsvTableSource does not support reading SqlTimeTypeInfo types
[
https://issues.apache.org/jira/browse/FLINK-4248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15509456#comment-15509456
]
ASF GitHub Bot commented on FLINK-4248:
---
Github user fhueske commented on a diff in the pull request:
https://github.com/apache/flink/pull/2303#discussion_r79799726
--- Diff:
flink-core/src/main/java/org/apache/flink/types/parser/FieldParser.java ---
@@ -174,6 +174,62 @@ protected void setErrorState(ParseErrorState error) {
public ParseErrorState getErrorState() {
return this.errorState;
}
+
+ /**
+* Returns the end position of a string with a numeric format (like
-XX-XX). Sets the error state if the
--- End diff --
Maybe reduce the scope of the method to only identify the end of the field
and do the whitespace check in each parser separately?
> CsvTableSource does not support reading SqlTimeTypeInfo types
> -
>
> Key: FLINK-4248
> URL: https://issues.apache.org/jira/browse/FLINK-4248
> Project: Flink
> Issue Type: Improvement
> Components: Table API & SQL
>Affects Versions: 1.1.0
>Reporter: Till Rohrmann
>Assignee: Timo Walther
>
> The Table API's {{CsvTableSource}} does not support to read all Table API
> supported data types. For example, it is not possible to read
> {{SqlTimeTypeInfo}} types via the {{CsvTableSource}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-4248) CsvTableSource does not support reading SqlTimeTypeInfo types
[
https://issues.apache.org/jira/browse/FLINK-4248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15509452#comment-15509452
]
ASF GitHub Bot commented on FLINK-4248:
---
Github user fhueske commented on a diff in the pull request:
https://github.com/apache/flink/pull/2303#discussion_r79799395
--- Diff:
flink-core/src/main/java/org/apache/flink/types/parser/FieldParser.java ---
@@ -174,6 +174,62 @@ protected void setErrorState(ParseErrorState error) {
public ParseErrorState getErrorState() {
return this.errorState;
}
+
+ /**
+* Returns the end position of a string with a numeric format (like
-XX-XX). Sets the error state if the
+* string contains leading/trailing whitespaces or if the column is
empty.
+*
+* @return the end position of the string or -1 if an error occurred
+*/
+ public final int formattedStringEndPos(byte[] bytes, int startPos, int
limit, byte[] delimiter) {
--- End diff --
rename to something like `findNextFieldEndPos()`?
> CsvTableSource does not support reading SqlTimeTypeInfo types
> -
>
> Key: FLINK-4248
> URL: https://issues.apache.org/jira/browse/FLINK-4248
> Project: Flink
> Issue Type: Improvement
> Components: Table API & SQL
>Affects Versions: 1.1.0
>Reporter: Till Rohrmann
>Assignee: Timo Walther
>
> The Table API's {{CsvTableSource}} does not support to read all Table API
> supported data types. For example, it is not possible to read
> {{SqlTimeTypeInfo}} types via the {{CsvTableSource}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-4248) CsvTableSource does not support reading SqlTimeTypeInfo types
[
https://issues.apache.org/jira/browse/FLINK-4248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15509453#comment-15509453
]
ASF GitHub Bot commented on FLINK-4248:
---
Github user fhueske commented on a diff in the pull request:
https://github.com/apache/flink/pull/2303#discussion_r79799234
--- Diff:
flink-core/src/main/java/org/apache/flink/types/parser/FieldParser.java ---
@@ -174,6 +174,62 @@ protected void setErrorState(ParseErrorState error) {
public ParseErrorState getErrorState() {
return this.errorState;
}
+
+ /**
+* Returns the end position of a string with a numeric format (like
-XX-XX). Sets the error state if the
--- End diff --
Why numeric format?
> CsvTableSource does not support reading SqlTimeTypeInfo types
> -
>
> Key: FLINK-4248
> URL: https://issues.apache.org/jira/browse/FLINK-4248
> Project: Flink
> Issue Type: Improvement
> Components: Table API & SQL
>Affects Versions: 1.1.0
>Reporter: Till Rohrmann
>Assignee: Timo Walther
>
> The Table API's {{CsvTableSource}} does not support to read all Table API
> supported data types. For example, it is not possible to read
> {{SqlTimeTypeInfo}} types via the {{CsvTableSource}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-4248) CsvTableSource does not support reading SqlTimeTypeInfo types
[
https://issues.apache.org/jira/browse/FLINK-4248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15509445#comment-15509445
]
ASF GitHub Bot commented on FLINK-4248:
---
Github user fhueske commented on a diff in the pull request:
https://github.com/apache/flink/pull/2303#discussion_r79798925
--- Diff:
flink-core/src/main/java/org/apache/flink/types/parser/FieldParser.java ---
@@ -174,6 +174,62 @@ protected void setErrorState(ParseErrorState error) {
public ParseErrorState getErrorState() {
return this.errorState;
}
+
+ /**
+* Returns the end position of a string with a numeric format (like
-XX-XX). Sets the error state if the
+* string contains leading/trailing whitespaces or if the column is
empty.
+*
+* @return the end position of the string or -1 if an error occurred
+*/
+ public final int formattedStringEndPos(byte[] bytes, int startPos, int
limit, byte[] delimiter) {
+ int len = startPos;
+
+ final int delimLimit = limit - delimiter.length + 1;
+
+ while (len < limit) {
+ if (len < delimLimit && delimiterNext(bytes, len,
delimiter)) {
+ if (len == startPos) {
+
setErrorState(ParseErrorState.EMPTY_COLUMN);
+ return -1;
+ }
+ break;
+ }
+ len++;
+ }
+
+ if (len > startPos &&
+ (Character.isWhitespace(bytes[startPos]) ||
Character.isWhitespace(bytes[(len - 1)]))) {
+
setErrorState(ParseErrorState.NUMERIC_VALUE_ILLEGAL_CHARACTER);
+ return -1;
+ }
+
+ return len;
+ }
+
+ /**
+* Returns a string with a numeric format (like -XX-XX). Throws an
exception if the
--- End diff --
Why is this method specific for numeric formats?
> CsvTableSource does not support reading SqlTimeTypeInfo types
> -
>
> Key: FLINK-4248
> URL: https://issues.apache.org/jira/browse/FLINK-4248
> Project: Flink
> Issue Type: Improvement
> Components: Table API & SQL
>Affects Versions: 1.1.0
>Reporter: Till Rohrmann
>Assignee: Timo Walther
>
> The Table API's {{CsvTableSource}} does not support to read all Table API
> supported data types. For example, it is not possible to read
> {{SqlTimeTypeInfo}} types via the {{CsvTableSource}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-4248) CsvTableSource does not support reading SqlTimeTypeInfo types
[
https://issues.apache.org/jira/browse/FLINK-4248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15509447#comment-15509447
]
ASF GitHub Bot commented on FLINK-4248:
---
Github user fhueske commented on a diff in the pull request:
https://github.com/apache/flink/pull/2303#discussion_r79798819
--- Diff:
flink-core/src/main/java/org/apache/flink/types/parser/FieldParser.java ---
@@ -174,6 +174,62 @@ protected void setErrorState(ParseErrorState error) {
public ParseErrorState getErrorState() {
return this.errorState;
}
+
+ /**
+* Returns the end position of a string with a numeric format (like
-XX-XX). Sets the error state if the
+* string contains leading/trailing whitespaces or if the column is
empty.
+*
+* @return the end position of the string or -1 if an error occurred
+*/
+ public final int formattedStringEndPos(byte[] bytes, int startPos, int
limit, byte[] delimiter) {
--- End diff --
make `protected`?
> CsvTableSource does not support reading SqlTimeTypeInfo types
> -
>
> Key: FLINK-4248
> URL: https://issues.apache.org/jira/browse/FLINK-4248
> Project: Flink
> Issue Type: Improvement
> Components: Table API & SQL
>Affects Versions: 1.1.0
>Reporter: Till Rohrmann
>Assignee: Timo Walther
>
> The Table API's {{CsvTableSource}} does not support to read all Table API
> supported data types. For example, it is not possible to read
> {{SqlTimeTypeInfo}} types via the {{CsvTableSource}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-4248) CsvTableSource does not support reading SqlTimeTypeInfo types
[
https://issues.apache.org/jira/browse/FLINK-4248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15509444#comment-15509444
]
ASF GitHub Bot commented on FLINK-4248:
---
Github user fhueske commented on a diff in the pull request:
https://github.com/apache/flink/pull/2303#discussion_r79798776
--- Diff:
flink-core/src/main/java/org/apache/flink/types/parser/FieldParser.java ---
@@ -174,6 +174,62 @@ protected void setErrorState(ParseErrorState error) {
public ParseErrorState getErrorState() {
return this.errorState;
}
+
+ /**
+* Returns the end position of a string with a numeric format (like
-XX-XX). Sets the error state if the
+* string contains leading/trailing whitespaces or if the column is
empty.
+*
+* @return the end position of the string or -1 if an error occurred
+*/
+ public final int formattedStringEndPos(byte[] bytes, int startPos, int
limit, byte[] delimiter) {
+ int len = startPos;
+
+ final int delimLimit = limit - delimiter.length + 1;
+
+ while (len < limit) {
+ if (len < delimLimit && delimiterNext(bytes, len,
delimiter)) {
+ if (len == startPos) {
+
setErrorState(ParseErrorState.EMPTY_COLUMN);
+ return -1;
+ }
+ break;
+ }
+ len++;
+ }
+
+ if (len > startPos &&
+ (Character.isWhitespace(bytes[startPos]) ||
Character.isWhitespace(bytes[(len - 1)]))) {
+
setErrorState(ParseErrorState.NUMERIC_VALUE_ILLEGAL_CHARACTER);
+ return -1;
+ }
+
+ return len;
+ }
+
+ /**
+* Returns a string with a numeric format (like -XX-XX). Throws an
exception if the
+* string contains leading/trailing whitespaces or if the column is
empty.
+*
+* @return the parsed string
+*/
+ public static final String formattedString(byte[] bytes, int startPos,
int length, char delimiter) {
+ if (length <= 0) {
+ throw new NumberFormatException("Invalid input: Empty
string");
+ }
+ int i = 0;
+ final byte delByte = (byte) delimiter;
+
+ while (i < length && bytes[startPos + i] != delByte) {
+ i++;
+ }
+
+ if (i > 0 &&
+ (Character.isWhitespace(bytes[startPos]) ||
Character.isWhitespace(bytes[startPos + i - 1]))) {
+ throw new NumberFormatException("There is leading or
trailing whitespace in the numeric field.");
--- End diff --
Throw a `IllegalArgumentException`? Method might be used by parsers that do
not parse numeric values.
> CsvTableSource does not support reading SqlTimeTypeInfo types
> -
>
> Key: FLINK-4248
> URL: https://issues.apache.org/jira/browse/FLINK-4248
> Project: Flink
> Issue Type: Improvement
> Components: Table API & SQL
>Affects Versions: 1.1.0
>Reporter: Till Rohrmann
>Assignee: Timo Walther
>
> The Table API's {{CsvTableSource}} does not support to read all Table API
> supported data types. For example, it is not possible to read
> {{SqlTimeTypeInfo}} types via the {{CsvTableSource}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-4248) CsvTableSource does not support reading SqlTimeTypeInfo types
[
https://issues.apache.org/jira/browse/FLINK-4248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15509446#comment-15509446
]
ASF GitHub Bot commented on FLINK-4248:
---
Github user fhueske commented on a diff in the pull request:
https://github.com/apache/flink/pull/2303#discussion_r79799184
--- Diff:
flink-core/src/main/java/org/apache/flink/types/parser/FieldParser.java ---
@@ -174,6 +174,62 @@ protected void setErrorState(ParseErrorState error) {
public ParseErrorState getErrorState() {
return this.errorState;
}
+
+ /**
+* Returns the end position of a string with a numeric format (like
-XX-XX). Sets the error state if the
+* string contains leading/trailing whitespaces or if the column is
empty.
+*
+* @return the end position of the string or -1 if an error occurred
+*/
+ public final int formattedStringEndPos(byte[] bytes, int startPos, int
limit, byte[] delimiter) {
+ int len = startPos;
+
+ final int delimLimit = limit - delimiter.length + 1;
+
+ while (len < limit) {
+ if (len < delimLimit && delimiterNext(bytes, len,
delimiter)) {
+ if (len == startPos) {
+
setErrorState(ParseErrorState.EMPTY_COLUMN);
+ return -1;
+ }
+ break;
+ }
+ len++;
+ }
+
+ if (len > startPos &&
+ (Character.isWhitespace(bytes[startPos]) ||
Character.isWhitespace(bytes[(len - 1)]))) {
+
setErrorState(ParseErrorState.NUMERIC_VALUE_ILLEGAL_CHARACTER);
+ return -1;
+ }
+
+ return len;
+ }
+
+ /**
+* Returns a string with a numeric format (like -XX-XX). Throws an
exception if the
+* string contains leading/trailing whitespaces or if the column is
empty.
+*
+* @return the parsed string
+*/
+ public static final String formattedString(byte[] bytes, int startPos,
int length, char delimiter) {
--- End diff --
What is formatted about the returned string? How about renaming the method
to `nextFieldAsString()`?
> CsvTableSource does not support reading SqlTimeTypeInfo types
> -
>
> Key: FLINK-4248
> URL: https://issues.apache.org/jira/browse/FLINK-4248
> Project: Flink
> Issue Type: Improvement
> Components: Table API & SQL
>Affects Versions: 1.1.0
>Reporter: Till Rohrmann
>Assignee: Timo Walther
>
> The Table API's {{CsvTableSource}} does not support to read all Table API
> supported data types. For example, it is not possible to read
> {{SqlTimeTypeInfo}} types via the {{CsvTableSource}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-4248) CsvTableSource does not support reading SqlTimeTypeInfo types
[
https://issues.apache.org/jira/browse/FLINK-4248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15509448#comment-15509448
]
ASF GitHub Bot commented on FLINK-4248:
---
Github user fhueske commented on a diff in the pull request:
https://github.com/apache/flink/pull/2303#discussion_r79798968
--- Diff:
flink-core/src/main/java/org/apache/flink/types/parser/FieldParser.java ---
@@ -174,6 +174,62 @@ protected void setErrorState(ParseErrorState error) {
public ParseErrorState getErrorState() {
return this.errorState;
}
+
+ /**
+* Returns the end position of a string with a numeric format (like
-XX-XX). Sets the error state if the
+* string contains leading/trailing whitespaces or if the column is
empty.
+*
+* @return the end position of the string or -1 if an error occurred
+*/
+ public final int formattedStringEndPos(byte[] bytes, int startPos, int
limit, byte[] delimiter) {
+ int len = startPos;
+
+ final int delimLimit = limit - delimiter.length + 1;
+
+ while (len < limit) {
+ if (len < delimLimit && delimiterNext(bytes, len,
delimiter)) {
+ if (len == startPos) {
+
setErrorState(ParseErrorState.EMPTY_COLUMN);
+ return -1;
+ }
+ break;
+ }
+ len++;
+ }
+
+ if (len > startPos &&
+ (Character.isWhitespace(bytes[startPos]) ||
Character.isWhitespace(bytes[(len - 1)]))) {
+
setErrorState(ParseErrorState.NUMERIC_VALUE_ILLEGAL_CHARACTER);
+ return -1;
+ }
+
+ return len;
+ }
+
+ /**
+* Returns a string with a numeric format (like -XX-XX). Throws an
exception if the
+* string contains leading/trailing whitespaces or if the column is
empty.
+*
+* @return the parsed string
+*/
+ public static final String formattedString(byte[] bytes, int startPos,
int length, char delimiter) {
+ if (length <= 0) {
+ throw new NumberFormatException("Invalid input: Empty
string");
--- End diff --
Change to `IllegalArgumentException`?
> CsvTableSource does not support reading SqlTimeTypeInfo types
> -
>
> Key: FLINK-4248
> URL: https://issues.apache.org/jira/browse/FLINK-4248
> Project: Flink
> Issue Type: Improvement
> Components: Table API & SQL
>Affects Versions: 1.1.0
>Reporter: Till Rohrmann
>Assignee: Timo Walther
>
> The Table API's {{CsvTableSource}} does not support to read all Table API
> supported data types. For example, it is not possible to read
> {{SqlTimeTypeInfo}} types via the {{CsvTableSource}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-4248) CsvTableSource does not support reading SqlTimeTypeInfo types
[
https://issues.apache.org/jira/browse/FLINK-4248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15509443#comment-15509443
]
ASF GitHub Bot commented on FLINK-4248:
---
Github user fhueske commented on a diff in the pull request:
https://github.com/apache/flink/pull/2303#discussion_r79798804
--- Diff:
flink-core/src/main/java/org/apache/flink/types/parser/FieldParser.java ---
@@ -174,6 +174,62 @@ protected void setErrorState(ParseErrorState error) {
public ParseErrorState getErrorState() {
return this.errorState;
}
+
+ /**
+* Returns the end position of a string with a numeric format (like
-XX-XX). Sets the error state if the
+* string contains leading/trailing whitespaces or if the column is
empty.
+*
+* @return the end position of the string or -1 if an error occurred
+*/
+ public final int formattedStringEndPos(byte[] bytes, int startPos, int
limit, byte[] delimiter) {
+ int len = startPos;
+
+ final int delimLimit = limit - delimiter.length + 1;
+
+ while (len < limit) {
+ if (len < delimLimit && delimiterNext(bytes, len,
delimiter)) {
+ if (len == startPos) {
+
setErrorState(ParseErrorState.EMPTY_COLUMN);
+ return -1;
+ }
+ break;
+ }
+ len++;
+ }
+
+ if (len > startPos &&
+ (Character.isWhitespace(bytes[startPos]) ||
Character.isWhitespace(bytes[(len - 1)]))) {
+
setErrorState(ParseErrorState.NUMERIC_VALUE_ILLEGAL_CHARACTER);
+ return -1;
+ }
+
+ return len;
+ }
+
+ /**
+* Returns a string with a numeric format (like -XX-XX). Throws an
exception if the
+* string contains leading/trailing whitespaces or if the column is
empty.
+*
+* @return the parsed string
+*/
+ public static final String formattedString(byte[] bytes, int startPos,
int length, char delimiter) {
--- End diff --
make `protected`?
> CsvTableSource does not support reading SqlTimeTypeInfo types
> -
>
> Key: FLINK-4248
> URL: https://issues.apache.org/jira/browse/FLINK-4248
> Project: Flink
> Issue Type: Improvement
> Components: Table API & SQL
>Affects Versions: 1.1.0
>Reporter: Till Rohrmann
>Assignee: Timo Walther
>
> The Table API's {{CsvTableSource}} does not support to read all Table API
> supported data types. For example, it is not possible to read
> {{SqlTimeTypeInfo}} types via the {{CsvTableSource}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-4248) CsvTableSource does not support reading SqlTimeTypeInfo types
[
https://issues.apache.org/jira/browse/FLINK-4248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15509272#comment-15509272
]
ASF GitHub Bot commented on FLINK-4248:
---
Github user twalthr commented on the issue:
https://github.com/apache/flink/pull/2303
@fhueske thanks for reviewing my code. I addressed your feedback.
> CsvTableSource does not support reading SqlTimeTypeInfo types
> -
>
> Key: FLINK-4248
> URL: https://issues.apache.org/jira/browse/FLINK-4248
> Project: Flink
> Issue Type: Improvement
> Components: Table API & SQL
>Affects Versions: 1.1.0
>Reporter: Till Rohrmann
>Assignee: Timo Walther
>
> The Table API's {{CsvTableSource}} does not support to read all Table API
> supported data types. For example, it is not possible to read
> {{SqlTimeTypeInfo}} types via the {{CsvTableSource}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-4248) CsvTableSource does not support reading SqlTimeTypeInfo types
[
https://issues.apache.org/jira/browse/FLINK-4248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15509099#comment-15509099
]
ASF GitHub Bot commented on FLINK-4248:
---
Github user twalthr commented on a diff in the pull request:
https://github.com/apache/flink/pull/2303#discussion_r79776134
--- Diff:
flink-core/src/test/java/org/apache/flink/types/parser/SqlDateParserTest.java
---
@@ -0,0 +1,64 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+
+package org.apache.flink.types.parser;
+
+
+import java.sql.Date;
+
+public class SqlDateParserTest extends ParserTestBase {
+
+ @Override
+ public String[] getValidTestValues() {
+ return new String[] {
+ "1970-01-01", "1990-10-14", "2013-08-12", "2040-05-12",
"2040-5-12", "1970-1-1",
+ };
+ }
+
+ @Override
+ public Date[] getValidTestResults() {
+ return new Date[] {
+ Date.valueOf("1970-01-01"), Date.valueOf("1990-10-14"),
Date.valueOf("2013-08-12"),
+ Date.valueOf("2040-05-12"), Date.valueOf("2040-05-12"),
Date.valueOf("1970-01-01")
+ };
+ }
+
+ @Override
+ public String[] getInvalidTestValues() {
+ return new String[] {
+ " 2013-08-12", "2013-08-12 ", "2013-08--12",
"13-08-12", "2013/08/12", " ", "\t",
--- End diff --
That results in a `IllegalArgumentException`.
> CsvTableSource does not support reading SqlTimeTypeInfo types
> -
>
> Key: FLINK-4248
> URL: https://issues.apache.org/jira/browse/FLINK-4248
> Project: Flink
> Issue Type: Improvement
> Components: Table API & SQL
>Affects Versions: 1.1.0
>Reporter: Till Rohrmann
>Assignee: Timo Walther
>
> The Table API's {{CsvTableSource}} does not support to read all Table API
> supported data types. For example, it is not possible to read
> {{SqlTimeTypeInfo}} types via the {{CsvTableSource}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-4248) CsvTableSource does not support reading SqlTimeTypeInfo types
[
https://issues.apache.org/jira/browse/FLINK-4248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15507021#comment-15507021
]
ASF GitHub Bot commented on FLINK-4248:
---
Github user twalthr commented on a diff in the pull request:
https://github.com/apache/flink/pull/2303#discussion_r79652411
--- Diff:
flink-core/src/test/java/org/apache/flink/types/parser/SqlTimestampParserTest.java
---
@@ -0,0 +1,69 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+
+package org.apache.flink.types.parser;
+
+
+import java.sql.Timestamp;
+
+public class SqlTimestampParserTest extends ParserTestBase {
+
+ @Override
+ public String[] getValidTestValues() {
+ return new String[] {
+ "1970-01-01 00:00:00.000", "1990-10-14 02:42:25.123",
"1990-10-14 02:42:25.12301",
+ "1990-10-14 02:42:25.12302", "2013-08-12
14:15:59.478", "2013-08-12 14:15:59.47",
+ "-01-01 00:00:00.000",
+ };
+ }
+
+ @Override
+ public Timestamp[] getValidTestResults() {
+ return new Timestamp[] {
+ Timestamp.valueOf("1970-01-01 00:00:00.000"),
Timestamp.valueOf("1990-10-14 02:42:25.123"),
+ Timestamp.valueOf("1990-10-14 02:42:25.12301"),
Timestamp.valueOf("1990-10-14 02:42:25.12302"),
+ Timestamp.valueOf("2013-08-12 14:15:59.478"),
Timestamp.valueOf("2013-08-12 14:15:59.47"),
+ Timestamp.valueOf("-01-01 00:00:00.000")
+ };
+ }
+
+ @Override
+ public String[] getInvalidTestValues() {
+ return new String[] {
+ " 2013-08-12 14:15:59.479", "2013-08-12 14:15:59.479 ",
"1970-01-01 00:00::00",
+ "00x00:00", "2013/08/12", "-01-01 00:00:00.f00",
"2013-08-12 14:15:59.4788",
--- End diff --
Yes, the maximum precision is 9 digits.
The timestamp you suggest is also valid.
> CsvTableSource does not support reading SqlTimeTypeInfo types
> -
>
> Key: FLINK-4248
> URL: https://issues.apache.org/jira/browse/FLINK-4248
> Project: Flink
> Issue Type: Improvement
> Components: Table API & SQL
>Affects Versions: 1.1.0
>Reporter: Till Rohrmann
>Assignee: Timo Walther
>
> The Table API's {{CsvTableSource}} does not support to read all Table API
> supported data types. For example, it is not possible to read
> {{SqlTimeTypeInfo}} types via the {{CsvTableSource}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-4248) CsvTableSource does not support reading SqlTimeTypeInfo types
[
https://issues.apache.org/jira/browse/FLINK-4248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15507023#comment-15507023
]
ASF GitHub Bot commented on FLINK-4248:
---
Github user twalthr commented on a diff in the pull request:
https://github.com/apache/flink/pull/2303#discussion_r79652535
--- Diff:
flink-core/src/main/java/org/apache/flink/types/parser/SqlDateParser.java ---
@@ -0,0 +1,134 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+
+package org.apache.flink.types.parser;
+
+import java.sql.Date;
+import org.apache.flink.annotation.PublicEvolving;
+
+/**
+ * Parses a text field into a {@link java.sql.Date}.
+ */
+@PublicEvolving
+public class SqlDateParser extends FieldParser {
+
+ private static final Date DATE_INSTANCE = new Date(0L);
+
+ private Date result;
+
+ @Override
+ public int parseField(byte[] bytes, int startPos, int limit, byte[]
delimiter, Date reusable) {
+ int i = startPos;
+
+ final int delimLimit = limit - delimiter.length + 1;
--- End diff --
I will try to generalize it a bit.
> CsvTableSource does not support reading SqlTimeTypeInfo types
> -
>
> Key: FLINK-4248
> URL: https://issues.apache.org/jira/browse/FLINK-4248
> Project: Flink
> Issue Type: Improvement
> Components: Table API & SQL
>Affects Versions: 1.1.0
>Reporter: Till Rohrmann
>Assignee: Timo Walther
>
> The Table API's {{CsvTableSource}} does not support to read all Table API
> supported data types. For example, it is not possible to read
> {{SqlTimeTypeInfo}} types via the {{CsvTableSource}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-4248) CsvTableSource does not support reading SqlTimeTypeInfo types
[
https://issues.apache.org/jira/browse/FLINK-4248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15507001#comment-15507001
]
ASF GitHub Bot commented on FLINK-4248:
---
Github user twalthr commented on a diff in the pull request:
https://github.com/apache/flink/pull/2303#discussion_r79651336
--- Diff:
flink-core/src/test/java/org/apache/flink/types/parser/SqlTimeParserTest.java
---
@@ -0,0 +1,63 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+
+package org.apache.flink.types.parser;
+
+
+import java.sql.Time;
+
+public class SqlTimeParserTest extends ParserTestBase {
+
+ @Override
+ public String[] getValidTestValues() {
+ return new String[] {
+ "00:00:00", "02:42:25", "14:15:51", "18:00:45",
"23:59:58", "0:0:0"
+ };
+ }
+
+ @Override
+ public Time[] getValidTestResults() {
+ return new Time[] {
+ Time.valueOf("00:00:00"), Time.valueOf("02:42:25"),
Time.valueOf("14:15:51"),
+ Time.valueOf("18:00:45"), Time.valueOf("23:59:58"),
Time.valueOf("0:0:0")
+ };
+ }
+
+ @Override
+ public String[] getInvalidTestValues() {
+ return new String[] {
+ " 00:00:00", "00:00:00 ", "00:00::00", "00x00:00",
"2013/08/12", " ", "\t"
--- End diff --
This is also not invalid. The seconds are converted to minutes.
> CsvTableSource does not support reading SqlTimeTypeInfo types
> -
>
> Key: FLINK-4248
> URL: https://issues.apache.org/jira/browse/FLINK-4248
> Project: Flink
> Issue Type: Improvement
> Components: Table API & SQL
>Affects Versions: 1.1.0
>Reporter: Till Rohrmann
>Assignee: Timo Walther
>
> The Table API's {{CsvTableSource}} does not support to read all Table API
> supported data types. For example, it is not possible to read
> {{SqlTimeTypeInfo}} types via the {{CsvTableSource}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-4248) CsvTableSource does not support reading SqlTimeTypeInfo types
[
https://issues.apache.org/jira/browse/FLINK-4248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15506986#comment-15506986
]
ASF GitHub Bot commented on FLINK-4248:
---
Github user fhueske commented on a diff in the pull request:
https://github.com/apache/flink/pull/2303#discussion_r79650020
--- Diff:
flink-core/src/test/java/org/apache/flink/types/parser/SqlDateParserTest.java
---
@@ -0,0 +1,64 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+
+package org.apache.flink.types.parser;
+
+
+import java.sql.Date;
+
+public class SqlDateParserTest extends ParserTestBase {
+
+ @Override
+ public String[] getValidTestValues() {
+ return new String[] {
+ "1970-01-01", "1990-10-14", "2013-08-12", "2040-05-12",
"2040-5-12", "1970-1-1",
+ };
+ }
+
+ @Override
+ public Date[] getValidTestResults() {
+ return new Date[] {
+ Date.valueOf("1970-01-01"), Date.valueOf("1990-10-14"),
Date.valueOf("2013-08-12"),
+ Date.valueOf("2040-05-12"), Date.valueOf("2040-05-12"),
Date.valueOf("1970-01-01")
+ };
+ }
+
+ @Override
+ public String[] getInvalidTestValues() {
+ return new String[] {
+ " 2013-08-12", "2013-08-12 ", "2013-08--12",
"13-08-12", "2013/08/12", " ", "\t",
--- End diff --
What does it make out of `"2000-14-01"`?
> CsvTableSource does not support reading SqlTimeTypeInfo types
> -
>
> Key: FLINK-4248
> URL: https://issues.apache.org/jira/browse/FLINK-4248
> Project: Flink
> Issue Type: Improvement
> Components: Table API & SQL
>Affects Versions: 1.1.0
>Reporter: Till Rohrmann
>Assignee: Timo Walther
>
> The Table API's {{CsvTableSource}} does not support to read all Table API
> supported data types. For example, it is not possible to read
> {{SqlTimeTypeInfo}} types via the {{CsvTableSource}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-4248) CsvTableSource does not support reading SqlTimeTypeInfo types
[
https://issues.apache.org/jira/browse/FLINK-4248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15506812#comment-15506812
]
ASF GitHub Bot commented on FLINK-4248:
---
Github user twalthr commented on a diff in the pull request:
https://github.com/apache/flink/pull/2303#discussion_r79632225
--- Diff:
flink-core/src/test/java/org/apache/flink/types/parser/SqlDateParserTest.java
---
@@ -0,0 +1,64 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+
+package org.apache.flink.types.parser;
+
+
+import java.sql.Date;
+
+public class SqlDateParserTest extends ParserTestBase {
+
+ @Override
+ public String[] getValidTestValues() {
+ return new String[] {
+ "1970-01-01", "1990-10-14", "2013-08-12", "2040-05-12",
"2040-5-12", "1970-1-1",
+ };
+ }
+
+ @Override
+ public Date[] getValidTestResults() {
+ return new Date[] {
+ Date.valueOf("1970-01-01"), Date.valueOf("1990-10-14"),
Date.valueOf("2013-08-12"),
+ Date.valueOf("2040-05-12"), Date.valueOf("2040-05-12"),
Date.valueOf("1970-01-01")
+ };
+ }
+
+ @Override
+ public String[] getInvalidTestValues() {
+ return new String[] {
+ " 2013-08-12", "2013-08-12 ", "2013-08--12",
"13-08-12", "2013/08/12", " ", "\t",
--- End diff --
The Java SQL date parser supports this and converts this into "2000-03-01".
> CsvTableSource does not support reading SqlTimeTypeInfo types
> -
>
> Key: FLINK-4248
> URL: https://issues.apache.org/jira/browse/FLINK-4248
> Project: Flink
> Issue Type: Improvement
> Components: Table API & SQL
>Affects Versions: 1.1.0
>Reporter: Till Rohrmann
>Assignee: Timo Walther
>
> The Table API's {{CsvTableSource}} does not support to read all Table API
> supported data types. For example, it is not possible to read
> {{SqlTimeTypeInfo}} types via the {{CsvTableSource}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-4248) CsvTableSource does not support reading SqlTimeTypeInfo types
[
https://issues.apache.org/jira/browse/FLINK-4248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15401747#comment-15401747
]
ASF GitHub Bot commented on FLINK-4248:
---
Github user twalthr commented on the issue:
https://github.com/apache/flink/pull/2303
@gallenvara I thought the same. But I'm not sure if it is worth it. You
could save ~25 lines of code per class maybe and if we want to make the parsing
more efficient (do not use the `valueOf` methods, but own implementation) the
super class becomes obsolete anyway.
> CsvTableSource does not support reading SqlTimeTypeInfo types
> -
>
> Key: FLINK-4248
> URL: https://issues.apache.org/jira/browse/FLINK-4248
> Project: Flink
> Issue Type: Improvement
> Components: Table API & SQL
>Affects Versions: 1.1.0
>Reporter: Till Rohrmann
>Assignee: Timo Walther
>
> The Table API's {{CsvTableSource}} does not support to read all Table API
> supported data types. For example, it is not possible to read
> {{SqlTimeTypeInfo}} types via the {{CsvTableSource}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-4248) CsvTableSource does not support reading SqlTimeTypeInfo types
[
https://issues.apache.org/jira/browse/FLINK-4248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15396991#comment-15396991
]
ASF GitHub Bot commented on FLINK-4248:
---
Github user gallenvara commented on the issue:
https://github.com/apache/flink/pull/2303
Hi @twalthr , the logical of parserFields method in Date/Time/TimeStamp
(also in the lastest PR for BigDecimal/BigInteger) is same, is it better to
refactor by creating super class for them to reduce the duplicate codes?
> CsvTableSource does not support reading SqlTimeTypeInfo types
> -
>
> Key: FLINK-4248
> URL: https://issues.apache.org/jira/browse/FLINK-4248
> Project: Flink
> Issue Type: Improvement
> Components: Table API & SQL
>Affects Versions: 1.1.0
>Reporter: Till Rohrmann
>Assignee: Timo Walther
>
> The Table API's {{CsvTableSource}} does not support to read all Table API
> supported data types. For example, it is not possible to read
> {{SqlTimeTypeInfo}} types via the {{CsvTableSource}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
