[jira] [Reopened] (FLINK-25508) Flink Batch mode, cluster shutdown early.
[ https://issues.apache.org/jira/browse/FLINK-25508?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] todd reopened FLINK-25508: -- > Flink Batch mode, cluster shutdown early. > - > > Key: FLINK-25508 > URL: https://issues.apache.org/jira/browse/FLINK-25508 > Project: Flink > Issue Type: Bug > Components: API / Core >Affects Versions: 1.13.3 >Reporter: todd >Priority: Major > > I encountered this problem while using hudi。 > [https://github.com/apache/hudi/issues/4469] > > OperatorCoordinator#handleEventFromOperator is in progress, but the flink > cluster has already started to stop. > > > > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Closed] (FLINK-25508) Flink Batch mode, cluster shutdown early.
[ https://issues.apache.org/jira/browse/FLINK-25508?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] todd closed FLINK-25508. Resolution: Invalid > Flink Batch mode, cluster shutdown early. > - > > Key: FLINK-25508 > URL: https://issues.apache.org/jira/browse/FLINK-25508 > Project: Flink > Issue Type: Bug > Components: API / Core >Affects Versions: 1.13.3 >Reporter: todd >Priority: Major > > I encountered this problem while using hudi。 > [https://github.com/apache/hudi/issues/4469] > > OperatorCoordinator#handleEventFromOperator is in progress, but the flink > cluster has already started to stop. > > > > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Closed] (FLINK-25508) Flink Batch mode, cluster shutdown early.
[ https://issues.apache.org/jira/browse/FLINK-25508?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] todd closed FLINK-25508. Resolution: Won't Do > Flink Batch mode, cluster shutdown early. > - > > Key: FLINK-25508 > URL: https://issues.apache.org/jira/browse/FLINK-25508 > Project: Flink > Issue Type: Bug > Components: API / Core >Affects Versions: 1.13.3 >Reporter: todd >Priority: Major > > I encountered this problem while using hudi。 > [https://github.com/apache/hudi/issues/4469] > > OperatorCoordinator#handleEventFromOperator is in progress, but the flink > cluster has already started to stop. > > > > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (FLINK-25508) Flink Batch mode, cluster shutdown early.
[ https://issues.apache.org/jira/browse/FLINK-25508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17468595#comment-17468595 ] todd commented on FLINK-25508: -- [~MartijnVisser] I have not subscribed to the dev mailing list, can you please send me a message > Flink Batch mode, cluster shutdown early. > - > > Key: FLINK-25508 > URL: https://issues.apache.org/jira/browse/FLINK-25508 > Project: Flink > Issue Type: Bug > Components: API / Core >Affects Versions: 1.13.3 >Reporter: todd >Priority: Major > > I encountered this problem while using hudi。 > [https://github.com/apache/hudi/issues/4469] > > OperatorCoordinator#handleEventFromOperator is in progress, but the flink > cluster has already started to stop. > > > > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Created] (FLINK-25508) Flink Batch mode, cluster shutdown early.
todd created FLINK-25508: Summary: Flink Batch mode, cluster shutdown early. Key: FLINK-25508 URL: https://issues.apache.org/jira/browse/FLINK-25508 Project: Flink Issue Type: Bug Components: API / Core Affects Versions: 1.13.3 Reporter: todd I encountered this problem while using hudi。 [https://github.com/apache/hudi/issues/4469] OperatorCoordinator#handleEventFromOperator is in progress, but the flink cluster has already started to stop. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (FLINK-23725) HadoopFsCommitter, file rename failure
[
https://issues.apache.org/jira/browse/FLINK-23725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17413898#comment-17413898
]
todd commented on FLINK-23725:
--
[~sewen] Do you have time to read this question?
> HadoopFsCommitter, file rename failure
> --
>
> Key: FLINK-23725
> URL: https://issues.apache.org/jira/browse/FLINK-23725
> Project: Flink
> Issue Type: Bug
> Components: Connectors / FileSystem, Connectors / Hadoop
> Compatibility, FileSystems
>Affects Versions: 1.11.1, 1.12.1
>Reporter: todd
>Priority: Major
>
> When the HDFS file is written, if the part file exists, only false will be
> returned if the duplicate name fails.Whether to throw an exception that
> already exists in the part, or print related logs.
>
> ```
> org.apache.flink.runtime.fs.hdfs.HadoopRecoverableFsDataOutputStream.HadoopFsCommitter#commit
> public void commit() throws IOException {
> final Path src = recoverable.tempFile();
> final Path dest = recoverable.targetFile();
> final long expectedLength = recoverable.offset();
> try {
> //always return false or ture
> fs.rename(src, dest);
> } catch (IOException e) {
> throw new IOException(
> "Committing file by rename failed: " + src + " to " + dest, e);
> }
> }
>
>
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-23725) HadoopFsCommitter, file rename failure
[
https://issues.apache.org/jira/browse/FLINK-23725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17403698#comment-17403698
]
todd commented on FLINK-23725:
--
[~sewen]
The target part file already exists. If you restore from the old ck, the old
part file will be used. Because the rename operation will only return true or
fasle, and will not overwrite the old file content. If the task is restored
from an earlier state snapshot, it is difficult to ensure data consistency.
For example, the current part-0-10 file is generated by ck-10, and my task is
restored from ck-0-5. The inprogress file generated by the new task will not be
renamed to the final part file, because it already exists in the current
directory. Part files that are reserved at one time.
> HadoopFsCommitter, file rename failure
> --
>
> Key: FLINK-23725
> URL: https://issues.apache.org/jira/browse/FLINK-23725
> Project: Flink
> Issue Type: Bug
> Components: Connectors / FileSystem, Connectors / Hadoop
> Compatibility, FileSystems
>Affects Versions: 1.11.1, 1.12.1
>Reporter: todd
>Priority: Major
>
> When the HDFS file is written, if the part file exists, only false will be
> returned if the duplicate name fails.Whether to throw an exception that
> already exists in the part, or print related logs.
>
> ```
> org.apache.flink.runtime.fs.hdfs.HadoopRecoverableFsDataOutputStream.HadoopFsCommitter#commit
> public void commit() throws IOException {
> final Path src = recoverable.tempFile();
> final Path dest = recoverable.targetFile();
> final long expectedLength = recoverable.offset();
> try {
> //always return false or ture
> fs.rename(src, dest);
> } catch (IOException e) {
> throw new IOException(
> "Committing file by rename failed: " + src + " to " + dest, e);
> }
> }
>
>
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-23725) HadoopFsCommitter, file rename failure
[
https://issues.apache.org/jira/browse/FLINK-23725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17403526#comment-17403526
]
todd commented on FLINK-23725:
--
I think that would be a problem if two files are written by different
executions without consistent states (e.g start a new job without savepoints
while keeping the files of previous executions). [~Paul Lin] You're right.
However, if you throw it out, you need to ask the user to manually come up with
the problem. This will encounter such troubles.
> HadoopFsCommitter, file rename failure
> --
>
> Key: FLINK-23725
> URL: https://issues.apache.org/jira/browse/FLINK-23725
> Project: Flink
> Issue Type: Bug
> Components: Connectors / FileSystem, Connectors / Hadoop
> Compatibility, FileSystems
>Affects Versions: 1.11.1, 1.12.1
>Reporter: todd
>Priority: Major
>
> When the HDFS file is written, if the part file exists, only false will be
> returned if the duplicate name fails.Whether to throw an exception that
> already exists in the part, or print related logs.
>
> ```
> org.apache.flink.runtime.fs.hdfs.HadoopRecoverableFsDataOutputStream.HadoopFsCommitter#commit
> public void commit() throws IOException {
> final Path src = recoverable.tempFile();
> final Path dest = recoverable.targetFile();
> final long expectedLength = recoverable.offset();
> try {
> //always return false or ture
> fs.rename(src, dest);
> } catch (IOException e) {
> throw new IOException(
> "Committing file by rename failed: " + src + " to " + dest, e);
> }
> }
>
>
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-23725) HadoopFsCommitter, file rename failure
[
https://issues.apache.org/jira/browse/FLINK-23725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17403523#comment-17403523
]
todd commented on FLINK-23725:
--
[~sewen] CC
> HadoopFsCommitter, file rename failure
> --
>
> Key: FLINK-23725
> URL: https://issues.apache.org/jira/browse/FLINK-23725
> Project: Flink
> Issue Type: Bug
> Components: Connectors / FileSystem, Connectors / Hadoop
> Compatibility, FileSystems
>Affects Versions: 1.11.1, 1.12.1
>Reporter: todd
>Priority: Major
>
> When the HDFS file is written, if the part file exists, only false will be
> returned if the duplicate name fails.Whether to throw an exception that
> already exists in the part, or print related logs.
>
> ```
> org.apache.flink.runtime.fs.hdfs.HadoopRecoverableFsDataOutputStream.HadoopFsCommitter#commit
> public void commit() throws IOException {
> final Path src = recoverable.tempFile();
> final Path dest = recoverable.targetFile();
> final long expectedLength = recoverable.offset();
> try {
> //always return false or ture
> fs.rename(src, dest);
> } catch (IOException e) {
> throw new IOException(
> "Committing file by rename failed: " + src + " to " + dest, e);
> }
> }
>
>
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-23725) HadoopFsCommitter, file rename failure
[
https://issues.apache.org/jira/browse/FLINK-23725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17403457#comment-17403457
]
todd commented on FLINK-23725:
--
[~Paul Lin] I think the target file should be deleted first instead of
throwing an exception. If an exception is thrown, it is difficult to recover
from the old ck, and you can only manually delete the existing HDFS files. If
it is up to me to modify this question, I will adopt this approach.
> HadoopFsCommitter, file rename failure
> --
>
> Key: FLINK-23725
> URL: https://issues.apache.org/jira/browse/FLINK-23725
> Project: Flink
> Issue Type: Bug
> Components: Connectors / FileSystem, Connectors / Hadoop
> Compatibility, FileSystems
>Affects Versions: 1.11.1, 1.12.1
>Reporter: todd
>Priority: Major
>
> When the HDFS file is written, if the part file exists, only false will be
> returned if the duplicate name fails.Whether to throw an exception that
> already exists in the part, or print related logs.
>
> ```
> org.apache.flink.runtime.fs.hdfs.HadoopRecoverableFsDataOutputStream.HadoopFsCommitter#commit
> public void commit() throws IOException {
> final Path src = recoverable.tempFile();
> final Path dest = recoverable.targetFile();
> final long expectedLength = recoverable.offset();
> try {
> //always return false or ture
> fs.rename(src, dest);
> } catch (IOException e) {
> throw new IOException(
> "Committing file by rename failed: " + src + " to " + dest, e);
> }
> }
>
>
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-23725) HadoopFsCommitter, file rename failure
[
https://issues.apache.org/jira/browse/FLINK-23725?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
todd updated FLINK-23725:
-
Summary: HadoopFsCommitter, file rename failure (was: HadoopFsCommitter，
file rename failure)
> HadoopFsCommitter, file rename failure
> --
>
> Key: FLINK-23725
> URL: https://issues.apache.org/jira/browse/FLINK-23725
> Project: Flink
> Issue Type: Bug
> Components: Connectors / FileSystem, Connectors / Hadoop
> Compatibility, FileSystems
>Affects Versions: 1.11.1, 1.12.1
>Reporter: todd
>Priority: Major
>
> When the HDFS file is written, if the part file exists, only false will be
> returned if the duplicate name fails.Whether to throw an exception that
> already exists in the part, or print related logs.
>
> ```
> org.apache.flink.runtime.fs.hdfs.HadoopRecoverableFsDataOutputStream.HadoopFsCommitter#commit
> public void commit() throws IOException {
> final Path src = recoverable.tempFile();
> final Path dest = recoverable.targetFile();
> final long expectedLength = recoverable.offset();
> try {
> //always return false or ture
> fs.rename(src, dest);
> } catch (IOException e) {
> throw new IOException(
> "Committing file by rename failed: " + src + " to " + dest, e);
> }
> }
>
>
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-23725) HadoopFsCommitter， file rename failure
[
https://issues.apache.org/jira/browse/FLINK-23725?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
todd updated FLINK-23725:
-
Summary: HadoopFsCommitter， file rename failure (was: HadoopFsCommitter，
Prompt for file rename failure)
> HadoopFsCommitter， file rename failure
> ---
>
> Key: FLINK-23725
> URL: https://issues.apache.org/jira/browse/FLINK-23725
> Project: Flink
> Issue Type: Bug
> Components: Connectors / FileSystem, Connectors / Hadoop
> Compatibility, FileSystems
>Affects Versions: 1.11.1, 1.12.1
>Reporter: todd
>Priority: Major
>
> When the HDFS file is written, if the part file exists, only false will be
> returned if the duplicate name fails.Whether to throw an exception that
> already exists in the part, or print related logs.
>
> ```
> org.apache.flink.runtime.fs.hdfs.HadoopRecoverableFsDataOutputStream.HadoopFsCommitter#commit
> public void commit() throws IOException {
> final Path src = recoverable.tempFile();
> final Path dest = recoverable.targetFile();
> final long expectedLength = recoverable.offset();
> try {
> //always return false or ture
> fs.rename(src, dest);
> } catch (IOException e) {
> throw new IOException(
> "Committing file by rename failed: " + src + " to " + dest, e);
> }
> }
>
>
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-23725) HadoopFsCommitter， Prompt for file rename failure
[
https://issues.apache.org/jira/browse/FLINK-23725?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
todd updated FLINK-23725:
-
Component/s: FileSystems
> HadoopFsCommitter， Prompt for file rename failure
> -
>
> Key: FLINK-23725
> URL: https://issues.apache.org/jira/browse/FLINK-23725
> Project: Flink
> Issue Type: Bug
> Components: Connectors / FileSystem, Connectors / Hadoop
> Compatibility, FileSystems
>Affects Versions: 1.11.1, 1.12.1
>Reporter: todd
>Priority: Major
>
> When the HDFS file is written, if the part file exists, only false will be
> returned if the duplicate name fails.Whether to throw an exception that
> already exists in the part, or print related logs.
>
> ```
> org.apache.flink.runtime.fs.hdfs.HadoopRecoverableFsDataOutputStream.HadoopFsCommitter#commit
> public void commit() throws IOException {
> final Path src = recoverable.tempFile();
> final Path dest = recoverable.targetFile();
> final long expectedLength = recoverable.offset();
> try {
> //always return false or ture
> fs.rename(src, dest);
> } catch (IOException e) {
> throw new IOException(
> "Committing file by rename failed: " + src + " to " + dest, e);
> }
> }
>
>
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-23725) HadoopFsCommitter， Prompt for file rename failure
[
https://issues.apache.org/jira/browse/FLINK-23725?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
todd updated FLINK-23725:
-
Description:
When the HDFS file is written, if the part file exists, only false will be
returned if the duplicate name fails.Whether to throw an exception that already
exists in the part, or print related logs.
```
org.apache.flink.runtime.fs.hdfs.HadoopRecoverableFsDataOutputStream.HadoopFsCommitter#commit
public void commit() throws IOException {
final Path src = recoverable.tempFile();
final Path dest = recoverable.targetFile();
final long expectedLength = recoverable.offset();
try {
//always return false or ture
fs.rename(src, dest);
} catch (IOException e) {
throw new IOException(
"Committing file by rename failed: " + src + " to " + dest, e);
}
}
was:
When the HDFS file is written, if the part file exists, only false will be
returned if the duplicate name fails.Whether to throw an exception that already
exists in the part, or print related logs.
```
org.apache.flink.runtime.fs.hdfs.HadoopRecoverableFsDataOutputStream.HadoopFsCommitter#commit
public void commit() throws IOException {
final Path src = recoverable.tempFile();
final Path dest = recoverable.targetFile();
final long expectedLength = recoverable.offset();
final FileStatus srcStatus;
try {
srcStatus = fs.getFileStatus(src);
} catch (IOException e) {
throw new IOException("Cannot clean commit: Staging file does not exist.");
}
if (srcStatus.getLen() != expectedLength) {
// something was done to this file since the committer was created.
// this is not the "clean" case
throw new IOException("Cannot clean commit: File has trailing junk data.");
}
try {
// return false or ture
fs.rename(src, dest);
} catch (IOException e) {
throw new IOException(
"Committing file by rename failed: " + src + " to " + dest, e);
}
}
> HadoopFsCommitter， Prompt for file rename failure
> -
>
> Key: FLINK-23725
> URL: https://issues.apache.org/jira/browse/FLINK-23725
> Project: Flink
> Issue Type: Bug
> Components: Connectors / FileSystem, Connectors / Hadoop
> Compatibility
>Affects Versions: 1.11.1, 1.12.1
>Reporter: todd
>Priority: Major
>
> When the HDFS file is written, if the part file exists, only false will be
> returned if the duplicate name fails.Whether to throw an exception that
> already exists in the part, or print related logs.
>
> ```
> org.apache.flink.runtime.fs.hdfs.HadoopRecoverableFsDataOutputStream.HadoopFsCommitter#commit
> public void commit() throws IOException {
> final Path src = recoverable.tempFile();
> final Path dest = recoverable.targetFile();
> final long expectedLength = recoverable.offset();
> try {
> //always return false or ture
> fs.rename(src, dest);
> } catch (IOException e) {
> throw new IOException(
> "Committing file by rename failed: " + src + " to " + dest, e);
> }
> }
>
>
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-23725) HadoopFsCommitter， Prompt for file rename failure
todd created FLINK-23725:
Summary: HadoopFsCommitter， Prompt for file rename failure
Key: FLINK-23725
URL: https://issues.apache.org/jira/browse/FLINK-23725
Project: Flink
Issue Type: Bug
Components: Connectors / FileSystem, Connectors / Hadoop Compatibility
Affects Versions: 1.12.1, 1.11.1
Reporter: todd
When the HDFS file is written, if the part file exists, only false will be
returned if the duplicate name fails.Whether to throw an exception that already
exists in the part, or print related logs.
```
org.apache.flink.runtime.fs.hdfs.HadoopRecoverableFsDataOutputStream.HadoopFsCommitter#commit
public void commit() throws IOException {
final Path src = recoverable.tempFile();
final Path dest = recoverable.targetFile();
final long expectedLength = recoverable.offset();
final FileStatus srcStatus;
try {
srcStatus = fs.getFileStatus(src);
} catch (IOException e) {
throw new IOException("Cannot clean commit: Staging file does not exist.");
}
if (srcStatus.getLen() != expectedLength) {
// something was done to this file since the committer was created.
// this is not the "clean" case
throw new IOException("Cannot clean commit: File has trailing junk data.");
}
try {
// return false or ture
fs.rename(src, dest);
} catch (IOException e) {
throw new IOException(
"Committing file by rename failed: " + src + " to " + dest, e);
}
}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (FLINK-22318) Support RENAME column name for ALTER TABLE statement
[
https://issues.apache.org/jira/browse/FLINK-22318?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1751#comment-1751
]
todd edited comment on FLINK-22318 at 4/27/21, 4:01 PM:
hi, [~jark]
Through today’s discussion in Pr, i think there are two points that need to be
adjusted.
1. validation of expressions of computed and watermark.
When processing convertRenameColumn , I used the following method to verify the
original expression.
```
HashMap newNameToTypeMap = new LinkedHashMap<>();
nameToTypeMap.putAll(newPhysicalFieldNamesToTypes);
nameToTypeMap.putAll(newMetadataFieldNamesToTypes);
try
{
// validate origin expression
sqlValidator.validateParameterizedExpression(expression, newNameToTypeMap);
} catch (SqlValidatorException exception) {
throw new ValidationException("rename column name error, the field
information is referenced in the %s expression.",expression );
}
```
2. build AlterTableRenameColumnOperation instead of AlterTableSchemaOperation.
was (Author: todd5167):
hi, [~jark]
Through today’s discussion in Pr, i think there are two points that need to be
adjusted.
1. validation of expressions of computed and watermark.
When processing convertRenameColumn , I used the following method to verify the
original expression.
```
HashMap newNameToTypeMap = new LinkedHashMap<>();
nameToTypeMap.putAll(newPhysicalFieldNamesToTypes);
nameToTypeMap.putAll(newMetadataFieldNamesToTypes);
try {
// validate origin expression
sqlValidator.validateParameterizedExpression(expression, newNameToTypeMap);
} catch (SqlValidatorException exception) {
throw new ValidationException("rename column name error, the field information
is referenced in the %s expression.",expression );
}
```
2. build AlterTableRenameColumnOperation instead of AlterTableSchemaOperation.
> Support RENAME column name for ALTER TABLE statement
>
>
> Key: FLINK-22318
> URL: https://issues.apache.org/jira/browse/FLINK-22318
> Project: Flink
> Issue Type: Sub-task
> Components: Table SQL / API
>Reporter: Jark Wu
>Priority: Major
> Labels: pull-request-available
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-22318) Support RENAME column name for ALTER TABLE statement
[
https://issues.apache.org/jira/browse/FLINK-22318?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1751#comment-1751
]
todd commented on FLINK-22318:
--
hi, [~jark]
Through today’s discussion in Pr, i think there are two points that need to be
adjusted.
1. validation of expressions of computed and watermark.
When processing convertRenameColumn , I used the following method to verify the
original expression.
```
HashMap newNameToTypeMap = new LinkedHashMap<>();
nameToTypeMap.putAll(newPhysicalFieldNamesToTypes);
nameToTypeMap.putAll(newMetadataFieldNamesToTypes);
try {
// validate origin expression
sqlValidator.validateParameterizedExpression(expression, newNameToTypeMap);
} catch (SqlValidatorException exception) {
throw new ValidationException("rename column name error, the field information
is referenced in the %s expression.",expression );
}
```
2. build AlterTableRenameColumnOperation instead of AlterTableSchemaOperation.
> Support RENAME column name for ALTER TABLE statement
>
>
> Key: FLINK-22318
> URL: https://issues.apache.org/jira/browse/FLINK-22318
> Project: Flink
> Issue Type: Sub-task
> Components: Table SQL / API
>Reporter: Jark Wu
>Priority: Major
> Labels: pull-request-available
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-22318) Support RENAME column name for ALTER TABLE statement
[ https://issues.apache.org/jira/browse/FLINK-22318?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17331973#comment-17331973 ] todd commented on FLINK-22318: -- [~jark] code review > Support RENAME column name for ALTER TABLE statement > > > Key: FLINK-22318 > URL: https://issues.apache.org/jira/browse/FLINK-22318 > Project: Flink > Issue Type: Sub-task > Components: Table SQL / API >Reporter: Jark Wu >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-22318) Support RENAME column name for ALTER TABLE statement
[ https://issues.apache.org/jira/browse/FLINK-22318?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17331781#comment-17331781 ] todd commented on FLINK-22318: -- Hi [~jark] mysql allows primary key column renaming，indexes and foreign keys that refer to the old column , are automatically renamed. I think when changing the column name, RowtimeAttribute of Watermark, primary key column, and partition column should also be renamed together. Modifying the column name has less impact on the job. > Support RENAME column name for ALTER TABLE statement > > > Key: FLINK-22318 > URL: https://issues.apache.org/jira/browse/FLINK-22318 > Project: Flink > Issue Type: Sub-task > Components: Table SQL / API >Reporter: Jark Wu >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-22318) Support RENAME column name for ALTER TABLE statement
[ https://issues.apache.org/jira/browse/FLINK-22318?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] todd updated FLINK-22318: - Attachment: (was: image-2021-04-26-13-58-19-302.png) > Support RENAME column name for ALTER TABLE statement > > > Key: FLINK-22318 > URL: https://issues.apache.org/jira/browse/FLINK-22318 > Project: Flink > Issue Type: Sub-task > Components: Table SQL / API >Reporter: Jark Wu >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-22318) Support RENAME column name for ALTER TABLE statement
[ https://issues.apache.org/jira/browse/FLINK-22318?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] todd updated FLINK-22318: - Attachment: image-2021-04-26-13-58-19-302.png > Support RENAME column name for ALTER TABLE statement > > > Key: FLINK-22318 > URL: https://issues.apache.org/jira/browse/FLINK-22318 > Project: Flink > Issue Type: Sub-task > Components: Table SQL / API >Reporter: Jark Wu >Priority: Major > Labels: pull-request-available > Attachments: image-2021-04-26-13-58-19-302.png > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-22318) Support RENAME column name for ALTER TABLE statement
[ https://issues.apache.org/jira/browse/FLINK-22318?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17331559#comment-17331559 ] todd commented on FLINK-22318: -- [~jark] I will try Mysql's grammar later, can you see the code related to grammar expansion, and whether it is correct. > Support RENAME column name for ALTER TABLE statement > > > Key: FLINK-22318 > URL: https://issues.apache.org/jira/browse/FLINK-22318 > Project: Flink > Issue Type: Sub-task > Components: Table SQL / API >Reporter: Jark Wu >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (FLINK-22318) Support RENAME column name for ALTER TABLE statement
[ https://issues.apache.org/jira/browse/FLINK-22318?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17331559#comment-17331559 ] todd edited comment on FLINK-22318 at 4/25/21, 3:28 PM: [~jark] I will try Mysql's grammar later, can you see the code related to grammar expansion, and whether it is correct. https://github.com/apache/flink/pull/15755/files was (Author: todd5167): [~jark] I will try Mysql's grammar later, can you see the code related to grammar expansion, and whether it is correct. > Support RENAME column name for ALTER TABLE statement > > > Key: FLINK-22318 > URL: https://issues.apache.org/jira/browse/FLINK-22318 > Project: Flink > Issue Type: Sub-task > Components: Table SQL / API >Reporter: Jark Wu >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-22318) Support RENAME column name for ALTER TABLE statement
[ https://issues.apache.org/jira/browse/FLINK-22318?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17331356#comment-17331356 ] todd commented on FLINK-22318: -- I think it should be done in two steps, the first step is to extend the SQL syntax, and the second step is to modify Schema#TableColumn#columnName. I'm not sure, PrimaryKey, RowtimeAttribute of Watermark, whether it is allowed to modify the column name. > Support RENAME column name for ALTER TABLE statement > > > Key: FLINK-22318 > URL: https://issues.apache.org/jira/browse/FLINK-22318 > Project: Flink > Issue Type: Sub-task > Components: Table SQL / API >Reporter: Jark Wu >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Issue Comment Deleted] (FLINK-22318) Support RENAME column name for ALTER TABLE statement
[ https://issues.apache.org/jira/browse/FLINK-22318?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] todd updated FLINK-22318: - Comment: was deleted (was: I think it should be done in two steps, the first step is to extend the SQL syntax, and the second step is to modify Schema#UnresolvedColumn#columnName. The current columnName is of final type, I need to remove the flinal keyword and add the setColumnName method at the same time.) > Support RENAME column name for ALTER TABLE statement > > > Key: FLINK-22318 > URL: https://issues.apache.org/jira/browse/FLINK-22318 > Project: Flink > Issue Type: Sub-task > Components: Table SQL / API >Reporter: Jark Wu >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-22318) Support RENAME column name for ALTER TABLE statement
[ https://issues.apache.org/jira/browse/FLINK-22318?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17329230#comment-17329230 ] todd commented on FLINK-22318: -- I think it should be done in two steps, the first step is to extend the SQL syntax, and the second step is to modify Schema#UnresolvedColumn#columnName. The current columnName is of final type, I need to remove the flinal keyword and add the setColumnName method at the same time. > Support RENAME column name for ALTER TABLE statement > > > Key: FLINK-22318 > URL: https://issues.apache.org/jira/browse/FLINK-22318 > Project: Flink > Issue Type: Sub-task > Components: Table SQL / API >Reporter: Jark Wu >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-22318) Support RENAME column name for ALTER TABLE statement
[ https://issues.apache.org/jira/browse/FLINK-22318?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17329224#comment-17329224 ] todd commented on FLINK-22318: -- Hi [~jark], I'm interested in this issue, could you assign it to me? > Support RENAME column name for ALTER TABLE statement > > > Key: FLINK-22318 > URL: https://issues.apache.org/jira/browse/FLINK-22318 > Project: Flink > Issue Type: Sub-task > Components: Table SQL / API >Reporter: Jark Wu >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Issue Comment Deleted] (FLINK-22318) Support RENAME column name for ALTER TABLE statement
[ https://issues.apache.org/jira/browse/FLINK-22318?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] todd updated FLINK-22318: - Comment: was deleted (was: Hi [~jark], I'm interested in this issue, could you assign it to me?) > Support RENAME column name for ALTER TABLE statement > > > Key: FLINK-22318 > URL: https://issues.apache.org/jira/browse/FLINK-22318 > Project: Flink > Issue Type: Sub-task > Components: Table SQL / API >Reporter: Jark Wu >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-22318) Support RENAME column name for ALTER TABLE statement
[ https://issues.apache.org/jira/browse/FLINK-22318?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17329112#comment-17329112 ] todd commented on FLINK-22318: -- Hi [~jark], I'm interested in this issue, could you assign it to me? > Support RENAME column name for ALTER TABLE statement > > > Key: FLINK-22318 > URL: https://issues.apache.org/jira/browse/FLINK-22318 > Project: Flink > Issue Type: Sub-task > Components: Table SQL / API >Reporter: Jark Wu >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (FLINK-22361) flink run -c not upload udf jar
[ https://issues.apache.org/jira/browse/FLINK-22361?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] todd closed FLINK-22361. Resolution: Invalid > flink run -c not upload udf jar > --- > > Key: FLINK-22361 > URL: https://issues.apache.org/jira/browse/FLINK-22361 > Project: Flink > Issue Type: Bug > Components: Command Line Client >Affects Versions: 1.11.1 >Reporter: todd >Priority: Minor > > flink run \ > -m yarn-cluster \ > -C file:////flink-demo-1.0.jar \ > x > > flink-demo-1.0.jar not in classpath, will throw class not find error > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-22361) flink run -c not upload udf jar
[ https://issues.apache.org/jira/browse/FLINK-22361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17325567#comment-17325567 ] todd commented on FLINK-22361: -- Thank you very much, I can execute the program by using -yD yarn.ship-directories combined with the -C command. But I think the parameter setting related documents are not detailed, I hope to add it. > flink run -c not upload udf jar > --- > > Key: FLINK-22361 > URL: https://issues.apache.org/jira/browse/FLINK-22361 > Project: Flink > Issue Type: Bug > Components: Command Line Client >Affects Versions: 1.11.1 >Reporter: todd >Priority: Minor > > flink run \ > -m yarn-cluster \ > -C file:////flink-demo-1.0.jar \ > x > > flink-demo-1.0.jar not in classpath, will throw class not find error > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-22361) flink run -c not upload udf jar
[ https://issues.apache.org/jira/browse/FLINK-22361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17325462#comment-17325462 ] todd commented on FLINK-22361: -- yarn.ship-files is a parameter of version 1.12. In addition, yarn.ship-files cannot be dynamically replaced. I use -Dyarn.ship-files = xxx.jar parameter does not take effect > flink run -c not upload udf jar > --- > > Key: FLINK-22361 > URL: https://issues.apache.org/jira/browse/FLINK-22361 > Project: Flink > Issue Type: Bug > Components: Command Line Client >Affects Versions: 1.11.1 >Reporter: todd >Priority: Minor > > flink run \ > -m yarn-cluster \ > -C file:////flink-demo-1.0.jar \ > x > > flink-demo-1.0.jar not in classpath, will throw class not find error > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-22361) flink run -c not upload udf jar
todd created FLINK-22361: Summary: flink run -c not upload udf jar Key: FLINK-22361 URL: https://issues.apache.org/jira/browse/FLINK-22361 Project: Flink Issue Type: Bug Components: Command Line Client Affects Versions: 1.11.1 Reporter: todd flink run \ -m yarn-cluster \ -C file:////flink-demo-1.0.jar \ x flink-demo-1.0.jar not in classpath, will throw class not find error -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (FLINK-16325) A connection check is required, and it needs to be reopened when the JDBC connection is interrupted
[ https://issues.apache.org/jira/browse/FLINK-16325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] todd closed FLINK-16325. Resolution: Invalid > A connection check is required, and it needs to be reopened when the JDBC > connection is interrupted > > > Key: FLINK-16325 > URL: https://issues.apache.org/jira/browse/FLINK-16325 > Project: Flink > Issue Type: Improvement > Components: Connectors / JDBC >Affects Versions: 1.10.0 >Reporter: todd >Priority: Minor > > JDBCOutputFormat#writeRecord. > When writing data, if the JDBC connection has been disconnected, the data > will be lost.Therefore, a connectivity judgment is required in the > writeRecord method. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (FLINK-22279) Upload the pipeline.classpaths file in yarn application mode
[ https://issues.apache.org/jira/browse/FLINK-22279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] todd closed FLINK-22279. Resolution: Invalid > Upload the pipeline.classpaths file in yarn application mode > > > Key: FLINK-22279 > URL: https://issues.apache.org/jira/browse/FLINK-22279 > Project: Flink > Issue Type: Bug > Components: Command Line Client >Affects Versions: 1.12.0 >Reporter: todd >Priority: Major > > pipeline.classpaths is a local resource package. If this file is used, all > nodes must include this file. I think that pipeline.jars should be uploaded > from the client. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-22279) Upload the pipeline.classpaths file in yarn application mode
[ https://issues.apache.org/jira/browse/FLINK-22279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17321854#comment-17321854 ] todd commented on FLINK-22279: -- Thank you very much, what you said is right. I did not pay attention to this parameter. [~fly_in_gis] > Upload the pipeline.classpaths file in yarn application mode > > > Key: FLINK-22279 > URL: https://issues.apache.org/jira/browse/FLINK-22279 > Project: Flink > Issue Type: Bug > Components: Command Line Client >Affects Versions: 1.12.0 >Reporter: todd >Priority: Major > > pipeline.classpaths is a local resource package. If this file is used, all > nodes must include this file. I think that pipeline.jars should be uploaded > from the client. -- This message was sent by Atlassian Jira (v8.3.4#803005)
