[GitHub] [flink] StephanEwen commented on issue #10560: FLINK-15174 added certificate pinning for SSL mutual auth to further protect cluster
StephanEwen commented on issue #10560: FLINK-15174 added certificate pinning for SSL mutual auth to further protect cluster URL: https://github.com/apache/flink/pull/10560#issuecomment-573546080 Rebased, waiting for a final CI run and merging then... This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] dawidwys commented on issue #10838: [hotfix][table-common] Simplify formatted exceptions
dawidwys commented on issue #10838: [hotfix][table-common] Simplify formatted
exceptions
URL: https://github.com/apache/flink/pull/10838#issuecomment-573546069
This solution has one caveat that it is not very intuitive. I think we all
(java coders) are used to passing the exception at the end. I'm afraid this
solution might be surprising for a lot of newcomers.
How about we rather add a builder:
```
public static TableExceptionBuilder tableException(String format, Object...
args);
class TableExceptionBuilder {
TableException cause(Throwable cause);
TableException build();
}
```
Or if we want to get rid off the `build` method:
```
class TableException {
public static TableException tableException(String format, Object...
args);
public TableException cause(Throwable cause) {
return new TableException(getMessage(), cause);
}
}
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot commented on issue #10838: [hotfix][table-common] Simplify formatted exceptions
flinkbot commented on issue #10838: [hotfix][table-common] Simplify formatted exceptions URL: https://github.com/apache/flink/pull/10838#issuecomment-573544312 ## CI report: * 6d7cd665ed96239643ccbc81bafacc8fd0e12e80 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-15559) Missing {{site.baseurl}} in some docs
[
https://issues.apache.org/jira/browse/FLINK-15559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17014096#comment-17014096
]
Benchao Li commented on FLINK-15559:
I know it's already in nightly tests. What I mean is we can add a {{baseurl}}
to {{build_docs.sh}}, then we can find bad links due to missing {{baseurl}}
earlier. What's your option?
> Missing {{site.baseurl}} in some docs
> -
>
> Key: FLINK-15559
> URL: https://issues.apache.org/jira/browse/FLINK-15559
> Project: Flink
> Issue Type: Improvement
> Components: Documentation
>Affects Versions: 1.10.0
>Reporter: Benchao Li
>Assignee: Benchao Li
>Priority: Major
>
> For example, in
> * dev/table/functions/systemFunctions.md
> * dev/table/functions/systemFunctions.zh.md
> * dev/table/sourceSinks.md
> * dev/table/sourceSinks.zh.md
> * dev/table/sql/queries.zh.md
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot commented on issue #10837: [FLINK-15541] Fix unstable case FlinkKinesisConsumerTest
flinkbot commented on issue #10837: [FLINK-15541] Fix unstable case FlinkKinesisConsumerTest URL: https://github.com/apache/flink/pull/10837#issuecomment-573544281 ## CI report: * 4d045d632eb72a162e5d56aa52a6819042d2f1aa UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #10825: [FLINK-15522][runtime] JobResult takes ExecutionGraph failure cause as its failure cause only if job is FAILED
flinkbot edited a comment on issue #10825: [FLINK-15522][runtime] JobResult takes ExecutionGraph failure cause as its failure cause only if job is FAILED URL: https://github.com/apache/flink/pull/10825#issuecomment-572937976 ## CI report: * 7d66fc50c8749017788386efb2908ae2087ad7fa Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/143859136) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4251) * 9b83f5d4bdb8a48a3960551a3be34c2c8c2c9cf7 Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/144099911) Azure: [PENDING](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4280) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #10787: [FLINK-15495][sql-cli] Set default planner to Blink planner for SQL Client
flinkbot edited a comment on issue #10787: [FLINK-15495][sql-cli] Set default planner to Blink planner for SQL Client URL: https://github.com/apache/flink/pull/10787#issuecomment-571493639 ## CI report: * a1b145997f6ba4834eca41304b3c9aeaf968442a Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/143359498) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4156) * 3d49191722026e32f48d1b2ef63bb63ef1675b0e Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/144097316) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4279) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-15559) Missing {{site.baseurl}} in some docs
[
https://issues.apache.org/jira/browse/FLINK-15559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17014093#comment-17014093
]
Jark Wu commented on FLINK-15559:
-
Thanks for the reporting, I assigend this issue to you. Btw, {{check_links.sh}}
is already in nightly tests.
> Missing {{site.baseurl}} in some docs
> -
>
> Key: FLINK-15559
> URL: https://issues.apache.org/jira/browse/FLINK-15559
> Project: Flink
> Issue Type: Improvement
> Components: Documentation
>Affects Versions: 1.10.0
>Reporter: Benchao Li
>Assignee: Benchao Li
>Priority: Major
>
> For example, in
> * dev/table/functions/systemFunctions.md
> * dev/table/functions/systemFunctions.zh.md
> * dev/table/sourceSinks.md
> * dev/table/sourceSinks.zh.md
> * dev/table/sql/queries.zh.md
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (FLINK-15559) Missing {{site.baseurl}} in some docs
[
https://issues.apache.org/jira/browse/FLINK-15559?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jark Wu reassigned FLINK-15559:
---
Assignee: Benchao Li
> Missing {{site.baseurl}} in some docs
> -
>
> Key: FLINK-15559
> URL: https://issues.apache.org/jira/browse/FLINK-15559
> Project: Flink
> Issue Type: Improvement
> Components: Documentation
>Affects Versions: 1.10.0
>Reporter: Benchao Li
>Assignee: Benchao Li
>Priority: Major
>
> For example, in
> * dev/table/functions/systemFunctions.md
> * dev/table/functions/systemFunctions.zh.md
> * dev/table/sourceSinks.md
> * dev/table/sourceSinks.zh.md
> * dev/table/sql/queries.zh.md
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-14163) Execution#producedPartitions is possibly not assigned when used
[
https://issues.apache.org/jira/browse/FLINK-14163?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17014085#comment-17014085
]
zhijiang commented on FLINK-14163:
--
Thanks for the above good suggestions from you guys! Sorry for coming back this
issue a bit late, especially for the PR already ready.
My previous guessing was that the formal support of async way would bring big
trouble for scheduler, or it may be conflict with new scheduler direction in
long term. Also considering the shuffle async way a bit over design then and no
real users atm, so I mentioned before that I can accept the way of adjusting
into the sync way to stop loss early. Although I also thought in general it is
not a good way to break compatibility for exposed public interface. If it is
not a problem for scheduler for handling the async way in future, I am happy to
retain the async shuffle way.
If we decide to retain the async way and work around it in scheduler
temporarily, it might be better to not fail directly after checking the future
not completed. I mean we can step forward to bear a timeout before failing.
This timeout is not only used for waiting future completion, also used for
waiting for the future return by shuffle master while calling to avoid main
thread stuck long time.
> Execution#producedPartitions is possibly not assigned when used
> ---
>
> Key: FLINK-14163
> URL: https://issues.apache.org/jira/browse/FLINK-14163
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination
>Affects Versions: 1.9.0, 1.10.0
>Reporter: Zhu Zhu
>Assignee: Yuan Mei
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.10.0
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Currently {{Execution#producedPartitions}} is assigned after the partitions
> have completed the registration to shuffle master in
> {{Execution#registerProducedPartitions(...)}}.
> The partition registration is an async interface
> ({{ShuffleMaster#registerPartitionWithProducer(...)}}), so
> {{Execution#producedPartitions}} is possible[1] not set when used.
> Usages includes:
> 1. deploying this task, so that the task may be deployed without its result
> partitions assigned, and the job would hang. (DefaultScheduler issue only,
> since legacy scheduler handled this case)
> 2. generating input descriptors for downstream tasks:
> 3. retrieve {{ResultPartitionID}} for partition releasing:
> [1] If a user uses Flink default shuffle master {{NettyShuffleMaster}}, it is
> not problematic at the moment since it returns a completed future on
> registration, so that it would be a synchronized process. However, if users
> implement their own shuffle service in which the
> {{ShuffleMaster#registerPartitionWithProducer}} returns an pending future, it
> can be a problem. This is possible since customizable shuffle service is open
> to users since 1.9 (via config "shuffle-service-factory.class").
> To avoid issues to happen, we may either
> 1. fix all the usages of {{Execution#producedPartitions}} regarding the async
> assigning, or
> 2. change {{ShuffleMaster#registerPartitionWithProducer(...)}} to a sync
> interface
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot commented on issue #10837: [FLINK-15541] Fix unstable case FlinkKinesisConsumerTest
flinkbot commented on issue #10837: [FLINK-15541] Fix unstable case FlinkKinesisConsumerTest URL: https://github.com/apache/flink/pull/10837#issuecomment-573541515 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit 4d045d632eb72a162e5d56aa52a6819042d2f1aa (Mon Jan 13 07:40:30 UTC 2020) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! * **This pull request references an unassigned [Jira ticket](https://issues.apache.org/jira/browse/FLINK-15541).** According to the [code contribution guide](https://flink.apache.org/contributing/contribute-code.html), tickets need to be assigned before starting with the implementation work. Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Closed] (FLINK-15512) Refactor the mechanism of how to constructure the cache and write buffer manager shared across RocksDB instances
[
https://issues.apache.org/jira/browse/FLINK-15512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yu Li closed FLINK-15512.
-
Resolution: Implemented
Merged in
master via: 346e2e02af385d7482376c25d2c3de09b89c
release-1.10 via: 1cd7cee8fc02061945220dcd8d83abf0f04cdaf6
> Refactor the mechanism of how to constructure the cache and write buffer
> manager shared across RocksDB instances
> -
>
> Key: FLINK-15512
> URL: https://issues.apache.org/jira/browse/FLINK-15512
> Project: Flink
> Issue Type: Sub-task
> Components: Runtime / State Backends
>Reporter: Yun Tang
>Assignee: Yun Tang
>Priority: Blocker
> Labels: pull-request-available
> Fix For: 1.10.0
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> FLINK-14484 introduce a {{LRUCache}} to share among RocksDB instances, so
> that the memory usage by RocksDB could be controlled well. However, due to
> the implementation and some bugs in RocksDB
> ([issue-6247|https://github.com/facebook/rocksdb/issues/6247]), we cannot
> limit the memory strictly.
> The way to walk around this issue is to consider the buffer which memtable
> would overuse (1/2 write buffer manager size). By introducing this, the
> actual cache size for user to share is not the same as the managed off-heap
> memory or user configured memory.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot commented on issue #10838: [hotfix][table-common] Simplify formatted exceptions
flinkbot commented on issue #10838: [hotfix][table-common] Simplify formatted exceptions URL: https://github.com/apache/flink/pull/10838#issuecomment-573541506 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit 6d7cd665ed96239643ccbc81bafacc8fd0e12e80 (Mon Jan 13 07:40:27 UTC 2020) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] wuchong commented on issue #10815: [FLINK-15537][table-planner-blink] Type of keys should be `BinaryRow`…
wuchong commented on issue #10815: [FLINK-15537][table-planner-blink] Type of keys should be `BinaryRow`… URL: https://github.com/apache/flink/pull/10815#issuecomment-573541256 Tests are failed, please have a look. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] twalthr opened a new pull request #10838: [hotfix][table-common] Simplify formatted exceptions
twalthr opened a new pull request #10838: [hotfix][table-common] Simplify formatted exceptions URL: https://github.com/apache/flink/pull/10838 ## What is the purpose of the change Makes formatted exceptions easier and helps by providing some simple coding guidelines in the JavaDocs. ## Brief change log - New constructors for `TableException` and `ValidationException` ## Verifying this change This change is a trivial rework / code cleanup without any test coverage. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): no - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: yes - The serializers: no - The runtime per-record code paths (performance sensitive): no - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: no - The S3 file system connector: no ## Documentation - Does this pull request introduce a new feature? no - If yes, how is the feature documented? JavaDocs This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-15541) FlinkKinesisConsumerTest.testSourceSynchronization is unstable on travis.
[
https://issues.apache.org/jira/browse/FLINK-15541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17014083#comment-17014083
]
Biao Liu commented on FLINK-15541:
--
This is an obvious bug of unstable test case. Please check the PR, it's easy to
understand.
> FlinkKinesisConsumerTest.testSourceSynchronization is unstable on travis.
> -
>
> Key: FLINK-15541
> URL: https://issues.apache.org/jira/browse/FLINK-15541
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kinesis
>Affects Versions: 1.10.0
>Reporter: Xintong Song
>Priority: Blocker
> Labels: test-stability
> Fix For: 1.10.0
>
>
> [https://api.travis-ci.org/v3/job/634712405/log.txt]
> {code:java}
> 13:16:19.144 [ERROR] Tests run: 11, Failures: 1, Errors: 0, Skipped: 0, Time
> elapsed: 4.338 s <<< FAILURE! - in
> org.apache.flink.streaming.connectors.kinesis.FlinkKinesisConsumerTest
> 13:16:19.144 [ERROR]
> testSourceSynchronization(org.apache.flink.streaming.connectors.kinesis.FlinkKinesisConsumerTest)
> Time elapsed: 1.001 s <<< FAILURE!
> java.lang.AssertionError: expected null, but was: expected>
> at
> org.apache.flink.streaming.connectors.kinesis.FlinkKinesisConsumerTest.testSourceSynchronization(FlinkKinesisConsumerTest.java:1018)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-15399) Join with a LookupableTableSource：java.lang.RuntimeException: while converting XXXX Caused by: java.lang.AssertionError: Field ordinal 26 is invalid for type
[
https://issues.apache.org/jira/browse/FLINK-15399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17014081#comment-17014081
]
Jark Wu commented on FLINK-15399:
-
Moved it to 1.11.
> Join with a LookupableTableSource：java.lang.RuntimeException: while
> converting Caused by: java.lang.AssertionError: Field ordinal 26 is
> invalid for type
> ---
>
> Key: FLINK-15399
> URL: https://issues.apache.org/jira/browse/FLINK-15399
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / API
>Affects Versions: 1.9.1
> Environment: jdk1.8.0_211
>Reporter: Rockey Cui
>Priority: Major
> Fix For: 1.11.0
>
> Attachments: JoinTest-1.0-SNAPSHOT.jar
>
>
>
> {code:java}
> //代码占位符
> public static void main(String[] args) throws Exception {
> StreamExecutionEnvironment env =
> StreamExecutionEnvironment.getExecutionEnvironment();
> EnvironmentSettings settings =
> EnvironmentSettings.newInstance().useBlinkPlanner().inStreamingMode().build();
> StreamTableEnvironment tableEnv = StreamTableEnvironment.create(env,
> settings);
> env.setParallelism(1);
> DataStreamSource stringDataStreamSource1 = env.fromElements(
> "HA"
> );
> String[] fields1 = new String[]{"ORD_ID", "PS_PARTKEY", "PS_SUPPKEY",
> "PS_AVAILQTY", "PS_SUPPLYCOST", "PS_COMMENT"
> // key
> , "PS_INT", "PS_LONG"
> , "PS_DOUBLE8", "PS_DOUBLE14", "PS_DOUBLE15"
> , "PS_NUMBER1", "PS_NUMBER2", "PS_NUMBER3", "PS_NUMBER4"
> , "PS_DATE", "PS_TIMESTAMP", "PS_DATE_EVENT",
> "PS_TIMESTAMP_EVENT"};
> TypeInformation[] types1 = new TypeInformation[]{Types.STRING,
> Types.INT, Types.LONG, Types.LONG, Types.DOUBLE, Types.STRING
> // key
> , Types.INT, Types.LONG
> , Types.DOUBLE, Types.DOUBLE, Types.DOUBLE
> , Types.LONG, Types.LONG, Types.DOUBLE, Types.DOUBLE
> , Types.SQL_DATE, Types.SQL_TIMESTAMP, Types.SQL_DATE,
> Types.SQL_TIMESTAMP};
> RowTypeInfo typeInformation1 = new RowTypeInfo(types1, fields1);
> DataStream stream1 = stringDataStreamSource1.map(new
> MapFunction() {
> private static final long serialVersionUID = 2349572544179673356L;
> @Override
> public Row map(String s) {
> return new Row(typeInformation1.getArity());
> }
> }).returns(typeInformation1);
> tableEnv.registerDataStream("FUN_1", stream1, String.join(",",
> typeInformation1.getFieldNames()) + ",PROCTIME.proctime");
> DataStreamSource stringDataStreamSource2 = env.fromElements(
> "HA"
> );
> String[] fields2 = new String[]{"C_NAME", "C_ADDRESS", "C_NATIONKEY"
> // key
> , "C_INT", "C_LONG"
> , "C_DOUBLE8", "C_DOUBLE14"
> , "C_DATE_EVENT", "C_TIMESTAMP_EVENT"};
> TypeInformation[] types2 = new TypeInformation[]{Types.STRING,
> Types.STRING, Types.LONG
> // key
> , Types.INT, Types.LONG
> , Types.DOUBLE, Types.DOUBLE
> , Types.SQL_DATE, Types.SQL_TIMESTAMP};
> RowTypeInfo typeInformation2 = new RowTypeInfo(types2, fields2);
> DataStream stream2 = stringDataStreamSource2.map(new
> MapFunction() {
> private static final long serialVersionUID = 2349572544179673349L;
> @Override
> public Row map(String s) {
> return new Row(typeInformation2.getArity());
> }
> }).returns(typeInformation2);
> tableEnv.registerDataStream("FUN_2", stream2, String.join(",",
> typeInformation2.getFieldNames()) + ",PROCTIME.proctime");
> MyLookupTableSource tableSource = MyLookupTableSource.newBuilder()
> .withFieldNames(new String[]{
> "S_NAME", "S_ADDRESS", "S_PHONE"
> , "S_ACCTBAL", "S_COMMENT"
> // key
> , "S_INT", "S_LONG"
> , "S_DOUBLE8", "S_DOUBLE14"
> , "S_DOUBLE15", "S_DATE_EVENT", "S_TIMESTAMP_EVENT"})
> .withFieldTypes(new TypeInformation[]{
> Types.STRING, Types.STRING, Types.STRING
> , Types.DOUBLE, Types.STRING
> // key
> , Types.INT, Types.LONG
> , Types.DOUBLE, Types.DOUBLE
> , Types.DOUBLE, Types.SQL_DATE, Types.SQL_TIMESTAMP})
> .build();
> tableEnv.registerTableSource("INFO", tableSource);
> String sql = "SELECT LN(F.PS_INT),LOG(F2.C_INT,1)\n" +
> " FROM (SELECT *\n" +
> " FROM FUN_1 F1\n" +
> " JOIN INFO FOR SYSTEM_TIME AS OF
[jira] [Updated] (FLINK-15399) Join with a LookupableTableSource：java.lang.RuntimeException: while converting XXXX Caused by: java.lang.AssertionError: Field ordinal 26 is invalid for type
[
https://issues.apache.org/jira/browse/FLINK-15399?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jark Wu updated FLINK-15399:
Fix Version/s: 1.11.0
> Join with a LookupableTableSource：java.lang.RuntimeException: while
> converting Caused by: java.lang.AssertionError: Field ordinal 26 is
> invalid for type
> ---
>
> Key: FLINK-15399
> URL: https://issues.apache.org/jira/browse/FLINK-15399
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / API
>Affects Versions: 1.9.1
> Environment: jdk1.8.0_211
>Reporter: Rockey Cui
>Priority: Major
> Fix For: 1.11.0
>
> Attachments: JoinTest-1.0-SNAPSHOT.jar
>
>
>
> {code:java}
> //代码占位符
> public static void main(String[] args) throws Exception {
> StreamExecutionEnvironment env =
> StreamExecutionEnvironment.getExecutionEnvironment();
> EnvironmentSettings settings =
> EnvironmentSettings.newInstance().useBlinkPlanner().inStreamingMode().build();
> StreamTableEnvironment tableEnv = StreamTableEnvironment.create(env,
> settings);
> env.setParallelism(1);
> DataStreamSource stringDataStreamSource1 = env.fromElements(
> "HA"
> );
> String[] fields1 = new String[]{"ORD_ID", "PS_PARTKEY", "PS_SUPPKEY",
> "PS_AVAILQTY", "PS_SUPPLYCOST", "PS_COMMENT"
> // key
> , "PS_INT", "PS_LONG"
> , "PS_DOUBLE8", "PS_DOUBLE14", "PS_DOUBLE15"
> , "PS_NUMBER1", "PS_NUMBER2", "PS_NUMBER3", "PS_NUMBER4"
> , "PS_DATE", "PS_TIMESTAMP", "PS_DATE_EVENT",
> "PS_TIMESTAMP_EVENT"};
> TypeInformation[] types1 = new TypeInformation[]{Types.STRING,
> Types.INT, Types.LONG, Types.LONG, Types.DOUBLE, Types.STRING
> // key
> , Types.INT, Types.LONG
> , Types.DOUBLE, Types.DOUBLE, Types.DOUBLE
> , Types.LONG, Types.LONG, Types.DOUBLE, Types.DOUBLE
> , Types.SQL_DATE, Types.SQL_TIMESTAMP, Types.SQL_DATE,
> Types.SQL_TIMESTAMP};
> RowTypeInfo typeInformation1 = new RowTypeInfo(types1, fields1);
> DataStream stream1 = stringDataStreamSource1.map(new
> MapFunction() {
> private static final long serialVersionUID = 2349572544179673356L;
> @Override
> public Row map(String s) {
> return new Row(typeInformation1.getArity());
> }
> }).returns(typeInformation1);
> tableEnv.registerDataStream("FUN_1", stream1, String.join(",",
> typeInformation1.getFieldNames()) + ",PROCTIME.proctime");
> DataStreamSource stringDataStreamSource2 = env.fromElements(
> "HA"
> );
> String[] fields2 = new String[]{"C_NAME", "C_ADDRESS", "C_NATIONKEY"
> // key
> , "C_INT", "C_LONG"
> , "C_DOUBLE8", "C_DOUBLE14"
> , "C_DATE_EVENT", "C_TIMESTAMP_EVENT"};
> TypeInformation[] types2 = new TypeInformation[]{Types.STRING,
> Types.STRING, Types.LONG
> // key
> , Types.INT, Types.LONG
> , Types.DOUBLE, Types.DOUBLE
> , Types.SQL_DATE, Types.SQL_TIMESTAMP};
> RowTypeInfo typeInformation2 = new RowTypeInfo(types2, fields2);
> DataStream stream2 = stringDataStreamSource2.map(new
> MapFunction() {
> private static final long serialVersionUID = 2349572544179673349L;
> @Override
> public Row map(String s) {
> return new Row(typeInformation2.getArity());
> }
> }).returns(typeInformation2);
> tableEnv.registerDataStream("FUN_2", stream2, String.join(",",
> typeInformation2.getFieldNames()) + ",PROCTIME.proctime");
> MyLookupTableSource tableSource = MyLookupTableSource.newBuilder()
> .withFieldNames(new String[]{
> "S_NAME", "S_ADDRESS", "S_PHONE"
> , "S_ACCTBAL", "S_COMMENT"
> // key
> , "S_INT", "S_LONG"
> , "S_DOUBLE8", "S_DOUBLE14"
> , "S_DOUBLE15", "S_DATE_EVENT", "S_TIMESTAMP_EVENT"})
> .withFieldTypes(new TypeInformation[]{
> Types.STRING, Types.STRING, Types.STRING
> , Types.DOUBLE, Types.STRING
> // key
> , Types.INT, Types.LONG
> , Types.DOUBLE, Types.DOUBLE
> , Types.DOUBLE, Types.SQL_DATE, Types.SQL_TIMESTAMP})
> .build();
> tableEnv.registerTableSource("INFO", tableSource);
> String sql = "SELECT LN(F.PS_INT),LOG(F2.C_INT,1)\n" +
> " FROM (SELECT *\n" +
> " FROM FUN_1 F1\n" +
> " JOIN INFO FOR SYSTEM_TIME AS OF F1.PROCTIME D1\n" +
>
[jira] [Updated] (FLINK-15399) Join with a LookupableTableSource：java.lang.RuntimeException: while converting XXXX Caused by: java.lang.AssertionError: Field ordinal 26 is invalid for type
[
https://issues.apache.org/jira/browse/FLINK-15399?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jark Wu updated FLINK-15399:
Fix Version/s: (was: 1.10.0)
> Join with a LookupableTableSource：java.lang.RuntimeException: while
> converting Caused by: java.lang.AssertionError: Field ordinal 26 is
> invalid for type
> ---
>
> Key: FLINK-15399
> URL: https://issues.apache.org/jira/browse/FLINK-15399
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / API
>Affects Versions: 1.9.1
> Environment: jdk1.8.0_211
>Reporter: Rockey Cui
>Priority: Major
> Attachments: JoinTest-1.0-SNAPSHOT.jar
>
>
>
> {code:java}
> //代码占位符
> public static void main(String[] args) throws Exception {
> StreamExecutionEnvironment env =
> StreamExecutionEnvironment.getExecutionEnvironment();
> EnvironmentSettings settings =
> EnvironmentSettings.newInstance().useBlinkPlanner().inStreamingMode().build();
> StreamTableEnvironment tableEnv = StreamTableEnvironment.create(env,
> settings);
> env.setParallelism(1);
> DataStreamSource stringDataStreamSource1 = env.fromElements(
> "HA"
> );
> String[] fields1 = new String[]{"ORD_ID", "PS_PARTKEY", "PS_SUPPKEY",
> "PS_AVAILQTY", "PS_SUPPLYCOST", "PS_COMMENT"
> // key
> , "PS_INT", "PS_LONG"
> , "PS_DOUBLE8", "PS_DOUBLE14", "PS_DOUBLE15"
> , "PS_NUMBER1", "PS_NUMBER2", "PS_NUMBER3", "PS_NUMBER4"
> , "PS_DATE", "PS_TIMESTAMP", "PS_DATE_EVENT",
> "PS_TIMESTAMP_EVENT"};
> TypeInformation[] types1 = new TypeInformation[]{Types.STRING,
> Types.INT, Types.LONG, Types.LONG, Types.DOUBLE, Types.STRING
> // key
> , Types.INT, Types.LONG
> , Types.DOUBLE, Types.DOUBLE, Types.DOUBLE
> , Types.LONG, Types.LONG, Types.DOUBLE, Types.DOUBLE
> , Types.SQL_DATE, Types.SQL_TIMESTAMP, Types.SQL_DATE,
> Types.SQL_TIMESTAMP};
> RowTypeInfo typeInformation1 = new RowTypeInfo(types1, fields1);
> DataStream stream1 = stringDataStreamSource1.map(new
> MapFunction() {
> private static final long serialVersionUID = 2349572544179673356L;
> @Override
> public Row map(String s) {
> return new Row(typeInformation1.getArity());
> }
> }).returns(typeInformation1);
> tableEnv.registerDataStream("FUN_1", stream1, String.join(",",
> typeInformation1.getFieldNames()) + ",PROCTIME.proctime");
> DataStreamSource stringDataStreamSource2 = env.fromElements(
> "HA"
> );
> String[] fields2 = new String[]{"C_NAME", "C_ADDRESS", "C_NATIONKEY"
> // key
> , "C_INT", "C_LONG"
> , "C_DOUBLE8", "C_DOUBLE14"
> , "C_DATE_EVENT", "C_TIMESTAMP_EVENT"};
> TypeInformation[] types2 = new TypeInformation[]{Types.STRING,
> Types.STRING, Types.LONG
> // key
> , Types.INT, Types.LONG
> , Types.DOUBLE, Types.DOUBLE
> , Types.SQL_DATE, Types.SQL_TIMESTAMP};
> RowTypeInfo typeInformation2 = new RowTypeInfo(types2, fields2);
> DataStream stream2 = stringDataStreamSource2.map(new
> MapFunction() {
> private static final long serialVersionUID = 2349572544179673349L;
> @Override
> public Row map(String s) {
> return new Row(typeInformation2.getArity());
> }
> }).returns(typeInformation2);
> tableEnv.registerDataStream("FUN_2", stream2, String.join(",",
> typeInformation2.getFieldNames()) + ",PROCTIME.proctime");
> MyLookupTableSource tableSource = MyLookupTableSource.newBuilder()
> .withFieldNames(new String[]{
> "S_NAME", "S_ADDRESS", "S_PHONE"
> , "S_ACCTBAL", "S_COMMENT"
> // key
> , "S_INT", "S_LONG"
> , "S_DOUBLE8", "S_DOUBLE14"
> , "S_DOUBLE15", "S_DATE_EVENT", "S_TIMESTAMP_EVENT"})
> .withFieldTypes(new TypeInformation[]{
> Types.STRING, Types.STRING, Types.STRING
> , Types.DOUBLE, Types.STRING
> // key
> , Types.INT, Types.LONG
> , Types.DOUBLE, Types.DOUBLE
> , Types.DOUBLE, Types.SQL_DATE, Types.SQL_TIMESTAMP})
> .build();
> tableEnv.registerTableSource("INFO", tableSource);
> String sql = "SELECT LN(F.PS_INT),LOG(F2.C_INT,1)\n" +
> " FROM (SELECT *\n" +
> " FROM FUN_1 F1\n" +
> " JOIN INFO FOR SYSTEM_TIME AS OF F1.PROCTIME D1\n" +
> " ON F1.PS_INT =
[jira] [Commented] (FLINK-15559) Missing {{site.baseurl}} in some docs
[
https://issues.apache.org/jira/browse/FLINK-15559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17014080#comment-17014080
]
Benchao Li commented on FLINK-15559:
Do you think it's beneficial that we add an option in {{build_docs.sh}}, which
adds a baseurl?
Then we can check bad links due to missing {{site.baseurl}} easily. A step
further, we can add this to nightly tests.
> Missing {{site.baseurl}} in some docs
> -
>
> Key: FLINK-15559
> URL: https://issues.apache.org/jira/browse/FLINK-15559
> Project: Flink
> Issue Type: Improvement
> Components: Documentation
>Affects Versions: 1.10.0
>Reporter: Benchao Li
>Priority: Major
>
> For example, in
> * dev/table/functions/systemFunctions.md
> * dev/table/functions/systemFunctions.zh.md
> * dev/table/sourceSinks.md
> * dev/table/sourceSinks.zh.md
> * dev/table/sql/queries.zh.md
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] ifndef-SleePy opened a new pull request #10837: [hotfix] Fix unstable case FlinkKinesisConsumerTest
ifndef-SleePy opened a new pull request #10837: [hotfix] Fix unstable case FlinkKinesisConsumerTest URL: https://github.com/apache/flink/pull/10837 ## What is the purpose of the change * It's a quick fix for `FlinkKinesisConsumerTest#testSourceSynchronization`. This checking should happen before `throwOnCollect` is set. ## Brief change log * Fix the bug of wrong checking introduced by FLINK-15301 ## Verifying this change * This is a fixing of test case ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): no - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: no - The serializers: no - The runtime per-record code paths (performance sensitive): no - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: no - The S3 file system connector: no ## Documentation - Does this pull request introduce a new feature? no - If yes, how is the feature documented? not applicable This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-15559) Missing {{site.baseurl}} in some docs
[
https://issues.apache.org/jira/browse/FLINK-15559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17014078#comment-17014078
]
Benchao Li commented on FLINK-15559:
[~jark] Yes.
I changed baseurl in local Jekyll proxy, and run {{check_links.sh}}, and found
some bad links due to missing {{site.baseurl}}.
> Missing {{site.baseurl}} in some docs
> -
>
> Key: FLINK-15559
> URL: https://issues.apache.org/jira/browse/FLINK-15559
> Project: Flink
> Issue Type: Improvement
> Components: Documentation
>Affects Versions: 1.10.0
>Reporter: Benchao Li
>Priority: Major
>
> For example, in
> * dev/table/functions/systemFunctions.md
> * dev/table/functions/systemFunctions.zh.md
> * dev/table/sourceSinks.md
> * dev/table/sourceSinks.zh.md
> * dev/table/sql/queries.zh.md
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-15553) Create table ddl support comment after computed column
[
https://issues.apache.org/jira/browse/FLINK-15553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17014077#comment-17014077
]
Jark Wu commented on FLINK-15553:
-
cc [~danny0405]
> Create table ddl support comment after computed column
> ---
>
> Key: FLINK-15553
> URL: https://issues.apache.org/jira/browse/FLINK-15553
> Project: Flink
> Issue Type: Improvement
> Components: Table SQL / API
>Affects Versions: 1.10.0
>Reporter: hailong wang
>Priority: Major
> Fix For: 1.11.0
>
>
> For now, we can define computed column in create table ddl, but we can not
> add comment after it just like regular table column, So we should support it,
> it's grammar as follows:
> {code:java}
> col_name AS expr [COMMENT 'string']
> {code}
> My idea is, we can introduce class
> {code:java}
> SqlTableComputedColumn{code}
> to wrap name, expression and comment, And just get the element from it will
> be ok.
> As for parserImpls.ftl, it can be like as follows:
> {code:java}
> identifier = SimpleIdentifier()
>
> expr = Expression(ExprContext.ACCEPT_NON_QUERY)
> [ {
> String p = SqlParserUtil.parseString(token.image);
> comment = SqlLiteral.createCharString(p, getPos());
> }]
> {
> SqlTableComputedColumn tableComputedColumn =
> new SqlTableComputedColumn(identifier, expr, comment, getPos());
> context.columnList.add(tableComputedColumn);
> }{code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (FLINK-14313) Add Gojek to Chinese Powered By page
[ https://issues.apache.org/jira/browse/FLINK-14313?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jark Wu reassigned FLINK-14313: --- Assignee: Jiawei Wu > Add Gojek to Chinese Powered By page > > > Key: FLINK-14313 > URL: https://issues.apache.org/jira/browse/FLINK-14313 > Project: Flink > Issue Type: Task > Components: chinese-translation, Project Website >Reporter: Fabian Hueske >Assignee: Jiawei Wu >Priority: Minor > > Add Gojek to Chinese Powered By Flink page. > The relevant commit is: 7fc857030998ea8ce6366bfec63850e08e24c563 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-14684) Add Pinterest to Chinese Powered By page
[ https://issues.apache.org/jira/browse/FLINK-14684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17014074#comment-17014074 ] Jark Wu commented on FLINK-14684: - Hi [~Jiawei Wu], would be better to separate them in different PRs. I also assigned FLINK-14313 to you as looks like you are interested in it. > Add Pinterest to Chinese Powered By page > > > Key: FLINK-14684 > URL: https://issues.apache.org/jira/browse/FLINK-14684 > Project: Flink > Issue Type: New Feature > Components: chinese-translation >Reporter: Hequn Cheng >Assignee: Jiawei Wu >Priority: Minor > > Pinterest was added to the English Powered By page with commit: > [51f7e3ced85b94dcbe3c051069379d22c88fbc5c|https://github.com/apache/flink-web/pull/281] > It should be added to the Chinese Powered By (and index.html) page as well. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on issue #10825: [FLINK-15522][runtime] JobResult takes ExecutionGraph failure cause as its failure cause only if job is FAILED
flinkbot edited a comment on issue #10825: [FLINK-15522][runtime] JobResult takes ExecutionGraph failure cause as its failure cause only if job is FAILED URL: https://github.com/apache/flink/pull/10825#issuecomment-572937976 ## CI report: * 7d66fc50c8749017788386efb2908ae2087ad7fa Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/143859136) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4251) * 9b83f5d4bdb8a48a3960551a3be34c2c8c2c9cf7 Travis: [PENDING](https://travis-ci.com/flink-ci/flink/builds/144099911) Azure: [PENDING](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4280) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-15556) Add a switch for PushProjectIntoTableSourceScanRule
[ https://issues.apache.org/jira/browse/FLINK-15556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17014073#comment-17014073 ] Jark Wu commented on FLINK-15556: - I think in most cases reusing sources will save more resources than push projects down? [~godfreyhe] > Add a switch for PushProjectIntoTableSourceScanRule > --- > > Key: FLINK-15556 > URL: https://issues.apache.org/jira/browse/FLINK-15556 > Project: Flink > Issue Type: Improvement > Components: Table SQL / Planner >Affects Versions: 1.10.0 >Reporter: hailong wang >Priority: Minor > Fix For: 1.11.0 > > > In some case, For the project push down, The digest of tableSource maybe > different. For example, If we create a jdbc tablesource, and is used twice > after it for different column, the source can not be reused for the different > digest. In this case, the query io will more precious, So we can add a switch > to turn off project push down, and the reused the source. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-15559) Missing {{site.baseurl}} in some docs
[
https://issues.apache.org/jira/browse/FLINK-15559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17014071#comment-17014071
]
Jark Wu commented on FLINK-15559:
-
Hi [~libenchao], do you mean these links are broken because of missing
{{site.baseurl}} ?
> Missing {{site.baseurl}} in some docs
> -
>
> Key: FLINK-15559
> URL: https://issues.apache.org/jira/browse/FLINK-15559
> Project: Flink
> Issue Type: Improvement
> Components: Documentation
>Affects Versions: 1.10.0
>Reporter: Benchao Li
>Priority: Major
>
> For example, in
> * dev/table/functions/systemFunctions.md
> * dev/table/functions/systemFunctions.zh.md
> * dev/table/sourceSinks.md
> * dev/table/sourceSinks.zh.md
> * dev/table/sql/queries.zh.md
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] yanghua commented on issue #10833: [FLINK-15535][documentation] Add usage of ProcessFunctionTestHarnesses for testing documentation
yanghua commented on issue #10833: [FLINK-15535][documentation] Add usage of ProcessFunctionTestHarnesses for testing documentation URL: https://github.com/apache/flink/pull/10833#issuecomment-573534012 cc @aljoscha This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-15552) SQL Client can not correctly create kafka table using --library to indicate a kafka connector directory
[
https://issues.apache.org/jira/browse/FLINK-15552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17014069#comment-17014069
]
Jark Wu commented on FLINK-15552:
-
Hi [~Terry1897], are you sure there is kafka sql jar under
{{/xx/connectors/kafka/}} ? Because AFAIK, SQL CLI e2e tests also uses
--library and --jars to test.
> SQL Client can not correctly create kafka table using --library to indicate a
> kafka connector directory
> ---
>
> Key: FLINK-15552
> URL: https://issues.apache.org/jira/browse/FLINK-15552
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Client, Table SQL / Runtime
>Reporter: Terry Wang
>Priority: Major
>
> How to Reproduce:
> first, I start a sql client and using `-l` to point to a kafka connector
> directory.
> `
> bin/sql-client.sh embedded -l /xx/connectors/kafka/
> `
> Then, I create a Kafka Table like following
> `
> Flink SQL> CREATE TABLE MyUserTable (
> > content String
> > ) WITH (
> > 'connector.type' = 'kafka',
> > 'connector.version' = 'universal',
> > 'connector.topic' = 'test',
> > 'connector.properties.zookeeper.connect' = 'localhost:2181',
> > 'connector.properties.bootstrap.servers' = 'localhost:9092',
> > 'connector.properties.group.id' = 'testGroup',
> > 'connector.startup-mode' = 'earliest-offset',
> > 'format.type' = 'csv'
> > );
> [INFO] Table has been created.
> `
> Then I select from just created table and an exception been thrown:
> `
> Flink SQL> select * from MyUserTable;
> [ERROR] Could not execute SQL statement. Reason:
> org.apache.flink.table.api.NoMatchingTableFactoryException: Could not find a
> suitable table factory for
> 'org.apache.flink.table.factories.TableSourceFactory' in
> the classpath.
> Reason: Required context properties mismatch.
> The matching candidates:
> org.apache.flink.table.sources.CsvBatchTableSourceFactory
> Mismatched properties:
> 'connector.type' expects 'filesystem', but is 'kafka'
> The following properties are requested:
> connector.properties.bootstrap.servers=localhost:9092
> connector.properties.group.id=testGroup
> connector.properties.zookeeper.connect=localhost:2181
> connector.startup-mode=earliest-offset
> connector.topic=test
> connector.type=kafka
> connector.version=universal
> format.type=csv
> schema.0.data-type=VARCHAR(2147483647)
> schema.0.name=content
> The following factories have been considered:
> org.apache.flink.table.sources.CsvBatchTableSourceFactory
> org.apache.flink.table.sources.CsvAppendTableSourceFactory
> `
> Potential Reasons:
> Now we use `TableFactoryUtil#findAndCreateTableSource` to convert a
> CatalogTable to TableSource, but when call `TableFactoryService.find` we
> don't pass current classLoader to this method, the default loader will be
> BootStrapClassLoader, which can not find our factory.
> I verified in my box, it's truly caused by this behavior.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (FLINK-15548) Make KeyedCoProcessOperatorWithWatermarkDelay extends KeyedCoProcessOperator instead of LegacyKeyedCoProcessOperator
[ https://issues.apache.org/jira/browse/FLINK-15548?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jark Wu resolved FLINK-15548. - Fix Version/s: (was: 1.10.0) 1.11.0 Resolution: Fixed 1.11.0: e155909907a55fa04dfec7a186fd9be992bf7c86 > Make KeyedCoProcessOperatorWithWatermarkDelay extends KeyedCoProcessOperator > instead of LegacyKeyedCoProcessOperator > > > Key: FLINK-15548 > URL: https://issues.apache.org/jira/browse/FLINK-15548 > Project: Flink > Issue Type: Improvement > Components: Table SQL / Runtime >Affects Versions: 1.10.0 >Reporter: wangsan >Assignee: wangsan >Priority: Minor > Labels: pull-request-available > Fix For: 1.11.0 > > Time Spent: 20m > Remaining Estimate: 0h > > `LegacyKeyedCoProcessOperator` is marked as deprecated, we should make > `KeyedCoProcessOperatorWithWatermarkDelay` extends `KeyedCoProcessOperator` > instead. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] wuchong merged pull request #10827: [FLINK-15548] [table-runtime-blink] Make KeyedCoProcessOperatorWithWatermarkDelay extends KeyedCoProcessOperator instead of LegacyKeyedCoProcessOpe
wuchong merged pull request #10827: [FLINK-15548] [table-runtime-blink] Make KeyedCoProcessOperatorWithWatermarkDelay extends KeyedCoProcessOperator instead of LegacyKeyedCoProcessOperator URL: https://github.com/apache/flink/pull/10827 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] zhijiangW commented on a change in pull request #10736: [FLINK-15010][Network] Add shutdown hook to ensure cleanup netty shuffle directories
zhijiangW commented on a change in pull request #10736: [FLINK-15010][Network]
Add shutdown hook to ensure cleanup netty shuffle directories
URL: https://github.com/apache/flink/pull/10736#discussion_r365663716
##
File path:
flink-runtime/src/test/java/org/apache/flink/runtime/io/network/NettyShuffleEnvironmentCleanupTest.java
##
@@ -0,0 +1,181 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.io.network;
+
+import org.apache.flink.api.common.time.Deadline;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.configuration.CoreOptions;
+import org.apache.flink.configuration.HighAvailabilityOptions;
+import org.apache.flink.configuration.JobManagerOptions;
+import org.apache.flink.configuration.NettyShuffleEnvironmentOptions;
+import org.apache.flink.configuration.RestOptions;
+import org.apache.flink.configuration.TaskManagerOptions;
+import org.apache.flink.runtime.entrypoint.StandaloneSessionClusterEntrypoint;
+import org.apache.flink.runtime.taskexecutor.TaskManagerRunner;
+import org.apache.flink.runtime.testutils.TestJvmProcess;
+import org.apache.flink.runtime.zookeeper.ZooKeeperResource;
+import org.apache.flink.util.OperatingSystem;
+
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.TemporaryFolder;
+
+import java.io.BufferedWriter;
+import java.io.File;
+import java.io.FileOutputStream;
+import java.io.IOException;
+import java.io.OutputStreamWriter;
+import java.time.Duration;
+import java.util.Map;
+
+import static
org.apache.flink.runtime.testutils.CommonTestUtils.createTemporaryLog4JProperties;
+import static
org.apache.flink.runtime.testutils.CommonTestUtils.getJavaCommandPath;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.fail;
+import static org.junit.Assume.assumeTrue;
+
+/**
+ * Verifies whether netty shuffle releases all the resources on shutdown, like
the temporary directories.
+ */
+public class NettyShuffleEnvironmentCleanupTest {
+ private static final Duration TEST_TIMEOUT = Duration.ofSeconds(10);
+
+ @Rule
+ public final TemporaryFolder temporaryFolder = new TemporaryFolder();
+
+ @Rule
+ public final ZooKeeperResource zooKeeperResource = new
ZooKeeperResource();
+
+ @Test
+ public void testRemovingTmpDirectoriesOnSignals() throws Exception {
+ assumeTrue(OperatingSystem.isLinux()
+ || OperatingSystem.isFreeBSD()
+ || OperatingSystem.isSolaris()
+ || OperatingSystem.isMac());
+
+ File confDir = temporaryFolder.newFolder();
+ File confFile = new File(confDir + "/flink-conf.yaml");
+
+ File taskManagerTmpDir = temporaryFolder.newFolder();
+
+ Configuration config = new Configuration();
+ config.setString(JobManagerOptions.ADDRESS, "localhost");
+ config.setString(RestOptions.BIND_PORT, "0");
+ config.setString(HighAvailabilityOptions.HA_MODE, "zookeeper");
+ config.setString(HighAvailabilityOptions.HA_ZOOKEEPER_QUORUM,
zooKeeperResource.getConnectString());
+ config.setString(HighAvailabilityOptions.HA_STORAGE_PATH,
temporaryFolder.newFolder().getAbsolutePath());
+ config.setString(CoreOptions.TMP_DIRS,
taskManagerTmpDir.getAbsolutePath());
+ config.setString(TaskManagerOptions.MANAGED_MEMORY_SIZE, "4m");
+
config.setInteger(NettyShuffleEnvironmentOptions.NETWORK_NUM_BUFFERS, 100);
+ config.setString(TaskManagerOptions.TOTAL_FLINK_MEMORY, "512m");
+
+ try (FileOutputStream fos = new FileOutputStream(confFile);
+ BufferedWriter writer = new BufferedWriter(new
OutputStreamWriter(fos))) {
+ for (Map.Entry e :
config.toMap().entrySet()) {
+ writer.write(e.getKey());
+ writer.write(": ");
+ writer.write(e.getValue());
+ writer.newLine();
+ }
+
+
[GitHub] [flink] zhijiangW commented on a change in pull request #10736: [FLINK-15010][Network] Add shutdown hook to ensure cleanup netty shuffle directories
zhijiangW commented on a change in pull request #10736: [FLINK-15010][Network]
Add shutdown hook to ensure cleanup netty shuffle directories
URL: https://github.com/apache/flink/pull/10736#discussion_r365663503
##
File path:
flink-runtime/src/test/java/org/apache/flink/runtime/io/network/NettyShuffleEnvironmentCleanupTest.java
##
@@ -0,0 +1,181 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.io.network;
+
+import org.apache.flink.api.common.time.Deadline;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.configuration.CoreOptions;
+import org.apache.flink.configuration.HighAvailabilityOptions;
+import org.apache.flink.configuration.JobManagerOptions;
+import org.apache.flink.configuration.NettyShuffleEnvironmentOptions;
+import org.apache.flink.configuration.RestOptions;
+import org.apache.flink.configuration.TaskManagerOptions;
+import org.apache.flink.runtime.entrypoint.StandaloneSessionClusterEntrypoint;
+import org.apache.flink.runtime.taskexecutor.TaskManagerRunner;
+import org.apache.flink.runtime.testutils.TestJvmProcess;
+import org.apache.flink.runtime.zookeeper.ZooKeeperResource;
+import org.apache.flink.util.OperatingSystem;
+
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.TemporaryFolder;
+
+import java.io.BufferedWriter;
+import java.io.File;
+import java.io.FileOutputStream;
+import java.io.IOException;
+import java.io.OutputStreamWriter;
+import java.time.Duration;
+import java.util.Map;
+
+import static
org.apache.flink.runtime.testutils.CommonTestUtils.createTemporaryLog4JProperties;
+import static
org.apache.flink.runtime.testutils.CommonTestUtils.getJavaCommandPath;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.fail;
+import static org.junit.Assume.assumeTrue;
+
+/**
+ * Verifies whether netty shuffle releases all the resources on shutdown, like
the temporary directories.
+ */
+public class NettyShuffleEnvironmentCleanupTest {
+ private static final Duration TEST_TIMEOUT = Duration.ofSeconds(10);
+
+ @Rule
+ public final TemporaryFolder temporaryFolder = new TemporaryFolder();
+
+ @Rule
+ public final ZooKeeperResource zooKeeperResource = new
ZooKeeperResource();
+
+ @Test
+ public void testRemovingTmpDirectoriesOnSignals() throws Exception {
+ assumeTrue(OperatingSystem.isLinux()
+ || OperatingSystem.isFreeBSD()
+ || OperatingSystem.isSolaris()
+ || OperatingSystem.isMac());
+
+ File confDir = temporaryFolder.newFolder();
+ File confFile = new File(confDir + "/flink-conf.yaml");
+
+ File taskManagerTmpDir = temporaryFolder.newFolder();
+
+ Configuration config = new Configuration();
+ config.setString(JobManagerOptions.ADDRESS, "localhost");
+ config.setString(RestOptions.BIND_PORT, "0");
+ config.setString(HighAvailabilityOptions.HA_MODE, "zookeeper");
+ config.setString(HighAvailabilityOptions.HA_ZOOKEEPER_QUORUM,
zooKeeperResource.getConnectString());
+ config.setString(HighAvailabilityOptions.HA_STORAGE_PATH,
temporaryFolder.newFolder().getAbsolutePath());
+ config.setString(CoreOptions.TMP_DIRS,
taskManagerTmpDir.getAbsolutePath());
+ config.setString(TaskManagerOptions.MANAGED_MEMORY_SIZE, "4m");
+
config.setInteger(NettyShuffleEnvironmentOptions.NETWORK_NUM_BUFFERS, 100);
+ config.setString(TaskManagerOptions.TOTAL_FLINK_MEMORY, "512m");
+
+ try (FileOutputStream fos = new FileOutputStream(confFile);
+ BufferedWriter writer = new BufferedWriter(new
OutputStreamWriter(fos))) {
+ for (Map.Entry e :
config.toMap().entrySet()) {
+ writer.write(e.getKey());
+ writer.write(": ");
+ writer.write(e.getValue());
+ writer.newLine();
+ }
+
+
[GitHub] [flink] flinkbot edited a comment on issue #10787: [FLINK-15495][sql-cli] Set default planner to Blink planner for SQL Client
flinkbot edited a comment on issue #10787: [FLINK-15495][sql-cli] Set default planner to Blink planner for SQL Client URL: https://github.com/apache/flink/pull/10787#issuecomment-571493639 ## CI report: * a1b145997f6ba4834eca41304b3c9aeaf968442a Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/143359498) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4156) * 3d49191722026e32f48d1b2ef63bb63ef1675b0e Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/144097316) Azure: [PENDING](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4279) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] zhijiangW commented on a change in pull request #10736: [FLINK-15010][Network] Add shutdown hook to ensure cleanup netty shuffle directories
zhijiangW commented on a change in pull request #10736: [FLINK-15010][Network]
Add shutdown hook to ensure cleanup netty shuffle directories
URL: https://github.com/apache/flink/pull/10736#discussion_r365663234
##
File path:
flink-runtime/src/test/java/org/apache/flink/runtime/io/network/NettyShuffleEnvironmentCleanupTest.java
##
@@ -0,0 +1,181 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.io.network;
+
+import org.apache.flink.api.common.time.Deadline;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.configuration.CoreOptions;
+import org.apache.flink.configuration.HighAvailabilityOptions;
+import org.apache.flink.configuration.JobManagerOptions;
+import org.apache.flink.configuration.NettyShuffleEnvironmentOptions;
+import org.apache.flink.configuration.RestOptions;
+import org.apache.flink.configuration.TaskManagerOptions;
+import org.apache.flink.runtime.entrypoint.StandaloneSessionClusterEntrypoint;
+import org.apache.flink.runtime.taskexecutor.TaskManagerRunner;
+import org.apache.flink.runtime.testutils.TestJvmProcess;
+import org.apache.flink.runtime.zookeeper.ZooKeeperResource;
+import org.apache.flink.util.OperatingSystem;
+
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.TemporaryFolder;
+
+import java.io.BufferedWriter;
+import java.io.File;
+import java.io.FileOutputStream;
+import java.io.IOException;
+import java.io.OutputStreamWriter;
+import java.time.Duration;
+import java.util.Map;
+
+import static
org.apache.flink.runtime.testutils.CommonTestUtils.createTemporaryLog4JProperties;
+import static
org.apache.flink.runtime.testutils.CommonTestUtils.getJavaCommandPath;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.fail;
+import static org.junit.Assume.assumeTrue;
+
+/**
+ * Verifies whether netty shuffle releases all the resources on shutdown, like
the temporary directories.
+ */
+public class NettyShuffleEnvironmentCleanupTest {
+ private static final Duration TEST_TIMEOUT = Duration.ofSeconds(10);
+
+ @Rule
+ public final TemporaryFolder temporaryFolder = new TemporaryFolder();
+
+ @Rule
+ public final ZooKeeperResource zooKeeperResource = new
ZooKeeperResource();
+
+ @Test
+ public void testRemovingTmpDirectoriesOnSignals() throws Exception {
+ assumeTrue(OperatingSystem.isLinux()
+ || OperatingSystem.isFreeBSD()
+ || OperatingSystem.isSolaris()
+ || OperatingSystem.isMac());
+
+ File confDir = temporaryFolder.newFolder();
+ File confFile = new File(confDir + "/flink-conf.yaml");
+
+ File taskManagerTmpDir = temporaryFolder.newFolder();
+
+ Configuration config = new Configuration();
+ config.setString(JobManagerOptions.ADDRESS, "localhost");
+ config.setString(RestOptions.BIND_PORT, "0");
+ config.setString(HighAvailabilityOptions.HA_MODE, "zookeeper");
+ config.setString(HighAvailabilityOptions.HA_ZOOKEEPER_QUORUM,
zooKeeperResource.getConnectString());
+ config.setString(HighAvailabilityOptions.HA_STORAGE_PATH,
temporaryFolder.newFolder().getAbsolutePath());
+ config.setString(CoreOptions.TMP_DIRS,
taskManagerTmpDir.getAbsolutePath());
+ config.setString(TaskManagerOptions.MANAGED_MEMORY_SIZE, "4m");
+
config.setInteger(NettyShuffleEnvironmentOptions.NETWORK_NUM_BUFFERS, 100);
+ config.setString(TaskManagerOptions.TOTAL_FLINK_MEMORY, "512m");
+
+ try (FileOutputStream fos = new FileOutputStream(confFile);
+ BufferedWriter writer = new BufferedWriter(new
OutputStreamWriter(fos))) {
+ for (Map.Entry e :
config.toMap().entrySet()) {
+ writer.write(e.getKey());
+ writer.write(": ");
+ writer.write(e.getValue());
+ writer.newLine();
+ }
+
+
[GitHub] [flink] zhijiangW commented on a change in pull request #10736: [FLINK-15010][Network] Add shutdown hook to ensure cleanup netty shuffle directories
zhijiangW commented on a change in pull request #10736: [FLINK-15010][Network]
Add shutdown hook to ensure cleanup netty shuffle directories
URL: https://github.com/apache/flink/pull/10736#discussion_r365662444
##
File path:
flink-runtime/src/test/java/org/apache/flink/runtime/io/network/NettyShuffleEnvironmentCleanupTest.java
##
@@ -0,0 +1,181 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.io.network;
+
+import org.apache.flink.api.common.time.Deadline;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.configuration.CoreOptions;
+import org.apache.flink.configuration.HighAvailabilityOptions;
+import org.apache.flink.configuration.JobManagerOptions;
+import org.apache.flink.configuration.NettyShuffleEnvironmentOptions;
+import org.apache.flink.configuration.RestOptions;
+import org.apache.flink.configuration.TaskManagerOptions;
+import org.apache.flink.runtime.entrypoint.StandaloneSessionClusterEntrypoint;
+import org.apache.flink.runtime.taskexecutor.TaskManagerRunner;
+import org.apache.flink.runtime.testutils.TestJvmProcess;
+import org.apache.flink.runtime.zookeeper.ZooKeeperResource;
+import org.apache.flink.util.OperatingSystem;
+
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.TemporaryFolder;
+
+import java.io.BufferedWriter;
+import java.io.File;
+import java.io.FileOutputStream;
+import java.io.IOException;
+import java.io.OutputStreamWriter;
+import java.time.Duration;
+import java.util.Map;
+
+import static
org.apache.flink.runtime.testutils.CommonTestUtils.createTemporaryLog4JProperties;
+import static
org.apache.flink.runtime.testutils.CommonTestUtils.getJavaCommandPath;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.fail;
+import static org.junit.Assume.assumeTrue;
+
+/**
+ * Verifies whether netty shuffle releases all the resources on shutdown, like
the temporary directories.
+ */
+public class NettyShuffleEnvironmentCleanupTest {
+ private static final Duration TEST_TIMEOUT = Duration.ofSeconds(10);
+
+ @Rule
+ public final TemporaryFolder temporaryFolder = new TemporaryFolder();
+
+ @Rule
+ public final ZooKeeperResource zooKeeperResource = new
ZooKeeperResource();
+
+ @Test
+ public void testRemovingTmpDirectoriesOnSignals() throws Exception {
+ assumeTrue(OperatingSystem.isLinux()
Review comment:
If the shutdown hook only supports the following systems, maybe it is better
to not fail for other systems, i mean this test can be ignored directly instead
of failing.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Assigned] (FLINK-15548) Make KeyedCoProcessOperatorWithWatermarkDelay extends KeyedCoProcessOperator instead of LegacyKeyedCoProcessOperator
[ https://issues.apache.org/jira/browse/FLINK-15548?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jark Wu reassigned FLINK-15548: --- Assignee: wangsan > Make KeyedCoProcessOperatorWithWatermarkDelay extends KeyedCoProcessOperator > instead of LegacyKeyedCoProcessOperator > > > Key: FLINK-15548 > URL: https://issues.apache.org/jira/browse/FLINK-15548 > Project: Flink > Issue Type: Improvement > Components: Table SQL / Runtime >Affects Versions: 1.10.0 >Reporter: wangsan >Assignee: wangsan >Priority: Minor > Labels: pull-request-available > Fix For: 1.10.0 > > Time Spent: 10m > Remaining Estimate: 0h > > `LegacyKeyedCoProcessOperator` is marked as deprecated, we should make > `KeyedCoProcessOperatorWithWatermarkDelay` extends `KeyedCoProcessOperator` > instead. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-15509) Use sql cilents create view occur Unexpected exception
[
https://issues.apache.org/jira/browse/FLINK-15509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17014059#comment-17014059
]
Jark Wu commented on FLINK-15509:
-
Hi [~lzljs3620320], +1 that's the point I mentioned above. If the connector
dosn't support source then it shouldn't implement {{StreamTableSourceFactory}}
interface.
> Use sql cilents create view occur Unexpected exception
> --
>
> Key: FLINK-15509
> URL: https://issues.apache.org/jira/browse/FLINK-15509
> Project: Flink
> Issue Type: Improvement
> Components: Table SQL / Client
>Affects Versions: 1.10.0
>Reporter: Xianxun Ye
>Priority: Major
> Fix For: 1.11.0
>
>
> version:master.
> Firstly I created a table sucessful by sql clients, and then throw an
> unexcepetd exp when created a view.
> My steps:
> Flink SQL> create table myTable (id int);
> *[INFO] Table has been created.*
> Flink SQL> show tables ;
> myTable
> Flink SQL> describe myTable ;
> root
> |-- id: INT
> Flink SQL> create view myView as select * from myTable;
>
> Exception in thread "main" org.apache.flink.table.client.SqlClientException:
> Unexpected exception. This is a bug. Please consider filing an issue.
> at org.apache.flink.table.client.SqlClient.main(SqlClient.java:190)
> Caused by: org.apache.flink.table.api.ValidationException: SQL validation
> failed. findAndCreateTableSource failed.
> at
> org.apache.flink.table.calcite.FlinkPlannerImpl.validateInternal(FlinkPlannerImpl.scala:130)
> at
> org.apache.flink.table.calcite.FlinkPlannerImpl.validate(FlinkPlannerImpl.scala:105)
> at
> org.apache.flink.table.sqlexec.SqlToOperationConverter.convert(SqlToOperationConverter.java:124)
> at org.apache.flink.table.planner.ParserImpl.parse(ParserImpl.java:66)
> at
> org.apache.flink.table.api.internal.TableEnvironmentImpl.sqlQuery(TableEnvironmentImpl.java:464)
> at
> org.apache.flink.table.client.gateway.local.LocalExecutor.addView(LocalExecutor.java:300)
> at
> org.apache.flink.table.client.cli.CliClient.callCreateView(CliClient.java:579)
> at
> org.apache.flink.table.client.cli.CliClient.callCommand(CliClient.java:308)
> at java.util.Optional.ifPresent(Optional.java:159)
> at org.apache.flink.table.client.cli.CliClient.open(CliClient.java:200)
> at org.apache.flink.table.client.SqlClient.openCli(SqlClient.java:125)
> at org.apache.flink.table.client.SqlClient.start(SqlClient.java:104)
> at org.apache.flink.table.client.SqlClient.main(SqlClient.java:178)
> Caused by: org.apache.flink.table.api.TableException:
> findAndCreateTableSource failed.
> at
> org.apache.flink.table.factories.TableFactoryUtil.findAndCreateTableSource(TableFactoryUtil.java:55)
> at
> org.apache.flink.table.factories.TableFactoryUtil.findAndCreateTableSource(TableFactoryUtil.java:92)
> at
> org.apache.flink.table.catalog.DatabaseCalciteSchema.convertCatalogTable(DatabaseCalciteSchema.java:138)
> at
> org.apache.flink.table.catalog.DatabaseCalciteSchema.convertTable(DatabaseCalciteSchema.java:97)
> at
> org.apache.flink.table.catalog.DatabaseCalciteSchema.lambda$getTable$0(DatabaseCalciteSchema.java:86)
> at java.util.Optional.map(Optional.java:215)
> at
> org.apache.flink.table.catalog.DatabaseCalciteSchema.getTable(DatabaseCalciteSchema.java:76)
> at
> org.apache.calcite.jdbc.SimpleCalciteSchema.getImplicitTable(SimpleCalciteSchema.java:83)
> at org.apache.calcite.jdbc.CalciteSchema.getTable(CalciteSchema.java:289)
> at org.apache.calcite.sql.validate.EmptyScope.resolve_(EmptyScope.java:143)
> at
> org.apache.calcite.sql.validate.EmptyScope.resolveTable(EmptyScope.java:99)
> at
> org.apache.calcite.sql.validate.DelegatingScope.resolveTable(DelegatingScope.java:203)
> at
> org.apache.calcite.sql.validate.IdentifierNamespace.resolveImpl(IdentifierNamespace.java:105)
> at
> org.apache.calcite.sql.validate.IdentifierNamespace.validateImpl(IdentifierNamespace.java:177)
> at
> org.apache.calcite.sql.validate.AbstractNamespace.validate(AbstractNamespace.java:84)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.validateNamespace(SqlValidatorImpl.java:1005)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.validateQuery(SqlValidatorImpl.java:965)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.validateFrom(SqlValidatorImpl.java:3125)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.validateFrom(SqlValidatorImpl.java:3107)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.validateSelect(SqlValidatorImpl.java:3379)
> at
> org.apache.calcite.sql.validate.SelectNamespace.validateImpl(SelectNamespace.java:60)
> at
> org.apache.calcite.sql.validate.AbstractNamespace.validate(AbstractNamespace.java:84)
> at
>
[jira] [Commented] (FLINK-15565) sql planner Incompatible
[ https://issues.apache.org/jira/browse/FLINK-15565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17014056#comment-17014056 ] Kurt Young commented on FLINK-15565: cc [~lzljs3620320] > sql planner Incompatible > > > Key: FLINK-15565 > URL: https://issues.apache.org/jira/browse/FLINK-15565 > Project: Flink > Issue Type: Bug > Components: Table SQL / Planner >Affects Versions: 1.10.0 >Reporter: xiaojin.wy >Priority: Major > Fix For: 1.10.0 > > > *The sql is:* > CREATE TABLE `over10k` ( > t tinyint, > si smallint, > i int, > b bigint, > f float, > d double, > bo boolean, > s varchar, > ts timestamp, > deci decimal(4,2), > bin varchar > ) WITH ( > > 'connector.path'='/daily_regression_batch_hive_1.10/test_window_with_specific_behavior/sources/over10k.csv', > 'format.empty-column-as-null'='true', > 'format.field-delimiter'='|', > 'connector.type'='filesystem', > 'format.derive-schema'='true', > 'format.type'='csv' > ); > select s, rank() over (partition by s order by si), sum(b) over (partition > by s order by si) from over10k limit 100; > *The data is :* > 109|277|65620|4294967305|97.25|7.80|true|nick quirinius|2013-03-01 > 09:11:58.703226|27.72|undecided > 93|263|65725|4294967341|6.06|4.12|false|calvin king|2013-03-01 > 09:11:58.703299|32.44|values clariffication > 108|383|65629|4294967510|39.55|47.67|false|jessica zipper|2013-03-01 > 09:11:58.703133|74.23|nap time > 89|463|65537|4294967493|64.82|13.79|true|ethan white|2013-03-01 > 09:11:58.703243|89.52|nap time > 88|372|65645|4294967358|34.48|11.18|true|quinn thompson|2013-03-01 > 09:11:58.703168|84.86|forestry > 123|432|65626|4294967435|2.39|16.49|true|david white|2013-03-01 > 09:11:58.703136|61.24|joggying > 57|486|65551|4294967397|36.11|9.88|true|katie xylophone|2013-03-01 > 09:11:58.703142|57.10|zync studies > 59|343|65787|4294967312|66.89|6.54|true|mike laertes|2013-03-01 > 09:11:58.703209|27.56|xylophone band > 74|267|65671|4294967409|21.14|14.64|true|priscilla miller|2013-03-01 > 09:11:58.703197|89.06|undecided > 25|336|65587|4294967336|71.01|14.90|true|tom ichabod|2013-03-01 > 09:11:58.703127|74.32|zync studies > 48|346|65712|4294967315|45.01|16.08|true|zach brown|2013-03-01 > 09:11:58.703108|21.68|zync studies > 84|385|65776|4294967452|35.80|32.13|false|xavier zipper|2013-03-01 > 09:11:58.703311|99.46|education > 58|389|65766|4294967416|95.55|20.62|false|sarah miller|2013-03-01 > 09:11:58.703215|70.92|history > 22|403|65565|4294967381|99.65|35.42|false|yuri johnson|2013-03-01 > 09:11:58.703154|94.47|geology > 55|428|65733|4294967535|99.54|5.35|false|jessica king|2013-03-01 > 09:11:58.703233|30.30|forestry > 117|410|65706|4294967391|50.15|0.21|false|quinn johnson|2013-03-01 > 09:11:58.703248|65.99|yard duty > 95|423|65573|4294967378|47.59|17.37|true|alice robinson|2013-03-01 > 09:11:58.703133|54.57|linguistics > 87|332|65748|4294967320|19.83|41.67|false|fred ellison|2013-03-01 > 09:11:58.703289|79.02|mathematics > 114|263|65674|4294967405|84.44|33.18|true|victor van buren|2013-03-01 > 09:11:58.703092|63.74|linguistics > 5|369|65780|4294967488|92.02|38.59|true|zach polk|2013-03-01 > 09:11:58.703271|67.29|yard duty > -3|430|65667|4294967469|65.50|40.46|true|yuri xylophone|2013-03-01 > 09:11:58.703258|30.94|american history > 120|264|65769|4294967486|89.97|41.18|false|xavier hernandez|2013-03-01 > 09:11:58.703140|66.89|philosophy > 107|317|65634|4294967488|5.68|18.89|false|priscilla ichabod|2013-03-01 > 09:11:58.703196|39.42|joggying > 29|386|65723|4294967328|71.48|6.13|false|ulysses ichabod|2013-03-01 > 09:11:58.703215|86.65|xylophone band > 22|434|65768|4294967543|44.25|27.56|false|tom polk|2013-03-01 > 09:11:58.703306|12.30|kindergarten > -1|274|65755|4294967300|22.01|35.52|false|oscar king|2013-03-01 > 09:11:58.703141|33.35|chemistry > 6|365|65603|4294967522|18.51|5.60|false|gabriella king|2013-03-01 > 09:11:58.703104|34.20|geology > 97|414|65757|4294967325|31.82|22.37|false|rachel nixon|2013-03-01 > 09:11:58.703127|61.00|nap time > 72|448|65538|4294967524|80.09|7.73|true|luke brown|2013-03-01 > 09:11:58.703090|95.81|american history > 51|280|65589|4294967486|57.46|23.35|false|zach xylophone|2013-03-01 > 09:11:58.703299|11.54|education > 12|447|65583|4294967389|0.98|29.79|true|yuri polk|2013-03-01 > 09:11:58.703305|1.89|wind surfing > -1|360|65539|4294967464|4.08|39.51|false|oscar davidson|2013-03-01 > 09:11:58.703144|59.47|nap time > 0|380|65569|4294967425|0.94|28.93|false|sarah robinson|2013-03-01 > 09:11:58.703176|88.81|xylophone band > 66|478|65669|4294967339|23.66|38.34|true|yuri carson|2013-03-01 > 09:11:58.703228|64.68|opthamology > 12|322|65771|4294967545|84.87|10.76|false|sarah allen|2013-03-01 > 09:11:58.703271|0.79|joggying >
[jira] [Updated] (FLINK-15565) Incompatible types of expression and result type thrown in codegen
[ https://issues.apache.org/jira/browse/FLINK-15565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kurt Young updated FLINK-15565: --- Summary: Incompatible types of expression and result type thrown in codegen (was: sql planner Incompatible) > Incompatible types of expression and result type thrown in codegen > -- > > Key: FLINK-15565 > URL: https://issues.apache.org/jira/browse/FLINK-15565 > Project: Flink > Issue Type: Bug > Components: Table SQL / Planner >Affects Versions: 1.10.0 >Reporter: xiaojin.wy >Priority: Major > Fix For: 1.10.0 > > > *The sql is:* > CREATE TABLE `over10k` ( > t tinyint, > si smallint, > i int, > b bigint, > f float, > d double, > bo boolean, > s varchar, > ts timestamp, > deci decimal(4,2), > bin varchar > ) WITH ( > > 'connector.path'='/daily_regression_batch_hive_1.10/test_window_with_specific_behavior/sources/over10k.csv', > 'format.empty-column-as-null'='true', > 'format.field-delimiter'='|', > 'connector.type'='filesystem', > 'format.derive-schema'='true', > 'format.type'='csv' > ); > select s, rank() over (partition by s order by si), sum(b) over (partition > by s order by si) from over10k limit 100; > *The data is :* > 109|277|65620|4294967305|97.25|7.80|true|nick quirinius|2013-03-01 > 09:11:58.703226|27.72|undecided > 93|263|65725|4294967341|6.06|4.12|false|calvin king|2013-03-01 > 09:11:58.703299|32.44|values clariffication > 108|383|65629|4294967510|39.55|47.67|false|jessica zipper|2013-03-01 > 09:11:58.703133|74.23|nap time > 89|463|65537|4294967493|64.82|13.79|true|ethan white|2013-03-01 > 09:11:58.703243|89.52|nap time > 88|372|65645|4294967358|34.48|11.18|true|quinn thompson|2013-03-01 > 09:11:58.703168|84.86|forestry > 123|432|65626|4294967435|2.39|16.49|true|david white|2013-03-01 > 09:11:58.703136|61.24|joggying > 57|486|65551|4294967397|36.11|9.88|true|katie xylophone|2013-03-01 > 09:11:58.703142|57.10|zync studies > 59|343|65787|4294967312|66.89|6.54|true|mike laertes|2013-03-01 > 09:11:58.703209|27.56|xylophone band > 74|267|65671|4294967409|21.14|14.64|true|priscilla miller|2013-03-01 > 09:11:58.703197|89.06|undecided > 25|336|65587|4294967336|71.01|14.90|true|tom ichabod|2013-03-01 > 09:11:58.703127|74.32|zync studies > 48|346|65712|4294967315|45.01|16.08|true|zach brown|2013-03-01 > 09:11:58.703108|21.68|zync studies > 84|385|65776|4294967452|35.80|32.13|false|xavier zipper|2013-03-01 > 09:11:58.703311|99.46|education > 58|389|65766|4294967416|95.55|20.62|false|sarah miller|2013-03-01 > 09:11:58.703215|70.92|history > 22|403|65565|4294967381|99.65|35.42|false|yuri johnson|2013-03-01 > 09:11:58.703154|94.47|geology > 55|428|65733|4294967535|99.54|5.35|false|jessica king|2013-03-01 > 09:11:58.703233|30.30|forestry > 117|410|65706|4294967391|50.15|0.21|false|quinn johnson|2013-03-01 > 09:11:58.703248|65.99|yard duty > 95|423|65573|4294967378|47.59|17.37|true|alice robinson|2013-03-01 > 09:11:58.703133|54.57|linguistics > 87|332|65748|4294967320|19.83|41.67|false|fred ellison|2013-03-01 > 09:11:58.703289|79.02|mathematics > 114|263|65674|4294967405|84.44|33.18|true|victor van buren|2013-03-01 > 09:11:58.703092|63.74|linguistics > 5|369|65780|4294967488|92.02|38.59|true|zach polk|2013-03-01 > 09:11:58.703271|67.29|yard duty > -3|430|65667|4294967469|65.50|40.46|true|yuri xylophone|2013-03-01 > 09:11:58.703258|30.94|american history > 120|264|65769|4294967486|89.97|41.18|false|xavier hernandez|2013-03-01 > 09:11:58.703140|66.89|philosophy > 107|317|65634|4294967488|5.68|18.89|false|priscilla ichabod|2013-03-01 > 09:11:58.703196|39.42|joggying > 29|386|65723|4294967328|71.48|6.13|false|ulysses ichabod|2013-03-01 > 09:11:58.703215|86.65|xylophone band > 22|434|65768|4294967543|44.25|27.56|false|tom polk|2013-03-01 > 09:11:58.703306|12.30|kindergarten > -1|274|65755|4294967300|22.01|35.52|false|oscar king|2013-03-01 > 09:11:58.703141|33.35|chemistry > 6|365|65603|4294967522|18.51|5.60|false|gabriella king|2013-03-01 > 09:11:58.703104|34.20|geology > 97|414|65757|4294967325|31.82|22.37|false|rachel nixon|2013-03-01 > 09:11:58.703127|61.00|nap time > 72|448|65538|4294967524|80.09|7.73|true|luke brown|2013-03-01 > 09:11:58.703090|95.81|american history > 51|280|65589|4294967486|57.46|23.35|false|zach xylophone|2013-03-01 > 09:11:58.703299|11.54|education > 12|447|65583|4294967389|0.98|29.79|true|yuri polk|2013-03-01 > 09:11:58.703305|1.89|wind surfing > -1|360|65539|4294967464|4.08|39.51|false|oscar davidson|2013-03-01 > 09:11:58.703144|59.47|nap time > 0|380|65569|4294967425|0.94|28.93|false|sarah robinson|2013-03-01 > 09:11:58.703176|88.81|xylophone band > 66|478|65669|4294967339|23.66|38.34|true|yuri carson|2013-03-01 >
[GitHub] [flink] flinkbot edited a comment on issue #10665: [FLINK-15354] Start and stop minikube only in kubernetes related e2e tests
flinkbot edited a comment on issue #10665: [FLINK-15354] Start and stop minikube only in kubernetes related e2e tests URL: https://github.com/apache/flink/pull/10665#issuecomment-568440738 ## CI report: * 5a0d5d3d499347ca216e19175ff5f066a6d9b458 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/142099952) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=3861) * 255d89be8069b36be2b980ea6dba4798568160bb Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/143507134) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4176) * 9168102e928bacaa8026407f77a33b80a8ddeae4 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/143514708) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4179) * 1465ba2465b0fba3c91377735481db17abb1c21f Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/143667456) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4203) * 76f3dadc5062279dda9df063901a548315a8b8da Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/143714109) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4219) * 677c4fcd1b82c766f79e296883f8464daafb6a44 Travis: [PENDING](https://travis-ci.com/flink-ci/flink/builds/144091754) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4278) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Resolved] (FLINK-15485) Reopen tests when blocking issue has been resolved
[ https://issues.apache.org/jira/browse/FLINK-15485?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jark Wu resolved FLINK-15485. - Resolution: Fixed 1.11.0: 328a428a7e1cb684e5b0e6229f9c54687ffdec7b 1.10.0: d1a11feb687be233f88be14c20d95f57a0f5a0a0 > Reopen tests when blocking issue has been resolved > -- > > Key: FLINK-15485 > URL: https://issues.apache.org/jira/browse/FLINK-15485 > Project: Flink > Issue Type: Bug > Components: Tests >Reporter: Leonard Xu >Assignee: Leonard Xu >Priority: Minor > Labels: pull-request-available > Fix For: 1.10.0 > > Time Spent: 20m > Remaining Estimate: 0h > > Sometimes we close test and left comment like 'TODO/when FLINK-xx is > closed/when FLINK-xx is merged' for various reasons and ready to reopen it > after they are really fixed. > Unfortunately we missed some of them. This issue aims to reopen tests that > close by > * FLINK-12088 > * FLINK-13740 > * CALCITE-1860 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] wuchong closed pull request #10774: [FLINK-15485][tests] Reopen tests when blocking issue has been resolved.
wuchong closed pull request #10774: [FLINK-15485][tests] Reopen tests when blocking issue has been resolved. URL: https://github.com/apache/flink/pull/10774 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #10825: [FLINK-15522][runtime] JobResult takes ExecutionGraph failure cause as its failure cause only if job is FAILED
flinkbot edited a comment on issue #10825: [FLINK-15522][runtime] JobResult takes ExecutionGraph failure cause as its failure cause only if job is FAILED URL: https://github.com/apache/flink/pull/10825#issuecomment-572937976 ## CI report: * 7d66fc50c8749017788386efb2908ae2087ad7fa Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/143859136) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4251) * 9b83f5d4bdb8a48a3960551a3be34c2c8c2c9cf7 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #10787: [FLINK-15495][sql-cli] Set default planner to Blink planner for SQL Client
flinkbot edited a comment on issue #10787: [FLINK-15495][sql-cli] Set default planner to Blink planner for SQL Client URL: https://github.com/apache/flink/pull/10787#issuecomment-571493639 ## CI report: * a1b145997f6ba4834eca41304b3c9aeaf968442a Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/143359498) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4156) * 3d49191722026e32f48d1b2ef63bb63ef1675b0e Travis: [PENDING](https://travis-ci.com/flink-ci/flink/builds/144097316) Azure: [PENDING](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4279) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] KarmaGYZ commented on issue #10746: [FLINK-15417] Remove the docker volume or mount when starting Mesos e…
KarmaGYZ commented on issue #10746: [FLINK-15417] Remove the docker volume or mount when starting Mesos e… URL: https://github.com/apache/flink/pull/10746#issuecomment-573521081 Kindly ping @GJL This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] zhuzhurk commented on issue #10825: [FLINK-15522][runtime] JobResult takes ExecutionGraph failure cause as its failure cause only if job is FAILED
zhuzhurk commented on issue #10825: [FLINK-15522][runtime] JobResult takes ExecutionGraph failure cause as its failure cause only if job is FAILED URL: https://github.com/apache/flink/pull/10825#issuecomment-573517594 Thanks @tillrohrmann. Will try to backport the fix to release-1.9. Merging. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #10787: [FLINK-15495][sql-cli] Set default planner to Blink planner for SQL Client
flinkbot edited a comment on issue #10787: [FLINK-15495][sql-cli] Set default planner to Blink planner for SQL Client URL: https://github.com/apache/flink/pull/10787#issuecomment-571493639 ## CI report: * a1b145997f6ba4834eca41304b3c9aeaf968442a Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/143359498) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4156) * 3d49191722026e32f48d1b2ef63bb63ef1675b0e UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #10797: [FLINK-15172][table-blink] Optimize the operator algorithm to lazily allocate memory
flinkbot edited a comment on issue #10797: [FLINK-15172][table-blink] Optimize the operator algorithm to lazily allocate memory URL: https://github.com/apache/flink/pull/10797#issuecomment-571963932 ## CI report: * e904ab518c4d3b26bf27a89c61b9df30af54b6ed Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/143532669) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4187) * 5628de7fba5004f793d14a16fff55186ce7f3d41 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/144087946) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4277) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #10745: [FLINK-15445][connectors/jdbc] JDBC Table Source didn't work for Type…
flinkbot edited a comment on issue #10745: [FLINK-15445][connectors/jdbc] JDBC Table Source didn't work for Type… URL: https://github.com/apache/flink/pull/10745#issuecomment-570165880 ## CI report: * 862f3c5fd9e77184af08aea1231cf3a00e03b193 Travis: [CANCELED](https://travis-ci.com/flink-ci/flink/builds/142855689) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4042) * 2cc4e7b6e88e9f655235b8c97707c41f45b6b238 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/142858894) Azure: [PENDING](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4044) * 67d02dcc57087af3626bcbc5cd1cd51117de2b83 Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/142952111) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4053) * 616c1a840307eeefb7fa7fa970d65a149ff7af99 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/143225362) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4123) * 30d7534882a4c84d906dce5d81e9807cce6c051d Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/143885991) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4257) * d915cd1e7d4efc5a2fb114be21483e249f2eca0b Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/144085481) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4276) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #10665: [FLINK-15354] Start and stop minikube only in kubernetes related e2e tests
flinkbot edited a comment on issue #10665: [FLINK-15354] Start and stop minikube only in kubernetes related e2e tests URL: https://github.com/apache/flink/pull/10665#issuecomment-568440738 ## CI report: * 5a0d5d3d499347ca216e19175ff5f066a6d9b458 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/142099952) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=3861) * 255d89be8069b36be2b980ea6dba4798568160bb Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/143507134) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4176) * 9168102e928bacaa8026407f77a33b80a8ddeae4 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/143514708) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4179) * 1465ba2465b0fba3c91377735481db17abb1c21f Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/143667456) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4203) * 76f3dadc5062279dda9df063901a548315a8b8da Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/143714109) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4219) * 677c4fcd1b82c766f79e296883f8464daafb6a44 Travis: [PENDING](https://travis-ci.com/flink-ci/flink/builds/144091754) Azure: [PENDING](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4278) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (FLINK-15565) sql planner Incompatible
[ https://issues.apache.org/jira/browse/FLINK-15565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] xiaojin.wy updated FLINK-15565: --- Description: *The sql is:* CREATE TABLE `over10k` ( t tinyint, si smallint, i int, b bigint, f float, d double, bo boolean, s varchar, ts timestamp, deci decimal(4,2), bin varchar ) WITH ( 'connector.path'='/daily_regression_batch_hive_1.10/test_window_with_specific_behavior/sources/over10k.csv', 'format.empty-column-as-null'='true', 'format.field-delimiter'='|', 'connector.type'='filesystem', 'format.derive-schema'='true', 'format.type'='csv' ); select s, rank() over (partition by s order by si), sum(b) over (partition by s order by si) from over10k limit 100; *The data is :* 109|277|65620|4294967305|97.25|7.80|true|nick quirinius|2013-03-01 09:11:58.703226|27.72|undecided 93|263|65725|4294967341|6.06|4.12|false|calvin king|2013-03-01 09:11:58.703299|32.44|values clariffication 108|383|65629|4294967510|39.55|47.67|false|jessica zipper|2013-03-01 09:11:58.703133|74.23|nap time 89|463|65537|4294967493|64.82|13.79|true|ethan white|2013-03-01 09:11:58.703243|89.52|nap time 88|372|65645|4294967358|34.48|11.18|true|quinn thompson|2013-03-01 09:11:58.703168|84.86|forestry 123|432|65626|4294967435|2.39|16.49|true|david white|2013-03-01 09:11:58.703136|61.24|joggying 57|486|65551|4294967397|36.11|9.88|true|katie xylophone|2013-03-01 09:11:58.703142|57.10|zync studies 59|343|65787|4294967312|66.89|6.54|true|mike laertes|2013-03-01 09:11:58.703209|27.56|xylophone band 74|267|65671|4294967409|21.14|14.64|true|priscilla miller|2013-03-01 09:11:58.703197|89.06|undecided 25|336|65587|4294967336|71.01|14.90|true|tom ichabod|2013-03-01 09:11:58.703127|74.32|zync studies 48|346|65712|4294967315|45.01|16.08|true|zach brown|2013-03-01 09:11:58.703108|21.68|zync studies 84|385|65776|4294967452|35.80|32.13|false|xavier zipper|2013-03-01 09:11:58.703311|99.46|education 58|389|65766|4294967416|95.55|20.62|false|sarah miller|2013-03-01 09:11:58.703215|70.92|history 22|403|65565|4294967381|99.65|35.42|false|yuri johnson|2013-03-01 09:11:58.703154|94.47|geology 55|428|65733|4294967535|99.54|5.35|false|jessica king|2013-03-01 09:11:58.703233|30.30|forestry 117|410|65706|4294967391|50.15|0.21|false|quinn johnson|2013-03-01 09:11:58.703248|65.99|yard duty 95|423|65573|4294967378|47.59|17.37|true|alice robinson|2013-03-01 09:11:58.703133|54.57|linguistics 87|332|65748|4294967320|19.83|41.67|false|fred ellison|2013-03-01 09:11:58.703289|79.02|mathematics 114|263|65674|4294967405|84.44|33.18|true|victor van buren|2013-03-01 09:11:58.703092|63.74|linguistics 5|369|65780|4294967488|92.02|38.59|true|zach polk|2013-03-01 09:11:58.703271|67.29|yard duty -3|430|65667|4294967469|65.50|40.46|true|yuri xylophone|2013-03-01 09:11:58.703258|30.94|american history 120|264|65769|4294967486|89.97|41.18|false|xavier hernandez|2013-03-01 09:11:58.703140|66.89|philosophy 107|317|65634|4294967488|5.68|18.89|false|priscilla ichabod|2013-03-01 09:11:58.703196|39.42|joggying 29|386|65723|4294967328|71.48|6.13|false|ulysses ichabod|2013-03-01 09:11:58.703215|86.65|xylophone band 22|434|65768|4294967543|44.25|27.56|false|tom polk|2013-03-01 09:11:58.703306|12.30|kindergarten -1|274|65755|4294967300|22.01|35.52|false|oscar king|2013-03-01 09:11:58.703141|33.35|chemistry 6|365|65603|4294967522|18.51|5.60|false|gabriella king|2013-03-01 09:11:58.703104|34.20|geology 97|414|65757|4294967325|31.82|22.37|false|rachel nixon|2013-03-01 09:11:58.703127|61.00|nap time 72|448|65538|4294967524|80.09|7.73|true|luke brown|2013-03-01 09:11:58.703090|95.81|american history 51|280|65589|4294967486|57.46|23.35|false|zach xylophone|2013-03-01 09:11:58.703299|11.54|education 12|447|65583|4294967389|0.98|29.79|true|yuri polk|2013-03-01 09:11:58.703305|1.89|wind surfing -1|360|65539|4294967464|4.08|39.51|false|oscar davidson|2013-03-01 09:11:58.703144|59.47|nap time 0|380|65569|4294967425|0.94|28.93|false|sarah robinson|2013-03-01 09:11:58.703176|88.81|xylophone band 66|478|65669|4294967339|23.66|38.34|true|yuri carson|2013-03-01 09:11:58.703228|64.68|opthamology 12|322|65771|4294967545|84.87|10.76|false|sarah allen|2013-03-01 09:11:58.703271|0.79|joggying 79|308|65563|4294967347|4.06|44.84|false|nick underhill|2013-03-01 09:11:58.703097|76.53|industrial engineering 4|382|65719|4294967329|7.26|39.92|true|fred polk|2013-03-01 09:11:58.703073|73.64|mathematics 10|448|65675|4294967392|26.20|16.30|true|rachel laertes|2013-03-01 09:11:58.703200|18.01|xylophone band 45|281|65685|4294967513|81.33|32.22|true|oscar allen|2013-03-01 09:11:58.703285|71.38|religion 57|288|65599|4294967422|90.33|44.25|false|bob young|2013-03-01 09:11:58.703185|11.16|biology 77|452|65706|4294967512|22.90|5.35|true|bob van buren|2013-03-01 09:11:58.703290|14.58|debate
[jira] [Created] (FLINK-15565) sql planner Incompatible
xiaojin.wy created FLINK-15565: -- Summary: sql planner Incompatible Key: FLINK-15565 URL: https://issues.apache.org/jira/browse/FLINK-15565 Project: Flink Issue Type: Bug Components: Table SQL / Planner Affects Versions: 1.10.0 Reporter: xiaojin.wy Fix For: 1.10.0 *The sql is: * CREATE TABLE `over10k` ( t tinyint, si smallint, i int, b bigint, f float, d double, bo boolean, s varchar, ts timestamp, deci decimal(4,2), bin varchar ) WITH ( 'connector.path'='/daily_regression_batch_hive_1.10/test_window_with_specific_behavior/sources/over10k.csv', 'format.empty-column-as-null'='true', 'format.field-delimiter'='|', 'connector.type'='filesystem', 'format.derive-schema'='true', 'format.type'='csv' ); select s, rank() over (partition by s order by si), sum(b) over (partition by s order by si) from over10k limit 100; *The data is :* 109|277|65620|4294967305|97.25|7.80|true|nick quirinius|2013-03-01 09:11:58.703226|27.72|undecided 93|263|65725|4294967341|6.06|4.12|false|calvin king|2013-03-01 09:11:58.703299|32.44|values clariffication 108|383|65629|4294967510|39.55|47.67|false|jessica zipper|2013-03-01 09:11:58.703133|74.23|nap time 89|463|65537|4294967493|64.82|13.79|true|ethan white|2013-03-01 09:11:58.703243|89.52|nap time 88|372|65645|4294967358|34.48|11.18|true|quinn thompson|2013-03-01 09:11:58.703168|84.86|forestry 123|432|65626|4294967435|2.39|16.49|true|david white|2013-03-01 09:11:58.703136|61.24|joggying 57|486|65551|4294967397|36.11|9.88|true|katie xylophone|2013-03-01 09:11:58.703142|57.10|zync studies 59|343|65787|4294967312|66.89|6.54|true|mike laertes|2013-03-01 09:11:58.703209|27.56|xylophone band 74|267|65671|4294967409|21.14|14.64|true|priscilla miller|2013-03-01 09:11:58.703197|89.06|undecided 25|336|65587|4294967336|71.01|14.90|true|tom ichabod|2013-03-01 09:11:58.703127|74.32|zync studies 48|346|65712|4294967315|45.01|16.08|true|zach brown|2013-03-01 09:11:58.703108|21.68|zync studies 84|385|65776|4294967452|35.80|32.13|false|xavier zipper|2013-03-01 09:11:58.703311|99.46|education 58|389|65766|4294967416|95.55|20.62|false|sarah miller|2013-03-01 09:11:58.703215|70.92|history 22|403|65565|4294967381|99.65|35.42|false|yuri johnson|2013-03-01 09:11:58.703154|94.47|geology 55|428|65733|4294967535|99.54|5.35|false|jessica king|2013-03-01 09:11:58.703233|30.30|forestry 117|410|65706|4294967391|50.15|0.21|false|quinn johnson|2013-03-01 09:11:58.703248|65.99|yard duty 95|423|65573|4294967378|47.59|17.37|true|alice robinson|2013-03-01 09:11:58.703133|54.57|linguistics 87|332|65748|4294967320|19.83|41.67|false|fred ellison|2013-03-01 09:11:58.703289|79.02|mathematics 114|263|65674|4294967405|84.44|33.18|true|victor van buren|2013-03-01 09:11:58.703092|63.74|linguistics 5|369|65780|4294967488|92.02|38.59|true|zach polk|2013-03-01 09:11:58.703271|67.29|yard duty -3|430|65667|4294967469|65.50|40.46|true|yuri xylophone|2013-03-01 09:11:58.703258|30.94|american history 120|264|65769|4294967486|89.97|41.18|false|xavier hernandez|2013-03-01 09:11:58.703140|66.89|philosophy 107|317|65634|4294967488|5.68|18.89|false|priscilla ichabod|2013-03-01 09:11:58.703196|39.42|joggying 29|386|65723|4294967328|71.48|6.13|false|ulysses ichabod|2013-03-01 09:11:58.703215|86.65|xylophone band 22|434|65768|4294967543|44.25|27.56|false|tom polk|2013-03-01 09:11:58.703306|12.30|kindergarten -1|274|65755|4294967300|22.01|35.52|false|oscar king|2013-03-01 09:11:58.703141|33.35|chemistry 6|365|65603|4294967522|18.51|5.60|false|gabriella king|2013-03-01 09:11:58.703104|34.20|geology 97|414|65757|4294967325|31.82|22.37|false|rachel nixon|2013-03-01 09:11:58.703127|61.00|nap time 72|448|65538|4294967524|80.09|7.73|true|luke brown|2013-03-01 09:11:58.703090|95.81|american history 51|280|65589|4294967486|57.46|23.35|false|zach xylophone|2013-03-01 09:11:58.703299|11.54|education 12|447|65583|4294967389|0.98|29.79|true|yuri polk|2013-03-01 09:11:58.703305|1.89|wind surfing -1|360|65539|4294967464|4.08|39.51|false|oscar davidson|2013-03-01 09:11:58.703144|59.47|nap time 0|380|65569|4294967425|0.94|28.93|false|sarah robinson|2013-03-01 09:11:58.703176|88.81|xylophone band 66|478|65669|4294967339|23.66|38.34|true|yuri carson|2013-03-01 09:11:58.703228|64.68|opthamology 12|322|65771|4294967545|84.87|10.76|false|sarah allen|2013-03-01 09:11:58.703271|0.79|joggying 79|308|65563|4294967347|4.06|44.84|false|nick underhill|2013-03-01 09:11:58.703097|76.53|industrial engineering 4|382|65719|4294967329|7.26|39.92|true|fred polk|2013-03-01 09:11:58.703073|73.64|mathematics 10|448|65675|4294967392|26.20|16.30|true|rachel laertes|2013-03-01 09:11:58.703200|18.01|xylophone band 45|281|65685|4294967513|81.33|32.22|true|oscar
[GitHub] [flink] flinkbot edited a comment on issue #10797: [FLINK-15172][table-blink] Optimize the operator algorithm to lazily allocate memory
flinkbot edited a comment on issue #10797: [FLINK-15172][table-blink] Optimize the operator algorithm to lazily allocate memory URL: https://github.com/apache/flink/pull/10797#issuecomment-571963932 ## CI report: * e904ab518c4d3b26bf27a89c61b9df30af54b6ed Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/143532669) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4187) * 5628de7fba5004f793d14a16fff55186ce7f3d41 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/144087946) Azure: [PENDING](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4277) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #10745: [FLINK-15445][connectors/jdbc] JDBC Table Source didn't work for Type…
flinkbot edited a comment on issue #10745: [FLINK-15445][connectors/jdbc] JDBC Table Source didn't work for Type… URL: https://github.com/apache/flink/pull/10745#issuecomment-570165880 ## CI report: * 862f3c5fd9e77184af08aea1231cf3a00e03b193 Travis: [CANCELED](https://travis-ci.com/flink-ci/flink/builds/142855689) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4042) * 2cc4e7b6e88e9f655235b8c97707c41f45b6b238 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/142858894) Azure: [PENDING](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4044) * 67d02dcc57087af3626bcbc5cd1cd51117de2b83 Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/142952111) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4053) * 616c1a840307eeefb7fa7fa970d65a149ff7af99 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/143225362) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4123) * 30d7534882a4c84d906dce5d81e9807cce6c051d Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/143885991) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4257) * d915cd1e7d4efc5a2fb114be21483e249f2eca0b Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/144085481) Azure: [PENDING](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4276) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #10665: [FLINK-15354] Start and stop minikube only in kubernetes related e2e tests
flinkbot edited a comment on issue #10665: [FLINK-15354] Start and stop minikube only in kubernetes related e2e tests URL: https://github.com/apache/flink/pull/10665#issuecomment-568440738 ## CI report: * 5a0d5d3d499347ca216e19175ff5f066a6d9b458 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/142099952) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=3861) * 255d89be8069b36be2b980ea6dba4798568160bb Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/143507134) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4176) * 9168102e928bacaa8026407f77a33b80a8ddeae4 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/143514708) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4179) * 1465ba2465b0fba3c91377735481db17abb1c21f Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/143667456) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4203) * 76f3dadc5062279dda9df063901a548315a8b8da Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/143714109) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4219) * 677c4fcd1b82c766f79e296883f8464daafb6a44 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-15527) can not control the number of container on yarn single job module
[
https://issues.apache.org/jira/browse/FLINK-15527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17014009#comment-17014009
]
Jingsong Lee commented on FLINK-15527:
--
Hi [~xintongsong]
You are right about using yarn queue with limited resource quota. This is an
inconvenient solution.
At present, our default behavior is batch rather than pipeline, which has many
advantages and does not affect performance. Generating more parallelisms means
that the granularity of fault tolerance can be smaller. The typical mode of
batch job is limited resources and large parallelisms. Batch by batch
parallelisms runs slowly. So generally speaking, the amount of parallelisms has
little to do with the current resources.
> can not control the number of container on yarn single job module
> -
>
> Key: FLINK-15527
> URL: https://issues.apache.org/jira/browse/FLINK-15527
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Hive
>Affects Versions: 1.10.0
>Reporter: chenchencc
>Priority: Major
> Fix For: 1.10.0
>
> Attachments: flink-conf.yaml, image-2020-01-09-14-30-46-973.png,
> yarn_application.png
>
>
> when run yarn single job run many container but paralism set 4
> *scripts:*
> ./bin/flink run -m yarn-cluster -ys 3 -p 4 -yjm 1024m -ytm 4096m -yqu bi -c
> com.cc.test.HiveTest2 ./cc_jars/hive-1.0-SNAPSHOT.jar 11.txt test61 6
> _notes_: in 1.9.1 has cli paramter -yn to control the number of containers
> and in 1.10 remove it
> *result:*
> the number of containers is 500+
>
> *code use:*
> query the table and save it to the hdfs text
>
> the storge of table is 200g+
>
>
>
>
> *code:*
> com.cc.test.HiveTest2
> public static void main(String[] args) throws Exception
> { EnvironmentSettings settings =
> EnvironmentSettings.newInstance().useBlinkPlanner().inStreamingMode().build();
> StreamExecutionEnvironment settings2 =
> StreamExecutionEnvironment.getExecutionEnvironment();
> settings2.setParallelism(Integer.valueOf(args[2]));
> StreamTableEnvironment tableEnv = StreamTableEnvironment.create(settings2,
> settings); String name = "myhive"; String defaultDatabase = "test"; String
> hiveConfDir = "/etc/hive/conf"; String version = "1.2.1"; // or 1.2.1 2.3.4
> HiveCatalog hive = new HiveCatalog(name, defaultDatabase, hiveConfDir,
> version); tableEnv.registerCatalog("myhive", hive);
> tableEnv.useCatalog("myhive"); tableEnv.listTables(); Table table =
> tableEnv.sqlQuery("select id from orderparent_test2 where id =
> 'A21204170176'"); tableEnv.toAppendStream(table, Row.class).print();
> tableEnv.toAppendStream(table, Row.class)
> .writeAsText("hdfs:///user/chenchao1/"+ args[0],
> FileSystem.WriteMode.OVERWRITE); tableEnv.execute(args[1]); }
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] zhijiangW commented on a change in pull request #10736: [FLINK-15010][Network] Add shutdown hook to ensure cleanup netty shuffle directories
zhijiangW commented on a change in pull request #10736: [FLINK-15010][Network]
Add shutdown hook to ensure cleanup netty shuffle directories
URL: https://github.com/apache/flink/pull/10736#discussion_r365642499
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/io/disk/FileChannelManagerImpl.java
##
@@ -51,12 +52,30 @@
/** The number of the next path to use. */
private volatile int nextPath;
+ private final String prefix;
+
+ private final Thread shutdownHook;
+
public FileChannelManagerImpl(String[] tempDirs, String prefix) {
+ this(tempDirs, prefix, false);
+ }
+
+ public FileChannelManagerImpl(String[] tempDirs, String prefix, boolean
deleteOnShutdown) {
checkNotNull(tempDirs, "The temporary directories must not be
null.");
checkArgument(tempDirs.length > 0, "The temporary directories
must not be empty.");
this.random = new Random();
this.nextPath = 0;
+ this.prefix = prefix;
+
+ if (deleteOnShutdown) {
+ shutdownHook = ShutdownHookUtil.addShutdownHook(this,
String.format("%s-%s", getClass().getSimpleName(), prefix), LOG);
Review comment:
I got the point now.
I think it is better to make `FileChannelManagerImpl` always have the
`shutdownHook`, and then it does not need judge whether it is `null` in other
usages. Also it does not need to care about this argument `deleteOnShutdown` to
true or false for upper component.
In order to avoid repeated `close` calling by `IOManager` usage, we can add
the atomic boolean var in `FileChannelManagerImpl` and check this state in
other usages as well.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #10797: [FLINK-15172][table-blink] Optimize the operator algorithm to lazily allocate memory
flinkbot edited a comment on issue #10797: [FLINK-15172][table-blink] Optimize the operator algorithm to lazily allocate memory URL: https://github.com/apache/flink/pull/10797#issuecomment-571963932 ## CI report: * e904ab518c4d3b26bf27a89c61b9df30af54b6ed Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/143532669) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4187) * 5628de7fba5004f793d14a16fff55186ce7f3d41 Travis: [PENDING](https://travis-ci.com/flink-ci/flink/builds/144087946) Azure: [PENDING](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4277) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Comment Edited] (FLINK-15527) can not control the number of container on yarn single job module
[
https://issues.apache.org/jira/browse/FLINK-15527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17014006#comment-17014006
]
Xintong Song edited comment on FLINK-15527 at 1/13/20 3:58 AM:
---
{quote}> Hive setting too high job parallelism
It is not a problem, that's how it should be. This is not a bug.
I agree that we can introduce max containers in 1.11.
For 1.10, user can control their hive source with
"table.exec.hive.infer-source-parallelism.max" to limit max parallelism. But
not work for complex jobs. If a job have a lot of stages, user are hard to
control the total parallelism in SQL world.
{quote}
Jut trying to understand the problem. If this is not a but, you mean the job is
suppose to have a very high parallelism. But these parallel tasks do not need
to all run at the same time. The user would prefer to use a smaller set or yarn
cluster resources to run these tasks in a pipelined (one after another) manner.
Am I understanding it correctly? [~lzljs3620320]
If that is indeed the case, one can also try to limit the job resource
consumption by configuring a dedicated yarn queue with limited resource quota
for the job. Of course that would be much more inconvenient compared to control
it with flink directly, and may not always be doable (say if the flink user do
not have admin access to the yarn cluster). I'm just trying to provide an idea
of potential workaround for Flink 1.10 as it is.
was (Author: xintongsong):
{quote}> Hive setting too high job parallelism
It is not a problem, that's how it should be. This is not a bug.
I agree that we can introduce max containers in 1.11.
For 1.10, user can control their hive source with
"table.exec.hive.infer-source-parallelism.max" to limit max parallelism. But
not work for complex jobs. If a job have a lot of stages, user are hard to
control the total parallelism in SQL world.
{quote}
Jut trying to understand the problem. If this is not a but, you mean the job is
suppose to have a very high parallelism. But these parallel tasks do not need
to all run at the same time. The user would prefer to use a smaller set or yarn
cluster resources to run these tasks in a pipelined (one after another) manner.
Am I understanding it correctly? [~lzljs3620320]
If that is indeed the case, one can also try to limit the job resource
consumption by configuring a dedicated queue with limited resource quota for
the job. Of course that would be much more inconvenient compared to control it
with flink directly, and may not always be doable (say if the flink user do not
have admin access to the yarn cluster). I'm just trying to provide an idea of
potential workaround for Flink 1.10 as it is.
> can not control the number of container on yarn single job module
> -
>
> Key: FLINK-15527
> URL: https://issues.apache.org/jira/browse/FLINK-15527
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Hive
>Affects Versions: 1.10.0
>Reporter: chenchencc
>Priority: Major
> Fix For: 1.10.0
>
> Attachments: flink-conf.yaml, image-2020-01-09-14-30-46-973.png,
> yarn_application.png
>
>
> when run yarn single job run many container but paralism set 4
> *scripts:*
> ./bin/flink run -m yarn-cluster -ys 3 -p 4 -yjm 1024m -ytm 4096m -yqu bi -c
> com.cc.test.HiveTest2 ./cc_jars/hive-1.0-SNAPSHOT.jar 11.txt test61 6
> _notes_: in 1.9.1 has cli paramter -yn to control the number of containers
> and in 1.10 remove it
> *result:*
> the number of containers is 500+
>
> *code use:*
> query the table and save it to the hdfs text
>
> the storge of table is 200g+
>
>
>
>
> *code:*
> com.cc.test.HiveTest2
> public static void main(String[] args) throws Exception
> { EnvironmentSettings settings =
> EnvironmentSettings.newInstance().useBlinkPlanner().inStreamingMode().build();
> StreamExecutionEnvironment settings2 =
> StreamExecutionEnvironment.getExecutionEnvironment();
> settings2.setParallelism(Integer.valueOf(args[2]));
> StreamTableEnvironment tableEnv = StreamTableEnvironment.create(settings2,
> settings); String name = "myhive"; String defaultDatabase = "test"; String
> hiveConfDir = "/etc/hive/conf"; String version = "1.2.1"; // or 1.2.1 2.3.4
> HiveCatalog hive = new HiveCatalog(name, defaultDatabase, hiveConfDir,
> version); tableEnv.registerCatalog("myhive", hive);
> tableEnv.useCatalog("myhive"); tableEnv.listTables(); Table table =
> tableEnv.sqlQuery("select id from orderparent_test2 where id =
> 'A21204170176'"); tableEnv.toAppendStream(table, Row.class).print();
> tableEnv.toAppendStream(table, Row.class)
> .writeAsText("hdfs:///user/chenchao1/"+ args[0],
> FileSystem.WriteMode.OVERWRITE); tableEnv.execute(args[1]); }
>
--
This message was sent by
[jira] [Commented] (FLINK-15527) can not control the number of container on yarn single job module
[
https://issues.apache.org/jira/browse/FLINK-15527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17014006#comment-17014006
]
Xintong Song commented on FLINK-15527:
--
{quote}> Hive setting too high job parallelism
It is not a problem, that's how it should be. This is not a bug.
I agree that we can introduce max containers in 1.11.
For 1.10, user can control their hive source with
"table.exec.hive.infer-source-parallelism.max" to limit max parallelism. But
not work for complex jobs. If a job have a lot of stages, user are hard to
control the total parallelism in SQL world.
{quote}
Jut trying to understand the problem. If this is not a but, you mean the job is
suppose to have a very high parallelism. But these parallel tasks do not need
to all run at the same time. The user would prefer to use a smaller set or yarn
cluster resources to run these tasks in a pipelined (one after another) manner.
Am I understanding it correctly? [~lzljs3620320]
If that is indeed the case, one can also try to limit the job resource
consumption by configuring a dedicated queue with limited resource quota for
the job. Of course that would be much more inconvenient compared to control it
with flink directly, and may not always be doable (say if the flink user do not
have admin access to the yarn cluster). I'm just trying to provide an idea of
potential workaround for Flink 1.10 as it is.
> can not control the number of container on yarn single job module
> -
>
> Key: FLINK-15527
> URL: https://issues.apache.org/jira/browse/FLINK-15527
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Hive
>Affects Versions: 1.10.0
>Reporter: chenchencc
>Priority: Major
> Fix For: 1.10.0
>
> Attachments: flink-conf.yaml, image-2020-01-09-14-30-46-973.png,
> yarn_application.png
>
>
> when run yarn single job run many container but paralism set 4
> *scripts:*
> ./bin/flink run -m yarn-cluster -ys 3 -p 4 -yjm 1024m -ytm 4096m -yqu bi -c
> com.cc.test.HiveTest2 ./cc_jars/hive-1.0-SNAPSHOT.jar 11.txt test61 6
> _notes_: in 1.9.1 has cli paramter -yn to control the number of containers
> and in 1.10 remove it
> *result:*
> the number of containers is 500+
>
> *code use:*
> query the table and save it to the hdfs text
>
> the storge of table is 200g+
>
>
>
>
> *code:*
> com.cc.test.HiveTest2
> public static void main(String[] args) throws Exception
> { EnvironmentSettings settings =
> EnvironmentSettings.newInstance().useBlinkPlanner().inStreamingMode().build();
> StreamExecutionEnvironment settings2 =
> StreamExecutionEnvironment.getExecutionEnvironment();
> settings2.setParallelism(Integer.valueOf(args[2]));
> StreamTableEnvironment tableEnv = StreamTableEnvironment.create(settings2,
> settings); String name = "myhive"; String defaultDatabase = "test"; String
> hiveConfDir = "/etc/hive/conf"; String version = "1.2.1"; // or 1.2.1 2.3.4
> HiveCatalog hive = new HiveCatalog(name, defaultDatabase, hiveConfDir,
> version); tableEnv.registerCatalog("myhive", hive);
> tableEnv.useCatalog("myhive"); tableEnv.listTables(); Table table =
> tableEnv.sqlQuery("select id from orderparent_test2 where id =
> 'A21204170176'"); tableEnv.toAppendStream(table, Row.class).print();
> tableEnv.toAppendStream(table, Row.class)
> .writeAsText("hdfs:///user/chenchao1/"+ args[0],
> FileSystem.WriteMode.OVERWRITE); tableEnv.execute(args[1]); }
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] carp84 merged pull request #10820: [FLINK-15512][statebackend] Refactor the mechanism to calculate the cache capacity shared among RocksDB instance(s)
carp84 merged pull request #10820: [FLINK-15512][statebackend] Refactor the mechanism to calculate the cache capacity shared among RocksDB instance(s) URL: https://github.com/apache/flink/pull/10820 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] wangyang0918 commented on a change in pull request #10665: [FLINK-15354] Start and stop minikube only in kubernetes related e2e tests
wangyang0918 commented on a change in pull request #10665: [FLINK-15354] Start
and stop minikube only in kubernetes related e2e tests
URL: https://github.com/apache/flink/pull/10665#discussion_r365635600
##
File path: flink-end-to-end-tests/test-scripts/common_kubernetes.sh
##
@@ -26,26 +26,56 @@ MINIKUBE_START_RETRIES=3
MINIKUBE_START_BACKOFF=5
RESULT_HASH="e682ec6622b5e83f2eb614617d5ab2cf"
+# If running tests on non-linux os, the kubectl and minikube should be
installed manually
+function setup_kubernetes_for_linux {
+# Download kubectl, which is a requirement for using minikube.
+if ! [ -x "$(command -v kubectl)" ]; then
+local version=$(curl -s
https://storage.googleapis.com/kubernetes-release/release/stable.txt)
+curl -Lo kubectl
https://storage.googleapis.com/kubernetes-release/release/$version/bin/linux/amd64/kubectl
&& \
+chmod +x kubectl && sudo mv kubectl /usr/local/bin/
+fi
+# Download minikube.
+if ! [ -x "$(command -v minikube)" ]; then
+curl -Lo minikube
https://storage.googleapis.com/minikube/releases/latest/minikube-linux-amd64 &&
\
+chmod +x minikube && sudo mv minikube /usr/local/bin/
+fi
+}
+
function check_kubernetes_status {
minikube status
return $?
}
function start_kubernetes_if_not_running {
if ! check_kubernetes_status; then
-minikube start
+start_command="minikube start"
+# We need sudo permission to set vm-driver to none in linux os.
+[[ "${OS_TYPE}" = "linux" ]] && start_command="sudo ${start_command}
--vm-driver=none"
Review comment:
When running this test in non-linux environment(e.g. Mac Os), we could not
start the minikube under `--vm-driver=none` mode. So we will not need sudo
permission. Of course we could use `sudo minikube start` on MacOS, then we need
to provide the password interactively. I suggest to run on `non-sudo` mode in
non-linux environment.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] wangyang0918 commented on a change in pull request #10665: [FLINK-15354] Start and stop minikube only in kubernetes related e2e tests
wangyang0918 commented on a change in pull request #10665: [FLINK-15354] Start
and stop minikube only in kubernetes related e2e tests
URL: https://github.com/apache/flink/pull/10665#discussion_r365636347
##
File path: flink-end-to-end-tests/test-scripts/test_kubernetes_session.sh
##
@@ -29,6 +29,7 @@ ARGS="--output ${OUTPUT_PATH}"
function cleanup {
kubectl delete service ${CLUSTER_ID}
kubectl delete clusterrolebinding ${CLUSTER_ROLE_BINDING}
+stop_kubernetes
Review comment:
Nice suggestion.
If we use `--vm-driver=none` when starting a minikube in linux-like system,
it is very fast(less than 5s). Most of the time cost at building the flink
image and then running the test. The advantage is we always get a new and clean
Kubernetes cluster environment for every test.
BTW, i think the Yarn/Mesos/Kubernetes share the logics. Start a new cluster
for each e2e test.
So i will create a follow-up ticket tracking the optimization and it is not
urgent.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Commented] (FLINK-15527) can not control the number of container on yarn single job module
[
https://issues.apache.org/jira/browse/FLINK-15527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17014002#comment-17014002
]
Jingsong Lee commented on FLINK-15527:
--
> Hive setting too high job parallelism
It is not a problem, that's how it should be. This is not a bug.
I agree that we can introduce max containers in 1.11.
For 1.10, user can control their hive source with
"table.exec.hive.infer-source-parallelism.max" to limit max parallelism. But
not work for complex jobs. If a job have a lot of stages, user are hard to
control the total parallelism in SQL world.
> can not control the number of container on yarn single job module
> -
>
> Key: FLINK-15527
> URL: https://issues.apache.org/jira/browse/FLINK-15527
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Hive
>Affects Versions: 1.10.0
>Reporter: chenchencc
>Priority: Major
> Fix For: 1.10.0
>
> Attachments: flink-conf.yaml, image-2020-01-09-14-30-46-973.png,
> yarn_application.png
>
>
> when run yarn single job run many container but paralism set 4
> *scripts:*
> ./bin/flink run -m yarn-cluster -ys 3 -p 4 -yjm 1024m -ytm 4096m -yqu bi -c
> com.cc.test.HiveTest2 ./cc_jars/hive-1.0-SNAPSHOT.jar 11.txt test61 6
> _notes_: in 1.9.1 has cli paramter -yn to control the number of containers
> and in 1.10 remove it
> *result:*
> the number of containers is 500+
>
> *code use:*
> query the table and save it to the hdfs text
>
> the storge of table is 200g+
>
>
>
>
> *code:*
> com.cc.test.HiveTest2
> public static void main(String[] args) throws Exception
> { EnvironmentSettings settings =
> EnvironmentSettings.newInstance().useBlinkPlanner().inStreamingMode().build();
> StreamExecutionEnvironment settings2 =
> StreamExecutionEnvironment.getExecutionEnvironment();
> settings2.setParallelism(Integer.valueOf(args[2]));
> StreamTableEnvironment tableEnv = StreamTableEnvironment.create(settings2,
> settings); String name = "myhive"; String defaultDatabase = "test"; String
> hiveConfDir = "/etc/hive/conf"; String version = "1.2.1"; // or 1.2.1 2.3.4
> HiveCatalog hive = new HiveCatalog(name, defaultDatabase, hiveConfDir,
> version); tableEnv.registerCatalog("myhive", hive);
> tableEnv.useCatalog("myhive"); tableEnv.listTables(); Table table =
> tableEnv.sqlQuery("select id from orderparent_test2 where id =
> 'A21204170176'"); tableEnv.toAppendStream(table, Row.class).print();
> tableEnv.toAppendStream(table, Row.class)
> .writeAsText("hdfs:///user/chenchao1/"+ args[0],
> FileSystem.WriteMode.OVERWRITE); tableEnv.execute(args[1]); }
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (FLINK-15527) can not control the number of container on yarn single job module
[
https://issues.apache.org/jira/browse/FLINK-15527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17014002#comment-17014002
]
Jingsong Lee edited comment on FLINK-15527 at 1/13/20 3:43 AM:
---
Thanks [~karmagyz] and [~xintongsong]
> Hive setting too high job parallelism
It is not a problem, that's how it should be. This is not a bug.
I agree that we can introduce max containers in 1.11.
For 1.10, user can control their hive source with
"table.exec.hive.infer-source-parallelism.max" to limit max parallelism. But
not work for complex jobs. If a job have a lot of stages, user are hard to
control the total parallelism in SQL world.
was (Author: lzljs3620320):
> Hive setting too high job parallelism
It is not a problem, that's how it should be. This is not a bug.
I agree that we can introduce max containers in 1.11.
For 1.10, user can control their hive source with
"table.exec.hive.infer-source-parallelism.max" to limit max parallelism. But
not work for complex jobs. If a job have a lot of stages, user are hard to
control the total parallelism in SQL world.
> can not control the number of container on yarn single job module
> -
>
> Key: FLINK-15527
> URL: https://issues.apache.org/jira/browse/FLINK-15527
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Hive
>Affects Versions: 1.10.0
>Reporter: chenchencc
>Priority: Major
> Fix For: 1.10.0
>
> Attachments: flink-conf.yaml, image-2020-01-09-14-30-46-973.png,
> yarn_application.png
>
>
> when run yarn single job run many container but paralism set 4
> *scripts:*
> ./bin/flink run -m yarn-cluster -ys 3 -p 4 -yjm 1024m -ytm 4096m -yqu bi -c
> com.cc.test.HiveTest2 ./cc_jars/hive-1.0-SNAPSHOT.jar 11.txt test61 6
> _notes_: in 1.9.1 has cli paramter -yn to control the number of containers
> and in 1.10 remove it
> *result:*
> the number of containers is 500+
>
> *code use:*
> query the table and save it to the hdfs text
>
> the storge of table is 200g+
>
>
>
>
> *code:*
> com.cc.test.HiveTest2
> public static void main(String[] args) throws Exception
> { EnvironmentSettings settings =
> EnvironmentSettings.newInstance().useBlinkPlanner().inStreamingMode().build();
> StreamExecutionEnvironment settings2 =
> StreamExecutionEnvironment.getExecutionEnvironment();
> settings2.setParallelism(Integer.valueOf(args[2]));
> StreamTableEnvironment tableEnv = StreamTableEnvironment.create(settings2,
> settings); String name = "myhive"; String defaultDatabase = "test"; String
> hiveConfDir = "/etc/hive/conf"; String version = "1.2.1"; // or 1.2.1 2.3.4
> HiveCatalog hive = new HiveCatalog(name, defaultDatabase, hiveConfDir,
> version); tableEnv.registerCatalog("myhive", hive);
> tableEnv.useCatalog("myhive"); tableEnv.listTables(); Table table =
> tableEnv.sqlQuery("select id from orderparent_test2 where id =
> 'A21204170176'"); tableEnv.toAppendStream(table, Row.class).print();
> tableEnv.toAppendStream(table, Row.class)
> .writeAsText("hdfs:///user/chenchao1/"+ args[0],
> FileSystem.WriteMode.OVERWRITE); tableEnv.execute(args[1]); }
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on issue #10745: [FLINK-15445][connectors/jdbc] JDBC Table Source didn't work for Type…
flinkbot edited a comment on issue #10745: [FLINK-15445][connectors/jdbc] JDBC Table Source didn't work for Type… URL: https://github.com/apache/flink/pull/10745#issuecomment-570165880 ## CI report: * 862f3c5fd9e77184af08aea1231cf3a00e03b193 Travis: [CANCELED](https://travis-ci.com/flink-ci/flink/builds/142855689) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4042) * 2cc4e7b6e88e9f655235b8c97707c41f45b6b238 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/142858894) Azure: [PENDING](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4044) * 67d02dcc57087af3626bcbc5cd1cd51117de2b83 Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/142952111) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4053) * 616c1a840307eeefb7fa7fa970d65a149ff7af99 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/143225362) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4123) * 30d7534882a4c84d906dce5d81e9807cce6c051d Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/143885991) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4257) * d915cd1e7d4efc5a2fb114be21483e249f2eca0b UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-15527) can not control the number of container on yarn single job module
[
https://issues.apache.org/jira/browse/FLINK-15527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17013999#comment-17013999
]
Yangze Guo commented on FLINK-15527:
I'm not quite familiar with the contract of yarn session mode in early version,
but it seems to be changed. You could follow [~fly_in_gis]'s
[reply|https://issues.apache.org/jira/browse/FLINK-15527?focusedCommentId=17011456=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17011456].
+1 to provide a mechanism to let user controls the resource usage of Flink
cluster. Regarding the solution, two proposals emerge in this thread:
- limit the number of containers(or total resource) through configuration
- implement reactive resource manager
Also agreed with [~xintongsong], this feature should not be add to release-1.10.
> can not control the number of container on yarn single job module
> -
>
> Key: FLINK-15527
> URL: https://issues.apache.org/jira/browse/FLINK-15527
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Hive
>Affects Versions: 1.10.0
>Reporter: chenchencc
>Priority: Major
> Fix For: 1.10.0
>
> Attachments: flink-conf.yaml, image-2020-01-09-14-30-46-973.png,
> yarn_application.png
>
>
> when run yarn single job run many container but paralism set 4
> *scripts:*
> ./bin/flink run -m yarn-cluster -ys 3 -p 4 -yjm 1024m -ytm 4096m -yqu bi -c
> com.cc.test.HiveTest2 ./cc_jars/hive-1.0-SNAPSHOT.jar 11.txt test61 6
> _notes_: in 1.9.1 has cli paramter -yn to control the number of containers
> and in 1.10 remove it
> *result:*
> the number of containers is 500+
>
> *code use:*
> query the table and save it to the hdfs text
>
> the storge of table is 200g+
>
>
>
>
> *code:*
> com.cc.test.HiveTest2
> public static void main(String[] args) throws Exception
> { EnvironmentSettings settings =
> EnvironmentSettings.newInstance().useBlinkPlanner().inStreamingMode().build();
> StreamExecutionEnvironment settings2 =
> StreamExecutionEnvironment.getExecutionEnvironment();
> settings2.setParallelism(Integer.valueOf(args[2]));
> StreamTableEnvironment tableEnv = StreamTableEnvironment.create(settings2,
> settings); String name = "myhive"; String defaultDatabase = "test"; String
> hiveConfDir = "/etc/hive/conf"; String version = "1.2.1"; // or 1.2.1 2.3.4
> HiveCatalog hive = new HiveCatalog(name, defaultDatabase, hiveConfDir,
> version); tableEnv.registerCatalog("myhive", hive);
> tableEnv.useCatalog("myhive"); tableEnv.listTables(); Table table =
> tableEnv.sqlQuery("select id from orderparent_test2 where id =
> 'A21204170176'"); tableEnv.toAppendStream(table, Row.class).print();
> tableEnv.toAppendStream(table, Row.class)
> .writeAsText("hdfs:///user/chenchao1/"+ args[0],
> FileSystem.WriteMode.OVERWRITE); tableEnv.execute(args[1]); }
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-15527) can not control the number of container on yarn single job module
[
https://issues.apache.org/jira/browse/FLINK-15527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17014000#comment-17014000
]
Xintong Song commented on FLINK-15527:
--
I would suggest we use this ticket to track and fix the problem Hive setting
too high job parallelism (in 1.10.0 I assume), and create another ticket for
control total resource of flink cluster and try to improve this in the next
release. WDYT?
> can not control the number of container on yarn single job module
> -
>
> Key: FLINK-15527
> URL: https://issues.apache.org/jira/browse/FLINK-15527
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Hive
>Affects Versions: 1.10.0
>Reporter: chenchencc
>Priority: Major
> Fix For: 1.10.0
>
> Attachments: flink-conf.yaml, image-2020-01-09-14-30-46-973.png,
> yarn_application.png
>
>
> when run yarn single job run many container but paralism set 4
> *scripts:*
> ./bin/flink run -m yarn-cluster -ys 3 -p 4 -yjm 1024m -ytm 4096m -yqu bi -c
> com.cc.test.HiveTest2 ./cc_jars/hive-1.0-SNAPSHOT.jar 11.txt test61 6
> _notes_: in 1.9.1 has cli paramter -yn to control the number of containers
> and in 1.10 remove it
> *result:*
> the number of containers is 500+
>
> *code use:*
> query the table and save it to the hdfs text
>
> the storge of table is 200g+
>
>
>
>
> *code:*
> com.cc.test.HiveTest2
> public static void main(String[] args) throws Exception
> { EnvironmentSettings settings =
> EnvironmentSettings.newInstance().useBlinkPlanner().inStreamingMode().build();
> StreamExecutionEnvironment settings2 =
> StreamExecutionEnvironment.getExecutionEnvironment();
> settings2.setParallelism(Integer.valueOf(args[2]));
> StreamTableEnvironment tableEnv = StreamTableEnvironment.create(settings2,
> settings); String name = "myhive"; String defaultDatabase = "test"; String
> hiveConfDir = "/etc/hive/conf"; String version = "1.2.1"; // or 1.2.1 2.3.4
> HiveCatalog hive = new HiveCatalog(name, defaultDatabase, hiveConfDir,
> version); tableEnv.registerCatalog("myhive", hive);
> tableEnv.useCatalog("myhive"); tableEnv.listTables(); Table table =
> tableEnv.sqlQuery("select id from orderparent_test2 where id =
> 'A21204170176'"); tableEnv.toAppendStream(table, Row.class).print();
> tableEnv.toAppendStream(table, Row.class)
> .writeAsText("hdfs:///user/chenchao1/"+ args[0],
> FileSystem.WriteMode.OVERWRITE); tableEnv.execute(args[1]); }
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-15527) can not control the number of container on yarn single job module
[
https://issues.apache.org/jira/browse/FLINK-15527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17013997#comment-17013997
]
Xintong Song commented on FLINK-15527:
--
Thanks for involving me, [~lzljs3620320].
If I understand correctly, for this specific case, the problem is that Hive
component overlooked user's configuration '-p 4' and set the job with too high
parallelism.
I also agree that it would be helpful to control the number of task executors
not relying on the parallelism, e.g., something like [~trohrmann] mentioned. I
think at least for batch jobs this approach will allow us to pipeline the tasks
and run them with much fewer slots. However, I'm not sure whether this should
be fixed in 1.10.0, since it sound like a feature improvement rather than a bug
fix to me, and we have far beyond the feature freeze.
> can not control the number of container on yarn single job module
> -
>
> Key: FLINK-15527
> URL: https://issues.apache.org/jira/browse/FLINK-15527
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Hive
>Affects Versions: 1.10.0
>Reporter: chenchencc
>Priority: Major
> Fix For: 1.10.0
>
> Attachments: flink-conf.yaml, image-2020-01-09-14-30-46-973.png,
> yarn_application.png
>
>
> when run yarn single job run many container but paralism set 4
> *scripts:*
> ./bin/flink run -m yarn-cluster -ys 3 -p 4 -yjm 1024m -ytm 4096m -yqu bi -c
> com.cc.test.HiveTest2 ./cc_jars/hive-1.0-SNAPSHOT.jar 11.txt test61 6
> _notes_: in 1.9.1 has cli paramter -yn to control the number of containers
> and in 1.10 remove it
> *result:*
> the number of containers is 500+
>
> *code use:*
> query the table and save it to the hdfs text
>
> the storge of table is 200g+
>
>
>
>
> *code:*
> com.cc.test.HiveTest2
> public static void main(String[] args) throws Exception
> { EnvironmentSettings settings =
> EnvironmentSettings.newInstance().useBlinkPlanner().inStreamingMode().build();
> StreamExecutionEnvironment settings2 =
> StreamExecutionEnvironment.getExecutionEnvironment();
> settings2.setParallelism(Integer.valueOf(args[2]));
> StreamTableEnvironment tableEnv = StreamTableEnvironment.create(settings2,
> settings); String name = "myhive"; String defaultDatabase = "test"; String
> hiveConfDir = "/etc/hive/conf"; String version = "1.2.1"; // or 1.2.1 2.3.4
> HiveCatalog hive = new HiveCatalog(name, defaultDatabase, hiveConfDir,
> version); tableEnv.registerCatalog("myhive", hive);
> tableEnv.useCatalog("myhive"); tableEnv.listTables(); Table table =
> tableEnv.sqlQuery("select id from orderparent_test2 where id =
> 'A21204170176'"); tableEnv.toAppendStream(table, Row.class).print();
> tableEnv.toAppendStream(table, Row.class)
> .writeAsText("hdfs:///user/chenchao1/"+ args[0],
> FileSystem.WriteMode.OVERWRITE); tableEnv.execute(args[1]); }
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-15318) RocksDBWriteBatchPerformanceTest.benchMark fails on ppc64le
[ https://issues.apache.org/jira/browse/FLINK-15318?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17013995#comment-17013995 ] Yun Tang commented on FLINK-15318: -- [~redmark-ibm] you can try this [commit|https://github.com/Myasuka/flink/commit/484fffb08620ab177175405c53d64faaeb585d01]. > RocksDBWriteBatchPerformanceTest.benchMark fails on ppc64le > --- > > Key: FLINK-15318 > URL: https://issues.apache.org/jira/browse/FLINK-15318 > Project: Flink > Issue Type: Bug > Components: Benchmarks, Runtime / State Backends > Environment: arch: ppc64le > os: rhel7.6, ubuntu 18.04 > jdk: 8, 11 > mvn: 3.3.9, 3.6.2 >Reporter: Siddhesh Ghadi >Priority: Major > Attachments: surefire-report.txt > > > RocksDBWriteBatchPerformanceTest.benchMark fails due to TestTimedOut, however > when test-timeout is increased from 2s to 5s in > org/apache/flink/contrib/streaming/state/benchmark/RocksDBWriteBatchPerformanceTest.java:75, > it passes. Is this acceptable solution? > Note: Tests are ran inside a container. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on issue #10797: [FLINK-15172][table-blink] Optimize the operator algorithm to lazily allocate memory
flinkbot edited a comment on issue #10797: [FLINK-15172][table-blink] Optimize the operator algorithm to lazily allocate memory URL: https://github.com/apache/flink/pull/10797#issuecomment-571963932 ## CI report: * e904ab518c4d3b26bf27a89c61b9df30af54b6ed Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/143532669) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4187) * 5628de7fba5004f793d14a16fff55186ce7f3d41 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-15563) When using ParquetTableSource, The program hangs until OOM
[
https://issues.apache.org/jira/browse/FLINK-15563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17013990#comment-17013990
]
Kurt Young commented on FLINK-15563:
cc [~godfreyhe]
> When using ParquetTableSource, The program hangs until OOM
> ---
>
> Key: FLINK-15563
> URL: https://issues.apache.org/jira/browse/FLINK-15563
> Project: Flink
> Issue Type: Bug
> Components: Connectors / FileSystem
>Affects Versions: 1.8.1, 1.9.1
>Reporter: sujun
>Priority: Critical
> Labels: Parquet
>
> This is my code:
> {code:java}
> def main(args: Array[String]): Unit = {
> val env = StreamExecutionEnvironment.getExecutionEnvironment
> env.setStreamTimeCharacteristic(TimeCharacteristic.EventTime)
> val tableEnv = StreamTableEnvironment.create(env)val schema =
> "{\"type\":\"record\",\"name\":\"root\",\"fields\":[{\"name\":\"log_id\",\"type\":[\"null\",\"string\"],\"default\":null},{\"name\":\"city\",\"type\":[\"null\",\"string\"],\"default\":null},{\"name\":\"log_from\",\"type\":[\"null\",\"string\"],\"default\":null},{\"name\":\"ip\",\"type\":[\"null\",\"string\"],\"default\":null},{\"name\":\"type\",\"type\":[\"null\",\"string\"],\"default\":null},{\"name\":\"data_source\",\"type\":[\"null\",\"string\"],\"default\":null},{\"name\":\"is_scan\",\"type\":[\"null\",\"string\"],\"default\":null},{\"name\":\"result\",\"type\":[\"null\",\"string\"],\"default\":null},{\"name\":\"timelong\",\"type\":[\"null\",\"long\"],\"default\":null},{\"name\":\"is_sec\",\"type\":[\"null\",\"string\"],\"default\":null},{\"name\":\"event_name\",\"type\":[\"null\",\"string\"],\"default\":null},{\"name\":\"id\",\"type\":[\"null\",\"string\"],\"default\":null},{\"name\":\"time_string\",\"type\":[\"null\",\"string\"],\"default\":null},{\"name\":\"device\",\"type\":[\"null\",\"string\"],\"default\":null},{\"name\":\"timestamp_string\",\"type\":[\"null\",\"string\"],\"default\":null},{\"name\":\"occur_time\",\"type\":[\"null\",{\"type\":\"long\",\"logicalType\":\"timestamp-millis\"}],\"default\":null},{\"name\":\"row_time\",\"type\":[\"null\",{\"type\":\"long\",\"logicalType\":\"timestamp-millis\"}],\"default\":null}]}"
>
> val parquetTableSource: ParquetTableSource = ParquetTableSource
> .builder
> .forParquetSchema(new
> org.apache.parquet.avro.AvroSchemaConverter().convert(
> org.apache.avro.Schema.parse(schema, true)))
> .path("/Users/sujun/Documents/tmp/login_data")
> .build
> tableEnv.registerTableSource("source",parquetTableSource)
> val t1 = tableEnv.sqlQuery("select log_id,city from source where city =
> '274' ")
> tableEnv.registerTable("t1",t1)
> val t2 = tableEnv.sqlQuery("select * from t1 where
> log_id='5927070661978133'")
> t2.toAppendStream[Row].print()
> env.execute()}
> {code}
>
> When the two SQLS each have a where condition, the main program will hang
> until OOM. When the filter push down code of ParquetTableSource is deleted,
> the program runs normally.
>
> Through my debugging, I found that the program hangs in the
> org.apache.calcite.plan.volcano.VolcanoPlanner.findBestExp method
>
> May be a bug in the calcite optimizer caused by filter push down code
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-15527) can not control the number of container on yarn single job module
[
https://issues.apache.org/jira/browse/FLINK-15527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17013989#comment-17013989
]
Kurt Young commented on FLINK-15527:
IIRC yarn session mode is just a standalone cluster but automatically deployed
by yarn, so this have been changed? IMO this mode is very useful for users,
they can easily control the max resource usage of the flink cluster. If user
only deploy one job into this cluster, it can also simulate the usage like
controlling the max resource usage of a single job.
> can not control the number of container on yarn single job module
> -
>
> Key: FLINK-15527
> URL: https://issues.apache.org/jira/browse/FLINK-15527
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Hive
>Affects Versions: 1.10.0
>Reporter: chenchencc
>Priority: Major
> Fix For: 1.10.0
>
> Attachments: flink-conf.yaml, image-2020-01-09-14-30-46-973.png,
> yarn_application.png
>
>
> when run yarn single job run many container but paralism set 4
> *scripts:*
> ./bin/flink run -m yarn-cluster -ys 3 -p 4 -yjm 1024m -ytm 4096m -yqu bi -c
> com.cc.test.HiveTest2 ./cc_jars/hive-1.0-SNAPSHOT.jar 11.txt test61 6
> _notes_: in 1.9.1 has cli paramter -yn to control the number of containers
> and in 1.10 remove it
> *result:*
> the number of containers is 500+
>
> *code use:*
> query the table and save it to the hdfs text
>
> the storge of table is 200g+
>
>
>
>
> *code:*
> com.cc.test.HiveTest2
> public static void main(String[] args) throws Exception
> { EnvironmentSettings settings =
> EnvironmentSettings.newInstance().useBlinkPlanner().inStreamingMode().build();
> StreamExecutionEnvironment settings2 =
> StreamExecutionEnvironment.getExecutionEnvironment();
> settings2.setParallelism(Integer.valueOf(args[2]));
> StreamTableEnvironment tableEnv = StreamTableEnvironment.create(settings2,
> settings); String name = "myhive"; String defaultDatabase = "test"; String
> hiveConfDir = "/etc/hive/conf"; String version = "1.2.1"; // or 1.2.1 2.3.4
> HiveCatalog hive = new HiveCatalog(name, defaultDatabase, hiveConfDir,
> version); tableEnv.registerCatalog("myhive", hive);
> tableEnv.useCatalog("myhive"); tableEnv.listTables(); Table table =
> tableEnv.sqlQuery("select id from orderparent_test2 where id =
> 'A21204170176'"); tableEnv.toAppendStream(table, Row.class).print();
> tableEnv.toAppendStream(table, Row.class)
> .writeAsText("hdfs:///user/chenchao1/"+ args[0],
> FileSystem.WriteMode.OVERWRITE); tableEnv.execute(args[1]); }
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-15527) can not control the number of container on yarn single job module
[
https://issues.apache.org/jira/browse/FLINK-15527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17013986#comment-17013986
]
Jingsong Lee commented on FLINK-15527:
--
CC: [~xintongsong]
> can not control the number of container on yarn single job module
> -
>
> Key: FLINK-15527
> URL: https://issues.apache.org/jira/browse/FLINK-15527
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Hive
>Affects Versions: 1.10.0
>Reporter: chenchencc
>Priority: Major
> Fix For: 1.10.0
>
> Attachments: flink-conf.yaml, image-2020-01-09-14-30-46-973.png,
> yarn_application.png
>
>
> when run yarn single job run many container but paralism set 4
> *scripts:*
> ./bin/flink run -m yarn-cluster -ys 3 -p 4 -yjm 1024m -ytm 4096m -yqu bi -c
> com.cc.test.HiveTest2 ./cc_jars/hive-1.0-SNAPSHOT.jar 11.txt test61 6
> _notes_: in 1.9.1 has cli paramter -yn to control the number of containers
> and in 1.10 remove it
> *result:*
> the number of containers is 500+
>
> *code use:*
> query the table and save it to the hdfs text
>
> the storge of table is 200g+
>
>
>
>
> *code:*
> com.cc.test.HiveTest2
> public static void main(String[] args) throws Exception
> { EnvironmentSettings settings =
> EnvironmentSettings.newInstance().useBlinkPlanner().inStreamingMode().build();
> StreamExecutionEnvironment settings2 =
> StreamExecutionEnvironment.getExecutionEnvironment();
> settings2.setParallelism(Integer.valueOf(args[2]));
> StreamTableEnvironment tableEnv = StreamTableEnvironment.create(settings2,
> settings); String name = "myhive"; String defaultDatabase = "test"; String
> hiveConfDir = "/etc/hive/conf"; String version = "1.2.1"; // or 1.2.1 2.3.4
> HiveCatalog hive = new HiveCatalog(name, defaultDatabase, hiveConfDir,
> version); tableEnv.registerCatalog("myhive", hive);
> tableEnv.useCatalog("myhive"); tableEnv.listTables(); Table table =
> tableEnv.sqlQuery("select id from orderparent_test2 where id =
> 'A21204170176'"); tableEnv.toAppendStream(table, Row.class).print();
> tableEnv.toAppendStream(table, Row.class)
> .writeAsText("hdfs:///user/chenchao1/"+ args[0],
> FileSystem.WriteMode.OVERWRITE); tableEnv.execute(args[1]); }
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-15563) When using ParquetTableSource, The program hangs until OOM
[
https://issues.apache.org/jira/browse/FLINK-15563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
sujun updated FLINK-15563:
--
Description:
This is my code:
{code:java}
def main(args: Array[String]): Unit = {
val env = StreamExecutionEnvironment.getExecutionEnvironment
env.setStreamTimeCharacteristic(TimeCharacteristic.EventTime)
val tableEnv = StreamTableEnvironment.create(env)val schema =
"{\"type\":\"record\",\"name\":\"root\",\"fields\":[{\"name\":\"log_id\",\"type\":[\"null\",\"string\"],\"default\":null},{\"name\":\"city\",\"type\":[\"null\",\"string\"],\"default\":null},{\"name\":\"log_from\",\"type\":[\"null\",\"string\"],\"default\":null},{\"name\":\"ip\",\"type\":[\"null\",\"string\"],\"default\":null},{\"name\":\"type\",\"type\":[\"null\",\"string\"],\"default\":null},{\"name\":\"data_source\",\"type\":[\"null\",\"string\"],\"default\":null},{\"name\":\"is_scan\",\"type\":[\"null\",\"string\"],\"default\":null},{\"name\":\"result\",\"type\":[\"null\",\"string\"],\"default\":null},{\"name\":\"timelong\",\"type\":[\"null\",\"long\"],\"default\":null},{\"name\":\"is_sec\",\"type\":[\"null\",\"string\"],\"default\":null},{\"name\":\"event_name\",\"type\":[\"null\",\"string\"],\"default\":null},{\"name\":\"id\",\"type\":[\"null\",\"string\"],\"default\":null},{\"name\":\"time_string\",\"type\":[\"null\",\"string\"],\"default\":null},{\"name\":\"device\",\"type\":[\"null\",\"string\"],\"default\":null},{\"name\":\"timestamp_string\",\"type\":[\"null\",\"string\"],\"default\":null},{\"name\":\"occur_time\",\"type\":[\"null\",{\"type\":\"long\",\"logicalType\":\"timestamp-millis\"}],\"default\":null},{\"name\":\"row_time\",\"type\":[\"null\",{\"type\":\"long\",\"logicalType\":\"timestamp-millis\"}],\"default\":null}]}"
val parquetTableSource: ParquetTableSource = ParquetTableSource
.builder
.forParquetSchema(new
org.apache.parquet.avro.AvroSchemaConverter().convert(
org.apache.avro.Schema.parse(schema, true)))
.path("/Users/sujun/Documents/tmp/login_data")
.build
tableEnv.registerTableSource("source",parquetTableSource)
val t1 = tableEnv.sqlQuery("select log_id,city from source where city =
'274' ")
tableEnv.registerTable("t1",t1)
val t2 = tableEnv.sqlQuery("select * from t1 where
log_id='5927070661978133'")
t2.toAppendStream[Row].print()
env.execute()}
{code}
When the two SQLS each have a where condition, the main program will hang
until OOM. When the filter push down code of ParquetTableSource is deleted, the
program runs normally.
Through my debugging, I found that the program hangs in the
org.apache.calcite.plan.volcano.VolcanoPlanner.findBestExp method
May be a bug in the calcite optimizer caused by filter push down code
was:
This is my code:
{code:java}
def main(args: Array[String]): Unit = {
val env = StreamExecutionEnvironment.getExecutionEnvironment
env.setStreamTimeCharacteristic(TimeCharacteristic.EventTime)
val tableEnv = StreamTableEnvironment.create(env)val schema =
"{\"type\":\"record\",\"name\":\"root\",\"fields\":[{\"name\":\"log_id\",\"type\":[\"null\",\"string\"],\"default\":null},{\"name\":\"city\",\"type\":[\"null\",\"string\"],\"default\":null},{\"name\":\"log_from\",\"type\":[\"null\",\"string\"],\"default\":null},{\"name\":\"ip\",\"type\":[\"null\",\"string\"],\"default\":null},{\"name\":\"type\",\"type\":[\"null\",\"string\"],\"default\":null},{\"name\":\"data_source\",\"type\":[\"null\",\"string\"],\"default\":null},{\"name\":\"is_scan\",\"type\":[\"null\",\"string\"],\"default\":null},{\"name\":\"result\",\"type\":[\"null\",\"string\"],\"default\":null},{\"name\":\"timelong\",\"type\":[\"null\",\"long\"],\"default\":null},{\"name\":\"is_sec\",\"type\":[\"null\",\"string\"],\"default\":null},{\"name\":\"event_name\",\"type\":[\"null\",\"string\"],\"default\":null},{\"name\":\"id\",\"type\":[\"null\",\"string\"],\"default\":null},{\"name\":\"time_string\",\"type\":[\"null\",\"string\"],\"default\":null},{\"name\":\"device\",\"type\":[\"null\",\"string\"],\"default\":null},{\"name\":\"timestamp_string\",\"type\":[\"null\",\"string\"],\"default\":null},{\"name\":\"occur_time\",\"type\":[\"null\",{\"type\":\"long\",\"logicalType\":\"timestamp-millis\"}],\"default\":null},{\"name\":\"row_time\",\"type\":[\"null\",{\"type\":\"long\",\"logicalType\":\"timestamp-millis\"}],\"default\":null}]}"
val parquetTableSource: ParquetTableSource = ParquetTableSource
.builder
.forParquetSchema(new
org.apache.parquet.avro.AvroSchemaConverter().convert(
org.apache.avro.Schema.parse(schema, true)))
.path("/Users/sujun/Documents/tmp/login_data")
.build
tableEnv.registerTableSource("source",parquetTableSource)
val t1 = tableEnv.sqlQuery("select log_id,city from source where city =
[jira] [Updated] (FLINK-15563) When using ParquetTableSource, The program hangs until OOM
[
https://issues.apache.org/jira/browse/FLINK-15563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
sujun updated FLINK-15563:
--
Description:
This is my code:
{code:java}
def main(args: Array[String]): Unit = {
val env = StreamExecutionEnvironment.getExecutionEnvironment
env.setStreamTimeCharacteristic(TimeCharacteristic.EventTime)
val tableEnv = StreamTableEnvironment.create(env)val schema =
"{\"type\":\"record\",\"name\":\"root\",\"fields\":[{\"name\":\"log_id\",\"type\":[\"null\",\"string\"],\"default\":null},{\"name\":\"city\",\"type\":[\"null\",\"string\"],\"default\":null},{\"name\":\"log_from\",\"type\":[\"null\",\"string\"],\"default\":null},{\"name\":\"ip\",\"type\":[\"null\",\"string\"],\"default\":null},{\"name\":\"type\",\"type\":[\"null\",\"string\"],\"default\":null},{\"name\":\"data_source\",\"type\":[\"null\",\"string\"],\"default\":null},{\"name\":\"is_scan\",\"type\":[\"null\",\"string\"],\"default\":null},{\"name\":\"result\",\"type\":[\"null\",\"string\"],\"default\":null},{\"name\":\"timelong\",\"type\":[\"null\",\"long\"],\"default\":null},{\"name\":\"is_sec\",\"type\":[\"null\",\"string\"],\"default\":null},{\"name\":\"event_name\",\"type\":[\"null\",\"string\"],\"default\":null},{\"name\":\"id\",\"type\":[\"null\",\"string\"],\"default\":null},{\"name\":\"time_string\",\"type\":[\"null\",\"string\"],\"default\":null},{\"name\":\"device\",\"type\":[\"null\",\"string\"],\"default\":null},{\"name\":\"timestamp_string\",\"type\":[\"null\",\"string\"],\"default\":null},{\"name\":\"occur_time\",\"type\":[\"null\",{\"type\":\"long\",\"logicalType\":\"timestamp-millis\"}],\"default\":null},{\"name\":\"row_time\",\"type\":[\"null\",{\"type\":\"long\",\"logicalType\":\"timestamp-millis\"}],\"default\":null}]}"
val parquetTableSource: ParquetTableSource = ParquetTableSource
.builder
.forParquetSchema(new
org.apache.parquet.avro.AvroSchemaConverter().convert(
org.apache.avro.Schema.parse(schema, true)))
.path("/Users/sujun/Documents/tmp/login_data")
.build
tableEnv.registerTableSource("source",parquetTableSource)
val t1 = tableEnv.sqlQuery("select log_id,city from source where city =
'274' ")
tableEnv.registerTable("t1",t1)
val t2 = tableEnv.sqlQuery("select * from t1 where
log_id='5927070661978133'")
t2.toAppendStream[Row].print()
env.execute()}
{code}
When the two SQLS each have a where condition, the main program will hang until
OOM. When the filter push down code of ParquetTableSource is deleted, the
program runs normally.
Through my debugging, I found that the program hangs in the
org.apache.calcite.plan.volcano.VolcanoPlanner.findBestExp method
May be a bug in the calcul optimizer caused by filter push down code
was:
def main(args: Array[String]): Unit = \{
val env = StreamExecutionEnvironment.getExecutionEnvironment
env.setStreamTimeCharacteristic(TimeCharacteristic.EventTime)
val tableEnv = StreamTableEnvironment.create(env)
val schema =
"{\"type\":\"record\",\"name\":\"root\",\"fields\":[{\"name\":\"log_id\",\"type\":[\"null\",\"string\"],\"default\":null},\{\"name\":\"city\",\"type\":[\"null\",\"string\"],\"default\":null},\{\"name\":\"log_from\",\"type\":[\"null\",\"string\"],\"default\":null},\{\"name\":\"ip\",\"type\":[\"null\",\"string\"],\"default\":null},\{\"name\":\"type\",\"type\":[\"null\",\"string\"],\"default\":null},\{\"name\":\"data_source\",\"type\":[\"null\",\"string\"],\"default\":null},\{\"name\":\"is_scan\",\"type\":[\"null\",\"string\"],\"default\":null},\{\"name\":\"result\",\"type\":[\"null\",\"string\"],\"default\":null},\{\"name\":\"timelong\",\"type\":[\"null\",\"long\"],\"default\":null},\{\"name\":\"is_sec\",\"type\":[\"null\",\"string\"],\"default\":null},\{\"name\":\"event_name\",\"type\":[\"null\",\"string\"],\"default\":null},\{\"name\":\"id\",\"type\":[\"null\",\"string\"],\"default\":null},\{\"name\":\"time_string\",\"type\":[\"null\",\"string\"],\"default\":null},\{\"name\":\"device\",\"type\":[\"null\",\"string\"],\"default\":null},\{\"name\":\"timestamp_string\",\"type\":[\"null\",\"string\"],\"default\":null},\{\"name\":\"occur_time\",\"type\":[\"null\",{\"type\":\"long\",\"logicalType\":\"timestamp-millis\"}],\"default\":null},\{\"name\":\"row_time\",\"type\":[\"null\",{\"type\":\"long\",\"logicalType\":\"timestamp-millis\"}],\"default\":null}]}"**
val parquetTableSource: ParquetTableSource = ParquetTableSource
.builder
.forParquetSchema(new org.apache.parquet.avro.AvroSchemaConverter().convert(
org.apache.avro.Schema.parse(schema, true)))
.path("/path/to/login_data")
.build
tableEnv.registerTableSource("source",parquetTableSource)
val t1 = tableEnv.sqlQuery("select log_id,city from source where city = '274'
")
tableEnv.registerTable("t1",t1)
val t4 = tableEnv.sqlQuery("select * from t1 where log_id='5927070661978133'")
[jira] [Created] (FLINK-15564) YarnClusterDescriptorTest failed to validate the original intended behavior
Xintong Song created FLINK-15564:
Summary: YarnClusterDescriptorTest failed to validate the original
intended behavior
Key: FLINK-15564
URL: https://issues.apache.org/jira/browse/FLINK-15564
Project: Flink
Issue Type: Bug
Components: Deployment / YARN
Reporter: Xintong Song
As discovered in [PR#10834|https://github.com/apache/flink/pull/10834], the
following test cases of {{YarnClusterDescriptorTest}} have failed to validate
the original intended behavior and are temporally skipped by PR#10834.
- {{testFailIfTaskSlotsHigherThanMaxVcores}}
- {{testConfigOverwrite}}
The original purpose of these two test cases was to verify the validation logic
against yarn max allocation vcores (in
{{5836f7eddb4849b95d4860cf20045bc61d061918}}).
These two cases should have failed when we change the validation logic to get
yarn max allocation vcores from yarnClient instead of configuration (in
{{e959e6d0cd42f0c5b21c0f03ce547f2025ac58d5}}), because there are no yarn
cluster (neither {{MiniYARNCluster}}) started in these cases, thus
{{yarnClient#getNodeReports}} will never return.
The cases have not failed because another {{IllegalConfigurationException}} was
thrown in {{validateClusterSpecification}}, because of memory validation
failure. The memory validation failure was by design, and in order to verify
the original purpose these two test cases should have been updated with
reasonable memory sizes, which is unfortunately overlooked.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] wangyang0918 commented on a change in pull request #10665: [FLINK-15354] Start and stop minikube only in kubernetes related e2e tests
wangyang0918 commented on a change in pull request #10665: [FLINK-15354] Start
and stop minikube only in kubernetes related e2e tests
URL: https://github.com/apache/flink/pull/10665#discussion_r365636347
##
File path: flink-end-to-end-tests/test-scripts/test_kubernetes_session.sh
##
@@ -29,6 +29,7 @@ ARGS="--output ${OUTPUT_PATH}"
function cleanup {
kubectl delete service ${CLUSTER_ID}
kubectl delete clusterrolebinding ${CLUSTER_ROLE_BINDING}
+stop_kubernetes
Review comment:
Nice suggestion. I will create a follow-up ticket to optimize the start/stop
of minikube.
BTW, i think the Yarn/Mesos/Kubernetes share the logics. Start a new cluster
for each e2e test.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] wangyang0918 commented on a change in pull request #10665: [FLINK-15354] Start and stop minikube only in kubernetes related e2e tests
wangyang0918 commented on a change in pull request #10665: [FLINK-15354] Start
and stop minikube only in kubernetes related e2e tests
URL: https://github.com/apache/flink/pull/10665#discussion_r365635974
##
File path: flink-end-to-end-tests/test-scripts/test_kubernetes_embedded_job.sh
##
@@ -30,7 +30,7 @@ function cleanup {
kubectl delete job flink-job-cluster
kubectl delete service flink-job-cluster
kubectl delete deployment flink-task-manager
-rm -rf ${OUTPUT_VOLUME}
Review comment:
The `${OUTPUT_VOLUME}` is under `${TEST_DATA_DIR}`. And `${TEST_DATA_DIR}`
will be cleaned up automatically when the e2e test finished. See
`test-runner-common.sh#cleanup`.
I will add a hotfix to remove this line. Then it will be more clear.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Comment Edited] (FLINK-15527) can not control the number of container on yarn single job module
[
https://issues.apache.org/jira/browse/FLINK-15527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17013959#comment-17013959
]
Jingsong Lee edited comment on FLINK-15527 at 1/13/20 3:01 AM:
---
Hi [~trohrmann], thanks for your involving. I have provided
"table.exec.hive.infer-source-parallelism.max" to limit max parallelism for
hive source. But still not satisfy user's requirement. [~chenchencc]
was (Author: lzljs3620320):
Hi [~trohrmann], thanks for your involving. I have provided
"table.exec.hive.infer-source-parallelism.max" to limit max parallelism for
hive source. But not still not satisfy user's requirement. [~chenchencc]
> can not control the number of container on yarn single job module
> -
>
> Key: FLINK-15527
> URL: https://issues.apache.org/jira/browse/FLINK-15527
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Hive
>Affects Versions: 1.10.0
>Reporter: chenchencc
>Priority: Major
> Fix For: 1.10.0
>
> Attachments: flink-conf.yaml, image-2020-01-09-14-30-46-973.png,
> yarn_application.png
>
>
> when run yarn single job run many container but paralism set 4
> *scripts:*
> ./bin/flink run -m yarn-cluster -ys 3 -p 4 -yjm 1024m -ytm 4096m -yqu bi -c
> com.cc.test.HiveTest2 ./cc_jars/hive-1.0-SNAPSHOT.jar 11.txt test61 6
> _notes_: in 1.9.1 has cli paramter -yn to control the number of containers
> and in 1.10 remove it
> *result:*
> the number of containers is 500+
>
> *code use:*
> query the table and save it to the hdfs text
>
> the storge of table is 200g+
>
>
>
>
> *code:*
> com.cc.test.HiveTest2
> public static void main(String[] args) throws Exception
> { EnvironmentSettings settings =
> EnvironmentSettings.newInstance().useBlinkPlanner().inStreamingMode().build();
> StreamExecutionEnvironment settings2 =
> StreamExecutionEnvironment.getExecutionEnvironment();
> settings2.setParallelism(Integer.valueOf(args[2]));
> StreamTableEnvironment tableEnv = StreamTableEnvironment.create(settings2,
> settings); String name = "myhive"; String defaultDatabase = "test"; String
> hiveConfDir = "/etc/hive/conf"; String version = "1.2.1"; // or 1.2.1 2.3.4
> HiveCatalog hive = new HiveCatalog(name, defaultDatabase, hiveConfDir,
> version); tableEnv.registerCatalog("myhive", hive);
> tableEnv.useCatalog("myhive"); tableEnv.listTables(); Table table =
> tableEnv.sqlQuery("select id from orderparent_test2 where id =
> 'A21204170176'"); tableEnv.toAppendStream(table, Row.class).print();
> tableEnv.toAppendStream(table, Row.class)
> .writeAsText("hdfs:///user/chenchao1/"+ args[0],
> FileSystem.WriteMode.OVERWRITE); tableEnv.execute(args[1]); }
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] wangyang0918 commented on a change in pull request #10665: [FLINK-15354] Start and stop minikube only in kubernetes related e2e tests
wangyang0918 commented on a change in pull request #10665: [FLINK-15354] Start
and stop minikube only in kubernetes related e2e tests
URL: https://github.com/apache/flink/pull/10665#discussion_r365635600
##
File path: flink-end-to-end-tests/test-scripts/common_kubernetes.sh
##
@@ -26,26 +26,56 @@ MINIKUBE_START_RETRIES=3
MINIKUBE_START_BACKOFF=5
RESULT_HASH="e682ec6622b5e83f2eb614617d5ab2cf"
+# If running tests on non-linux os, the kubectl and minikube should be
installed manually
+function setup_kubernetes_for_linux {
+# Download kubectl, which is a requirement for using minikube.
+if ! [ -x "$(command -v kubectl)" ]; then
+local version=$(curl -s
https://storage.googleapis.com/kubernetes-release/release/stable.txt)
+curl -Lo kubectl
https://storage.googleapis.com/kubernetes-release/release/$version/bin/linux/amd64/kubectl
&& \
+chmod +x kubectl && sudo mv kubectl /usr/local/bin/
+fi
+# Download minikube.
+if ! [ -x "$(command -v minikube)" ]; then
+curl -Lo minikube
https://storage.googleapis.com/minikube/releases/latest/minikube-linux-amd64 &&
\
+chmod +x minikube && sudo mv minikube /usr/local/bin/
+fi
+}
+
function check_kubernetes_status {
minikube status
return $?
}
function start_kubernetes_if_not_running {
if ! check_kubernetes_status; then
-minikube start
+start_command="minikube start"
+# We need sudo permission to set vm-driver to none in linux os.
+[[ "${OS_TYPE}" = "linux" ]] && start_command="sudo ${start_command}
--vm-driver=none"
Review comment:
When running this test on non-linux environment(e.g. Mac Os), we could not
start the minikube under `--vm-driver=none` mode. So we will not need sudo
permission. Of course we could use `sudo minikube start` on MacOS, then we need
to provide the password interactively.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Commented] (FLINK-15527) can not control the number of container on yarn single job module
[
https://issues.apache.org/jira/browse/FLINK-15527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17013983#comment-17013983
]
Yangze Guo commented on FLINK-15527:
[~ykt836] I think this approach won't solve this problem since we now using the
same "Active" resource manager in session mode and per-job mode. We'd better
add some primitives for container limitation or introduce Reactive resource
manager.
> can not control the number of container on yarn single job module
> -
>
> Key: FLINK-15527
> URL: https://issues.apache.org/jira/browse/FLINK-15527
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Hive
>Affects Versions: 1.10.0
>Reporter: chenchencc
>Priority: Major
> Fix For: 1.10.0
>
> Attachments: flink-conf.yaml, image-2020-01-09-14-30-46-973.png,
> yarn_application.png
>
>
> when run yarn single job run many container but paralism set 4
> *scripts:*
> ./bin/flink run -m yarn-cluster -ys 3 -p 4 -yjm 1024m -ytm 4096m -yqu bi -c
> com.cc.test.HiveTest2 ./cc_jars/hive-1.0-SNAPSHOT.jar 11.txt test61 6
> _notes_: in 1.9.1 has cli paramter -yn to control the number of containers
> and in 1.10 remove it
> *result:*
> the number of containers is 500+
>
> *code use:*
> query the table and save it to the hdfs text
>
> the storge of table is 200g+
>
>
>
>
> *code:*
> com.cc.test.HiveTest2
> public static void main(String[] args) throws Exception
> { EnvironmentSettings settings =
> EnvironmentSettings.newInstance().useBlinkPlanner().inStreamingMode().build();
> StreamExecutionEnvironment settings2 =
> StreamExecutionEnvironment.getExecutionEnvironment();
> settings2.setParallelism(Integer.valueOf(args[2]));
> StreamTableEnvironment tableEnv = StreamTableEnvironment.create(settings2,
> settings); String name = "myhive"; String defaultDatabase = "test"; String
> hiveConfDir = "/etc/hive/conf"; String version = "1.2.1"; // or 1.2.1 2.3.4
> HiveCatalog hive = new HiveCatalog(name, defaultDatabase, hiveConfDir,
> version); tableEnv.registerCatalog("myhive", hive);
> tableEnv.useCatalog("myhive"); tableEnv.listTables(); Table table =
> tableEnv.sqlQuery("select id from orderparent_test2 where id =
> 'A21204170176'"); tableEnv.toAppendStream(table, Row.class).print();
> tableEnv.toAppendStream(table, Row.class)
> .writeAsText("hdfs:///user/chenchao1/"+ args[0],
> FileSystem.WriteMode.OVERWRITE); tableEnv.execute(args[1]); }
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-15563) When using ParquetTableSource, The program hangs until OOM
sujun created FLINK-15563:
-
Summary: When using ParquetTableSource, The program hangs until
OOM
Key: FLINK-15563
URL: https://issues.apache.org/jira/browse/FLINK-15563
Project: Flink
Issue Type: Bug
Components: Connectors / FileSystem
Affects Versions: 1.9.1, 1.8.1
Reporter: sujun
def main(args: Array[String]): Unit = \{
val env = StreamExecutionEnvironment.getExecutionEnvironment
env.setStreamTimeCharacteristic(TimeCharacteristic.EventTime)
val tableEnv = StreamTableEnvironment.create(env)
val schema =
"{\"type\":\"record\",\"name\":\"root\",\"fields\":[{\"name\":\"log_id\",\"type\":[\"null\",\"string\"],\"default\":null},\{\"name\":\"city\",\"type\":[\"null\",\"string\"],\"default\":null},\{\"name\":\"log_from\",\"type\":[\"null\",\"string\"],\"default\":null},\{\"name\":\"ip\",\"type\":[\"null\",\"string\"],\"default\":null},\{\"name\":\"type\",\"type\":[\"null\",\"string\"],\"default\":null},\{\"name\":\"data_source\",\"type\":[\"null\",\"string\"],\"default\":null},\{\"name\":\"is_scan\",\"type\":[\"null\",\"string\"],\"default\":null},\{\"name\":\"result\",\"type\":[\"null\",\"string\"],\"default\":null},\{\"name\":\"timelong\",\"type\":[\"null\",\"long\"],\"default\":null},\{\"name\":\"is_sec\",\"type\":[\"null\",\"string\"],\"default\":null},\{\"name\":\"event_name\",\"type\":[\"null\",\"string\"],\"default\":null},\{\"name\":\"id\",\"type\":[\"null\",\"string\"],\"default\":null},\{\"name\":\"time_string\",\"type\":[\"null\",\"string\"],\"default\":null},\{\"name\":\"device\",\"type\":[\"null\",\"string\"],\"default\":null},\{\"name\":\"timestamp_string\",\"type\":[\"null\",\"string\"],\"default\":null},\{\"name\":\"occur_time\",\"type\":[\"null\",{\"type\":\"long\",\"logicalType\":\"timestamp-millis\"}],\"default\":null},\{\"name\":\"row_time\",\"type\":[\"null\",{\"type\":\"long\",\"logicalType\":\"timestamp-millis\"}],\"default\":null}]}"**
val parquetTableSource: ParquetTableSource = ParquetTableSource
.builder
.forParquetSchema(new org.apache.parquet.avro.AvroSchemaConverter().convert(
org.apache.avro.Schema.parse(schema, true)))
.path("/path/to/login_data")
.build
tableEnv.registerTableSource("source",parquetTableSource)
val t1 = tableEnv.sqlQuery("select log_id,city from source where city = '274'
")
tableEnv.registerTable("t1",t1)
val t4 = tableEnv.sqlQuery("select * from t1 where log_id='5927070661978133'")
t1.toAppendStream[Row].print()
env.execute()
}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-15530) Replace process memory with flink memory for TMs in default flink-conf.yaml
[ https://issues.apache.org/jira/browse/FLINK-15530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xintong Song updated FLINK-15530: - Priority: Blocker (was: Major) > Replace process memory with flink memory for TMs in default flink-conf.yaml > --- > > Key: FLINK-15530 > URL: https://issues.apache.org/jira/browse/FLINK-15530 > Project: Flink > Issue Type: Improvement > Components: Runtime / Configuration >Reporter: Xintong Song >Assignee: Xintong Song >Priority: Blocker > Labels: pull-request-available > Fix For: 1.10.0 > > Time Spent: 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] wangyang0918 commented on a change in pull request #10665: [FLINK-15354] Start and stop minikube only in kubernetes related e2e tests
wangyang0918 commented on a change in pull request #10665: [FLINK-15354] Start
and stop minikube only in kubernetes related e2e tests
URL: https://github.com/apache/flink/pull/10665#discussion_r365635088
##
File path: flink-end-to-end-tests/test-scripts/common_kubernetes.sh
##
@@ -26,26 +26,56 @@ MINIKUBE_START_RETRIES=3
MINIKUBE_START_BACKOFF=5
RESULT_HASH="e682ec6622b5e83f2eb614617d5ab2cf"
+# If running tests on non-linux os, the kubectl and minikube should be
installed manually
+function setup_kubernetes_for_linux {
+# Download kubectl, which is a requirement for using minikube.
+if ! [ -x "$(command -v kubectl)" ]; then
+local version=$(curl -s
https://storage.googleapis.com/kubernetes-release/release/stable.txt)
+curl -Lo kubectl
https://storage.googleapis.com/kubernetes-release/release/$version/bin/linux/amd64/kubectl
&& \
+chmod +x kubectl && sudo mv kubectl /usr/local/bin/
Review comment:
Yeah, we have the `sudo: required` in `.travis.yml`. So we will have the
sudo permission without providing password.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Commented] (FLINK-14684) Add Pinterest to Chinese Powered By page
[ https://issues.apache.org/jira/browse/FLINK-14684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17013977#comment-17013977 ] Jiawei Wu commented on FLINK-14684: --- [~fhueske] Thanks! I will try to finish it in the next 3 days. BTW, could I combine this with [FLINK-14313|https://issues.apache.org/jira/browse/FLINK-14313] and send only 1 PR? > Add Pinterest to Chinese Powered By page > > > Key: FLINK-14684 > URL: https://issues.apache.org/jira/browse/FLINK-14684 > Project: Flink > Issue Type: New Feature > Components: chinese-translation >Reporter: Hequn Cheng >Assignee: Jiawei Wu >Priority: Minor > > Pinterest was added to the English Powered By page with commit: > [51f7e3ced85b94dcbe3c051069379d22c88fbc5c|https://github.com/apache/flink-web/pull/281] > It should be added to the Chinese Powered By (and index.html) page as well. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] JingsongLi commented on a change in pull request #10797: [FLINK-15172][table-blink] Optimize the operator algorithm to lazily allocate memory
JingsongLi commented on a change in pull request #10797:
[FLINK-15172][table-blink] Optimize the operator algorithm to lazily allocate
memory
URL: https://github.com/apache/flink/pull/10797#discussion_r365632450
##
File path:
flink-table/flink-table-runtime-blink/src/main/java/org/apache/flink/table/runtime/util/LazyMemorySegmentPool.java
##
@@ -0,0 +1,118 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.util;
+
+import org.apache.flink.core.memory.MemorySegment;
+import org.apache.flink.runtime.memory.MemoryAllocationException;
+import org.apache.flink.runtime.memory.MemoryManager;
+
+import java.io.Closeable;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+
+/**
+ * {@link MemorySegmentPool} that lazy allocate memory pages from {@link
MemoryManager}.
+ */
+public class LazyMemorySegmentPool implements MemorySegmentPool, Closeable {
+
+ private static final long PER_REQUEST_MEMORY_SIZE = 16 * 1024 * 1024;
+
+ private final Object owner;
+ private final MemoryManager memoryManager;
+ private final boolean freeMore;
+ private final ArrayList cachePages;
+ private final int maxPages;
+ private final int perRequestPages;
+
+ private int pageUsage;
+
+ public LazyMemorySegmentPool(Object owner, MemoryManager memoryManager,
int maxPages) {
+ this(owner, memoryManager, maxPages, false);
+ }
+
+ public LazyMemorySegmentPool(Object owner, MemoryManager memoryManager,
int maxPages, boolean freeMore) {
+ this.owner = owner;
+ this.memoryManager = memoryManager;
+ this.freeMore = freeMore;
+ this.cachePages = new ArrayList<>();
+ this.maxPages = maxPages;
+ this.pageUsage = 0;
+ this.perRequestPages = Math.max(1, (int)
(PER_REQUEST_MEMORY_SIZE / memoryManager.getPageSize()));
+ }
+
+ @Override
+ public int pageSize() {
+ return this.memoryManager.getPageSize();
+ }
+
+ @Override
+ public void returnAll(List memory) {
+ this.pageUsage -= memory.size();
+ if (this.pageUsage < 0) {
+ throw new RuntimeException("Return too more memories.");
+ }
+ this.cachePages.addAll(memory);
+
+ if (this.freeMore && this.cachePages.size() >=
this.perRequestPages) {
Review comment:
not used now, I'll delete it.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Comment Edited] (FLINK-15562) Unable to create walk through project
[
https://issues.apache.org/jira/browse/FLINK-15562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17013961#comment-17013961
]
Yangze Guo edited comment on FLINK-15562 at 1/13/20 2:11 AM:
-
_With maven-archetype-plugin 3.x it is not possible anymore to specify the
repository as commandline argument (for consistency and security reasons)._[1]
There is already a note to clarify it from FLINK-7839. I think we can also give
another suggestion, locking the version of the plugin to 2.4, in the document.
If it makes sense to you, I'd like to work for it.
cc [~chesnay] [~gjy] [~aljoscha]
[1]
https://stackoverflow.com/questions/43570594/error-executing-maven-archetype-generate-command-to-create-a-opendaylight-projec
was (Author: karmagyz):
_With maven-archetype-plugin 3.x it is not possible anymore to specify the
repository as commandline argument (for consistency and security reasons)._[1]
There is already a note to clarify it from FLINK-7839. I think we can also give
another suggestion, lock the version of the plugin to 2.4, in the document. If
it make sense to you, I'd like work for it.
cc [~chesnay] [~gjy]
[1]
https://stackoverflow.com/questions/43570594/error-executing-maven-archetype-generate-command-to-create-a-opendaylight-projec
> Unable to create walk through project
> -
>
> Key: FLINK-15562
> URL: https://issues.apache.org/jira/browse/FLINK-15562
> Project: Flink
> Issue Type: Bug
> Components: Documentation
>Affects Versions: 1.10.0
>Reporter: Jeff Zhang
>Priority: Major
>
> I try to follow the instruction here to create flink walk though project, but
> hit the following errors.
> [https://ci.apache.org/projects/flink/flink-docs-master/getting-started/walkthroughs/table_api.html]
> {code:java}
> ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-archetype-plugin:3.0.1:generate (default-cli)
> on project standalone-pom: archetypeCatalog
> 'https://repository.apache.org/content/repositories/snapshots/' is not
> supported anymore. Please read the plugin documentation for details. -> [Help
> 1]
> [ERROR]
> [ERROR] To see the full stack trace of the errors, re-run Maven with the -e
> switch.
> [ERROR] Re-run Maven using the -X switch to enable full debug logging.
> [ERROR]
> [ERROR] For more information about the errors and possible solutions, please
> read the following articles:
> [ERROR] [Help 1]
> http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-15527) can not control the number of container on yarn single job module
[
https://issues.apache.org/jira/browse/FLINK-15527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17013963#comment-17013963
]
Kurt Young commented on FLINK-15527:
[~chenchencc] can you use yarn session mode to achieve what you need? To me,
yarn per job mode is somewhat like map reduce, we might launch as many
containers as we needed after infer the job parallelism. With session mode, you
can strictly control the number of TMs, aka containers, and no matter how high
the parallelism if for your source operator, they will be finished step by
step.
> can not control the number of container on yarn single job module
> -
>
> Key: FLINK-15527
> URL: https://issues.apache.org/jira/browse/FLINK-15527
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Hive
>Affects Versions: 1.10.0
>Reporter: chenchencc
>Priority: Major
> Fix For: 1.10.0
>
> Attachments: flink-conf.yaml, image-2020-01-09-14-30-46-973.png,
> yarn_application.png
>
>
> when run yarn single job run many container but paralism set 4
> *scripts:*
> ./bin/flink run -m yarn-cluster -ys 3 -p 4 -yjm 1024m -ytm 4096m -yqu bi -c
> com.cc.test.HiveTest2 ./cc_jars/hive-1.0-SNAPSHOT.jar 11.txt test61 6
> _notes_: in 1.9.1 has cli paramter -yn to control the number of containers
> and in 1.10 remove it
> *result:*
> the number of containers is 500+
>
> *code use:*
> query the table and save it to the hdfs text
>
> the storge of table is 200g+
>
>
>
>
> *code:*
> com.cc.test.HiveTest2
> public static void main(String[] args) throws Exception
> { EnvironmentSettings settings =
> EnvironmentSettings.newInstance().useBlinkPlanner().inStreamingMode().build();
> StreamExecutionEnvironment settings2 =
> StreamExecutionEnvironment.getExecutionEnvironment();
> settings2.setParallelism(Integer.valueOf(args[2]));
> StreamTableEnvironment tableEnv = StreamTableEnvironment.create(settings2,
> settings); String name = "myhive"; String defaultDatabase = "test"; String
> hiveConfDir = "/etc/hive/conf"; String version = "1.2.1"; // or 1.2.1 2.3.4
> HiveCatalog hive = new HiveCatalog(name, defaultDatabase, hiveConfDir,
> version); tableEnv.registerCatalog("myhive", hive);
> tableEnv.useCatalog("myhive"); tableEnv.listTables(); Table table =
> tableEnv.sqlQuery("select id from orderparent_test2 where id =
> 'A21204170176'"); tableEnv.toAppendStream(table, Row.class).print();
> tableEnv.toAppendStream(table, Row.class)
> .writeAsText("hdfs:///user/chenchao1/"+ args[0],
> FileSystem.WriteMode.OVERWRITE); tableEnv.execute(args[1]); }
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (FLINK-15562) Unable to create walk through project
[
https://issues.apache.org/jira/browse/FLINK-15562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17013961#comment-17013961
]
Yangze Guo edited comment on FLINK-15562 at 1/13/20 2:09 AM:
-
_With maven-archetype-plugin 3.x it is not possible anymore to specify the
repository as commandline argument (for consistency and security reasons)._[1]
There is already a note to clarify it from FLINK-7839. I think we can also give
another suggestion, lock the version of the plugin to 2.4, in the document. If
it make sense to you, I'd like work for it.
cc [~chesnay] [~gjy]
[1]
https://stackoverflow.com/questions/43570594/error-executing-maven-archetype-generate-command-to-create-a-opendaylight-projec
was (Author: karmagyz):
_With maven-archetype-plugin 3.x it is not possible anymore to specify the
repository as commandline argument (for consistency and security reasons)._[1]
There is already a note to clarify it from FLINK-7839. I think we can also give
another suggestion, lock the version of the plugin to 2.4, in the document. If
it make sense to you, I'd like work for it.
cc [~chesnay]
[1]
https://stackoverflow.com/questions/43570594/error-executing-maven-archetype-generate-command-to-create-a-opendaylight-projec
> Unable to create walk through project
> -
>
> Key: FLINK-15562
> URL: https://issues.apache.org/jira/browse/FLINK-15562
> Project: Flink
> Issue Type: Bug
> Components: Documentation
>Affects Versions: 1.10.0
>Reporter: Jeff Zhang
>Priority: Major
>
> I try to follow the instruction here to create flink walk though project, but
> hit the following errors.
> [https://ci.apache.org/projects/flink/flink-docs-master/getting-started/walkthroughs/table_api.html]
> {code:java}
> ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-archetype-plugin:3.0.1:generate (default-cli)
> on project standalone-pom: archetypeCatalog
> 'https://repository.apache.org/content/repositories/snapshots/' is not
> supported anymore. Please read the plugin documentation for details. -> [Help
> 1]
> [ERROR]
> [ERROR] To see the full stack trace of the errors, re-run Maven with the -e
> switch.
> [ERROR] Re-run Maven using the -X switch to enable full debug logging.
> [ERROR]
> [ERROR] For more information about the errors and possible solutions, please
> read the following articles:
> [ERROR] [Help 1]
> http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-15562) Unable to create walk through project
[
https://issues.apache.org/jira/browse/FLINK-15562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17013961#comment-17013961
]
Yangze Guo commented on FLINK-15562:
_With maven-archetype-plugin 3.x it is not possible anymore to specify the
repository as commandline argument (for consistency and security reasons)._[1]
There is already a note to clarify it from FLINK-7839. I think we can also give
another suggestion, lock the version of the plugin to 2.4, in the document. If
it make sense to you, I'd like work for it.
cc [~chesnay]
[1]
https://stackoverflow.com/questions/43570594/error-executing-maven-archetype-generate-command-to-create-a-opendaylight-projec
> Unable to create walk through project
> -
>
> Key: FLINK-15562
> URL: https://issues.apache.org/jira/browse/FLINK-15562
> Project: Flink
> Issue Type: Bug
> Components: Documentation
>Affects Versions: 1.10.0
>Reporter: Jeff Zhang
>Priority: Major
>
> I try to follow the instruction here to create flink walk though project, but
> hit the following errors.
> [https://ci.apache.org/projects/flink/flink-docs-master/getting-started/walkthroughs/table_api.html]
> {code:java}
> ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-archetype-plugin:3.0.1:generate (default-cli)
> on project standalone-pom: archetypeCatalog
> 'https://repository.apache.org/content/repositories/snapshots/' is not
> supported anymore. Please read the plugin documentation for details. -> [Help
> 1]
> [ERROR]
> [ERROR] To see the full stack trace of the errors, re-run Maven with the -e
> switch.
> [ERROR] Re-run Maven using the -X switch to enable full debug logging.
> [ERROR]
> [ERROR] For more information about the errors and possible solutions, please
> read the following articles:
> [ERROR] [Help 1]
> http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-15527) can not control the number of container on yarn single job module
[
https://issues.apache.org/jira/browse/FLINK-15527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17013959#comment-17013959
]
Jingsong Lee commented on FLINK-15527:
--
Hi [~trohrmann], thanks for your involving. I have provided
"table.exec.hive.infer-source-parallelism.max" to limit max parallelism for
hive source. But not still not satisfy user's requirement. [~chenchencc]
> can not control the number of container on yarn single job module
> -
>
> Key: FLINK-15527
> URL: https://issues.apache.org/jira/browse/FLINK-15527
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Hive
>Affects Versions: 1.10.0
>Reporter: chenchencc
>Priority: Major
> Fix For: 1.10.0
>
> Attachments: flink-conf.yaml, image-2020-01-09-14-30-46-973.png,
> yarn_application.png
>
>
> when run yarn single job run many container but paralism set 4
> *scripts:*
> ./bin/flink run -m yarn-cluster -ys 3 -p 4 -yjm 1024m -ytm 4096m -yqu bi -c
> com.cc.test.HiveTest2 ./cc_jars/hive-1.0-SNAPSHOT.jar 11.txt test61 6
> _notes_: in 1.9.1 has cli paramter -yn to control the number of containers
> and in 1.10 remove it
> *result:*
> the number of containers is 500+
>
> *code use:*
> query the table and save it to the hdfs text
>
> the storge of table is 200g+
>
>
>
>
> *code:*
> com.cc.test.HiveTest2
> public static void main(String[] args) throws Exception
> { EnvironmentSettings settings =
> EnvironmentSettings.newInstance().useBlinkPlanner().inStreamingMode().build();
> StreamExecutionEnvironment settings2 =
> StreamExecutionEnvironment.getExecutionEnvironment();
> settings2.setParallelism(Integer.valueOf(args[2]));
> StreamTableEnvironment tableEnv = StreamTableEnvironment.create(settings2,
> settings); String name = "myhive"; String defaultDatabase = "test"; String
> hiveConfDir = "/etc/hive/conf"; String version = "1.2.1"; // or 1.2.1 2.3.4
> HiveCatalog hive = new HiveCatalog(name, defaultDatabase, hiveConfDir,
> version); tableEnv.registerCatalog("myhive", hive);
> tableEnv.useCatalog("myhive"); tableEnv.listTables(); Table table =
> tableEnv.sqlQuery("select id from orderparent_test2 where id =
> 'A21204170176'"); tableEnv.toAppendStream(table, Row.class).print();
> tableEnv.toAppendStream(table, Row.class)
> .writeAsText("hdfs:///user/chenchao1/"+ args[0],
> FileSystem.WriteMode.OVERWRITE); tableEnv.execute(args[1]); }
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-15562) Unable to create walk through project
[
https://issues.apache.org/jira/browse/FLINK-15562?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jeff Zhang updated FLINK-15562:
---
Component/s: Documentation
> Unable to create walk through project
> -
>
> Key: FLINK-15562
> URL: https://issues.apache.org/jira/browse/FLINK-15562
> Project: Flink
> Issue Type: Bug
> Components: Documentation
>Affects Versions: 1.10.0
>Reporter: Jeff Zhang
>Priority: Major
>
> I try to follow the instruction here to create flink walk though project, but
> hit the following errors.
> [https://ci.apache.org/projects/flink/flink-docs-master/getting-started/walkthroughs/table_api.html]
> {code:java}
> ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-archetype-plugin:3.0.1:generate (default-cli)
> on project standalone-pom: archetypeCatalog
> 'https://repository.apache.org/content/repositories/snapshots/' is not
> supported anymore. Please read the plugin documentation for details. -> [Help
> 1]
> [ERROR]
> [ERROR] To see the full stack trace of the errors, re-run Maven with the -e
> switch.
> [ERROR] Re-run Maven using the -X switch to enable full debug logging.
> [ERROR]
> [ERROR] For more information about the errors and possible solutions, please
> read the following articles:
> [ERROR] [Help 1]
> http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-15562) Unable to create walk through project
Jeff Zhang created FLINK-15562:
--
Summary: Unable to create walk through project
Key: FLINK-15562
URL: https://issues.apache.org/jira/browse/FLINK-15562
Project: Flink
Issue Type: Bug
Affects Versions: 1.10.0
Reporter: Jeff Zhang
I try to follow the instruction here to create flink walk though project, but
hit the following errors.
[https://ci.apache.org/projects/flink/flink-docs-master/getting-started/walkthroughs/table_api.html]
{code:java}
ERROR] Failed to execute goal
org.apache.maven.plugins:maven-archetype-plugin:3.0.1:generate (default-cli) on
project standalone-pom: archetypeCatalog
'https://repository.apache.org/content/repositories/snapshots/' is not
supported anymore. Please read the plugin documentation for details. -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e
switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please
read the following articles:
[ERROR] [Help 1]
http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] jiasheng55 commented on issue #8400: [FLINK-12472][yarn] Support setting attemptFailuresValidityInterval o…
jiasheng55 commented on issue #8400: [FLINK-12472][yarn] Support setting attemptFailuresValidityInterval o… URL: https://github.com/apache/flink/pull/8400#issuecomment-573483344 @langdangjushi 请平息一下情绪……你可以看一下这个“File Changed”中详细改动内容，在改动之前，`reflector.setAttemptFailuresValidityInterval`传入的值是`AkkaUtils.getTimeout(flinkConfiguration).toMillis()`，也就是10s； 在和社区讨论的时候，为了不因为这个PR改动原有的作业默认行为，保持了10s的默认值。 如果你对这个默认配置值有改动建议，可以在Flink issue或者邮件组里表达一下自己的建议，推动改动的落地:) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #10682: [FLINK-15247][Runtime] Wait for all slots to be free before task executor services shutdown upon stopping
flinkbot edited a comment on issue #10682: [FLINK-15247][Runtime] Wait for all slots to be free before task executor services shutdown upon stopping URL: https://github.com/apache/flink/pull/10682#issuecomment-568850864 ## CI report: * a7411a5d1a86381078670898c1f20be4fe3a88f4 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/142305023) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=3903) * 178357b40789d132e1db954a199bf1c62efff474 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/143770612) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4240) * 8d2db0ad7b2ce3c268cd8c52a6332a2bc86aca61 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/144064773) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4271) * 5625904d8f9e74bb7a7f1ff392094a8c1ea9b9c2 Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/144068497) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4272) * e3be286cd26d25b91e5805eed149bdf5a39aa292 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/144075835) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4275) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #10762: [FLINK-15115][kafka] Drop Kafka 0.8/0.9
flinkbot edited a comment on issue #10762: [FLINK-15115][kafka] Drop Kafka 0.8/0.9 URL: https://github.com/apache/flink/pull/10762#issuecomment-570591280 ## CI report: * 6bfb6d18af5fdfbddadf6da53adab78567d64aa9 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/143008336) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4082) * d61cca153ada8faf1ca0c59e9cb8ec8bd55ed39e Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/143046561) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4087) * 62027cd3e2e3237eebb24fb2ded311feb0ded645 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/144075826) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4274) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #10830: [FLINK-15554][azure] Bump jetty-util to 3.1.2
flinkbot edited a comment on issue #10830: [FLINK-15554][azure] Bump jetty-util to 3.1.2 URL: https://github.com/apache/flink/pull/10830#issuecomment-573052596 ## CI report: * 8aba8ccd54ff402b95b302453317d534febb5df9 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/143899788) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4260) * 600e84d3d506226d2361f5037f84210b97164e72 Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/144073760) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4273) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
