[jira] [Commented] (FLINK-15581) SpillingResettableMutableObjectIterator data overflow
[
https://issues.apache.org/jira/browse/FLINK-15581?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17014911#comment-17014911
]
Benchao Li commented on FLINK-15581:
Hi [~pnowojski], I noticed that [~caojian0613] has already created a issue to
track this: https://issues.apache.org/jira/browse/FLINK-15549
> SpillingResettableMutableObjectIterator data overflow
> -
>
> Key: FLINK-15581
> URL: https://issues.apache.org/jira/browse/FLINK-15581
> Project: Flink
> Issue Type: Bug
> Components: API / DataSet
>Affects Versions: 1.6.4, 1.7.2, 1.8.3, 1.9.1, 1.10.0
>Reporter: Piotr Nowojski
>Priority: Minor
>
> As [reported by a user on the mailing
> list|https://lists.apache.org/thread.html/r1e3c53eaddfd8050c94ee4e521da4fc96a119662937cf801801bde52%40%3Cuser.flink.apache.org%3E]
> {quote}
> SpillingResettableMutableObjectIterator has a data overflow problem if the
> number of elements in a single input exceeds Integer.MAX_VALUE.
> The reason is inside the SpillingResettableMutableObjectIterator, it track
> the total number of elements and the number of elements currently read with
> two int type fileds (elementCount and currentElementNum), and if the number
> of elements exceeds Integer.MAX_VALUE, it will overflow.
> If there is an overflow, then in the next iteration, after reset the input ,
> the data will not be read or only part of the data will be read.
> {quote}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-11899) Introduce vectorized parquet InputFormat for blink runtime
[ https://issues.apache.org/jira/browse/FLINK-11899?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17014909#comment-17014909 ] Jingsong Lee commented on FLINK-11899: -- Hi [~hpeter], another way is re-writing readers instead of re-using hive readers, just like: [https://github.com/flink-tpc-ds/flink/tree/tpcds-master/flink-table/flink-table-runtime-blink/src/main/java/org/apache/flink/table/runtime/parquet] I think that is better way, otherwise we will deal with multi hive versions again... Which is annoying. > Introduce vectorized parquet InputFormat for blink runtime > -- > > Key: FLINK-11899 > URL: https://issues.apache.org/jira/browse/FLINK-11899 > Project: Flink > Issue Type: Sub-task > Components: Table SQL / Runtime >Reporter: Jingsong Lee >Assignee: Zhenqiu Huang >Priority: Major > Fix For: 1.11.0 > > > VectorizedParquetInputFormat is introduced to read parquet data in batches. > When returning each row of data, instead of actually retrieving each field, > we use BaseRow's abstraction to return a Columnar Row-like view. > This will greatly improve the downstream filtered scenarios, so that there is > no need to access redundant fields on the filtered data. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (FLINK-15549) integer overflow in SpillingResettableMutableObjectIterator
[ https://issues.apache.org/jira/browse/FLINK-15549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Piotr Nowojski reassigned FLINK-15549: -- Assignee: caojian0613 > integer overflow in SpillingResettableMutableObjectIterator > --- > > Key: FLINK-15549 > URL: https://issues.apache.org/jira/browse/FLINK-15549 > Project: Flink > Issue Type: Bug > Components: API / DataSet >Affects Versions: 1.6.4, 1.7.2, 1.8.3, 1.9.1, 1.10.0 >Reporter: caojian0613 >Assignee: caojian0613 >Priority: Major > Labels: overflow > > The SpillingResettableMutableObjectIterator has a data overflow problem if > the number of elements in a single input exceeds Integer.MAX_VALUE. > The reason is inside the SpillingResettableMutableObjectIterator, it track > the total number of elements and the number of elements currently read with > two int type fileds (elementCount and currentElementNum), and if the number > of elements exceeds Integer.MAX_VALUE, it will overflow. > If there is an overflow, then in the next iteration, after reset the input , > the data will not be read or only part of the data will be read. > Therefore, we should changing the type of these two fields of > SpillingResettableIterator* from int to long, and we also need a pre-check > mechanism before such numerical. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-15549) integer overflow in SpillingResettableMutableObjectIterator
[ https://issues.apache.org/jira/browse/FLINK-15549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Piotr Nowojski updated FLINK-15549: --- Affects Version/s: 1.6.4 1.7.2 1.8.3 1.9.1 > integer overflow in SpillingResettableMutableObjectIterator > --- > > Key: FLINK-15549 > URL: https://issues.apache.org/jira/browse/FLINK-15549 > Project: Flink > Issue Type: Bug > Components: Runtime / Task >Affects Versions: 1.6.4, 1.7.2, 1.8.3, 1.9.1, 1.10.0 >Reporter: caojian0613 >Priority: Major > Labels: overflow > > The SpillingResettableMutableObjectIterator has a data overflow problem if > the number of elements in a single input exceeds Integer.MAX_VALUE. > The reason is inside the SpillingResettableMutableObjectIterator, it track > the total number of elements and the number of elements currently read with > two int type fileds (elementCount and currentElementNum), and if the number > of elements exceeds Integer.MAX_VALUE, it will overflow. > If there is an overflow, then in the next iteration, after reset the input , > the data will not be read or only part of the data will be read. > Therefore, we should changing the type of these two fields of > SpillingResettableIterator* from int to long, and we also need a pre-check > mechanism before such numerical. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (FLINK-15581) SpillingResettableMutableObjectIterator data overflow
[
https://issues.apache.org/jira/browse/FLINK-15581?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Piotr Nowojski closed FLINK-15581.
--
Resolution: Duplicate
> SpillingResettableMutableObjectIterator data overflow
> -
>
> Key: FLINK-15581
> URL: https://issues.apache.org/jira/browse/FLINK-15581
> Project: Flink
> Issue Type: Bug
> Components: API / DataSet
>Affects Versions: 1.6.4, 1.7.2, 1.8.3, 1.9.1, 1.10.0
>Reporter: Piotr Nowojski
>Priority: Minor
>
> As [reported by a user on the mailing
> list|https://lists.apache.org/thread.html/r1e3c53eaddfd8050c94ee4e521da4fc96a119662937cf801801bde52%40%3Cuser.flink.apache.org%3E]
> {quote}
> SpillingResettableMutableObjectIterator has a data overflow problem if the
> number of elements in a single input exceeds Integer.MAX_VALUE.
> The reason is inside the SpillingResettableMutableObjectIterator, it track
> the total number of elements and the number of elements currently read with
> two int type fileds (elementCount and currentElementNum), and if the number
> of elements exceeds Integer.MAX_VALUE, it will overflow.
> If there is an overflow, then in the next iteration, after reset the input ,
> the data will not be read or only part of the data will be read.
> {quote}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-15549) integer overflow in SpillingResettableMutableObjectIterator
[ https://issues.apache.org/jira/browse/FLINK-15549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Piotr Nowojski updated FLINK-15549: --- Component/s: (was: Runtime / Task) API / DataSet > integer overflow in SpillingResettableMutableObjectIterator > --- > > Key: FLINK-15549 > URL: https://issues.apache.org/jira/browse/FLINK-15549 > Project: Flink > Issue Type: Bug > Components: API / DataSet >Affects Versions: 1.6.4, 1.7.2, 1.8.3, 1.9.1, 1.10.0 >Reporter: caojian0613 >Priority: Major > Labels: overflow > > The SpillingResettableMutableObjectIterator has a data overflow problem if > the number of elements in a single input exceeds Integer.MAX_VALUE. > The reason is inside the SpillingResettableMutableObjectIterator, it track > the total number of elements and the number of elements currently read with > two int type fileds (elementCount and currentElementNum), and if the number > of elements exceeds Integer.MAX_VALUE, it will overflow. > If there is an overflow, then in the next iteration, after reset the input , > the data will not be read or only part of the data will be read. > Therefore, we should changing the type of these two fields of > SpillingResettableIterator* from int to long, and we also need a pre-check > mechanism before such numerical. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-15581) SpillingResettableMutableObjectIterator data overflow
Piotr Nowojski created FLINK-15581:
--
Summary: SpillingResettableMutableObjectIterator data overflow
Key: FLINK-15581
URL: https://issues.apache.org/jira/browse/FLINK-15581
Project: Flink
Issue Type: Bug
Components: API / DataSet
Affects Versions: 1.9.1, 1.8.3, 1.7.2, 1.6.4, 1.10.0
Reporter: Piotr Nowojski
As [reported by a user on the mailing
list|https://lists.apache.org/thread.html/r1e3c53eaddfd8050c94ee4e521da4fc96a119662937cf801801bde52%40%3Cuser.flink.apache.org%3E]
{quote}
SpillingResettableMutableObjectIterator has a data overflow problem if the
number of elements in a single input exceeds Integer.MAX_VALUE.
The reason is inside the SpillingResettableMutableObjectIterator, it track the
total number of elements and the number of elements currently read with two int
type fileds (elementCount and currentElementNum), and if the number of elements
exceeds Integer.MAX_VALUE, it will overflow.
If there is an overflow, then in the next iteration, after reset the input ,
the data will not be read or only part of the data will be read.
{quote}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot commented on issue #10849: [hotfix] [javadocs] fix typo in KafkaTopicPartitionStateWithPunctuatedWatermarks class introduction
flinkbot commented on issue #10849: [hotfix] [javadocs] fix typo in KafkaTopicPartitionStateWithPunctuatedWatermarks class introduction URL: https://github.com/apache/flink/pull/10849#issuecomment-574047859 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit c780c93aad6554dab17acb7110246b957d6daf77 (Tue Jan 14 07:48:09 UTC 2020) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] guliziduo opened a new pull request #10849: [hotfix] [javadocs] fix typo in KafkaTopicPartitionStateWithPunctuatedWatermarks class introduction
guliziduo opened a new pull request #10849: [hotfix] [javadocs] fix typo in KafkaTopicPartitionStateWithPunctuatedWatermarks class introduction URL: https://github.com/apache/flink/pull/10849 ## What is the purpose of the change fix typo in KafkaTopicPartitionStateWithPunctuatedWatermarks class introduction ## Brief change log - * fix typo in KafkaTopicPartitionStateWithPunctuatedWatermarks class introduction* ## Verifying this change This change is a trivial rework / code cleanup without any test coverage. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (no) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (no) - The serializers: (no) - The runtime per-record code paths (performance sensitive): (no) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: (no ) - The S3 file system connector: (no) ## Documentation - Does this pull request introduce a new feature? (no) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] dawidwys commented on issue #10674: [FLINK-15220][Connector/Kafka][Table] Add startFromTimestamp in KafkaTableSource
dawidwys commented on issue #10674: [FLINK-15220][Connector/Kafka][Table] Add startFromTimestamp in KafkaTableSource URL: https://github.com/apache/flink/pull/10674#issuecomment-574046614 Thank you for the update @link3280 . Will merge it later today. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-11899) Introduce vectorized parquet InputFormat for blink runtime
[ https://issues.apache.org/jira/browse/FLINK-11899?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17014906#comment-17014906 ] Zhenqiu Huang commented on FLINK-11899: --- [~lzljs3620320] I want to leverage existing org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedPrimitiveColumnReader. So ParquetColumnarRowSplitReader will need a wrap on hive column vector as ORC. Not sure what's the efficient way of reading data directly to Flink Vector. > Introduce vectorized parquet InputFormat for blink runtime > -- > > Key: FLINK-11899 > URL: https://issues.apache.org/jira/browse/FLINK-11899 > Project: Flink > Issue Type: Sub-task > Components: Table SQL / Runtime >Reporter: Jingsong Lee >Assignee: Zhenqiu Huang >Priority: Major > Fix For: 1.11.0 > > > VectorizedParquetInputFormat is introduced to read parquet data in batches. > When returning each row of data, instead of actually retrieving each field, > we use BaseRow's abstraction to return a Columnar Row-like view. > This will greatly improve the downstream filtered scenarios, so that there is > no need to access redundant fields on the filtered data. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on issue #10845: [FLINK-15355][plugins] Classloader avoids loading unrelated services.
flinkbot edited a comment on issue #10845: [FLINK-15355][plugins] Classloader avoids loading unrelated services. URL: https://github.com/apache/flink/pull/10845#issuecomment-573736294 ## CI report: * 0942cb8b913ba50ecf8d7ca28832c4c92bf78e6c Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/144178023) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4303) * ce62e6539d1394a5d27d8ef51db010104852e433 Travis: [PENDING](https://travis-ci.com/flink-ci/flink/builds/144207965) Azure: [PENDING](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4304) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-15552) SQL Client can not correctly create kafka table using --library to indicate a kafka connector directory
[ https://issues.apache.org/jira/browse/FLINK-15552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17014903#comment-17014903 ] Terry Wang commented on FLINK-15552: Hi, [~jark]. I'm sure about it. As for SQL CLI e2e tests, maybe there is kafka jar under /lib directory. > SQL Client can not correctly create kafka table using --library to indicate a > kafka connector directory > --- > > Key: FLINK-15552 > URL: https://issues.apache.org/jira/browse/FLINK-15552 > Project: Flink > Issue Type: Bug > Components: Table SQL / Client, Table SQL / Runtime >Reporter: Terry Wang >Priority: Major > > How to Reproduce: > first, I start a sql client and using `-l` to point to a kafka connector > directory. > ` > bin/sql-client.sh embedded -l /xx/connectors/kafka/ > ` > Then, I create a Kafka Table like following > ` > Flink SQL> CREATE TABLE MyUserTable ( > > content String > > ) WITH ( > > 'connector.type' = 'kafka', > > 'connector.version' = 'universal', > > 'connector.topic' = 'test', > > 'connector.properties.zookeeper.connect' = 'localhost:2181', > > 'connector.properties.bootstrap.servers' = 'localhost:9092', > > 'connector.properties.group.id' = 'testGroup', > > 'connector.startup-mode' = 'earliest-offset', > > 'format.type' = 'csv' > > ); > [INFO] Table has been created. > ` > Then I select from just created table and an exception been thrown: > ` > Flink SQL> select * from MyUserTable; > [ERROR] Could not execute SQL statement. Reason: > org.apache.flink.table.api.NoMatchingTableFactoryException: Could not find a > suitable table factory for > 'org.apache.flink.table.factories.TableSourceFactory' in > the classpath. > Reason: Required context properties mismatch. > The matching candidates: > org.apache.flink.table.sources.CsvBatchTableSourceFactory > Mismatched properties: > 'connector.type' expects 'filesystem', but is 'kafka' > The following properties are requested: > connector.properties.bootstrap.servers=localhost:9092 > connector.properties.group.id=testGroup > connector.properties.zookeeper.connect=localhost:2181 > connector.startup-mode=earliest-offset > connector.topic=test > connector.type=kafka > connector.version=universal > format.type=csv > schema.0.data-type=VARCHAR(2147483647) > schema.0.name=content > The following factories have been considered: > org.apache.flink.table.sources.CsvBatchTableSourceFactory > org.apache.flink.table.sources.CsvAppendTableSourceFactory > ` > Potential Reasons: > Now we use `TableFactoryUtil#findAndCreateTableSource` to convert a > CatalogTable to TableSource, but when call `TableFactoryService.find` we > don't pass current classLoader to this method, the default loader will be > BootStrapClassLoader, which can not find our factory. > I verified in my box, it's truly caused by this behavior. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-15577) WindowAggregate RelNodes missing Window specs in digest
[
https://issues.apache.org/jira/browse/FLINK-15577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17014902#comment-17014902
]
Benoit Hanotte commented on FLINK-15577:
[~ykt836] yes, I'll have a look at both
> WindowAggregate RelNodes missing Window specs in digest
> ---
>
> Key: FLINK-15577
> URL: https://issues.apache.org/jira/browse/FLINK-15577
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Legacy Planner
>Affects Versions: 1.9.1
>Reporter: Benoit Hanotte
>Assignee: Benoit Hanotte
>Priority: Critical
>
> The RelNode's digest (AbstractRelNode.getDigest()), along with its RowType,
> is used by the Calcite HepPlanner to avoid adding duplicate Vertices to the
> graph. If an equivalent vertex is already present in the graph, then that
> vertex is used in place of the newly generated one:
> https://github.com/apache/calcite/blob/branch-1.21/core/src/main/java/org/apache/calcite/plan/hep/HepPlanner.java#L828
> This means that *the digest needs to contain all the information necessary to
> identify a vertex and distinguish it from similar - but not equivalent -
> vertices*.
> In the case of `LogicalWindowAggregation` and
> `FlinkLogicalWindowAggregation`, the window specs are currently not in the
> digest, meaning that two aggregations with the same signatures and
> expressions but different windows are considered equivalent by the planner,
> which is not correct and will lead to an invalid Physical Plan.
> For instance, the following query would give an invalid plan:
> {code}
> WITH window_1h AS (
> SELECT HOP_ROWTIME(`timestamp`, INTERVAL '1' HOUR, INTERVAL '1' HOUR) as
> `timestamp`
> FROM my_table
> GROUP BY HOP(`timestamp`, INTERVAL '1' HOUR, INTERVAL '1' HOUR)
> ),
> window_2h AS (
> SELECT HOP_ROWTIME(`timestamp`, INTERVAL '1' HOUR, INTERVAL '2' HOUR) as
> `timestamp`
> FROM my_table
> GROUP BY HOP(`timestamp`, INTERVAL '1' HOUR, INTERVAL '2' HOUR)
> )
> (SELECT * FROM window_1h)
> UNION ALL
> (SELECT * FROM window_2h)
> {code}
> The invalid plan generated by the planner is the following (*Please note the
> windows in the two DataStreamGroupWindowAggregates nodes being the same when
> they should be different*):
> {code}
> DataStreamUnion(all=[true], union all=[timestamp]): rowcount = 200.0,
> cumulative cost = {800.0 rows, 802.0 cpu, 0.0 io}, id = 176
> DataStreamCalc(select=[w$rowtime AS timestamp]): rowcount = 100.0,
> cumulative cost = {300.0 rows, 301.0 cpu, 0.0 io}, id = 173
> DataStreamGroupWindowAggregate(window=[SlidingGroupWindow('w$,
> 'timestamp, 720.millis, 360.millis)], select=[start('w$) AS w$start,
> end('w$) AS w$end, rowtime('w$) AS w$rowtime, proctime('w$) AS w$proctime]):
> rowcount = 100.0, cumulative cost = {200.0 rows, 201.0 cpu, 0.0 io}, id = 172
> DataStreamScan(id=[1], fields=[timestamp]): rowcount = 100.0,
> cumulative cost = {100.0 rows, 101.0 cpu, 0.0 io}, id = 171
> DataStreamCalc(select=[w$rowtime AS timestamp]): rowcount = 100.0,
> cumulative cost = {300.0 rows, 301.0 cpu, 0.0 io}, id = 175
> DataStreamGroupWindowAggregate(window=[SlidingGroupWindow('w$,
> 'timestamp, 720.millis, 360.millis)], select=[start('w$) AS w$start,
> end('w$) AS w$end, rowtime('w$) AS w$rowtime, proctime('w$) AS w$proctime]):
> rowcount = 100.0, cumulative cost = {200.0 rows, 201.0 cpu, 0.0 io}, id = 174
> DataStreamScan(id=[1], fields=[timestamp]): rowcount = 100.0,
> cumulative cost = {100.0 rows, 101.0 cpu, 0.0 io}, id = 171
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-15249) Improve PipelinedRegions calculation with Union Set
[ https://issues.apache.org/jira/browse/FLINK-15249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhu Zhu updated FLINK-15249: Attachment: RegionFailoverPerfTest.java > Improve PipelinedRegions calculation with Union Set > --- > > Key: FLINK-15249 > URL: https://issues.apache.org/jira/browse/FLINK-15249 > Project: Flink > Issue Type: Improvement > Components: Runtime / Coordination >Reporter: Chongchen Chen >Priority: Major > Labels: pull-request-available > Attachments: PipelinedRegionComputeUtil.diff, > RegionFailoverPerfTest.java > > Time Spent: 10m > Remaining Estimate: 0h > > Union Set's Merge Set cost is O(1). current implementation is O(N). the > attachment is patch. > [Disjoint Set Data > Structure|[https://en.wikipedia.org/wiki/Disjoint-set_data_structure]] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-15249) Improve PipelinedRegions calculation with Union Set
[ https://issues.apache.org/jira/browse/FLINK-15249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17014900#comment-17014900 ] Zhu Zhu commented on FLINK-15249: - Hi [~nppoly], I tried your patch with a testing topology. The result turned out to be that region building is even slower with the patch. Here's the result (the value is region building time eplapsed in ms): Pure master(2020-01-14): build regions: 700 build regions: 675 build regions: 477 build regions: 447 build regions: 461 build regions: 430 build regions: 459 build regions: 482 build regions: 733 build regions: 455 Apply #10572 on master(2020-01-14): build regions: 1298 build regions: 1384 build regions: 1029 build regions: 867 build regions: 995 build regions: 1042 build regions: 965 build regions: 1161 build regions: 1050 build regions: 1021 The testing topology is attached. > Improve PipelinedRegions calculation with Union Set > --- > > Key: FLINK-15249 > URL: https://issues.apache.org/jira/browse/FLINK-15249 > Project: Flink > Issue Type: Improvement > Components: Runtime / Coordination >Reporter: Chongchen Chen >Priority: Major > Labels: pull-request-available > Attachments: PipelinedRegionComputeUtil.diff > > Time Spent: 10m > Remaining Estimate: 0h > > Union Set's Merge Set cost is O(1). current implementation is O(N). the > attachment is patch. > [Disjoint Set Data > Structure|[https://en.wikipedia.org/wiki/Disjoint-set_data_structure]] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on issue #10378: [FLINK-14919] [flink-end-to-end-perf-tests] Add performance e2e test suite for basic operations
flinkbot edited a comment on issue #10378: [FLINK-14919] [flink-end-to-end-perf-tests] Add performance e2e test suite for basic operations URL: https://github.com/apache/flink/pull/10378#issuecomment-560291002 ## CI report: * 5a9b8cc32f275d416e8358ee148ad4daff2626ba Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/138907609) * cc2ef5f9dfc57a895046c3f478f22cb508f2a716 Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/138941833) * 5b46ddf6c512ca856e30a4aae56e0be771b94393 Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/138957594) * abbb96dbf131d246b9697590824916a4590f4bf3 Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/139076775) * 496f4834a8345968ba0f988cd45af065b5f73db3 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/139083484) * b5e3b371ac86fa7dc6e24010f060d6ee8395f991 Travis: [CANCELED](https://travis-ci.com/flink-ci/flink/builds/139790553) * 286491ed4bb010c501cea2e5084d348290bf1357 Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/139795221) * b8b10dbdfa1f15379ba7dec6ee7a99d8eb73c3c5 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/139865418) * 0c419a072b4ae3448a5f21aa8622d0354e2f6193 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/141182278) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=3629) * 63a748011eeabfedcc4a52d0264a4780fb63ad3b Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/144267619) Azure: [PENDING](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4317) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] AHeise commented on issue #10845: [FLINK-15355][plugins] Classloader avoids loading unrelated services.
AHeise commented on issue #10845: [FLINK-15355][plugins] Classloader avoids loading unrelated services. URL: https://github.com/apache/flink/pull/10845#issuecomment-574032048 @flinkbot run travis This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] AHeise commented on issue #10845: [FLINK-15355][plugins] Classloader avoids loading unrelated services.
AHeise commented on issue #10845: [FLINK-15355][plugins] Classloader avoids loading unrelated services. URL: https://github.com/apache/flink/pull/10845#issuecomment-574032079 @flinkbot run azure This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-14163) Execution#producedPartitions is possibly not assigned when used
[
https://issues.apache.org/jira/browse/FLINK-14163?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17014886#comment-17014886
]
Yuan Mei commented on FLINK-14163:
--
Thanks [~azagrebin] and [~zhuzh]! I do not have a strong opinion on whether to
add a time-out or not. If I have to choose, I would probably prefer to going
with a simple check because this makes system behavior consistent and easy to
reason about.
I double checked with [~zjwang] before he left (he is on vacation right now),
and he is fine with either case as well.
> Execution#producedPartitions is possibly not assigned when used
> ---
>
> Key: FLINK-14163
> URL: https://issues.apache.org/jira/browse/FLINK-14163
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination
>Affects Versions: 1.9.0, 1.10.0
>Reporter: Zhu Zhu
>Assignee: Yuan Mei
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.10.0
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> Currently {{Execution#producedPartitions}} is assigned after the partitions
> have completed the registration to shuffle master in
> {{Execution#registerProducedPartitions(...)}}.
> The partition registration is an async interface
> ({{ShuffleMaster#registerPartitionWithProducer(...)}}), so
> {{Execution#producedPartitions}} is possible[1] not set when used.
> Usages includes:
> 1. deploying this task, so that the task may be deployed without its result
> partitions assigned, and the job would hang. (DefaultScheduler issue only,
> since legacy scheduler handled this case)
> 2. generating input descriptors for downstream tasks:
> 3. retrieve {{ResultPartitionID}} for partition releasing:
> [1] If a user uses Flink default shuffle master {{NettyShuffleMaster}}, it is
> not problematic at the moment since it returns a completed future on
> registration, so that it would be a synchronized process. However, if users
> implement their own shuffle service in which the
> {{ShuffleMaster#registerPartitionWithProducer}} returns an pending future, it
> can be a problem. This is possible since customizable shuffle service is open
> to users since 1.9 (via config "shuffle-service-factory.class").
> To avoid issues to happen, we may either
> 1. fix all the usages of {{Execution#producedPartitions}} regarding the async
> assigning, or
> 2. change {{ShuffleMaster#registerPartitionWithProducer(...)}} to a sync
> interface
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] KarmaGYZ commented on issue #10668: [hotfix] Align the parameter pattern of retry_times with retry_times_…
KarmaGYZ commented on issue #10668: [hotfix] Align the parameter pattern of retry_times with retry_times_… URL: https://github.com/apache/flink/pull/10668#issuecomment-574025389 Kindly ping @tillrohrmann This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #10378: [FLINK-14919] [flink-end-to-end-perf-tests] Add performance e2e test suite for basic operations
flinkbot edited a comment on issue #10378: [FLINK-14919] [flink-end-to-end-perf-tests] Add performance e2e test suite for basic operations URL: https://github.com/apache/flink/pull/10378#issuecomment-560291002 ## CI report: * 5a9b8cc32f275d416e8358ee148ad4daff2626ba Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/138907609) * cc2ef5f9dfc57a895046c3f478f22cb508f2a716 Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/138941833) * 5b46ddf6c512ca856e30a4aae56e0be771b94393 Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/138957594) * abbb96dbf131d246b9697590824916a4590f4bf3 Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/139076775) * 496f4834a8345968ba0f988cd45af065b5f73db3 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/139083484) * b5e3b371ac86fa7dc6e24010f060d6ee8395f991 Travis: [CANCELED](https://travis-ci.com/flink-ci/flink/builds/139790553) * 286491ed4bb010c501cea2e5084d348290bf1357 Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/139795221) * b8b10dbdfa1f15379ba7dec6ee7a99d8eb73c3c5 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/139865418) * 0c419a072b4ae3448a5f21aa8622d0354e2f6193 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/141182278) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=3629) * 63a748011eeabfedcc4a52d0264a4780fb63ad3b Travis: [PENDING](https://travis-ci.com/flink-ci/flink/builds/144267619) Azure: [PENDING](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4317) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Assigned] (FLINK-15577) WindowAggregate RelNodes missing Window specs in digest
[
https://issues.apache.org/jira/browse/FLINK-15577?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kurt Young reassigned FLINK-15577:
--
Assignee: Benoit Hanotte
> WindowAggregate RelNodes missing Window specs in digest
> ---
>
> Key: FLINK-15577
> URL: https://issues.apache.org/jira/browse/FLINK-15577
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Legacy Planner
>Affects Versions: 1.9.1
>Reporter: Benoit Hanotte
>Assignee: Benoit Hanotte
>Priority: Critical
>
> The RelNode's digest (AbstractRelNode.getDigest()), along with its RowType,
> is used by the Calcite HepPlanner to avoid adding duplicate Vertices to the
> graph. If an equivalent vertex is already present in the graph, then that
> vertex is used in place of the newly generated one:
> https://github.com/apache/calcite/blob/branch-1.21/core/src/main/java/org/apache/calcite/plan/hep/HepPlanner.java#L828
> This means that *the digest needs to contain all the information necessary to
> identify a vertex and distinguish it from similar - but not equivalent -
> vertices*.
> In the case of `LogicalWindowAggregation` and
> `FlinkLogicalWindowAggregation`, the window specs are currently not in the
> digest, meaning that two aggregations with the same signatures and
> expressions but different windows are considered equivalent by the planner,
> which is not correct and will lead to an invalid Physical Plan.
> For instance, the following query would give an invalid plan:
> {code}
> WITH window_1h AS (
> SELECT HOP_ROWTIME(`timestamp`, INTERVAL '1' HOUR, INTERVAL '1' HOUR) as
> `timestamp`
> FROM my_table
> GROUP BY HOP(`timestamp`, INTERVAL '1' HOUR, INTERVAL '1' HOUR)
> ),
> window_2h AS (
> SELECT HOP_ROWTIME(`timestamp`, INTERVAL '1' HOUR, INTERVAL '2' HOUR) as
> `timestamp`
> FROM my_table
> GROUP BY HOP(`timestamp`, INTERVAL '1' HOUR, INTERVAL '2' HOUR)
> )
> (SELECT * FROM window_1h)
> UNION ALL
> (SELECT * FROM window_2h)
> {code}
> The invalid plan generated by the planner is the following (*Please note the
> windows in the two DataStreamGroupWindowAggregates nodes being the same when
> they should be different*):
> {code}
> DataStreamUnion(all=[true], union all=[timestamp]): rowcount = 200.0,
> cumulative cost = {800.0 rows, 802.0 cpu, 0.0 io}, id = 176
> DataStreamCalc(select=[w$rowtime AS timestamp]): rowcount = 100.0,
> cumulative cost = {300.0 rows, 301.0 cpu, 0.0 io}, id = 173
> DataStreamGroupWindowAggregate(window=[SlidingGroupWindow('w$,
> 'timestamp, 720.millis, 360.millis)], select=[start('w$) AS w$start,
> end('w$) AS w$end, rowtime('w$) AS w$rowtime, proctime('w$) AS w$proctime]):
> rowcount = 100.0, cumulative cost = {200.0 rows, 201.0 cpu, 0.0 io}, id = 172
> DataStreamScan(id=[1], fields=[timestamp]): rowcount = 100.0,
> cumulative cost = {100.0 rows, 101.0 cpu, 0.0 io}, id = 171
> DataStreamCalc(select=[w$rowtime AS timestamp]): rowcount = 100.0,
> cumulative cost = {300.0 rows, 301.0 cpu, 0.0 io}, id = 175
> DataStreamGroupWindowAggregate(window=[SlidingGroupWindow('w$,
> 'timestamp, 720.millis, 360.millis)], select=[start('w$) AS w$start,
> end('w$) AS w$end, rowtime('w$) AS w$rowtime, proctime('w$) AS w$proctime]):
> rowcount = 100.0, cumulative cost = {200.0 rows, 201.0 cpu, 0.0 io}, id = 174
> DataStreamScan(id=[1], fields=[timestamp]): rowcount = 100.0,
> cumulative cost = {100.0 rows, 101.0 cpu, 0.0 io}, id = 171
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-15577) WindowAggregate RelNodes missing Window specs in digest
[
https://issues.apache.org/jira/browse/FLINK-15577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17014876#comment-17014876
]
Kurt Young commented on FLINK-15577:
[~b.hanotte] Thanks for the fix, looks like blink planner also has this issue,
could you fix both?
> WindowAggregate RelNodes missing Window specs in digest
> ---
>
> Key: FLINK-15577
> URL: https://issues.apache.org/jira/browse/FLINK-15577
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Legacy Planner
>Affects Versions: 1.9.1
>Reporter: Benoit Hanotte
>Priority: Critical
>
> The RelNode's digest (AbstractRelNode.getDigest()), along with its RowType,
> is used by the Calcite HepPlanner to avoid adding duplicate Vertices to the
> graph. If an equivalent vertex is already present in the graph, then that
> vertex is used in place of the newly generated one:
> https://github.com/apache/calcite/blob/branch-1.21/core/src/main/java/org/apache/calcite/plan/hep/HepPlanner.java#L828
> This means that *the digest needs to contain all the information necessary to
> identify a vertex and distinguish it from similar - but not equivalent -
> vertices*.
> In the case of `LogicalWindowAggregation` and
> `FlinkLogicalWindowAggregation`, the window specs are currently not in the
> digest, meaning that two aggregations with the same signatures and
> expressions but different windows are considered equivalent by the planner,
> which is not correct and will lead to an invalid Physical Plan.
> For instance, the following query would give an invalid plan:
> {code}
> WITH window_1h AS (
> SELECT HOP_ROWTIME(`timestamp`, INTERVAL '1' HOUR, INTERVAL '1' HOUR) as
> `timestamp`
> FROM my_table
> GROUP BY HOP(`timestamp`, INTERVAL '1' HOUR, INTERVAL '1' HOUR)
> ),
> window_2h AS (
> SELECT HOP_ROWTIME(`timestamp`, INTERVAL '1' HOUR, INTERVAL '2' HOUR) as
> `timestamp`
> FROM my_table
> GROUP BY HOP(`timestamp`, INTERVAL '1' HOUR, INTERVAL '2' HOUR)
> )
> (SELECT * FROM window_1h)
> UNION ALL
> (SELECT * FROM window_2h)
> {code}
> The invalid plan generated by the planner is the following (*Please note the
> windows in the two DataStreamGroupWindowAggregates nodes being the same when
> they should be different*):
> {code}
> DataStreamUnion(all=[true], union all=[timestamp]): rowcount = 200.0,
> cumulative cost = {800.0 rows, 802.0 cpu, 0.0 io}, id = 176
> DataStreamCalc(select=[w$rowtime AS timestamp]): rowcount = 100.0,
> cumulative cost = {300.0 rows, 301.0 cpu, 0.0 io}, id = 173
> DataStreamGroupWindowAggregate(window=[SlidingGroupWindow('w$,
> 'timestamp, 720.millis, 360.millis)], select=[start('w$) AS w$start,
> end('w$) AS w$end, rowtime('w$) AS w$rowtime, proctime('w$) AS w$proctime]):
> rowcount = 100.0, cumulative cost = {200.0 rows, 201.0 cpu, 0.0 io}, id = 172
> DataStreamScan(id=[1], fields=[timestamp]): rowcount = 100.0,
> cumulative cost = {100.0 rows, 101.0 cpu, 0.0 io}, id = 171
> DataStreamCalc(select=[w$rowtime AS timestamp]): rowcount = 100.0,
> cumulative cost = {300.0 rows, 301.0 cpu, 0.0 io}, id = 175
> DataStreamGroupWindowAggregate(window=[SlidingGroupWindow('w$,
> 'timestamp, 720.millis, 360.millis)], select=[start('w$) AS w$start,
> end('w$) AS w$end, rowtime('w$) AS w$rowtime, proctime('w$) AS w$proctime]):
> rowcount = 100.0, cumulative cost = {200.0 rows, 201.0 cpu, 0.0 io}, id = 174
> DataStreamScan(id=[1], fields=[timestamp]): rowcount = 100.0,
> cumulative cost = {100.0 rows, 101.0 cpu, 0.0 io}, id = 171
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-15579) Can not use jdbc connector on Blink batch mode
[
https://issues.apache.org/jira/browse/FLINK-15579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17014875#comment-17014875
]
Kurt Young commented on FLINK-15579:
So you want to use JDBC as a batch sink?
> Can not use jdbc connector on Blink batch mode
> ---
>
> Key: FLINK-15579
> URL: https://issues.apache.org/jira/browse/FLINK-15579
> Project: Flink
> Issue Type: Improvement
> Components: Table SQL / Planner
>Affects Versions: 1.9.0, 1.9.1
>Reporter: Shu Li Zheng
>Priority: Major
>
> Because JDBCTableSourceSinkFactory.createStreamTableSink() create
> JDBCUpsertTableSink. But BatchExecSink can not work with
> UpsertStreamTableSink.
> {code:scala}
> override protected def translateToPlanInternal(
> planner: BatchPlanner): Transformation[Any] = {
> val resultTransformation = sink match {
> case _: RetractStreamTableSink[T] | _: UpsertStreamTableSink[T] =>
> throw new TableException("RetractStreamTableSink and
> UpsertStreamTableSink is not" +
> " supported in Batch environment.")
> {code}
> DDL like:
> CREATE TABLE USER_RESULT(
> NAME VARCHAR,
> CITY VARCHAR,
> SCORE BIGINT
> ) WITH (
> 'connector.type' = 'jdbc',
> 'connector.url' = '',
> 'connector.table' = '',
> 'connector.driver' = 'com.mysql.cj.jdbc.Driver',
> 'connector.username' = 'root',
> 'connector.password' = '',
> 'connector.write.flush.interval' = '1s')
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-15579) Can not use jdbc connector on Blink batch mode
[
https://issues.apache.org/jira/browse/FLINK-15579?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kurt Young updated FLINK-15579:
---
Issue Type: Improvement (was: Bug)
> Can not use jdbc connector on Blink batch mode
> ---
>
> Key: FLINK-15579
> URL: https://issues.apache.org/jira/browse/FLINK-15579
> Project: Flink
> Issue Type: Improvement
> Components: Table SQL / Planner
>Affects Versions: 1.9.0, 1.9.1
>Reporter: Shu Li Zheng
>Priority: Major
>
> Because JDBCTableSourceSinkFactory.createStreamTableSink() create
> JDBCUpsertTableSink. But BatchExecSink can not work with
> UpsertStreamTableSink.
> {code:scala}
> override protected def translateToPlanInternal(
> planner: BatchPlanner): Transformation[Any] = {
> val resultTransformation = sink match {
> case _: RetractStreamTableSink[T] | _: UpsertStreamTableSink[T] =>
> throw new TableException("RetractStreamTableSink and
> UpsertStreamTableSink is not" +
> " supported in Batch environment.")
> {code}
> DDL like:
> CREATE TABLE USER_RESULT(
> NAME VARCHAR,
> CITY VARCHAR,
> SCORE BIGINT
> ) WITH (
> 'connector.type' = 'jdbc',
> 'connector.url' = '',
> 'connector.table' = '',
> 'connector.driver' = 'com.mysql.cj.jdbc.Driver',
> 'connector.username' = 'root',
> 'connector.password' = '',
> 'connector.write.flush.interval' = '1s')
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-15576) remove isTemporary property from CatalogFunction API
[ https://issues.apache.org/jira/browse/FLINK-15576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17014874#comment-17014874 ] Jingsong Lee commented on FLINK-15576: -- Hi [~phoenixjiangnan], looks like it is just an internal code refactor, no public api change, so it could not be a blocker? > remove isTemporary property from CatalogFunction API > > > Key: FLINK-15576 > URL: https://issues.apache.org/jira/browse/FLINK-15576 > Project: Flink > Issue Type: Bug > Components: Table SQL / API >Affects Versions: 1.10.0 >Reporter: Bowen Li >Assignee: Bowen Li >Priority: Blocker > Labels: pull-request-available > Fix For: 1.10.0 > > Time Spent: 10m > Remaining Estimate: 0h > > according to FLIP-79, CatalogFunction shouldn't have "isTemporary" property. > Moving that from CatalogFunction to Create/AlterCatalogFunctionOperation -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-15579) Can not use jdbc connector on Blink batch mode
[
https://issues.apache.org/jira/browse/FLINK-15579?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shu Li Zheng updated FLINK-15579:
-
Affects Version/s: 1.9.0
1.9.1
> Can not use jdbc connector on Blink batch mode
> ---
>
> Key: FLINK-15579
> URL: https://issues.apache.org/jira/browse/FLINK-15579
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Planner
>Affects Versions: 1.9.0, 1.9.1
>Reporter: Shu Li Zheng
>Priority: Major
>
> Because JDBCTableSourceSinkFactory.createStreamTableSink() create
> JDBCUpsertTableSink. But BatchExecSink can not work with
> UpsertStreamTableSink.
> {code:scala}
> override protected def translateToPlanInternal(
> planner: BatchPlanner): Transformation[Any] = {
> val resultTransformation = sink match {
> case _: RetractStreamTableSink[T] | _: UpsertStreamTableSink[T] =>
> throw new TableException("RetractStreamTableSink and
> UpsertStreamTableSink is not" +
> " supported in Batch environment.")
> {code}
> DDL like:
> CREATE TABLE USER_RESULT(
> NAME VARCHAR,
> CITY VARCHAR,
> SCORE BIGINT
> ) WITH (
> 'connector.type' = 'jdbc',
> 'connector.url' = '',
> 'connector.table' = '',
> 'connector.driver' = 'com.mysql.cj.jdbc.Driver',
> 'connector.username' = 'root',
> 'connector.password' = '',
> 'connector.write.flush.interval' = '1s')
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on issue #10378: [FLINK-14919] [flink-end-to-end-perf-tests] Add performance e2e test suite for basic operations
flinkbot edited a comment on issue #10378: [FLINK-14919] [flink-end-to-end-perf-tests] Add performance e2e test suite for basic operations URL: https://github.com/apache/flink/pull/10378#issuecomment-560291002 ## CI report: * 5a9b8cc32f275d416e8358ee148ad4daff2626ba Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/138907609) * cc2ef5f9dfc57a895046c3f478f22cb508f2a716 Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/138941833) * 5b46ddf6c512ca856e30a4aae56e0be771b94393 Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/138957594) * abbb96dbf131d246b9697590824916a4590f4bf3 Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/139076775) * 496f4834a8345968ba0f988cd45af065b5f73db3 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/139083484) * b5e3b371ac86fa7dc6e24010f060d6ee8395f991 Travis: [CANCELED](https://travis-ci.com/flink-ci/flink/builds/139790553) * 286491ed4bb010c501cea2e5084d348290bf1357 Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/139795221) * b8b10dbdfa1f15379ba7dec6ee7a99d8eb73c3c5 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/139865418) * 0c419a072b4ae3448a5f21aa8622d0354e2f6193 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/141182278) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=3629) * 63a748011eeabfedcc4a52d0264a4780fb63ad3b UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] Li-Aihua commented on issue #10378: [FLINK-14919] [flink-end-to-end-perf-tests] Add performance e2e test suite for basic operations
Li-Aihua commented on issue #10378: [FLINK-14919] [flink-end-to-end-perf-tests] Add performance e2e test suite for basic operations URL: https://github.com/apache/flink/pull/10378#issuecomment-574011447 > I'm not convinced that the `PerformanceTestJobBase` is the right approach. Could you expand a bit on the design and why you selected this one? > > I'm wondering what the point of all these setters are; do we expect some subclasses to override them? > Why are some methods returning values despite them not being used? > Why are _all_ methods public despite not being used? There were some problems with the previous implementation, I updated it again, you can review it again.thanks。 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Created] (FLINK-15580) Add baseurl to docs/build_docs.sh
Benchao Li created FLINK-15580:
--
Summary: Add baseurl to docs/build_docs.sh
Key: FLINK-15580
URL: https://issues.apache.org/jira/browse/FLINK-15580
Project: Flink
Issue Type: Improvement
Components: Documentation
Affects Versions: 1.9.1, 1.10.0
Reporter: Benchao Li
As discussed in https://issues.apache.org/jira/browse/FLINK-15559
Currently, we did not set {{baseurl}}, which made broken links due to missing
{{baseurl}} not easy to detect.
So I propose we add a {{baseurl}} to {{build_docs.sh}}'s -i & -p mode, for
example {{/projects/flink/local-baseurl}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on issue #10736: [FLINK-15010][Network] Add shutdown hook to ensure cleanup netty shuffle directories
flinkbot edited a comment on issue #10736: [FLINK-15010][Network] Add shutdown hook to ensure cleanup netty shuffle directories URL: https://github.com/apache/flink/pull/10736#issuecomment-570066854 ## CI report: * 4b605068e32d3eb15a51f838ed29918d1224959a Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/142804653) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4021) * 9430066683a67318f9685de8a58904972c5dbaca Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/142829633) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4026) * 2185007c824d21817356c9dfb9c9e09846e27f7e UNKNOWN * d7ab35b18c5964b837be1d52611623d7c271dc99 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/144257501) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4316) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #10848: [FLINK-15307][failover]Rename Subclasses of FailoverStrategy
flinkbot edited a comment on issue #10848: [FLINK-15307][failover]Rename Subclasses of FailoverStrategy URL: https://github.com/apache/flink/pull/10848#issuecomment-573958003 ## CI report: * a3f83df73af74cf8509f084e860ab88e63129ac8 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/144251297) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4310) * c2f939165d6d4aa4907591f57b1a14c9578c2122 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/144257490) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4315) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Comment Edited] (FLINK-15579) Can not use jdbc connector on Blink batch mode
[
https://issues.apache.org/jira/browse/FLINK-15579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17014848#comment-17014848
]
Shu Li Zheng edited comment on FLINK-15579 at 1/14/20 4:42 AM:
---
[~ykt836] Could you assign this bug to me? I would like to fix it.
was (Author: eric.zheng):
[~ykt836]Could you assign this bug to me? I would like to fix it.
> Can not use jdbc connector on Blink batch mode
> ---
>
> Key: FLINK-15579
> URL: https://issues.apache.org/jira/browse/FLINK-15579
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Planner
>Reporter: Shu Li Zheng
>Priority: Major
>
> Because JDBCTableSourceSinkFactory.createStreamTableSink() create
> JDBCUpsertTableSink. But BatchExecSink can not work with
> UpsertStreamTableSink.
> {code:scala}
> override protected def translateToPlanInternal(
> planner: BatchPlanner): Transformation[Any] = {
> val resultTransformation = sink match {
> case _: RetractStreamTableSink[T] | _: UpsertStreamTableSink[T] =>
> throw new TableException("RetractStreamTableSink and
> UpsertStreamTableSink is not" +
> " supported in Batch environment.")
> {code}
> DDL like:
> CREATE TABLE USER_RESULT(
> NAME VARCHAR,
> CITY VARCHAR,
> SCORE BIGINT
> ) WITH (
> 'connector.type' = 'jdbc',
> 'connector.url' = '',
> 'connector.table' = '',
> 'connector.driver' = 'com.mysql.cj.jdbc.Driver',
> 'connector.username' = 'root',
> 'connector.password' = '',
> 'connector.write.flush.interval' = '1s')
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-15577) WindowAggregate RelNodes missing Window specs in digest
[
https://issues.apache.org/jira/browse/FLINK-15577?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Benoit Hanotte updated FLINK-15577:
---
Description:
The RelNode's digest (AbstractRelNode.getDigest()), along with its RowType, is
used by the Calcite HepPlanner to avoid adding duplicate Vertices to the graph.
If an equivalent vertex is already present in the graph, then that vertex is
used in place of the newly generated one:
https://github.com/apache/calcite/blob/branch-1.21/core/src/main/java/org/apache/calcite/plan/hep/HepPlanner.java#L828
This means that *the digest needs to contain all the information necessary to
identify a vertex and distinguish it from similar - but not equivalent -
vertices*.
In the case of `LogicalWindowAggregation` and `FlinkLogicalWindowAggregation`,
the window specs are currently not in the digest, meaning that two aggregations
with the same signatures and expressions but different windows are considered
equivalent by the planner, which is not correct and will lead to an invalid
Physical Plan.
For instance, the following query would give an invalid plan:
{code}
WITH window_1h AS (
SELECT HOP_ROWTIME(`timestamp`, INTERVAL '1' HOUR, INTERVAL '1' HOUR) as
`timestamp`
FROM my_table
GROUP BY HOP(`timestamp`, INTERVAL '1' HOUR, INTERVAL '1' HOUR)
),
window_2h AS (
SELECT HOP_ROWTIME(`timestamp`, INTERVAL '1' HOUR, INTERVAL '2' HOUR) as
`timestamp`
FROM my_table
GROUP BY HOP(`timestamp`, INTERVAL '1' HOUR, INTERVAL '2' HOUR)
)
(SELECT * FROM window_1h)
UNION ALL
(SELECT * FROM window_2h)
{code}
The invalid plan generated by the planner is the following (*Please note the
windows in the two DataStreamGroupWindowAggregates nodes being the same when
they should be different*):
{code}
DataStreamUnion(all=[true], union all=[timestamp]): rowcount = 200.0,
cumulative cost = {800.0 rows, 802.0 cpu, 0.0 io}, id = 176
DataStreamCalc(select=[w$rowtime AS timestamp]): rowcount = 100.0, cumulative
cost = {300.0 rows, 301.0 cpu, 0.0 io}, id = 173
DataStreamGroupWindowAggregate(window=[SlidingGroupWindow('w$, 'timestamp,
720.millis, 360.millis)], select=[start('w$) AS w$start, end('w$) AS
w$end, rowtime('w$) AS w$rowtime, proctime('w$) AS w$proctime]): rowcount =
100.0, cumulative cost = {200.0 rows, 201.0 cpu, 0.0 io}, id = 172
DataStreamScan(id=[1], fields=[timestamp]): rowcount = 100.0, cumulative
cost = {100.0 rows, 101.0 cpu, 0.0 io}, id = 171
DataStreamCalc(select=[w$rowtime AS timestamp]): rowcount = 100.0, cumulative
cost = {300.0 rows, 301.0 cpu, 0.0 io}, id = 175
DataStreamGroupWindowAggregate(window=[SlidingGroupWindow('w$, 'timestamp,
720.millis, 360.millis)], select=[start('w$) AS w$start, end('w$) AS
w$end, rowtime('w$) AS w$rowtime, proctime('w$) AS w$proctime]): rowcount =
100.0, cumulative cost = {200.0 rows, 201.0 cpu, 0.0 io}, id = 174
DataStreamScan(id=[1], fields=[timestamp]): rowcount = 100.0, cumulative
cost = {100.0 rows, 101.0 cpu, 0.0 io}, id = 171
{code}
was:
The RelNode's digest (AbstractRelNode.getDigest()), along its RowType, is used
by the Calcite HepPlanner to avoid adding duplicate Vertices to the graph. If
an equivalent vertex was already present in the graph, then that vertex is used
in place of the new generated one:
https://github.com/apache/calcite/blob/branch-1.21/core/src/main/java/org/apache/calcite/plan/hep/HepPlanner.java#L828
This means that *the digest needs to contain all the information necessary to
identify a vertex and distinguish it from similar - but not equivalent -
vertices*.
In the case of `LogicalWindowAggregation` and `FlinkLogicalWindowAggregation`,
the window specs are currently not in the digest, meaning that two aggregations
with the same signatures and expressions but different windows are considered
equivalent by the planner, which is not correct and will lead to an invalid
Physical Plan.
For instance, the following query would give an invalid plan:
{code}
WITH window_1h AS (
SELECT HOP_ROWTIME(`timestamp`, INTERVAL '1' HOUR, INTERVAL '1' HOUR) as
`timestamp`
FROM my_table
GROUP BY HOP(`timestamp`, INTERVAL '1' HOUR, INTERVAL '1' HOUR)
),
window_2h AS (
SELECT HOP_ROWTIME(`timestamp`, INTERVAL '1' HOUR, INTERVAL '2' HOUR) as
`timestamp`
FROM my_table
GROUP BY HOP(`timestamp`, INTERVAL '1' HOUR, INTERVAL '2' HOUR)
)
(SELECT * FROM window_1h)
UNION ALL
(SELECT * FROM window_2h)
{code}
The invalid plan generated by the planner is the following (*Please note the
windows in the two DataStreamGroupWindowAggregates nodes being the same when
they should be different*):
{code}
DataStreamUnion(all=[true], union all=[timestamp]): rowcount = 200.0,
cumulative cost = {800.0 rows, 802.0 cpu, 0.0 io}, id = 176
DataStreamCalc(select=[w$rowtime AS timestamp]): rowcount = 100.0, cumulative
cost = {300.0 rows, 301.0 cpu, 0.0 io}, id = 173
Dat
[GitHub] [flink] flinkbot edited a comment on issue #10736: [FLINK-15010][Network] Add shutdown hook to ensure cleanup netty shuffle directories
flinkbot edited a comment on issue #10736: [FLINK-15010][Network] Add shutdown hook to ensure cleanup netty shuffle directories URL: https://github.com/apache/flink/pull/10736#issuecomment-570066854 ## CI report: * 4b605068e32d3eb15a51f838ed29918d1224959a Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/142804653) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4021) * 9430066683a67318f9685de8a58904972c5dbaca Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/142829633) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4026) * 2185007c824d21817356c9dfb9c9e09846e27f7e UNKNOWN * d7ab35b18c5964b837be1d52611623d7c271dc99 Travis: [PENDING](https://travis-ci.com/flink-ci/flink/builds/144257501) Azure: [PENDING](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4316) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-15577) WindowAggregate RelNodes missing Window specs in digest
[
https://issues.apache.org/jira/browse/FLINK-15577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17014856#comment-17014856
]
Benoit Hanotte commented on FLINK-15577:
Hi [~ykt836], yes, I'll push a PR today
> WindowAggregate RelNodes missing Window specs in digest
> ---
>
> Key: FLINK-15577
> URL: https://issues.apache.org/jira/browse/FLINK-15577
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Legacy Planner
>Affects Versions: 1.9.1
>Reporter: Benoit Hanotte
>Priority: Critical
>
> The RelNode's digest (AbstractRelNode.getDigest()), along its RowType, is
> used by the Calcite HepPlanner to avoid adding duplicate Vertices to the
> graph. If an equivalent vertex was already present in the graph, then that
> vertex is used in place of the new generated one:
> https://github.com/apache/calcite/blob/branch-1.21/core/src/main/java/org/apache/calcite/plan/hep/HepPlanner.java#L828
> This means that *the digest needs to contain all the information necessary to
> identify a vertex and distinguish it from similar - but not equivalent -
> vertices*.
> In the case of `LogicalWindowAggregation` and
> `FlinkLogicalWindowAggregation`, the window specs are currently not in the
> digest, meaning that two aggregations with the same signatures and
> expressions but different windows are considered equivalent by the planner,
> which is not correct and will lead to an invalid Physical Plan.
> For instance, the following query would give an invalid plan:
> {code}
> WITH window_1h AS (
> SELECT HOP_ROWTIME(`timestamp`, INTERVAL '1' HOUR, INTERVAL '1' HOUR) as
> `timestamp`
> FROM my_table
> GROUP BY HOP(`timestamp`, INTERVAL '1' HOUR, INTERVAL '1' HOUR)
> ),
> window_2h AS (
> SELECT HOP_ROWTIME(`timestamp`, INTERVAL '1' HOUR, INTERVAL '2' HOUR) as
> `timestamp`
> FROM my_table
> GROUP BY HOP(`timestamp`, INTERVAL '1' HOUR, INTERVAL '2' HOUR)
> )
> (SELECT * FROM window_1h)
> UNION ALL
> (SELECT * FROM window_2h)
> {code}
> The invalid plan generated by the planner is the following (*Please note the
> windows in the two DataStreamGroupWindowAggregates nodes being the same when
> they should be different*):
> {code}
> DataStreamUnion(all=[true], union all=[timestamp]): rowcount = 200.0,
> cumulative cost = {800.0 rows, 802.0 cpu, 0.0 io}, id = 176
> DataStreamCalc(select=[w$rowtime AS timestamp]): rowcount = 100.0,
> cumulative cost = {300.0 rows, 301.0 cpu, 0.0 io}, id = 173
> DataStreamGroupWindowAggregate(window=[SlidingGroupWindow('w$,
> 'timestamp, 720.millis, 360.millis)], select=[start('w$) AS w$start,
> end('w$) AS w$end, rowtime('w$) AS w$rowtime, proctime('w$) AS w$proctime]):
> rowcount = 100.0, cumulative cost = {200.0 rows, 201.0 cpu, 0.0 io}, id = 172
> DataStreamScan(id=[1], fields=[timestamp]): rowcount = 100.0,
> cumulative cost = {100.0 rows, 101.0 cpu, 0.0 io}, id = 171
> DataStreamCalc(select=[w$rowtime AS timestamp]): rowcount = 100.0,
> cumulative cost = {300.0 rows, 301.0 cpu, 0.0 io}, id = 175
> DataStreamGroupWindowAggregate(window=[SlidingGroupWindow('w$,
> 'timestamp, 720.millis, 360.millis)], select=[start('w$) AS w$start,
> end('w$) AS w$end, rowtime('w$) AS w$rowtime, proctime('w$) AS w$proctime]):
> rowcount = 100.0, cumulative cost = {200.0 rows, 201.0 cpu, 0.0 io}, id = 174
> DataStreamScan(id=[1], fields=[timestamp]): rowcount = 100.0,
> cumulative cost = {100.0 rows, 101.0 cpu, 0.0 io}, id = 171
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-15579) Can not use jdbc connector on Blink batch mode
[
https://issues.apache.org/jira/browse/FLINK-15579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17014848#comment-17014848
]
Shu Li Zheng commented on FLINK-15579:
--
[~ykt836]Could you assign this bug to me? I would like to fix it.
> Can not use jdbc connector on Blink batch mode
> ---
>
> Key: FLINK-15579
> URL: https://issues.apache.org/jira/browse/FLINK-15579
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Planner
>Reporter: Shu Li Zheng
>Priority: Major
>
> Because JDBCTableSourceSinkFactory.createStreamTableSink() create
> JDBCUpsertTableSink. But BatchExecSink can not work with
> UpsertStreamTableSink.
> {code:scala}
> override protected def translateToPlanInternal(
> planner: BatchPlanner): Transformation[Any] = {
> val resultTransformation = sink match {
> case _: RetractStreamTableSink[T] | _: UpsertStreamTableSink[T] =>
> throw new TableException("RetractStreamTableSink and
> UpsertStreamTableSink is not" +
> " supported in Batch environment.")
> {code}
> DDL like:
> CREATE TABLE USER_RESULT(
> NAME VARCHAR,
> CITY VARCHAR,
> SCORE BIGINT
> ) WITH (
> 'connector.type' = 'jdbc',
> 'connector.url' = '',
> 'connector.table' = '',
> 'connector.driver' = 'com.mysql.cj.jdbc.Driver',
> 'connector.username' = 'root',
> 'connector.password' = '',
> 'connector.write.flush.interval' = '1s')
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on issue #10815: [FLINK-15537][table-planner-blink] Type of keys should be `BinaryRow`…
flinkbot edited a comment on issue #10815: [FLINK-15537][table-planner-blink] Type of keys should be `BinaryRow`… URL: https://github.com/apache/flink/pull/10815#issuecomment-572566376 ## CI report: * 19a4290f709495491fe460037c8c31d106984ea8 Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/143732723) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4229) * c3ef5ea345a343170806de8112163edb7df31f69 Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/144110200) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4284) * 941a5d4725dee3317ca05f8ab16eb103f61d3fcb Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/144255612) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4312) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #10814: [FLINK-15434][Tests]Fix unstable tests in JobMasterTest
flinkbot edited a comment on issue #10814: [FLINK-15434][Tests]Fix unstable tests in JobMasterTest URL: https://github.com/apache/flink/pull/10814#issuecomment-572552628 ## CI report: * a814428b99636921abf55ab089e664cec9d13876 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/143727769) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4228) * 96a81a9b98ba96eb075c9b5f32fd15a6e08082ab Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/144255567) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4311) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (FLINK-15579) Can not use jdbc connector on Blink batch mode
[
https://issues.apache.org/jira/browse/FLINK-15579?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shu Li Zheng updated FLINK-15579:
-
Description:
Because JDBCTableSourceSinkFactory.createStreamTableSink() create
JDBCUpsertTableSink. But BatchExecSink can not work with UpsertStreamTableSink.
{code:scala}
override protected def translateToPlanInternal(
planner: BatchPlanner): Transformation[Any] = {
val resultTransformation = sink match {
case _: RetractStreamTableSink[T] | _: UpsertStreamTableSink[T] =>
throw new TableException("RetractStreamTableSink and
UpsertStreamTableSink is not" +
" supported in Batch environment.")
{code}
DDL like:
CREATE TABLE USER_RESULT(
NAME VARCHAR,
CITY VARCHAR,
SCORE BIGINT
) WITH (
'connector.type' = 'jdbc',
'connector.url' = '',
'connector.table' = '',
'connector.driver' = 'com.mysql.cj.jdbc.Driver',
'connector.username' = 'root',
'connector.password' = '',
'connector.write.flush.interval' = '1s')
was:
Because JDBCTableSourceSinkFactory.createStreamTableSink() create
JDBCUpsertTableSink. But BatchExecSink can not work with UpsertStreamTableSink.
{code:scala}
override protected def translateToPlanInternal(
planner: BatchPlanner): Transformation[Any] = {
val resultTransformation = sink match {
case _: RetractStreamTableSink[T] | _: UpsertStreamTableSink[T] =>
throw new TableException("RetractStreamTableSink and
UpsertStreamTableSink is not" +
" supported in Batch environment.")
{code}
> Can not use jdbc connector on Blink batch mode
> ---
>
> Key: FLINK-15579
> URL: https://issues.apache.org/jira/browse/FLINK-15579
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Planner
>Reporter: Shu Li Zheng
>Priority: Major
>
> Because JDBCTableSourceSinkFactory.createStreamTableSink() create
> JDBCUpsertTableSink. But BatchExecSink can not work with
> UpsertStreamTableSink.
> {code:scala}
> override protected def translateToPlanInternal(
> planner: BatchPlanner): Transformation[Any] = {
> val resultTransformation = sink match {
> case _: RetractStreamTableSink[T] | _: UpsertStreamTableSink[T] =>
> throw new TableException("RetractStreamTableSink and
> UpsertStreamTableSink is not" +
> " supported in Batch environment.")
> {code}
> DDL like:
> CREATE TABLE USER_RESULT(
> NAME VARCHAR,
> CITY VARCHAR,
> SCORE BIGINT
> ) WITH (
> 'connector.type' = 'jdbc',
> 'connector.url' = '',
> 'connector.table' = '',
> 'connector.driver' = 'com.mysql.cj.jdbc.Driver',
> 'connector.username' = 'root',
> 'connector.password' = '',
> 'connector.write.flush.interval' = '1s')
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-15579) Can not use jdbc connector on Blink batch mode
[
https://issues.apache.org/jira/browse/FLINK-15579?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kurt Young updated FLINK-15579:
---
Component/s: Table SQL / Planner
> Can not use jdbc connector on Blink batch mode
> ---
>
> Key: FLINK-15579
> URL: https://issues.apache.org/jira/browse/FLINK-15579
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Planner
>Reporter: Shu Li Zheng
>Priority: Major
>
> Because JDBCTableSourceSinkFactory.createStreamTableSink() create
> JDBCUpsertTableSink. But BatchExecSink can not work with
> UpsertStreamTableSink.
> {code:scala}
> override protected def translateToPlanInternal(
> planner: BatchPlanner): Transformation[Any] = {
> val resultTransformation = sink match {
> case _: RetractStreamTableSink[T] | _: UpsertStreamTableSink[T] =>
> throw new TableException("RetractStreamTableSink and
> UpsertStreamTableSink is not" +
> " supported in Batch environment.")
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-15579) Can not use jdbc connector on Blink batch mode
Shu Li Zheng created FLINK-15579:
Summary: Can not use jdbc connector on Blink batch mode
Key: FLINK-15579
URL: https://issues.apache.org/jira/browse/FLINK-15579
Project: Flink
Issue Type: Bug
Reporter: Shu Li Zheng
Because JDBCTableSourceSinkFactory.createStreamTableSink() create
JDBCUpsertTableSink. But BatchExecSink can not work with UpsertStreamTableSink.
{code:scala}
override protected def translateToPlanInternal(
planner: BatchPlanner): Transformation[Any] = {
val resultTransformation = sink match {
case _: RetractStreamTableSink[T] | _: UpsertStreamTableSink[T] =>
throw new TableException("RetractStreamTableSink and
UpsertStreamTableSink is not" +
" supported in Batch environment.")
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on issue #10736: [FLINK-15010][Network] Add shutdown hook to ensure cleanup netty shuffle directories
flinkbot edited a comment on issue #10736: [FLINK-15010][Network] Add shutdown hook to ensure cleanup netty shuffle directories URL: https://github.com/apache/flink/pull/10736#issuecomment-570066854 ## CI report: * 4b605068e32d3eb15a51f838ed29918d1224959a Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/142804653) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4021) * 9430066683a67318f9685de8a58904972c5dbaca Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/142829633) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4026) * 2185007c824d21817356c9dfb9c9e09846e27f7e UNKNOWN * d7ab35b18c5964b837be1d52611623d7c271dc99 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] zhuzhurk commented on a change in pull request #10848: [FLINK-15307][failover]Rename Subclasses of FailoverStrategy
zhuzhurk commented on a change in pull request #10848:
[FLINK-15307][failover]Rename Subclasses of FailoverStrategy
URL: https://github.com/apache/flink/pull/10848#discussion_r366143207
##
File path:
flink-runtime/src/test/java/org/apache/flink/runtime/executiongraph/AdaptedRestartPipelinedRegionFailoverStrategyNGConcurrentFailoverTest.java
##
@@ -51,7 +51,7 @@
* Tests for {@link AdaptedRestartPipelinedRegionStrategyNG} failover handling
when concurrent failovers happen.
* There can be local+local and local+global concurrent failovers.
*/
-public class AdaptedRestartPipelinedRegionStrategyNGConcurrentFailoverTest
extends TestLogger {
+public class
AdaptedRestartPipelinedRegionFailoverStrategyNGConcurrentFailoverTest extends
TestLogger {
Review comment:
It's related to a legacy failover strategy
`AdaptedRestartPipelinedRegionStrategyNG` so do not rename it in this PR.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] zhuzhurk commented on a change in pull request #10848: [FLINK-15307][failover]Rename Subclasses of FailoverStrategy
zhuzhurk commented on a change in pull request #10848:
[FLINK-15307][failover]Rename Subclasses of FailoverStrategy
URL: https://github.com/apache/flink/pull/10848#discussion_r366143882
##
File path:
flink-tests/src/test/java/org/apache/flink/test/checkpointing/ContinuousFileProcessingCheckpointITCase.java
##
@@ -77,9 +77,9 @@
@Before
public void createHDFS() throws IOException {
if
(failoverStrategy.equals(FailoverStrategy.RestartPipelinedRegionStrategy)) {
- // TODO the 'NO_OF_RETRIES' is useless for current
RestartPipelinedRegionStrategy,
- // for this ContinuousFileProcessingCheckpointITCase,
using RestartPipelinedRegionStrategy would result in endless running.
- throw new AssumptionViolatedException("ignored
ContinuousFileProcessingCheckpointITCase when using
RestartPipelinedRegionStrategy");
+ // TODO the 'NO_OF_RETRIES' is useless for current
RestartPipelinedRegionFailoverStrategy,
Review comment:
This is actually not `flip. RestartPipelinedRegionStrategy ` but
`StreamFaultToleranceTestBase#FailoverStrategy`.
I think we need to rename the enums in it as well.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] zhuzhurk commented on a change in pull request #10848: [FLINK-15307][failover]Rename Subclasses of FailoverStrategy
zhuzhurk commented on a change in pull request #10848:
[FLINK-15307][failover]Rename Subclasses of FailoverStrategy
URL: https://github.com/apache/flink/pull/10848#discussion_r366143327
##
File path:
flink-runtime/src/test/java/org/apache/flink/runtime/executiongraph/AdaptedRestartPipelinedRegionFailoverStrategyNGFailoverTest.java
##
@@ -63,7 +63,7 @@
/**
* Tests for {@link AdaptedRestartPipelinedRegionStrategyNG} failover handling.
*/
-public class AdaptedRestartPipelinedRegionStrategyNGFailoverTest extends
TestLogger {
+public class AdaptedRestartPipelinedRegionFailoverStrategyNGFailoverTest
extends TestLogger {
Review comment:
This change is not needed. See my other 2 comments.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] zhuzhurk commented on a change in pull request #10848: [FLINK-15307][failover]Rename Subclasses of FailoverStrategy
zhuzhurk commented on a change in pull request #10848:
[FLINK-15307][failover]Rename Subclasses of FailoverStrategy
URL: https://github.com/apache/flink/pull/10848#discussion_r366142860
##
File path:
flink-runtime/src/test/java/org/apache/flink/runtime/executiongraph/AdaptedRestartPipelinedRegionFailoverStrategyNGAbortPendingCheckpointsTest.java
##
@@ -59,7 +59,7 @@
/**
* Tests for {@link AdaptedRestartPipelinedRegionStrategyNG}.
*/
-public class
AdaptedRestartPipelinedRegionStrategyNGAbortPendingCheckpointsTest extends
TestLogger {
+public class
AdaptedRestartPipelinedRegionFailoverStrategyNGAbortPendingCheckpointsTest
extends TestLogger {
Review comment:
We should not do this since this PR does not rename
AdaptedRestartPipelinedRegionStrategyNG.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #10848: [FLINK-15307][failover]Rename Subclasses of FailoverStrategy
flinkbot edited a comment on issue #10848: [FLINK-15307][failover]Rename Subclasses of FailoverStrategy URL: https://github.com/apache/flink/pull/10848#issuecomment-573958003 ## CI report: * a3f83df73af74cf8509f084e860ab88e63129ac8 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/144251297) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4310) * c2f939165d6d4aa4907591f57b1a14c9578c2122 Travis: [PENDING](https://travis-ci.com/flink-ci/flink/builds/144257490) Azure: [PENDING](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4315) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #10814: [FLINK-15434][Tests]Fix unstable tests in JobMasterTest
flinkbot edited a comment on issue #10814: [FLINK-15434][Tests]Fix unstable tests in JobMasterTest URL: https://github.com/apache/flink/pull/10814#issuecomment-572552628 ## CI report: * a814428b99636921abf55ab089e664cec9d13876 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/143727769) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4228) * 96a81a9b98ba96eb075c9b5f32fd15a6e08082ab Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/144255567) Azure: [PENDING](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4311) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-13590) flink-on-yarn sometimes could create many little files that are xxx-taskmanager-conf.yaml
[ https://issues.apache.org/jira/browse/FLINK-13590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17014847#comment-17014847 ] Yang Wang commented on FLINK-13590: --- [~gjy] It we do not want to backport to 1.7. Then this ticket could be closed. Since this fix will be contained in 1.8.3, 1.9.2, 1.10.0. > flink-on-yarn sometimes could create many little files that are > xxx-taskmanager-conf.yaml > - > > Key: FLINK-13590 > URL: https://issues.apache.org/jira/browse/FLINK-13590 > Project: Flink > Issue Type: Bug > Components: Deployment / YARN >Affects Versions: 1.7.2, 1.8.0 >Reporter: shuwenjun >Priority: Major > Attachments: taskmanager-conf-yaml.png > > > Both of 1.7.2 and 1.8.0 are used, but they could create many little files. > These files are the configuration file of taskmanager and when the flink > session try to apply a new container, one of the files will be created. And I > don't know why sometimes the flink session apply container again and again? > Or when one container has lost, it could delete its taskmanager-conf.yaml > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-14163) Execution#producedPartitions is possibly not assigned when used
[
https://issues.apache.org/jira/browse/FLINK-14163?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17014845#comment-17014845
]
Zhu Zhu commented on FLINK-14163:
-
+1 to fix it with a simple check first. As long as we are call `get()` on the
future directly, a timeout may still lead to blocked main thread which we are
trying to avoid. And of course finally we will need a timeout to support the
async handling of the partition registration.
> Execution#producedPartitions is possibly not assigned when used
> ---
>
> Key: FLINK-14163
> URL: https://issues.apache.org/jira/browse/FLINK-14163
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination
>Affects Versions: 1.9.0, 1.10.0
>Reporter: Zhu Zhu
>Assignee: Yuan Mei
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.10.0
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> Currently {{Execution#producedPartitions}} is assigned after the partitions
> have completed the registration to shuffle master in
> {{Execution#registerProducedPartitions(...)}}.
> The partition registration is an async interface
> ({{ShuffleMaster#registerPartitionWithProducer(...)}}), so
> {{Execution#producedPartitions}} is possible[1] not set when used.
> Usages includes:
> 1. deploying this task, so that the task may be deployed without its result
> partitions assigned, and the job would hang. (DefaultScheduler issue only,
> since legacy scheduler handled this case)
> 2. generating input descriptors for downstream tasks:
> 3. retrieve {{ResultPartitionID}} for partition releasing:
> [1] If a user uses Flink default shuffle master {{NettyShuffleMaster}}, it is
> not problematic at the moment since it returns a completed future on
> registration, so that it would be a synchronized process. However, if users
> implement their own shuffle service in which the
> {{ShuffleMaster#registerPartitionWithProducer}} returns an pending future, it
> can be a problem. This is possible since customizable shuffle service is open
> to users since 1.9 (via config "shuffle-service-factory.class").
> To avoid issues to happen, we may either
> 1. fix all the usages of {{Execution#producedPartitions}} regarding the async
> assigning, or
> 2. change {{ShuffleMaster#registerPartitionWithProducer(...)}} to a sync
> interface
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] JingsongLi commented on a change in pull request #10846: [FLINK-15576] remove isTemporary property from CatalogFunction API
JingsongLi commented on a change in pull request #10846: [FLINK-15576] remove
isTemporary property from CatalogFunction API
URL: https://github.com/apache/flink/pull/10846#discussion_r366139545
##
File path:
flink-table/flink-table-api-java/src/main/java/org/apache/flink/table/operations/ddl/CreateCatalogFunctionOperation.java
##
@@ -34,14 +34,18 @@
private final ObjectIdentifier functionIdentifier;
private CatalogFunction catalogFunction;
private boolean ignoreIfExists;
+ private boolean isTemporary;
public CreateCatalogFunctionOperation(
ObjectIdentifier functionIdentifier,
CatalogFunction catalogFunction,
- boolean ignoreIfExists) {
+ boolean ignoreIfExists,
+ boolean isTemporary) {
this.functionIdentifier = functionIdentifier;
this.catalogFunction = catalogFunction;
this.ignoreIfExists = ignoreIfExists;
+ this.isTemporary = isTemporary;
+
Review comment:
remove empty line
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] wuchong commented on a change in pull request #10769: [FLINK-15479]Override explainSource method for JDBCTableSource
wuchong commented on a change in pull request #10769: [FLINK-15479]Override
explainSource method for JDBCTableSource
URL: https://github.com/apache/flink/pull/10769#discussion_r366136972
##
File path:
flink-connectors/flink-jdbc/src/test/java/org/apache/flink/api/java/io/jdbc/JDBCTableSourceITCase.java
##
@@ -0,0 +1,111 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.api.java.io.jdbc;
+
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.table.api.Table;
+import org.apache.flink.table.api.java.StreamTableEnvironment;
+import org.apache.flink.table.runtime.utils.StreamITCase;
+import org.apache.flink.types.Row;
+
+import org.junit.Test;
+
+import java.util.ArrayList;
+import java.util.List;
+
+/**
+ * IT case for {@link JDBCTableSource}.
+ */
+
Review comment:
remove empty line.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #10736: [FLINK-15010][Network] Add shutdown hook to ensure cleanup netty shuffle directories
flinkbot edited a comment on issue #10736: [FLINK-15010][Network] Add shutdown hook to ensure cleanup netty shuffle directories URL: https://github.com/apache/flink/pull/10736#issuecomment-570066854 ## CI report: * 4b605068e32d3eb15a51f838ed29918d1224959a Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/142804653) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4021) * 9430066683a67318f9685de8a58904972c5dbaca Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/142829633) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4026) * 2185007c824d21817356c9dfb9c9e09846e27f7e UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] KarmaGYZ commented on a change in pull request #10746: [FLINK-15417] Remove the docker volume or mount when starting Mesos e…
KarmaGYZ commented on a change in pull request #10746: [FLINK-15417] Remove the
docker volume or mount when starting Mesos e…
URL: https://github.com/apache/flink/pull/10746#discussion_r366138517
##
File path: flink-end-to-end-tests/test-scripts/common_mesos_docker.sh
##
@@ -53,11 +55,22 @@ function start_flink_cluster_with_mesos() {
set_config_key "jobmanager.rpc.address" "mesos-master"
set_config_key "rest.address" "mesos-master"
-docker exec -itd mesos-master bash -c "${FLINK_DIR}/bin/mesos-appmaster.sh
-Dmesos.master=mesos-master:5050"
+docker cp ${FLINK_DIR} mesos-master:$MESOS_FLINK_DIR
Review comment:
> all writes should go to a volume or bind mount
Regarding this concern, I think that the two approaches do not have
significant difference. Since We'll always write the log to the container's
storage. If you have a strong point here, I'm ok to bundle Flink and the test
data inside the image.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Commented] (FLINK-12362) Remove legacy container number config option for Flink on yarn
[
https://issues.apache.org/jira/browse/FLINK-12362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17014840#comment-17014840
]
liupengcheng commented on FLINK-12362:
--
[~trohrmann] [~yanghua] I still wondering how to set the a total fixed number
of task manager now in flink, I know the parallism can be computed by -p and
-ys config option, but that's for each JobVertex, one sql query can be mapping
to multiple JobVertex, how can we set the total number of task manager to a
fixed number? I think this is necessary when we need to do some benchmark
testing.
> Remove legacy container number config option for Flink on yarn
> --
>
> Key: FLINK-12362
> URL: https://issues.apache.org/jira/browse/FLINK-12362
> Project: Flink
> Issue Type: Improvement
> Components: Deployment / YARN
>Reporter: vinoyang
>Assignee: vinoyang
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.10.0
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> The {{-n}} config option for Flink on YARN has been deprecated and removed
> from documentation. However, there are still some legacy code exists in the
> codebase. We need to clean up them.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-11899) Introduce vectorized parquet InputFormat for blink runtime
[ https://issues.apache.org/jira/browse/FLINK-11899?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17014838#comment-17014838 ] Jingsong Lee commented on FLINK-11899: -- [~hpeter] Why share so much? What is your plan? IMO, Orc reader is wrapping Hive Vector to Flink vector, but parquet reader is reading data directly to Flink vector. > Introduce vectorized parquet InputFormat for blink runtime > -- > > Key: FLINK-11899 > URL: https://issues.apache.org/jira/browse/FLINK-11899 > Project: Flink > Issue Type: Sub-task > Components: Table SQL / Runtime >Reporter: Jingsong Lee >Assignee: Zhenqiu Huang >Priority: Major > Fix For: 1.11.0 > > > VectorizedParquetInputFormat is introduced to read parquet data in batches. > When returning each row of data, instead of actually retrieving each field, > we use BaseRow's abstraction to return a Columnar Row-like view. > This will greatly improve the downstream filtered scenarios, so that there is > no need to access redundant fields on the filtered data. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (FLINK-15567) Add documentation for INSERT statements for Flink SQL
[ https://issues.apache.org/jira/browse/FLINK-15567?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jark Wu resolved FLINK-15567. - Resolution: Fixed 1.11.0: b2e3967c15ca719fb599060c88ccad242f4095d3 1.10.0: dc0fff984caffaafe52e6e42b0357cbcc46e99c9 > Add documentation for INSERT statements for Flink SQL > - > > Key: FLINK-15567 > URL: https://issues.apache.org/jira/browse/FLINK-15567 > Project: Flink > Issue Type: Task > Components: Documentation >Reporter: Jark Wu >Assignee: Jark Wu >Priority: Major > Labels: pull-request-available > Fix For: 1.10.0 > > Time Spent: 20m > Remaining Estimate: 0h > > We missed to add documentation for INSERT statements which should be added > under "SQL" page. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] wuchong merged pull request #10839: [FLINK-15567][table][docs] Add documentation for INSERT statements for Flink SQL
wuchong merged pull request #10839: [FLINK-15567][table][docs] Add documentation for INSERT statements for Flink SQL URL: https://github.com/apache/flink/pull/10839 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] wuchong commented on issue #10839: [FLINK-15567][table][docs] Add documentation for INSERT statements for Flink SQL
wuchong commented on issue #10839: [FLINK-15567][table][docs] Add documentation for INSERT statements for Flink SQL URL: https://github.com/apache/flink/pull/10839#issuecomment-573980423 Thanks @JingsongLi , Merging... This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #10848: [FLINK-15307][failover]Rename Subclasses of FailoverStrategy
flinkbot edited a comment on issue #10848: [FLINK-15307][failover]Rename Subclasses of FailoverStrategy URL: https://github.com/apache/flink/pull/10848#issuecomment-573958003 ## CI report: * a3f83df73af74cf8509f084e860ab88e63129ac8 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/144251297) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4310) * c2f939165d6d4aa4907591f57b1a14c9578c2122 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #10844: [FLINK-15559][docs] Fix broken links due to missing baseurl
flinkbot edited a comment on issue #10844: [FLINK-15559][docs] Fix broken links due to missing baseurl URL: https://github.com/apache/flink/pull/10844#issuecomment-573736138 ## CI report: * a356df63a802f15c8a1e6982cf68c51da30b454c Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/144172942) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4300) * 2163c16cddb9ea165f201b3aeb1caa21cd989ead UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #10839: [FLINK-15567][table][docs] Add documentation for INSERT statements for Flink SQL
flinkbot edited a comment on issue #10839: [FLINK-15567][table][docs] Add documentation for INSERT statements for Flink SQL URL: https://github.com/apache/flink/pull/10839#issuecomment-573563076 ## CI report: * c5595ea0c994dcf664cb68ec45a6f500e705a3b7 Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/144113663) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4285) * 39d7478a9edb9ce3f592a02c6afa2aa69e49a5d5 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #10815: [FLINK-15537][table-planner-blink] Type of keys should be `BinaryRow`…
flinkbot edited a comment on issue #10815: [FLINK-15537][table-planner-blink] Type of keys should be `BinaryRow`… URL: https://github.com/apache/flink/pull/10815#issuecomment-572566376 ## CI report: * 19a4290f709495491fe460037c8c31d106984ea8 Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/143732723) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4229) * c3ef5ea345a343170806de8112163edb7df31f69 Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/144110200) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4284) * 941a5d4725dee3317ca05f8ab16eb103f61d3fcb Travis: [PENDING](https://travis-ci.com/flink-ci/flink/builds/144255612) Azure: [PENDING](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4312) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Comment Edited] (FLINK-15469) UpsertStreamTableSink should support new type system
[
https://issues.apache.org/jira/browse/FLINK-15469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17014828#comment-17014828
]
Zhenghua Gao edited comment on FLINK-15469 at 1/14/20 3:15 AM:
---
Hi [~lzljs3620320], after re-think the whole thing, we should bring the
physical data types of the sink and the type class(java tuple2 or scale tuple2)
to the planner so that our blink planner could handle the precision things. So
there is a proposal as following:
# deprecate getRecordType and introduce getRecordDataType instead
# remove getOutputType and introduce getConsumedDataType, which returns
ROW
# introduce getTypeClass interface, which returns type class for codegen
What do you think? I will file a PR soon if this works.
was (Author: docete):
Hi [~lzljs3620320], after re-think the whole thing, we should bring the
physical data types of the sink and the type class(java tuple2 or scale tuple2)
to the planner so that our blink planner could handle the precision things. So
there is a proposal as following:
# remove getRecordType and introduce getRecordDataType instead
# remove getOutputType and introduce getConsumedDataType, which returns
ROW
# introduce getTypeClass interface, which returns type class for codegen
What do you think? I will file a PR soon if this works.
> UpsertStreamTableSink should support new type system
>
>
> Key: FLINK-15469
> URL: https://issues.apache.org/jira/browse/FLINK-15469
> Project: Flink
> Issue Type: Improvement
> Components: Table SQL / API
>Affects Versions: 1.10.0
>Reporter: Zhenghua Gao
>Priority: Major
> Fix For: 1.11.0
>
>
> Currently *UpsertStreamTableSink* can only returns TypeInformation of the
> requested record, which can't support types with precision and scale, e.g.
> TIMESTAMP(p), DECIMAL(p,s).
> A proposal is deprecating the *getRecordType* API and adding a
> *getRecordDataType* API instead to return the data type of the requested
> record.
> {code:java}
> /**
> * Returns the requested record type.
> *
> * @Deprecated This method will be removed in future versions. It's
> recommended to use {@link #getRecordDataType()} instead.
> */
> @Deprecated
> TypeInformation getRecordType();
> /*
> * Returns the requested record data type.
> */
> DataType getRecordDataType();
> {code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on issue #10814: [FLINK-15434][Tests]Fix unstable tests in JobMasterTest
flinkbot edited a comment on issue #10814: [FLINK-15434][Tests]Fix unstable tests in JobMasterTest URL: https://github.com/apache/flink/pull/10814#issuecomment-572552628 ## CI report: * a814428b99636921abf55ab089e664cec9d13876 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/143727769) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4228) * 96a81a9b98ba96eb075c9b5f32fd15a6e08082ab Travis: [PENDING](https://travis-ci.com/flink-ci/flink/builds/144255567) Azure: [PENDING](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4311) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #10784: [FLINK-15494] Fix time field index wrong in LogicalWindowAggregateRul…
flinkbot edited a comment on issue #10784: [FLINK-15494] Fix time field index wrong in LogicalWindowAggregateRul… URL: https://github.com/apache/flink/pull/10784#issuecomment-571473399 ## CI report: * d0e5cb3e0d2cc118c98cd91e0470c5123219d1ec Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/143353476) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4149) * c58517317e7de5dda91742eaf5d0e6d23d3d14a0 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/144249050) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4309) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (FLINK-15469) UpsertStreamTableSink should support new type system
[
https://issues.apache.org/jira/browse/FLINK-15469?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jingsong Lee updated FLINK-15469:
-
Issue Type: Improvement (was: New Feature)
> UpsertStreamTableSink should support new type system
>
>
> Key: FLINK-15469
> URL: https://issues.apache.org/jira/browse/FLINK-15469
> Project: Flink
> Issue Type: Improvement
> Components: Table SQL / API
>Affects Versions: 1.10.0
>Reporter: Zhenghua Gao
>Priority: Major
> Fix For: 1.11.0
>
>
> Currently *UpsertStreamTableSink* can only returns TypeInformation of the
> requested record, which can't support types with precision and scale, e.g.
> TIMESTAMP(p), DECIMAL(p,s).
> A proposal is deprecating the *getRecordType* API and adding a
> *getRecordDataType* API instead to return the data type of the requested
> record.
> {code:java}
> /**
> * Returns the requested record type.
> *
> * @Deprecated This method will be removed in future versions. It's
> recommended to use {@link #getRecordDataType()} instead.
> */
> @Deprecated
> TypeInformation getRecordType();
> /*
> * Returns the requested record data type.
> */
> DataType getRecordDataType();
> {code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] JingsongLi commented on issue #10839: [FLINK-15567][table][docs] Add documentation for INSERT statements for Flink SQL
JingsongLi commented on issue #10839: [FLINK-15567][table][docs] Add documentation for INSERT statements for Flink SQL URL: https://github.com/apache/flink/pull/10839#issuecomment-573978938 LGTM This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-15469) UpsertStreamTableSink should support new type system
[
https://issues.apache.org/jira/browse/FLINK-15469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17014828#comment-17014828
]
Zhenghua Gao commented on FLINK-15469:
--
Hi [~lzljs3620320], after re-think the whole thing, we should bring the
physical data types of the sink and the type class(java tuple2 or scale tuple2)
to the planner so that our blink planner could handle the precision things. So
there is a proposal as following:
# remove getRecordType and introduce getRecordDataType instead
# remove getOutputType and introduce getConsumedDataType, which returns
ROW
# introduce getTypeClass interface, which returns type class for codegen
What do you think? I will file a PR soon if this works.
> UpsertStreamTableSink should support new type system
>
>
> Key: FLINK-15469
> URL: https://issues.apache.org/jira/browse/FLINK-15469
> Project: Flink
> Issue Type: New Feature
> Components: Table SQL / API
>Affects Versions: 1.10.0
>Reporter: Zhenghua Gao
>Priority: Major
> Fix For: 1.11.0
>
>
> Currently *UpsertStreamTableSink* can only returns TypeInformation of the
> requested record, which can't support types with precision and scale, e.g.
> TIMESTAMP(p), DECIMAL(p,s).
> A proposal is deprecating the *getRecordType* API and adding a
> *getRecordDataType* API instead to return the data type of the requested
> record.
> {code:java}
> /**
> * Returns the requested record type.
> *
> * @Deprecated This method will be removed in future versions. It's
> recommended to use {@link #getRecordDataType()} instead.
> */
> @Deprecated
> TypeInformation getRecordType();
> /*
> * Returns the requested record data type.
> */
> DataType getRecordDataType();
> {code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (FLINK-15559) Missing {{site.baseurl}} in some docs
[
https://issues.apache.org/jira/browse/FLINK-15559?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jark Wu resolved FLINK-15559.
-
Fix Version/s: 1.10.0
Resolution: Fixed
1.11.0: dd05f44d7d2b9bf6ea151489b66dc9cd2bd64eed
1.10.0: 3ba7070ff1f3d6b53180b904c64403aede803aa3
> Missing {{site.baseurl}} in some docs
> -
>
> Key: FLINK-15559
> URL: https://issues.apache.org/jira/browse/FLINK-15559
> Project: Flink
> Issue Type: Improvement
> Components: Documentation
>Affects Versions: 1.10.0
>Reporter: Benchao Li
>Assignee: Benchao Li
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.10.0
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> For example, in
> * dev/table/functions/systemFunctions.md
> * dev/table/functions/systemFunctions.zh.md
> * dev/table/sourceSinks.md
> * dev/table/sourceSinks.zh.md
> * dev/table/sql/queries.zh.md
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] gaoyunhaii commented on issue #10736: [FLINK-15010][Network] Add shutdown hook to ensure cleanup netty shuffle directories
gaoyunhaii commented on issue #10736: [FLINK-15010][Network] Add shutdown hook to ensure cleanup netty shuffle directories URL: https://github.com/apache/flink/pull/10736#issuecomment-573978662 Hi @zhijiangW very thanks for the review! I have updated the PR according to the comments. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] gaoyunhaii commented on a change in pull request #10736: [FLINK-15010][Network] Add shutdown hook to ensure cleanup netty shuffle directories
gaoyunhaii commented on a change in pull request #10736: [FLINK-15010][Network]
Add shutdown hook to ensure cleanup netty shuffle directories
URL: https://github.com/apache/flink/pull/10736#discussion_r366134565
##
File path:
flink-runtime/src/test/java/org/apache/flink/runtime/io/network/NettyShuffleEnvironmentCleanupTest.java
##
@@ -0,0 +1,181 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.io.network;
+
+import org.apache.flink.api.common.time.Deadline;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.configuration.CoreOptions;
+import org.apache.flink.configuration.HighAvailabilityOptions;
+import org.apache.flink.configuration.JobManagerOptions;
+import org.apache.flink.configuration.NettyShuffleEnvironmentOptions;
+import org.apache.flink.configuration.RestOptions;
+import org.apache.flink.configuration.TaskManagerOptions;
+import org.apache.flink.runtime.entrypoint.StandaloneSessionClusterEntrypoint;
+import org.apache.flink.runtime.taskexecutor.TaskManagerRunner;
+import org.apache.flink.runtime.testutils.TestJvmProcess;
+import org.apache.flink.runtime.zookeeper.ZooKeeperResource;
+import org.apache.flink.util.OperatingSystem;
+
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.TemporaryFolder;
+
+import java.io.BufferedWriter;
+import java.io.File;
+import java.io.FileOutputStream;
+import java.io.IOException;
+import java.io.OutputStreamWriter;
+import java.time.Duration;
+import java.util.Map;
+
+import static
org.apache.flink.runtime.testutils.CommonTestUtils.createTemporaryLog4JProperties;
+import static
org.apache.flink.runtime.testutils.CommonTestUtils.getJavaCommandPath;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.fail;
+import static org.junit.Assume.assumeTrue;
+
+/**
+ * Verifies whether netty shuffle releases all the resources on shutdown, like
the temporary directories.
+ */
+public class NettyShuffleEnvironmentCleanupTest {
+ private static final Duration TEST_TIMEOUT = Duration.ofSeconds(10);
+
+ @Rule
+ public final TemporaryFolder temporaryFolder = new TemporaryFolder();
+
+ @Rule
+ public final ZooKeeperResource zooKeeperResource = new
ZooKeeperResource();
+
+ @Test
+ public void testRemovingTmpDirectoriesOnSignals() throws Exception {
+ assumeTrue(OperatingSystem.isLinux()
+ || OperatingSystem.isFreeBSD()
+ || OperatingSystem.isSolaris()
+ || OperatingSystem.isMac());
+
+ File confDir = temporaryFolder.newFolder();
+ File confFile = new File(confDir + "/flink-conf.yaml");
+
+ File taskManagerTmpDir = temporaryFolder.newFolder();
+
+ Configuration config = new Configuration();
+ config.setString(JobManagerOptions.ADDRESS, "localhost");
+ config.setString(RestOptions.BIND_PORT, "0");
+ config.setString(HighAvailabilityOptions.HA_MODE, "zookeeper");
+ config.setString(HighAvailabilityOptions.HA_ZOOKEEPER_QUORUM,
zooKeeperResource.getConnectString());
+ config.setString(HighAvailabilityOptions.HA_STORAGE_PATH,
temporaryFolder.newFolder().getAbsolutePath());
+ config.setString(CoreOptions.TMP_DIRS,
taskManagerTmpDir.getAbsolutePath());
+ config.setString(TaskManagerOptions.MANAGED_MEMORY_SIZE, "4m");
+
config.setInteger(NettyShuffleEnvironmentOptions.NETWORK_NUM_BUFFERS, 100);
+ config.setString(TaskManagerOptions.TOTAL_FLINK_MEMORY, "512m");
+
+ try (FileOutputStream fos = new FileOutputStream(confFile);
+ BufferedWriter writer = new BufferedWriter(new
OutputStreamWriter(fos))) {
+ for (Map.Entry e :
config.toMap().entrySet()) {
+ writer.write(e.getKey());
+ writer.write(": ");
+ writer.write(e.getValue());
+ writer.newLine();
+ }
+
+
[GitHub] [flink] JingsongLi commented on issue #10769: [FLINK-15479]Override explainSource method for JDBCTableSource
JingsongLi commented on issue #10769: [FLINK-15479]Override explainSource method for JDBCTableSource URL: https://github.com/apache/flink/pull/10769#issuecomment-573978365 ping @wuchong This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] wuchong merged pull request #10844: [FLINK-15559][docs] Fix broken links due to missing baseurl
wuchong merged pull request #10844: [FLINK-15559][docs] Fix broken links due to missing baseurl URL: https://github.com/apache/flink/pull/10844 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Closed] (FLINK-15511) export org.apache.flink.table.api.TableException when flink 1.10 connect hive
[
https://issues.apache.org/jira/browse/FLINK-15511?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jingsong Lee closed FLINK-15511.
Resolution: Invalid
> export org.apache.flink.table.api.TableException when flink 1.10 connect hive
> --
>
> Key: FLINK-15511
> URL: https://issues.apache.org/jira/browse/FLINK-15511
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Hive
>Affects Versions: 1.10.0
> Environment: flink master
> hive 1.2.1
>
>Reporter: chenchencc
>Priority: Major
> Labels: flink, hive
> Fix For: 1.10.0
>
>
> *run scripts:*
> bin/start-scala-shell.sh yarn -qu bi -jm 1024m -tm 2048m
> import org.apache.flink.table.catalog.hive.HiveCatalog
> val name = "myhive"
> val defaultDatabase = "test"
> val hiveConfDir = "/etc/hive/conf"
> val version = "1.2.1" // or 1.2.1 2.3.4
> val hive = new HiveCatalog(name, defaultDatabase, hiveConfDir, version)
> stenv.registerCatalog("myhive", hive)
> stenv.useCatalog("myhive")
> stenv.listTables
> stenv.sqlQuery("select * from gsp_test3").toAppendStream[Row].print
> *gsp_test3 table columns:*
> id int
> name string
> *gsp_test3 table storage:*
> txt file
>
> *scripts run message*
> scala> import org.apache.flink.table.catalog.hive.HiveCatalog
> import org.apache.flink.table.catalog.hive.HiveCatalog
> scala> val name = "myhive"
> name: String = myhive
> scala> val defaultDatabase = "test"
> defaultDatabase: String = test
> scala> val hiveConfDir = "/etc/hive/conf"
> hiveConfDir: String = /etc/hive/conf
> scala> val version = "1.2.1" // or 1.2.1 2.3.4
> version: String = 1.2.1
> scala> val hive = new HiveCatalog(name, defaultDatabase, hiveConfDir, version)
> 20/01/08 14:36:10 INFO hive.HiveCatalog: Setting hive conf dir as
> /etc/hive/conf
> 20/01/08 14:36:10 WARN conf.HiveConf: HiveConf of name
> hive.server2.enable.impersonation does not exist
> 20/01/08 14:36:10 WARN conf.HiveConf: HiveConf of name
> hive.mapred.supports.subdirectories does not exist
> 20/01/08 14:36:10 INFO hive.HiveCatalog: Created HiveCatalog 'myhive'
> hive: org.apache.flink.table.catalog.hive.HiveCatalog =
> org.apache.flink.table.catalog.hive.HiveCatalog@60729135
> scala> stenv.registerCatalog("myhive", hive)
> 20/01/08 14:36:10 INFO hive.metastore: Trying to connect to metastore with
> URI thrift://bgnode4:9083
> 20/01/08 14:36:10 INFO hive.metastore: Connected to metastore.
> 20/01/08 14:36:10 INFO hive.HiveCatalog: Connected to Hive metastore
> scala> stenv.useCatalog("myhive")
> 20/01/08 14:36:10 INFO catalog.CatalogManager: Set the current default
> catalog as [myhive] and the current default database as [test].
> scala> stenv.listTables
> res6: Array[String] = Array(amazonproductscore_test,
> amazonproductscore_test_tmp, amazonshopmanagerkpi, bucketed_user,
> bulkload_spark_gross_profit_items_zcm, dim_date_test,
> dw_gross_profit_items_phoenix_test, dw_gross_profit_items_phoenix_test2,
> dw_gross_profit_items_phoenix_test3, dw_gross_profit_items_phoenix_test4,
> dw_gross_profit_items_phoenix_test5, gsp_test12, gsp_test2, gsp_test3,
> hive_phoenix, ni, orderparent_test, orderparent_test2,
> phoenix_orderparent_id_put_tb, phoenix_orderparent_id_put_tb2,
> phoenix_orderparent_id_tb, productdailysales, result20190404,
> result20190404_2, result20190404_3, result20190404_4_5_9, result20190404_5,
> result20190404vat, result20190404vat11, result20190404vat12,
> result20190404vat13, result20190404vat5, result20190404vat6_2, ...
> scala> stenv.sqlQuery("select * from gsp_test3").toAppendStream[Row].print
> 20/01/08 14:36:13 INFO typeutils.TypeExtractor: class
> org.apache.flink.types.Row does not contain a getter for field fields
> 20/01/08 14:36:13 INFO typeutils.TypeExtractor: class
> org.apache.flink.types.Row does not contain a setter for field fields
> 20/01/08 14:36:13 INFO typeutils.TypeExtractor: Class class
> org.apache.flink.types.Row cannot be used as a POJO type because not all
> fields are valid POJO fields, and must be processed as GenericType. Please
> read the Flink documentation on "Data Types & Serialization" for details of
> the effect on performance.
> 20/01/08 14:36:13 WARN conf.HiveConf: HiveConf of name
> hive.server2.enable.impersonation does not exist
> 20/01/08 14:36:13 WARN conf.HiveConf: HiveConf of name
> hive.mapred.supports.subdirectories does not exist
> 20/01/08 14:36:13 INFO hive.metastore: Trying to connect to metastore with
> URI thrift://bgnode3:9083
> 20/01/08 14:36:13 INFO hive.metastore: Connected to metastore.
> 20/01/08 14:36:13 INFO configuration.GlobalConfiguration: Loading
> configuration property: jobmanager.rpc.address, localhost
> 20/01/08 14:36:13 INFO configuration.GlobalConfiguration: Loading
> configu
[jira] [Commented] (FLINK-15497) Streaming TopN operator doesn't reduce outputs when rank number is not required
[
https://issues.apache.org/jira/browse/FLINK-15497?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17014827#comment-17014827
]
Jingsong Lee commented on FLINK-15497:
--
[~jinyu.zj] Please click "start in-progress", thanks.
> Streaming TopN operator doesn't reduce outputs when rank number is not
> required
>
>
> Key: FLINK-15497
> URL: https://issues.apache.org/jira/browse/FLINK-15497
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Runtime
>Affects Versions: 1.9.1
>Reporter: Kurt Young
>Assignee: Jing Zhang
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.9.2, 1.10.0
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> As we described in the doc:
> [https://ci.apache.org/projects/flink/flink-docs-release-1.9/dev/table/sql.html#top-n]
> when rank number is not required, we can reduce some output, like unnecessary
> retract messages.
> Here is an example which can re-produce:
> {code:java}
> val data = List(
> ("aaa", 97.0, 200.0),
> ("bbb", 67.0, 200.0),
> ("bbb", 162.0, 200.0)
> )
> val ds = failingDataSource(data).toTable(tEnv, 'guid, 'a, 'b)
> tEnv.registerTable("T", ds)
> val aggreagtedTable = tEnv.sqlQuery(
> """
> |select guid,
> |sum(a) as reached_score,
> |sum(b) as max_score,
> |sum(a) / sum(b) as score
> |from T group by guid
> |""".stripMargin
> )
> tEnv.registerTable("T2", aggreagtedTable)
> val sql =
> """
> |SELECT guid, reached_score, max_score, score
> |FROM (
> | SELECT *,
> | ROW_NUMBER() OVER (ORDER BY score DESC) as rank_num
> | FROM T2)
> |WHERE rank_num <= 5
> """.stripMargin
> val sink = new TestingRetractSink
> tEnv.sqlQuery(sql).toRetractStream[Row].addSink(sink).setParallelism(1)
> env.execute()
> {code}
> In this case, the output is:
> {code:java}
> (true,aaa,97.0,200.0,0.485)
> (true,bbb,67.0,200.0,0.335)
> (false,bbb,67.0,200.0,0.335)
> (true,bbb,229.0,400.0,0.5725)
> (false,aaa,97.0,200.0,0.485)
> (true,aaa,97.0,200.0,0.485)
> {code}
> But the last 2 messages are unnecessary.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-15511) export org.apache.flink.table.api.TableException when flink 1.10 connect hive
[
https://issues.apache.org/jira/browse/FLINK-15511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17014826#comment-17014826
]
Jingsong Lee commented on FLINK-15511:
--
Since this is not a bug. I linked this to FLINK-15533 and FLINK-15515 , and
close this one.
> export org.apache.flink.table.api.TableException when flink 1.10 connect hive
> --
>
> Key: FLINK-15511
> URL: https://issues.apache.org/jira/browse/FLINK-15511
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Hive
>Affects Versions: 1.10.0
> Environment: flink master
> hive 1.2.1
>
>Reporter: chenchencc
>Priority: Major
> Labels: flink, hive
> Fix For: 1.10.0
>
>
> *run scripts:*
> bin/start-scala-shell.sh yarn -qu bi -jm 1024m -tm 2048m
> import org.apache.flink.table.catalog.hive.HiveCatalog
> val name = "myhive"
> val defaultDatabase = "test"
> val hiveConfDir = "/etc/hive/conf"
> val version = "1.2.1" // or 1.2.1 2.3.4
> val hive = new HiveCatalog(name, defaultDatabase, hiveConfDir, version)
> stenv.registerCatalog("myhive", hive)
> stenv.useCatalog("myhive")
> stenv.listTables
> stenv.sqlQuery("select * from gsp_test3").toAppendStream[Row].print
> *gsp_test3 table columns:*
> id int
> name string
> *gsp_test3 table storage:*
> txt file
>
> *scripts run message*
> scala> import org.apache.flink.table.catalog.hive.HiveCatalog
> import org.apache.flink.table.catalog.hive.HiveCatalog
> scala> val name = "myhive"
> name: String = myhive
> scala> val defaultDatabase = "test"
> defaultDatabase: String = test
> scala> val hiveConfDir = "/etc/hive/conf"
> hiveConfDir: String = /etc/hive/conf
> scala> val version = "1.2.1" // or 1.2.1 2.3.4
> version: String = 1.2.1
> scala> val hive = new HiveCatalog(name, defaultDatabase, hiveConfDir, version)
> 20/01/08 14:36:10 INFO hive.HiveCatalog: Setting hive conf dir as
> /etc/hive/conf
> 20/01/08 14:36:10 WARN conf.HiveConf: HiveConf of name
> hive.server2.enable.impersonation does not exist
> 20/01/08 14:36:10 WARN conf.HiveConf: HiveConf of name
> hive.mapred.supports.subdirectories does not exist
> 20/01/08 14:36:10 INFO hive.HiveCatalog: Created HiveCatalog 'myhive'
> hive: org.apache.flink.table.catalog.hive.HiveCatalog =
> org.apache.flink.table.catalog.hive.HiveCatalog@60729135
> scala> stenv.registerCatalog("myhive", hive)
> 20/01/08 14:36:10 INFO hive.metastore: Trying to connect to metastore with
> URI thrift://bgnode4:9083
> 20/01/08 14:36:10 INFO hive.metastore: Connected to metastore.
> 20/01/08 14:36:10 INFO hive.HiveCatalog: Connected to Hive metastore
> scala> stenv.useCatalog("myhive")
> 20/01/08 14:36:10 INFO catalog.CatalogManager: Set the current default
> catalog as [myhive] and the current default database as [test].
> scala> stenv.listTables
> res6: Array[String] = Array(amazonproductscore_test,
> amazonproductscore_test_tmp, amazonshopmanagerkpi, bucketed_user,
> bulkload_spark_gross_profit_items_zcm, dim_date_test,
> dw_gross_profit_items_phoenix_test, dw_gross_profit_items_phoenix_test2,
> dw_gross_profit_items_phoenix_test3, dw_gross_profit_items_phoenix_test4,
> dw_gross_profit_items_phoenix_test5, gsp_test12, gsp_test2, gsp_test3,
> hive_phoenix, ni, orderparent_test, orderparent_test2,
> phoenix_orderparent_id_put_tb, phoenix_orderparent_id_put_tb2,
> phoenix_orderparent_id_tb, productdailysales, result20190404,
> result20190404_2, result20190404_3, result20190404_4_5_9, result20190404_5,
> result20190404vat, result20190404vat11, result20190404vat12,
> result20190404vat13, result20190404vat5, result20190404vat6_2, ...
> scala> stenv.sqlQuery("select * from gsp_test3").toAppendStream[Row].print
> 20/01/08 14:36:13 INFO typeutils.TypeExtractor: class
> org.apache.flink.types.Row does not contain a getter for field fields
> 20/01/08 14:36:13 INFO typeutils.TypeExtractor: class
> org.apache.flink.types.Row does not contain a setter for field fields
> 20/01/08 14:36:13 INFO typeutils.TypeExtractor: Class class
> org.apache.flink.types.Row cannot be used as a POJO type because not all
> fields are valid POJO fields, and must be processed as GenericType. Please
> read the Flink documentation on "Data Types & Serialization" for details of
> the effect on performance.
> 20/01/08 14:36:13 WARN conf.HiveConf: HiveConf of name
> hive.server2.enable.impersonation does not exist
> 20/01/08 14:36:13 WARN conf.HiveConf: HiveConf of name
> hive.mapred.supports.subdirectories does not exist
> 20/01/08 14:36:13 INFO hive.metastore: Trying to connect to metastore with
> URI thrift://bgnode3:9083
> 20/01/08 14:36:13 INFO hive.metastore: Connected to metastore.
> 20/01/08 14:36:13 INFO configuration.GlobalConfiguration: Loading
> configur
[jira] [Updated] (FLINK-15527) can not control the number of container on yarn single job module
[
https://issues.apache.org/jira/browse/FLINK-15527?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jingsong Lee updated FLINK-15527:
-
Fix Version/s: (was: 1.10.0)
1.11.0
> can not control the number of container on yarn single job module
> -
>
> Key: FLINK-15527
> URL: https://issues.apache.org/jira/browse/FLINK-15527
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Hive
>Affects Versions: 1.10.0
>Reporter: chenchencc
>Priority: Major
> Fix For: 1.11.0
>
> Attachments: flink-conf.yaml, image-2020-01-09-14-30-46-973.png,
> yarn_application.png
>
>
> when run yarn single job run many container but paralism set 4
> *scripts:*
> ./bin/flink run -m yarn-cluster -ys 3 -p 4 -yjm 1024m -ytm 4096m -yqu bi -c
> com.cc.test.HiveTest2 ./cc_jars/hive-1.0-SNAPSHOT.jar 11.txt test61 6
> _notes_: in 1.9.1 has cli paramter -yn to control the number of containers
> and in 1.10 remove it
> *result:*
> the number of containers is 500+
>
> *code use:*
> query the table and save it to the hdfs text
>
> the storge of table is 200g+
>
>
>
>
> *code:*
> com.cc.test.HiveTest2
> public static void main(String[] args) throws Exception
> { EnvironmentSettings settings =
> EnvironmentSettings.newInstance().useBlinkPlanner().inStreamingMode().build();
> StreamExecutionEnvironment settings2 =
> StreamExecutionEnvironment.getExecutionEnvironment();
> settings2.setParallelism(Integer.valueOf(args[2]));
> StreamTableEnvironment tableEnv = StreamTableEnvironment.create(settings2,
> settings); String name = "myhive"; String defaultDatabase = "test"; String
> hiveConfDir = "/etc/hive/conf"; String version = "1.2.1"; // or 1.2.1 2.3.4
> HiveCatalog hive = new HiveCatalog(name, defaultDatabase, hiveConfDir,
> version); tableEnv.registerCatalog("myhive", hive);
> tableEnv.useCatalog("myhive"); tableEnv.listTables(); Table table =
> tableEnv.sqlQuery("select id from orderparent_test2 where id =
> 'A21204170176'"); tableEnv.toAppendStream(table, Row.class).print();
> tableEnv.toAppendStream(table, Row.class)
> .writeAsText("hdfs:///user/chenchao1/"+ args[0],
> FileSystem.WriteMode.OVERWRITE); tableEnv.execute(args[1]); }
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] gaoyunhaii commented on a change in pull request #10736: [FLINK-15010][Network] Add shutdown hook to ensure cleanup netty shuffle directories
gaoyunhaii commented on a change in pull request #10736: [FLINK-15010][Network]
Add shutdown hook to ensure cleanup netty shuffle directories
URL: https://github.com/apache/flink/pull/10736#discussion_r366133522
##
File path:
flink-runtime/src/test/java/org/apache/flink/runtime/io/network/NettyShuffleEnvironmentCleanupTest.java
##
@@ -0,0 +1,181 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.io.network;
+
+import org.apache.flink.api.common.time.Deadline;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.configuration.CoreOptions;
+import org.apache.flink.configuration.HighAvailabilityOptions;
+import org.apache.flink.configuration.JobManagerOptions;
+import org.apache.flink.configuration.NettyShuffleEnvironmentOptions;
+import org.apache.flink.configuration.RestOptions;
+import org.apache.flink.configuration.TaskManagerOptions;
+import org.apache.flink.runtime.entrypoint.StandaloneSessionClusterEntrypoint;
+import org.apache.flink.runtime.taskexecutor.TaskManagerRunner;
+import org.apache.flink.runtime.testutils.TestJvmProcess;
+import org.apache.flink.runtime.zookeeper.ZooKeeperResource;
+import org.apache.flink.util.OperatingSystem;
+
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.TemporaryFolder;
+
+import java.io.BufferedWriter;
+import java.io.File;
+import java.io.FileOutputStream;
+import java.io.IOException;
+import java.io.OutputStreamWriter;
+import java.time.Duration;
+import java.util.Map;
+
+import static
org.apache.flink.runtime.testutils.CommonTestUtils.createTemporaryLog4JProperties;
+import static
org.apache.flink.runtime.testutils.CommonTestUtils.getJavaCommandPath;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.fail;
+import static org.junit.Assume.assumeTrue;
+
+/**
+ * Verifies whether netty shuffle releases all the resources on shutdown, like
the temporary directories.
+ */
+public class NettyShuffleEnvironmentCleanupTest {
+ private static final Duration TEST_TIMEOUT = Duration.ofSeconds(10);
+
+ @Rule
+ public final TemporaryFolder temporaryFolder = new TemporaryFolder();
+
+ @Rule
+ public final ZooKeeperResource zooKeeperResource = new
ZooKeeperResource();
+
+ @Test
+ public void testRemovingTmpDirectoriesOnSignals() throws Exception {
+ assumeTrue(OperatingSystem.isLinux()
+ || OperatingSystem.isFreeBSD()
+ || OperatingSystem.isSolaris()
+ || OperatingSystem.isMac());
+
+ File confDir = temporaryFolder.newFolder();
+ File confFile = new File(confDir + "/flink-conf.yaml");
+
+ File taskManagerTmpDir = temporaryFolder.newFolder();
+
+ Configuration config = new Configuration();
+ config.setString(JobManagerOptions.ADDRESS, "localhost");
+ config.setString(RestOptions.BIND_PORT, "0");
+ config.setString(HighAvailabilityOptions.HA_MODE, "zookeeper");
+ config.setString(HighAvailabilityOptions.HA_ZOOKEEPER_QUORUM,
zooKeeperResource.getConnectString());
+ config.setString(HighAvailabilityOptions.HA_STORAGE_PATH,
temporaryFolder.newFolder().getAbsolutePath());
+ config.setString(CoreOptions.TMP_DIRS,
taskManagerTmpDir.getAbsolutePath());
+ config.setString(TaskManagerOptions.MANAGED_MEMORY_SIZE, "4m");
+
config.setInteger(NettyShuffleEnvironmentOptions.NETWORK_NUM_BUFFERS, 100);
+ config.setString(TaskManagerOptions.TOTAL_FLINK_MEMORY, "512m");
+
+ try (FileOutputStream fos = new FileOutputStream(confFile);
+ BufferedWriter writer = new BufferedWriter(new
OutputStreamWriter(fos))) {
+ for (Map.Entry e :
config.toMap().entrySet()) {
+ writer.write(e.getKey());
+ writer.write(": ");
+ writer.write(e.getValue());
+ writer.newLine();
+ }
+
+
[GitHub] [flink] libenchao commented on a change in pull request #10844: [FLINK-15559][docs] Fix broken links due to missing baseurl
libenchao commented on a change in pull request #10844: [FLINK-15559][docs] Fix broken links due to missing baseurl URL: https://github.com/apache/flink/pull/10844#discussion_r366133539 ## File path: docs/release-notes/flink-1.8.md ## @@ -200,7 +200,7 @@ The `CompositeSerializerSnapshot` utility class has been removed. You should now use `CompositeTypeSerializerSnapshot` instead, for snapshots of composite serializers that delegate serialization to multiple nested serializers. Please see -[here](/dev/stream/state/custom_serialization.html#implementing-a-compositetypeserializersnapshot) +[here](/projects/flink/flink-docs-release-1.8/dev/stream/state/custom_serialization.html#implementing-a-compositetypeserializersnapshot) Review comment: Yes, it indeed does not work in local. Fixed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] gaoyunhaii commented on a change in pull request #10736: [FLINK-15010][Network] Add shutdown hook to ensure cleanup netty shuffle directories
gaoyunhaii commented on a change in pull request #10736: [FLINK-15010][Network]
Add shutdown hook to ensure cleanup netty shuffle directories
URL: https://github.com/apache/flink/pull/10736#discussion_r366133442
##
File path:
flink-runtime/src/test/java/org/apache/flink/runtime/io/network/NettyShuffleEnvironmentCleanupTest.java
##
@@ -0,0 +1,181 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.io.network;
+
+import org.apache.flink.api.common.time.Deadline;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.configuration.CoreOptions;
+import org.apache.flink.configuration.HighAvailabilityOptions;
+import org.apache.flink.configuration.JobManagerOptions;
+import org.apache.flink.configuration.NettyShuffleEnvironmentOptions;
+import org.apache.flink.configuration.RestOptions;
+import org.apache.flink.configuration.TaskManagerOptions;
+import org.apache.flink.runtime.entrypoint.StandaloneSessionClusterEntrypoint;
+import org.apache.flink.runtime.taskexecutor.TaskManagerRunner;
+import org.apache.flink.runtime.testutils.TestJvmProcess;
+import org.apache.flink.runtime.zookeeper.ZooKeeperResource;
+import org.apache.flink.util.OperatingSystem;
+
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.TemporaryFolder;
+
+import java.io.BufferedWriter;
+import java.io.File;
+import java.io.FileOutputStream;
+import java.io.IOException;
+import java.io.OutputStreamWriter;
+import java.time.Duration;
+import java.util.Map;
+
+import static
org.apache.flink.runtime.testutils.CommonTestUtils.createTemporaryLog4JProperties;
+import static
org.apache.flink.runtime.testutils.CommonTestUtils.getJavaCommandPath;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.fail;
+import static org.junit.Assume.assumeTrue;
+
+/**
+ * Verifies whether netty shuffle releases all the resources on shutdown, like
the temporary directories.
+ */
+public class NettyShuffleEnvironmentCleanupTest {
+ private static final Duration TEST_TIMEOUT = Duration.ofSeconds(10);
+
+ @Rule
+ public final TemporaryFolder temporaryFolder = new TemporaryFolder();
+
+ @Rule
+ public final ZooKeeperResource zooKeeperResource = new
ZooKeeperResource();
+
+ @Test
+ public void testRemovingTmpDirectoriesOnSignals() throws Exception {
+ assumeTrue(OperatingSystem.isLinux()
Review comment:
I think `assumeTrue` would skip the tests if conditions not satisfied, it
will not cause the test fails.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Resolved] (FLINK-15565) Incompatible types of expression and result type thrown in codegen
[ https://issues.apache.org/jira/browse/FLINK-15565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jark Wu resolved FLINK-15565. - Resolution: Fixed 1.11.0: a38d3f7420817785d6b62da3eb488f5dd0c8086c 1.10.0: 999756544d66756695eb7fafd99d88dd911a4474 > Incompatible types of expression and result type thrown in codegen > -- > > Key: FLINK-15565 > URL: https://issues.apache.org/jira/browse/FLINK-15565 > Project: Flink > Issue Type: Bug > Components: Table SQL / Planner >Affects Versions: 1.10.0 >Reporter: xiaojin.wy >Assignee: Jingsong Lee >Priority: Critical > Labels: pull-request-available > Fix For: 1.10.0 > > Time Spent: 0.5h > Remaining Estimate: 0h > > *The sql is:* > CREATE TABLE `over10k` ( > t tinyint, > si smallint, > i int, > b bigint, > f float, > d double, > bo boolean, > s varchar, > ts timestamp, > deci decimal(4,2), > bin varchar > ) WITH ( > > 'connector.path'='/daily_regression_batch_hive_1.10/test_window_with_specific_behavior/sources/over10k.csv', > 'format.empty-column-as-null'='true', > 'format.field-delimiter'='|', > 'connector.type'='filesystem', > 'format.derive-schema'='true', > 'format.type'='csv' > ); > select s, rank() over (partition by s order by si), sum(b) over (partition > by s order by si) from over10k limit 100; > *The data is :* > 109|277|65620|4294967305|97.25|7.80|true|nick quirinius|2013-03-01 > 09:11:58.703226|27.72|undecided > 93|263|65725|4294967341|6.06|4.12|false|calvin king|2013-03-01 > 09:11:58.703299|32.44|values clariffication > 108|383|65629|4294967510|39.55|47.67|false|jessica zipper|2013-03-01 > 09:11:58.703133|74.23|nap time > 89|463|65537|4294967493|64.82|13.79|true|ethan white|2013-03-01 > 09:11:58.703243|89.52|nap time > 88|372|65645|4294967358|34.48|11.18|true|quinn thompson|2013-03-01 > 09:11:58.703168|84.86|forestry > 123|432|65626|4294967435|2.39|16.49|true|david white|2013-03-01 > 09:11:58.703136|61.24|joggying > 57|486|65551|4294967397|36.11|9.88|true|katie xylophone|2013-03-01 > 09:11:58.703142|57.10|zync studies > 59|343|65787|4294967312|66.89|6.54|true|mike laertes|2013-03-01 > 09:11:58.703209|27.56|xylophone band > 74|267|65671|4294967409|21.14|14.64|true|priscilla miller|2013-03-01 > 09:11:58.703197|89.06|undecided > 25|336|65587|4294967336|71.01|14.90|true|tom ichabod|2013-03-01 > 09:11:58.703127|74.32|zync studies > 48|346|65712|4294967315|45.01|16.08|true|zach brown|2013-03-01 > 09:11:58.703108|21.68|zync studies > 84|385|65776|4294967452|35.80|32.13|false|xavier zipper|2013-03-01 > 09:11:58.703311|99.46|education > 58|389|65766|4294967416|95.55|20.62|false|sarah miller|2013-03-01 > 09:11:58.703215|70.92|history > 22|403|65565|4294967381|99.65|35.42|false|yuri johnson|2013-03-01 > 09:11:58.703154|94.47|geology > 55|428|65733|4294967535|99.54|5.35|false|jessica king|2013-03-01 > 09:11:58.703233|30.30|forestry > 117|410|65706|4294967391|50.15|0.21|false|quinn johnson|2013-03-01 > 09:11:58.703248|65.99|yard duty > 95|423|65573|4294967378|47.59|17.37|true|alice robinson|2013-03-01 > 09:11:58.703133|54.57|linguistics > 87|332|65748|4294967320|19.83|41.67|false|fred ellison|2013-03-01 > 09:11:58.703289|79.02|mathematics > 114|263|65674|4294967405|84.44|33.18|true|victor van buren|2013-03-01 > 09:11:58.703092|63.74|linguistics > 5|369|65780|4294967488|92.02|38.59|true|zach polk|2013-03-01 > 09:11:58.703271|67.29|yard duty > -3|430|65667|4294967469|65.50|40.46|true|yuri xylophone|2013-03-01 > 09:11:58.703258|30.94|american history > 120|264|65769|4294967486|89.97|41.18|false|xavier hernandez|2013-03-01 > 09:11:58.703140|66.89|philosophy > 107|317|65634|4294967488|5.68|18.89|false|priscilla ichabod|2013-03-01 > 09:11:58.703196|39.42|joggying > 29|386|65723|4294967328|71.48|6.13|false|ulysses ichabod|2013-03-01 > 09:11:58.703215|86.65|xylophone band > 22|434|65768|4294967543|44.25|27.56|false|tom polk|2013-03-01 > 09:11:58.703306|12.30|kindergarten > -1|274|65755|4294967300|22.01|35.52|false|oscar king|2013-03-01 > 09:11:58.703141|33.35|chemistry > 6|365|65603|4294967522|18.51|5.60|false|gabriella king|2013-03-01 > 09:11:58.703104|34.20|geology > 97|414|65757|4294967325|31.82|22.37|false|rachel nixon|2013-03-01 > 09:11:58.703127|61.00|nap time > 72|448|65538|4294967524|80.09|7.73|true|luke brown|2013-03-01 > 09:11:58.703090|95.81|american history > 51|280|65589|4294967486|57.46|23.35|false|zach xylophone|2013-03-01 > 09:11:58.703299|11.54|education > 12|447|65583|4294967389|0.98|29.79|true|yuri polk|2013-03-01 > 09:11:58.703305|1.89|wind surfing > -1|360|65539|4294967464|4.08|39.51|false|oscar davidson|2013-03-01 > 09:11:58.703144|59.47|nap time > 0|380|65569|4294967425|0.94|28.93|false|sarah robinson|2013
[jira] [Comment Edited] (FLINK-15307) Subclasses of FailoverStrategy are easily confused with implementation classes of RestartStrategy
[ https://issues.apache.org/jira/browse/FLINK-15307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17014810#comment-17014810 ] Andrew.D.lin edited comment on FLINK-15307 at 1/14/20 3:01 AM: --- Hi [~zhuzh], I modified it according to your suggestion( the name patterns discussed above), and only modified strategy under flip1. hope you help review and give me some suggestions. thank you very much! :D was (Author: andrew_lin): Hi [~zhuzh], I modified it according to your suggestion, only modified strategy under flip1. hope you help review and give me some suggestions. thank you very much! :D > Subclasses of FailoverStrategy are easily confused with implementation > classes of RestartStrategy > - > > Key: FLINK-15307 > URL: https://issues.apache.org/jira/browse/FLINK-15307 > Project: Flink > Issue Type: Improvement > Components: Runtime / Configuration >Reporter: Andrew.D.lin >Assignee: Andrew.D.lin >Priority: Minor > Labels: pull-request-available > Fix For: 1.11.0 > > Attachments: image-2019-12-18-14-59-03-181.png > > Time Spent: 10m > Remaining Estimate: 24h > > Subclasses of RestartStrategy > * FailingRestartStrategy > * FailureRateRestartStrategy > * FixedDelayRestartStrategy > * InfiniteDelayRestartStrategy > Implementation class of FailoverStrategy > * AdaptedRestartPipelinedRegionStrategyNG > * RestartAllStrategy > * RestartIndividualStrategy > * RestartPipelinedRegionStrategy > > FailoverStrategy describes how the job computation recovers from task > failures. > I think the following names may be easier to understand and easier to > distinguish： > Implementation class of FailoverStrategy > * AdaptedPipelinedRegionFailoverStrategyNG > * FailoverAllStrategy > * FailoverIndividualStrategy > * FailoverPipelinedRegionStrategy > FailoverStrategy is currently generated by configuration. If we change the > name of the implementation class, it will not affect compatibility. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] wuchong merged pull request #10841: [FLINK-15565][table-planner-blink] Fix error conversion of TinyInt, SmallInt literals between Flink and Calcite
wuchong merged pull request #10841: [FLINK-15565][table-planner-blink] Fix error conversion of TinyInt,SmallInt literals between Flink and Calcite URL: https://github.com/apache/flink/pull/10841 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Comment Edited] (FLINK-15307) Subclasses of FailoverStrategy are easily confused with implementation classes of RestartStrategy
[ https://issues.apache.org/jira/browse/FLINK-15307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17014810#comment-17014810 ] Andrew.D.lin edited comment on FLINK-15307 at 1/14/20 2:56 AM: --- Hi [~zhuzh], I modified it according to your suggestion, only modified strategy under flip1. hope you help review and give me some suggestions. thank you very much! :D was (Author: andrew_lin): [~zhuzh] sorry, I missed it, Let me modify it > Subclasses of FailoverStrategy are easily confused with implementation > classes of RestartStrategy > - > > Key: FLINK-15307 > URL: https://issues.apache.org/jira/browse/FLINK-15307 > Project: Flink > Issue Type: Improvement > Components: Runtime / Configuration >Reporter: Andrew.D.lin >Assignee: Andrew.D.lin >Priority: Minor > Labels: pull-request-available > Fix For: 1.11.0 > > Attachments: image-2019-12-18-14-59-03-181.png > > Time Spent: 10m > Remaining Estimate: 24h > > Subclasses of RestartStrategy > * FailingRestartStrategy > * FailureRateRestartStrategy > * FixedDelayRestartStrategy > * InfiniteDelayRestartStrategy > Implementation class of FailoverStrategy > * AdaptedRestartPipelinedRegionStrategyNG > * RestartAllStrategy > * RestartIndividualStrategy > * RestartPipelinedRegionStrategy > > FailoverStrategy describes how the job computation recovers from task > failures. > I think the following names may be easier to understand and easier to > distinguish： > Implementation class of FailoverStrategy > * AdaptedPipelinedRegionFailoverStrategyNG > * FailoverAllStrategy > * FailoverIndividualStrategy > * FailoverPipelinedRegionStrategy > FailoverStrategy is currently generated by configuration. If we change the > name of the implementation class, it will not affect compatibility. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-14684) Add Pinterest to Chinese Powered By page
[ https://issues.apache.org/jira/browse/FLINK-14684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17014823#comment-17014823 ] Jiawei Wu commented on FLINK-14684: --- [~jark] great! Thanks for that. > Add Pinterest to Chinese Powered By page > > > Key: FLINK-14684 > URL: https://issues.apache.org/jira/browse/FLINK-14684 > Project: Flink > Issue Type: New Feature > Components: chinese-translation >Reporter: Hequn Cheng >Assignee: Jiawei Wu >Priority: Minor > > Pinterest was added to the English Powered By page with commit: > [51f7e3ced85b94dcbe3c051069379d22c88fbc5c|https://github.com/apache/flink-web/pull/281] > It should be added to the Chinese Powered By (and index.html) page as well. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-15505) flink-clients test failing on Red Hat 7.6 PowerPC Linux
[
https://issues.apache.org/jira/browse/FLINK-15505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17014820#comment-17014820
]
wangxiyuan commented on FLINK-15505:
[~fly_in_gis]
1.8.3 doesn't contain my change, for example:
[https://github.com/apache/flink/blob/release-1.8.3/flink-table/flink-table-planner/src/main/scala/org/apache/flink/table/plan/cost/DataSetCost.scala#L112]
I guess my change is not released now?
the test failure on ppc64le is the same with aarch64. So use StrictMath can
solve this problem.
I just wrote a test :
[https://github.com/wangxiyuan/test_diff_arch/blob/master/TestMath.java]
the test result on diff arch:
[https://travis-ci.com/wangxiyuan/test_diff_arch/builds/144255686]
You can see that StrichMath does the same behavior on different arch.
[~redmark-ibm] Please try flink master branch for these two tests or wait
another new release IMO.
> flink-clients test failing on Red Hat 7.6 PowerPC Linux
> ---
>
> Key: FLINK-15505
> URL: https://issues.apache.org/jira/browse/FLINK-15505
> Project: Flink
> Issue Type: Bug
> Components: Tests
>Affects Versions: 1.8.3
> Environment: Red Hat 7.6 Linux ppc64le
> Java 1.8.0.232
> Maven 3.2.5
>Reporter: Ronald O. Edmark
>Priority: Critical
>
> git clone [https://github.com/apache/flink.git]
> cd flink
> git checkout release-1.8.3
> mvn clean package
>
> [ERROR] SqlExpressionTest>ExpressionTestBase.evaluateExprs:170 Wrong result
> for: [LOG(3,27)] optimized to: [3.0004440E0] expected:<3.0[]> but
> was:<3.0[004]>
> [ERROR] SqlExpressionTest>ExpressionTestBase.evaluateExprs:171 Wrong result
> for: [EXP(1)] optimized to: [2.7182818284590455348E0]
> expected:<2.718281828459045[]> but was:<2.718281828459045[5]>
>
> Edit file and removed the two tests worked.
> ./flink-table/flink-table-planner/src/test/scala/org/apache/flink/table/expressions/SqlExpressionTest.scala
> testSqlApi("LOG(3,27)", "3.0")
> testSqlApi("EXP(1)", "2.718281828459045")
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] JingsongLi commented on a change in pull request #10839: [FLINK-15567][table][docs] Add documentation for INSERT statements for Flink SQL
JingsongLi commented on a change in pull request #10839:
[FLINK-15567][table][docs] Add documentation for INSERT statements for Flink SQL
URL: https://github.com/apache/flink/pull/10839#discussion_r366131547
##
File path: docs/dev/table/sql/insert.md
##
@@ -125,20 +125,28 @@ part_spec:
### Examples
{% highlight sql %}
-
-- Creates a partitioned table
CREATE TABLE country_page_view (user STRING, cnt INT, date STRING, country
STRING)
PARTITIONED BY (date, country)
WITH (...)
--- Appends rows into the partition (date='2019-8-30', country='China')
+-- Appends rows into the static partition (date='2019-8-30', country='China')
INSERT INTO country_page_view PARTITION (date='2019-8-30', country='China')
SELECT user, cnt FROM page_view_source;
--- Overwrites the partition (date='2019-8-30', country='China') using rows in
page_view_source
+-- Appends rows into partition (date, country), where date is static partition
with value '2019-8-30',
+-- country is dynamic partition whose value is dynamic determined by each row.
INSERT INTO country_page_view PARTITION (date='2019-8-30', country='China')
Review comment:
Remove `, country='China'`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] KarmaGYZ edited a comment on issue #10746: [FLINK-15417] Remove the docker volume or mount when starting Mesos e…
KarmaGYZ edited a comment on issue #10746: [FLINK-15417] Remove the docker volume or mount when starting Mesos e… URL: https://github.com/apache/flink/pull/10746#issuecomment-573974153 > will we always run into permission issues Sorry, I'm not sure about this. At least so far, I could not figure out what happens in [FLINK-15377](https://issues.apache.org/jira/browse/FLINK-15377) cause the permission issue. It always runs well on my local environment(MacOS). But I think this PR could prevent that happening again. > Another approach I have found is described here It sounds good, I give a try on this approach. However, I meet `addgroup: The GID '20' is already in use` issue. It seems this approach does not run well with MacOS. I'm not sure about that. https://github.com/pyro-ppl/pyro/issues/700#issuecomment-359984295 https://github.com/docker/docker-py/issues/1714 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] KarmaGYZ commented on issue #10746: [FLINK-15417] Remove the docker volume or mount when starting Mesos e…
KarmaGYZ commented on issue #10746: [FLINK-15417] Remove the docker volume or mount when starting Mesos e… URL: https://github.com/apache/flink/pull/10746#issuecomment-573974153 > will we always run into permission issues Sorry, I'm not sure about this. At least so far, I could not figure out what happens in [FLINK-15377](https://issues.apache.org/jira/browse/FLINK-15377) cause the permission issue. It always runs well on my local environment(MacOS). But I think this PR could prevent that happening again. > Another approach I have found is described here It sounds good, I give a try on this approach. However, I meet `addgroup: The GID `20' is already in use` issue. It seems this approach does not run well with MacOS. I'm not sure about that. https://github.com/pyro-ppl/pyro/issues/700#issuecomment-359984295 https://github.com/docker/docker-py/issues/1714 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #10848: [FLINK-15307][failover]Rename Subclasses of FailoverStrategy
flinkbot edited a comment on issue #10848: [FLINK-15307][failover]Rename Subclasses of FailoverStrategy URL: https://github.com/apache/flink/pull/10848#issuecomment-573958003 ## CI report: * a3f83df73af74cf8509f084e860ab88e63129ac8 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/144251297) Azure: [PENDING](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4310) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #10815: [FLINK-15537][table-planner-blink] Type of keys should be `BinaryRow`…
flinkbot edited a comment on issue #10815: [FLINK-15537][table-planner-blink] Type of keys should be `BinaryRow`… URL: https://github.com/apache/flink/pull/10815#issuecomment-572566376 ## CI report: * 19a4290f709495491fe460037c8c31d106984ea8 Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/143732723) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4229) * c3ef5ea345a343170806de8112163edb7df31f69 Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/144110200) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4284) * 941a5d4725dee3317ca05f8ab16eb103f61d3fcb UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #10814: [FLINK-15434][Tests]Fix unstable tests in JobMasterTest
flinkbot edited a comment on issue #10814: [FLINK-15434][Tests]Fix unstable tests in JobMasterTest URL: https://github.com/apache/flink/pull/10814#issuecomment-572552628 ## CI report: * a814428b99636921abf55ab089e664cec9d13876 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/143727769) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4228) * 96a81a9b98ba96eb075c9b5f32fd15a6e08082ab UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-15546) Obscure error message from ScalarOperatorGens::generateCast
[
https://issues.apache.org/jira/browse/FLINK-15546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17014818#comment-17014818
]
Zhenghua Gao commented on FLINK-15546:
--
{code:java}
case (from, to) =>
throw new CodeGenException(s"Unsupported cast from '$from' to '$to'.")
{code}
I think we should use *operand.resultType* and *targetType* instead in the
error message since from and to are *LogicalTypeRoot*
> Obscure error message from ScalarOperatorGens::generateCast
> ---
>
> Key: FLINK-15546
> URL: https://issues.apache.org/jira/browse/FLINK-15546
> Project: Flink
> Issue Type: Improvement
> Components: Table SQL / API
>Reporter: Rui Li
>Priority: Minor
>
> Consider the following case:
> {noformat}
> Flink SQL> describe foo;
> root
> |-- x: ROW<`f1` DOUBLE, `f2` VARCHAR(10)>
> Flink SQL> insert into foo select row(1.1,'abc');
> [INFO] Submitting SQL update statement to the cluster...
> [ERROR] Could not execute SQL statement. Reason:
> org.apache.flink.table.planner.codegen.CodeGenException: Unsupported cast
> from 'ROW' to 'ROW'.
> {noformat}
> Users are unlikely to figure out what goes wrong from the above error
> message. Something like {{Unsupported cast from 'ROW'
> to 'ROW'}} will be more helpful.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-15307) Subclasses of FailoverStrategy are easily confused with implementation classes of RestartStrategy
[ https://issues.apache.org/jira/browse/FLINK-15307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17014810#comment-17014810 ] Andrew.D.lin commented on FLINK-15307: -- [~zhuzh] sorry, I missed it, Let me modify it > Subclasses of FailoverStrategy are easily confused with implementation > classes of RestartStrategy > - > > Key: FLINK-15307 > URL: https://issues.apache.org/jira/browse/FLINK-15307 > Project: Flink > Issue Type: Improvement > Components: Runtime / Configuration >Reporter: Andrew.D.lin >Assignee: Andrew.D.lin >Priority: Minor > Labels: pull-request-available > Fix For: 1.11.0 > > Attachments: image-2019-12-18-14-59-03-181.png > > Time Spent: 10m > Remaining Estimate: 24h > > Subclasses of RestartStrategy > * FailingRestartStrategy > * FailureRateRestartStrategy > * FixedDelayRestartStrategy > * InfiniteDelayRestartStrategy > Implementation class of FailoverStrategy > * AdaptedRestartPipelinedRegionStrategyNG > * RestartAllStrategy > * RestartIndividualStrategy > * RestartPipelinedRegionStrategy > > FailoverStrategy describes how the job computation recovers from task > failures. > I think the following names may be easier to understand and easier to > distinguish： > Implementation class of FailoverStrategy > * AdaptedPipelinedRegionFailoverStrategyNG > * FailoverAllStrategy > * FailoverIndividualStrategy > * FailoverPipelinedRegionStrategy > FailoverStrategy is currently generated by configuration. If we change the > name of the implementation class, it will not affect compatibility. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-15505) flink-clients test failing on Red Hat 7.6 PowerPC Linux
[
https://issues.apache.org/jira/browse/FLINK-15505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17014805#comment-17014805
]
wangxiyuan commented on FLINK-15505:
Oh, let me check it for ppc64le
> flink-clients test failing on Red Hat 7.6 PowerPC Linux
> ---
>
> Key: FLINK-15505
> URL: https://issues.apache.org/jira/browse/FLINK-15505
> Project: Flink
> Issue Type: Bug
> Components: Tests
>Affects Versions: 1.8.3
> Environment: Red Hat 7.6 Linux ppc64le
> Java 1.8.0.232
> Maven 3.2.5
>Reporter: Ronald O. Edmark
>Priority: Critical
>
> git clone [https://github.com/apache/flink.git]
> cd flink
> git checkout release-1.8.3
> mvn clean package
>
> [ERROR] SqlExpressionTest>ExpressionTestBase.evaluateExprs:170 Wrong result
> for: [LOG(3,27)] optimized to: [3.0004440E0] expected:<3.0[]> but
> was:<3.0[004]>
> [ERROR] SqlExpressionTest>ExpressionTestBase.evaluateExprs:171 Wrong result
> for: [EXP(1)] optimized to: [2.7182818284590455348E0]
> expected:<2.718281828459045[]> but was:<2.718281828459045[5]>
>
> Edit file and removed the two tests worked.
> ./flink-table/flink-table-planner/src/test/scala/org/apache/flink/table/expressions/SqlExpressionTest.scala
> testSqlApi("LOG(3,27)", "3.0")
> testSqlApi("EXP(1)", "2.718281828459045")
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-15574) DataType to LogicalType conversion issue
[
https://issues.apache.org/jira/browse/FLINK-15574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17014804#comment-17014804
]
Zhenghua Gao commented on FLINK-15574:
--
HI [~b.hanotte], a reason to keep this deprecated conversion logic is to bypass
conversion logic in flink-table-common module. In blink we introduce some
runtime typeinformation(e.g. DecimalTypeInfo, LegacyTimestampTypeInfo, etc) to
support things like precision.
Could you update your test to a meaningful scenarios so that we can check
whether there is a potential bug in blink's conversion logic?
> DataType to LogicalType conversion issue
>
>
> Key: FLINK-15574
> URL: https://issues.apache.org/jira/browse/FLINK-15574
> Project: Flink
> Issue Type: Improvement
> Components: Table SQL / Runtime
>Reporter: Benoit Hanotte
>Priority: Major
> Labels: pull-request-available
> Attachments: 0001-FLINK-15574-Add-unit-test-to-reproduce-issue.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> We seem to be encountering an issue with the conversion from DataType to
> LogicalType with the Blink planner (full stacktrace below):
> {code}
> org.apache.flink.table.api.ValidationException: Type
> LEGACY(BasicArrayTypeInfo) of table field 'my_array' does not match
> with type BasicArrayTypeInfo of the field 'my_array' of the
> TableSource return type.
> {code}
> It seems there exists 2 paths to do the conversion from DataType to
> LogicalType:
> 1. TypeConversions.fromLegacyInfoToDataType():
> used for instance when calling TableSchema.fromTypeInformation().
> 2. LogicalTypeDataTypeConverter.fromDataTypeToLogicalType():
> Deprecated but still used in TableSourceUtil and many other places.
> These 2 code paths can return a different LogicalType for the same input,
> leading to issues when the LogicalTypes are compared to ensure they are
> compatible. For instance, PlannerTypeUtils.isAssignable() returns false for
> a DataType created from BasicArrayTypeInfo (leading to the
> ValidationException above).
> The full stacktrace is the following:
> {code}
> org.apache.flink.table.api.ValidationException: Type
> LEGACY(BasicArrayTypeInfo) of table field 'my_array' does not match
> with type BasicArrayTypeInfo of the field 'my_array' of the
> TableSource return type.
> at
> org.apache.flink.table.planner.sources.TableSourceUtil$$anonfun$4.apply(TableSourceUtil.scala:121)
> at
> org.apache.flink.table.planner.sources.TableSourceUtil$$anonfun$4.apply(TableSourceUtil.scala:92)
> at
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
> at
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
> at
> scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
> at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:186)
> at scala.collection.TraversableLike$class.map(TraversableLike.scala:234)
> at scala.collection.mutable.ArrayOps$ofRef.map(ArrayOps.scala:186)
> at
> org.apache.flink.table.planner.sources.TableSourceUtil$.computeIndexMapping(TableSourceUtil.scala:92)
> at
> org.apache.flink.table.planner.plan.nodes.physical.stream.StreamExecTableSourceScan.translateToPlanInternal(StreamExecTableSourceScan.scala:100)
> at
> org.apache.flink.table.planner.plan.nodes.physical.stream.StreamExecTableSourceScan.translateToPlanInternal(StreamExecTableSourceScan.scala:55)
> at
> org.apache.flink.table.planner.plan.nodes.exec.ExecNode$class.translateToPlan(ExecNode.scala:54)
> at
> org.apache.flink.table.planner.plan.nodes.physical.stream.StreamExecTableSourceScan.translateToPlan(StreamExecTableSourceScan.scala:55)
> at
> org.apache.flink.table.planner.plan.nodes.physical.stream.StreamExecCalc.translateToPlanInternal(StreamExecCalc.scala:86)
> at
> org.apache.flink.table.planner.plan.nodes.physical.stream.StreamExecCalc.translateToPlanInternal(StreamExecCalc.scala:46)
> at
> org.apache.flink.table.planner.plan.nodes.exec.ExecNode$class.translateToPlan(ExecNode.scala:54)
> at
> org.apache.flink.table.planner.plan.nodes.physical.stream.StreamExecCalc.translateToPlan(StreamExecCalc.scala:46)
> at
> org.apache.flink.table.planner.plan.nodes.physical.stream.StreamExecUnion$$anonfun$2.apply(StreamExecUnion.scala:86)
> at
> org.apache.flink.table.planner.plan.nodes.physical.stream.StreamExecUnion$$anonfun$2.apply(StreamExecUnion.scala:86)
> at
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
> at
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
> at
> scala.collection.mutable.ResizableArray$class.foreach(ResizableArra
[GitHub] [flink] JingsongLi commented on a change in pull request #10839: [FLINK-15567][table][docs] Add documentation for INSERT statements for Flink SQL
JingsongLi commented on a change in pull request #10839:
[FLINK-15567][table][docs] Add documentation for INSERT statements for Flink SQL
URL: https://github.com/apache/flink/pull/10839#discussion_r366125955
##
File path: docs/dev/table/sql/insert.md
##
@@ -0,0 +1,173 @@
+---
+title: "INSERT Statement"
+nav-parent_id: sql
+nav-pos: 5
+---
+
+
+* This will be replaced by the TOC
+{:toc}
+
+INSERT statements are used to add rows to a table.
+
+## Run an INSERT statement
+
+INSERT statements are specified with the `sqlUpdate()` method of the
`TableEnvironment` or executed in [SQL CLI]({{ site.baseurl
}}/dev/table/sqlClient.html). The method `sqlUpdate()` for INSERT statements is
a lazy execution, they will be executed only when
`TableEnvironment.execute(jobName)` is invoked.
Review comment:
ok
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
