[jira] [Updated] (FLINK-14606) Simplify params of Execution#processFail
[
https://issues.apache.org/jira/browse/FLINK-14606?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhu Zhu updated FLINK-14606:
Fix Version/s: (was: 1.10.0)
1.11.0
> Simplify params of Execution#processFail
>
>
> Key: FLINK-14606
> URL: https://issues.apache.org/jira/browse/FLINK-14606
> Project: Flink
> Issue Type: Sub-task
> Components: Runtime / Coordination
>Affects Versions: 1.10.0
>Reporter: Zhu Zhu
>Priority: Major
> Fix For: 1.11.0
>
>
> The 3 params fromSchedulerNg/releasePartitions/isCallback of
> Execution#processFail are quite a mess while they seem to be correlated.
> I'd propose to simplify the prams of processFail by using a
> {{isInternalError}} to replace those 3 params. {{isInternalError}} is true
> iff the failure is from TM(strictly speaking, notified from SchedulerBase).
> This also hardens the handling of cases that a task is successfully deployed
> but JM does not realize it(see #3 below).
> Here's why these 3 params can be simplified:
> 1. {{fromSchedulerNg}}, true iff the failure is from TM and
> isLegacyScheduling==false.
> It's only used like this: {{if (!fromSchedulerNg &&
> !isLegacyScheduling()))}}. So it's the same to use {{!isInternalFailure}} to
> replace it.
> 2. {{releasePartitions}}, true iff the failure is from TM.
> Now the value is exactly the same as {{isInternalFailure}}, we can drop it
> and use {{isInternalFailure}} instead.
> 3. {{isCallback}}, true iff the failure is from TM or the task is not
> deployed.
> It's only used like this: {{(!isCallback && (current == RUNNING ||
> current == DEPLOYING))}}.
> So using {{!isInternalFailure}} to replace it would be enough. It is a
> bit different for the case that a task deployment to a task manager fails,
> which set {{isCallback}} to true previously. However, it would be safer to
> signal a cancel call, in case the deployment is actually a success but the
> response is lost on network.
> cc [~GJL]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-14234) All partition consumable events should be notified to SchedulingStrategy (SchedulerNG)
[
https://issues.apache.org/jira/browse/FLINK-14234?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhu Zhu updated FLINK-14234:
Fix Version/s: (was: 1.10.0)
1.11.0
> All partition consumable events should be notified to SchedulingStrategy
> (SchedulerNG)
> --
>
> Key: FLINK-14234
> URL: https://issues.apache.org/jira/browse/FLINK-14234
> Project: Flink
> Issue Type: Sub-task
> Components: Runtime / Coordination
>Affects Versions: 1.10.0
>Reporter: Zhu Zhu
>Priority: Major
> Fix For: 1.11.0
>
>
> {{SchedulingStrategy}} requires partition consumable notification to make
> scheduling decisions.
> According to {{SchedulingStrategy#onPartitionConsumable}} definition, all
> partition consumable events should be notified to {{SchedulingStrategy}},
> including those from TMs (pipelined partitions consumable for data produced)
> and from within JM(blocking partitions consumable for producer finished).
> More details see
> [here|https://github.com/apache/flink/pull/9663#discussion_r326540913].
> This can be helpful for a more flexible scheduling.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-14233) All task state changes should be notified to SchedulingStrategy (SchedulerNG)
[
https://issues.apache.org/jira/browse/FLINK-14233?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhu Zhu updated FLINK-14233:
Fix Version/s: (was: 1.10.0)
1.11.0
> All task state changes should be notified to SchedulingStrategy (SchedulerNG)
> -
>
> Key: FLINK-14233
> URL: https://issues.apache.org/jira/browse/FLINK-14233
> Project: Flink
> Issue Type: Sub-task
> Components: Runtime / Coordination
>Affects Versions: 1.10.0
>Reporter: Zhu Zhu
>Priority: Major
> Fix For: 1.11.0
>
>
> {{SchedulingStrategy}} requires task states to make scheduling decisions.
> According to {{SchedulingStrategy#onExecutionStateChange}} definition, all
> task state changes should be notified to {{SchedulingStrategy}}, including
> those from TMs and from within JM.
> This can be helpful for a more flexible scheduling.
> In the worst case that some states are not supposed to be notified for
> certain concerns, we should document it in the interface.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-14162) Unify SchedulerOperations#allocateSlotsAndDeploy implementation for all scheduling strategies
[
https://issues.apache.org/jira/browse/FLINK-14162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhu Zhu updated FLINK-14162:
Fix Version/s: (was: 1.10.0)
1.11.0
> Unify SchedulerOperations#allocateSlotsAndDeploy implementation for all
> scheduling strategies
> -
>
> Key: FLINK-14162
> URL: https://issues.apache.org/jira/browse/FLINK-14162
> Project: Flink
> Issue Type: Sub-task
> Components: Runtime / Coordination
>Affects Versions: 1.10.0
>Reporter: Zhu Zhu
>Priority: Major
> Fix For: 1.11.0
>
>
> In scheduler NG, scheduling strategies invokes
> {{SchedulerOperations#allocateSlotsAndDeploy(Collection)}}
> to trigger scheduling of tasks.
> However, {{EagerSchedulingStrategy}} and
> {{LazyFromSourcesSchedulingStrategy}} both invokes it by passing a batch of
> tasks, but requires the scheduling process to be conducted in 2 different
> ways:
> * {{EagerSchedulingStrategy}} requires the batch of tasks to deploy after
> all of them have acquired slots. This is essential to avoid partition update
> RPCs in streaming job scheduling.
> * {{LazyFromSourcesSchedulingStrategy}} requires tasks in the batch to
> allocate slot and get deployed individually, so that it can deploy a few
> tasks even if the slots is not enough for all tasks in the batch. This is
> helpful for batch job scheduling.
> The scheduler then have to decide the scheduling pattern based whether the
> scheduling strategy is a {{LazyFromSourcesSchedulingStrategy}}. This is not
> good, as there can be more strategies in the future, and even customized
> scheduling strategies.
> I think it's better to define the
> {{SchedulerOperations#allocateSlotsAndDeploy(Collection)}}
> to be that all tasks in the batch need to be assigned and deployed together,
> like what we do for {{EagerSchedulingStrategy}}.
> All scheduling strategies need to follow this rule. If tasks should be
> scheduled individually, the strategy should invoke {{allocateSlotsAndDeploy}}
> multiple times, one for each task. As a result, the
> {{LazyFromSourcesSchedulingStrategy}} needs to be adjusted for that.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-13437) Add Hive SQL E2E test
[ https://issues.apache.org/jira/browse/FLINK-13437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16992208#comment-16992208 ] Kurt Young commented on FLINK-13437: [~liyu] We will try to do this in 1.10 > Add Hive SQL E2E test > - > > Key: FLINK-13437 > URL: https://issues.apache.org/jira/browse/FLINK-13437 > Project: Flink > Issue Type: Test > Components: Connectors / Hive, Tests >Affects Versions: 1.9.0 >Reporter: Till Rohrmann >Assignee: Jingsong Lee >Priority: Major > Fix For: 1.10.0 > > > We should add an E2E test for the Hive integration: List all tables and read > some metadata, read from an existing table, register a new table in Hive, use > a registered function, write to an existing table, write to a new table. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-14032) Make the cache size of RocksDBPriorityQueueSetFactory configurable
[
https://issues.apache.org/jira/browse/FLINK-14032?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yu Li updated FLINK-14032:
--
Fix Version/s: (was: 1.10.0)
1.11.0
Change fix version to 1.11.0 since we already reached feature freeze for 1.10.0
> Make the cache size of RocksDBPriorityQueueSetFactory configurable
> --
>
> Key: FLINK-14032
> URL: https://issues.apache.org/jira/browse/FLINK-14032
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / State Backends
>Reporter: Yun Tang
>Priority: Major
> Fix For: 1.11.0
>
>
> Currently, the cache size of {{RocksDBPriorityQueueSetFactory}} has been set
> as 128 and no any ways to configure this to other value. (We could increase
> this to obtain better performance if necessary). Actually, this is also a
> TODO for quiet a long time.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (FLINK-13437) Add Hive SQL E2E test
[ https://issues.apache.org/jira/browse/FLINK-13437?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kurt Young reassigned FLINK-13437: -- Assignee: Jingsong Lee > Add Hive SQL E2E test > - > > Key: FLINK-13437 > URL: https://issues.apache.org/jira/browse/FLINK-13437 > Project: Flink > Issue Type: Test > Components: Connectors / Hive, Tests >Affects Versions: 1.9.0 >Reporter: Till Rohrmann >Assignee: Jingsong Lee >Priority: Major > > We should add an E2E test for the Hive integration: List all tables and read > some metadata, read from an existing table, register a new table in Hive, use > a registered function, write to an existing table, write to a new table. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-13437) Add Hive SQL E2E test
[ https://issues.apache.org/jira/browse/FLINK-13437?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kurt Young updated FLINK-13437: --- Fix Version/s: 1.10.0 > Add Hive SQL E2E test > - > > Key: FLINK-13437 > URL: https://issues.apache.org/jira/browse/FLINK-13437 > Project: Flink > Issue Type: Test > Components: Connectors / Hive, Tests >Affects Versions: 1.9.0 >Reporter: Till Rohrmann >Assignee: Jingsong Lee >Priority: Major > Fix For: 1.10.0 > > > We should add an E2E test for the Hive integration: List all tables and read > some metadata, read from an existing table, register a new table in Hive, use > a registered function, write to an existing table, write to a new table. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on issue #10458: [FLINK-14815][rest]Expose network metric in IOMetricsInfo
flinkbot edited a comment on issue #10458: [FLINK-14815][rest]Expose network metric in IOMetricsInfo URL: https://github.com/apache/flink/pull/10458#issuecomment-562482569 ## CI report: * 303f921f58ff61aa3e332c36ffdc07d5f9ceb352 Travis: [CANCELED](https://travis-ci.com/flink-ci/flink/builds/139659908) * 0122cc64d2c4a46ee8ba05ca67832a1589b024b1 Travis: [CANCELED](https://travis-ci.com/flink-ci/flink/builds/139667569) * 2b332510e6d2cd2dee836de532139241c20ce00c Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/139677676) * 30f1b29134a585e7d19c0de5e4ef000282a285b2 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/140158853) * 8a5207e384e995fb2f8929e5de3d87e89e801b3e Travis: [CANCELED](https://travis-ci.com/flink-ci/flink/builds/140199482) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=3348) * 54049f914f9ac88570f882317e7f497ccb09587e Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/140203351) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=3350) * 308229677cea17a602f48cce2ce9e289a4ede30a Travis: [CANCELED](https://travis-ci.com/flink-ci/flink/builds/140345632) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=3384) * 7aa5e9c2b36d3fde471ecf7d751c02a17f2b0ce4 Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/140348794) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=3386) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (FLINK-13905) Separate checkpoint triggering into stages
[
https://issues.apache.org/jira/browse/FLINK-13905?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yu Li updated FLINK-13905:
--
Fix Version/s: (was: 1.10.0)
1.11.0
Change the fix version to 1.11.0 since we already reached feature freeze for
1.10.0
> Separate checkpoint triggering into stages
> --
>
> Key: FLINK-13905
> URL: https://issues.apache.org/jira/browse/FLINK-13905
> Project: Flink
> Issue Type: Sub-task
> Components: Runtime / Checkpointing
>Reporter: Biao Liu
>Assignee: Biao Liu
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.11.0
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Currently {{CheckpointCoordinator#triggerCheckpoint}} includes some heavy IO
> operations. We plan to separate the triggering into different stages. The IO
> operations are executed in IO threads, while other on-memory operations are
> not.
> This is a preparation for making all on-memory operations of
> {{CheckpointCoordinator}} single threaded (in main thread).
> Note that we could not put on-memory operations of triggering into main
> thread directly now. Because there are still some operations on a heavy lock
> (coordinator-wide).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-14647) Improve the exception message when required property is not matched
[
https://issues.apache.org/jira/browse/FLINK-14647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16992206#comment-16992206
]
Nicholas Jiang commented on FLINK-14647:
[~jark]Yeah, my original intention was that requiredContext includes some
required properties, so add matchContext method to tell the comparsion with
required context. As your feel, your exposion of two interfaces is feasible,
and I will modify the TableFactory interface design, not match context in
interface concept.
> Improve the exception message when required property is not matched
> ---
>
> Key: FLINK-14647
> URL: https://issues.apache.org/jira/browse/FLINK-14647
> Project: Flink
> Issue Type: Sub-task
> Components: Table SQL / API
>Reporter: Jark Wu
>Priority: Major
>
> Currently, all the required properties should exist and match, otherwise,
> {{NoMatchingTableFactoryException}} will be thrown.
> For example, if we have {{connector.type=hbase, connector.versions=1.1.1}},
> the following exception will be thrown.
> {code}
> org.apache.flink.table.api.NoMatchingTableFactoryException: Could not find a
> suitable table factory for 'org.apache.flink.addons.hbase.HBaseTableFactory'
> in
> the classpath.
> Reason: No context matches.
> The following properties are requested:
> connector.type=hbase
> connector.version=1.1.1
> {code}
> It's hard to know the problem is the version is wrong. A quick fixing is move
> version out of {{requiredContext()}} if we only support one version and throw
> a readable exception in {{ConnectorDescriptorValidator#validate}}.
> However, for the multiple-version connectors, e.g. Kafka, maybe we should
> improve the design of {{TableFactory}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-13848) Support “scheduleAtFixedRate/scheduleAtFixedDelay” in RpcEndpoint#MainThreadExecutor
[
https://issues.apache.org/jira/browse/FLINK-13848?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yu Li updated FLINK-13848:
--
Fix Version/s: (was: 1.10.0)
1.11.0
Change the fix version to 1.11.0 since we already reached feature freeze for
1.10.0
> Support “scheduleAtFixedRate/scheduleAtFixedDelay” in
> RpcEndpoint#MainThreadExecutor
>
>
> Key: FLINK-13848
> URL: https://issues.apache.org/jira/browse/FLINK-13848
> Project: Flink
> Issue Type: Sub-task
> Components: Runtime / Coordination
>Reporter: Biao Liu
>Assignee: Biao Liu
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.11.0
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Currently the methods “scheduleAtFixedRate/scheduleAtFixedDelay" of
> {{RpcEndpoint#MainThreadExecutor}} are not implemented. Because there was no
> requirement on them before.
> Now we are planning to implement these methods to support periodic checkpoint
> triggering.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-15017) add a thrift server for Flink
[ https://issues.apache.org/jira/browse/FLINK-15017?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16992202#comment-16992202 ] Kurt Young commented on FLINK-15017: cc [~godfreyhe] [~docete] > add a thrift server for Flink > - > > Key: FLINK-15017 > URL: https://issues.apache.org/jira/browse/FLINK-15017 > Project: Flink > Issue Type: New Feature > Components: Client / Job Submission >Reporter: Bowen Li >Assignee: Kurt Young >Priority: Major > Fix For: 1.11.0 > > > allow jdbc/odbc clients to execute SQL queries over jdbc/odbc on Flink. > can refer to Spark thrift server, a port of Hive's HiveServer2, as example. > https://github.com/apache/spark/tree/master/sql/hive-thriftserver > https://cwiki.apache.org/confluence/display/Hive/HiveServer2+Overview -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-13876) Remove ExecutionConfig field from PojoSerializer
[
https://issues.apache.org/jira/browse/FLINK-13876?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yu Li updated FLINK-13876:
--
Fix Version/s: (was: 1.10.0)
1.11.0
Affects Version/s: 1.10.0
Since we already reached 1.10.0 feature freeze and this issue seems to be a big
change/improvement instead of bug fix or document change, change the fix
version to 1.11.0.
> Remove ExecutionConfig field from PojoSerializer
>
>
> Key: FLINK-13876
> URL: https://issues.apache.org/jira/browse/FLINK-13876
> Project: Flink
> Issue Type: Improvement
> Components: API / Type Serialization System
>Affects Versions: 1.7.2, 1.8.1, 1.9.0, 1.10.0
>Reporter: Dawid Wysakowicz
>Priority: Major
> Fix For: 1.11.0
>
>
> The PojoSerializers stores an instance of ExecutionConfig as internal field,
> even though that the only information it may ever need are the registered
> kryo serializers.
> This has a few drawbacks:
> * It blocks the evolution of {{ExecutionConfig}} as serializers where stored
> in a state. Therefore any change to ExecutionConfig must be backwards
> compatible in respect to java serialization
> * It probably already introduced a bug, as upon restore the Snapshot actually
> recreates the serializer with an empty ExecutionConfig (see
> org.apache.flink.api.java.typeutils.runtime.PojoSerializerSnapshot#restoreSerializer)
> I suggest to remove the field completely and adjust corresponding usages.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-15122) Reusing record object in StreamTaskNetworkInput
[ https://issues.apache.org/jira/browse/FLINK-15122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16992201#comment-16992201 ] Jingsong Lee commented on FLINK-15122: -- [~pnowojski] Yes, will create a benchmark to show the improvement. (Temporarily busy 1.10 testing/bugfix) [~jark] Thanks for your information, Yes, FLIP-21 completely hits this ticket. > Reusing record object in StreamTaskNetworkInput > --- > > Key: FLINK-15122 > URL: https://issues.apache.org/jira/browse/FLINK-15122 > Project: Flink > Issue Type: Sub-task > Components: Runtime / Network >Reporter: Jingsong Lee >Priority: Major > Fix For: 1.11.0 > > > Now blink's batch is forced to open object reusing, but the data read from > the network is not reused, which will lead to a large GC of the batch job. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-13866) develop testing plan for many Hive versions that we support
[ https://issues.apache.org/jira/browse/FLINK-13866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16992199#comment-16992199 ] Yu Li commented on FLINK-13866: --- [~phoenixjiangnan] [~xuefuz] Are we still targeting this work at 1.10? Thanks. > develop testing plan for many Hive versions that we support > --- > > Key: FLINK-13866 > URL: https://issues.apache.org/jira/browse/FLINK-13866 > Project: Flink > Issue Type: Test > Components: Connectors / Hive >Reporter: Bowen Li >Assignee: Xuefu Zhang >Priority: Major > Fix For: 1.10.0 > > > with FLINK-13841, we will start to support quite a few hive versions, let > alone other major versions like 1.1, 2.2, and 3.x. > We need to come up with a testing plan to cover all these Hive versions to > guarantee 1) help identify and fix breaking changes ASAP, 2) minimize > developers' efforts in manually test and maintain compatibilities of all > these Hive versions, and automate as much as possible. > Set it to 1.10.0 for now. > cc [~xuefuz] [~lirui] [~Terry1897] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-13857) Remove remaining UdfAnalyzer configurations
[ https://issues.apache.org/jira/browse/FLINK-13857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16992198#comment-16992198 ] Yu Li commented on FLINK-13857: --- [~dwysakowicz] Just to confirm, are we still targeting at completing this work in 1.10.0? Thanks. > Remove remaining UdfAnalyzer configurations > --- > > Key: FLINK-13857 > URL: https://issues.apache.org/jira/browse/FLINK-13857 > Project: Flink > Issue Type: Improvement > Components: Runtime / Configuration >Reporter: Dawid Wysakowicz >Assignee: Dawid Wysakowicz >Priority: Major > Fix For: 1.10.0 > > > The UdfAnalyzer code was dropped in 1.9 release. A few configuration > classes/options were marked as deprecated as part of this effort. Having in > mind that they take no effect at all and were deprecated in 1.9 release I > suggest to drop them in 1.10 release. > It also does not break binary compatibility as all the classes were marked > with PublicEvolving from the very beginning. > I suggest to drop: > * CodeAnalysisMode > * ExecutionConfig#get/setCodeAnalysisMode > * SkipCodeAnalysis -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] KarmaGYZ commented on issue #10348: [FLINK-14951][tests] Harden the thread safety of State TTL backend tests
KarmaGYZ commented on issue #10348: [FLINK-14951][tests] Harden the thread safety of State TTL backend tests URL: https://github.com/apache/flink/pull/10348#issuecomment-563847973 @flinkbot run travis This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (FLINK-13692) Make CompletedCheckpointStore backwards compatible?
[
https://issues.apache.org/jira/browse/FLINK-13692?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yu Li updated FLINK-13692:
--
Fix Version/s: (was: 1.10.0)
1.11.0
Change fix version to 1.11.0 since we already reached feature freeze for 1.10.0
> Make CompletedCheckpointStore backwards compatible?
> ---
>
> Key: FLINK-13692
> URL: https://issues.apache.org/jira/browse/FLINK-13692
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Checkpointing
>Affects Versions: 1.9.0, 1.10.0
>Reporter: Till Rohrmann
>Priority: Major
> Fix For: 1.9.2, 1.11.0
>
>
> With FLINK-11159 we changed the signature of
> {{CompletedCheckpointStore#getLatestCheckpoint}} from
> {{getLatestCheckpoint()}} to {{getLatestCheckpoint(boolean)}}. This is an API
> breaking change which causes users of this interface having to adapt their
> implementation and recompile it. Since {{CompletedCheckpointStore}} is no
> public API, this change is technically ok, but not so nice.
> If we decide to provide backwards compatibility, then we should still keep
> {{getLatestCheckpoint()}} and add a deprecation annotation to it. At least,
> we should update the JavaDocs of the
> {{CompletedCheckpointStore#getLatestCheckpoint}} method to reflect what the
> passed parameter states.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-13437) Add Hive SQL E2E test
[ https://issues.apache.org/jira/browse/FLINK-13437?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kurt Young updated FLINK-13437: --- Fix Version/s: (was: 1.10.0) > Add Hive SQL E2E test > - > > Key: FLINK-13437 > URL: https://issues.apache.org/jira/browse/FLINK-13437 > Project: Flink > Issue Type: Test > Components: Connectors / Hive, Tests >Affects Versions: 1.9.0 >Reporter: Till Rohrmann >Priority: Major > > We should add an E2E test for the Hive integration: List all tables and read > some metadata, read from an existing table, register a new table in Hive, use > a registered function, write to an existing table, write to a new table. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-13437) Add Hive SQL E2E test
[ https://issues.apache.org/jira/browse/FLINK-13437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16992194#comment-16992194 ] Yu Li commented on FLINK-13437: --- [~ykt836] [~xuefuz] Could you help confirm whether we still target at completing this work in 1.10.0? Thanks. > Add Hive SQL E2E test > - > > Key: FLINK-13437 > URL: https://issues.apache.org/jira/browse/FLINK-13437 > Project: Flink > Issue Type: Test > Components: Connectors / Hive, Tests >Affects Versions: 1.9.0 >Reporter: Till Rohrmann >Priority: Major > Fix For: 1.10.0 > > > We should add an E2E test for the Hive integration: List all tables and read > some metadata, read from an existing table, register a new table in Hive, use > a registered function, write to an existing table, write to a new table. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-13015) Create validators, strategies and transformations required for porting logical expressions
[ https://issues.apache.org/jira/browse/FLINK-13015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16992192#comment-16992192 ] Yu Li commented on FLINK-13015: --- [~dwysakowicz] It seems to me the relative PR is closed with changes merged. Is there any work left for this issue? If so, any concern if we change the fix version to 1.11.0? If not, could you close this JIRA? Thanks. > Create validators, strategies and transformations required for porting > logical expressions > -- > > Key: FLINK-13015 > URL: https://issues.apache.org/jira/browse/FLINK-13015 > Project: Flink > Issue Type: New Feature > Components: Table SQL / API >Reporter: Dawid Wysakowicz >Assignee: Dawid Wysakowicz >Priority: Major > Labels: pull-request-available > Fix For: 1.10.0 > > Time Spent: 20m > Remaining Estimate: 0h > > The goal of this task is to implement : > InputTypeValidator: > * by type root > TypeStrategies: > * cascade > * explicit > TypeTransformations: > * to_nullable > This set of classes will enable porting > AND/OR/NOT/IS_TRUE/IS_FALSE/IS_NOT_TRUE/IS_NOT_FALSE to new type inference > stack. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on issue #10458: [FLINK-14815][rest]Expose network metric in IOMetricsInfo
flinkbot edited a comment on issue #10458: [FLINK-14815][rest]Expose network metric in IOMetricsInfo URL: https://github.com/apache/flink/pull/10458#issuecomment-562482569 ## CI report: * 303f921f58ff61aa3e332c36ffdc07d5f9ceb352 Travis: [CANCELED](https://travis-ci.com/flink-ci/flink/builds/139659908) * 0122cc64d2c4a46ee8ba05ca67832a1589b024b1 Travis: [CANCELED](https://travis-ci.com/flink-ci/flink/builds/139667569) * 2b332510e6d2cd2dee836de532139241c20ce00c Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/139677676) * 30f1b29134a585e7d19c0de5e4ef000282a285b2 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/140158853) * 8a5207e384e995fb2f8929e5de3d87e89e801b3e Travis: [CANCELED](https://travis-ci.com/flink-ci/flink/builds/140199482) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=3348) * 54049f914f9ac88570f882317e7f497ccb09587e Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/140203351) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=3350) * 308229677cea17a602f48cce2ce9e289a4ede30a Travis: [CANCELED](https://travis-ci.com/flink-ci/flink/builds/140345632) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=3384) * 7aa5e9c2b36d3fde471ecf7d751c02a17f2b0ce4 Travis: [PENDING](https://travis-ci.com/flink-ci/flink/builds/140348794) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=3386) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-14953) Parquet table source should use schema type to build FilterPredicate
[ https://issues.apache.org/jira/browse/FLINK-14953?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16992190#comment-16992190 ] Zhenqiu Huang commented on FLINK-14953: --- [~ykt836] Would you please help to merge this PR https://github.com/apache/flink/pull/10371? > Parquet table source should use schema type to build FilterPredicate > > > Key: FLINK-14953 > URL: https://issues.apache.org/jira/browse/FLINK-14953 > Project: Flink > Issue Type: Bug > Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile) >Affects Versions: 1.8.0, 1.8.2, 1.9.0, 1.9.1 >Reporter: Zhenqiu Huang >Assignee: Zhenqiu Huang >Priority: Major > Fix For: 1.10.0, 1.9.2 > > > The issue happens when the data type of value in predicate inferred from SQL > doesn't match the parquet schema. For example, foo is a long type, foo < 1 is > the predicate. Literal will be recognized as an integration. It causes the > parquet FilterPredicate is mistakenly created for the column of Integer type. > Then, the exception comes. > java.lang.UnsupportedOperationException > at > org.apache.parquet.filter2.recordlevel.IncrementallyUpdatedFilterPredicate$ValueInspector.update(IncrementallyUpdatedFilterPredicate.java:71) > at > org.apache.parquet.filter2.recordlevel.FilteringPrimitiveConverter.addLong(FilteringPrimitiveConverter.java:105) > at > org.apache.parquet.column.impl.ColumnReaderImpl$2$4.writeValue(ColumnReaderImpl.java:268) > at > org.apache.parquet.column.impl.ColumnReaderImpl.writeCurrentValueToConverter(ColumnReaderImpl.java:367) > at > org.apache.parquet.io.RecordReaderImplementation.read(RecordReaderImplementation.java:406) > at > org.apache.flink.formats.parquet.utils.ParquetRecordReader.readNextRecord(ParquetRecordReader.java:235) > at > org.apache.flink.formats.parquet.utils.ParquetRecordReader.reachEnd(ParquetRecordReader.java:207) > at > org.apache.flink.formats.parquet.ParquetInputFormat.reachedEnd(ParquetInputFormat.java:233) > at > org.apache.flink.api.common.operators.GenericDataSourceBase.executeOnCollections(GenericDataSourceBase.java:231) > at > org.apache.flink.api.common.operators.CollectionExecutor.executeDataSource(CollectionExecutor.java:219) > at > org.apache.flink.api.common.operators.CollectionExecutor.execute(CollectionExecutor.java:155) > at > org.apache.flink.api.common.operators.CollectionExecutor.executeUnaryOperator(CollectionExecutor.java:229) > at > org.apache.flink.api.common.operators.CollectionExecutor.execute(CollectionExecutor.java:149) > at > org.apache.flink.api.common.operators.CollectionExecutor.execute(CollectionExecutor.java:131) > at > org.apache.flink.api.common.operators.CollectionExecutor.executeDataSink(CollectionExecutor.java:182) > at > org.apache.flink.api.common.operators.CollectionExecutor.execute(CollectionExecutor.java:158) > at > org.apache.flink.api.common.operators.CollectionExecutor.execute(CollectionExecutor.java:131) > at > org.apache.flink.api.common.operators.CollectionExecutor.execute(CollectionExecutor.java:115) > at > org.apache.flink.api.java.CollectionEnvironment.execute(CollectionEnvironment.java:38) > at > org.apache.flink.test.util.CollectionTestEnvironment.execute(CollectionTestEnvironment.java:52) > at > org.apache.flink.test.util.CollectionTestEnvironment.execute(CollectionTestEnvironment.java:47) > at org.apache.flink.api.java.DataSet.collect(DataSet.java:413) > at > org.apache.flink.formats.parquet.ParquetTableSourceITCase.testScanWithProjectionAndFilter(ParquetTableSourceITCase.java:91) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) > at > org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) > at > org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) > at > org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) > at > org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) > at > org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) > at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55) > at org.junit.rules.RunRules.evaluate(RunRules.java:20) > at
[GitHub] [flink] flinkbot edited a comment on issue #10504: [FLINK-15069][benchmark] Supplement the pipelined shuffle compression case for benchmark
flinkbot edited a comment on issue #10504: [FLINK-15069][benchmark] Supplement the pipelined shuffle compression case for benchmark URL: https://github.com/apache/flink/pull/10504#issuecomment-563761500 ## CI report: * 64f397f5a4fdb5cf37119c1251e7a553d5c4da0f Travis: [PENDING](https://travis-ci.com/flink-ci/flink/builds/140352443) Azure: [PENDING](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=3387) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #10503: [FLINK-15137][avro] Improve schema derivation for Avro format
flinkbot edited a comment on issue #10503: [FLINK-15137][avro] Improve schema derivation for Avro format URL: https://github.com/apache/flink/pull/10503#issuecomment-563692047 ## CI report: * 0a63ef8576f25cdec9fe106d7f69429fff6a4c7e Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/140348792) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=3385) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-15122) Reusing record object in StreamTaskNetworkInput
[ https://issues.apache.org/jira/browse/FLINK-15122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16992181#comment-16992181 ] Jark Wu commented on FLINK-15122: - In general object copy is very expensive. Providing a benchmark to show how much perfromance improvement can we get is great. This topic has already been discussed serveral times in the community, and there is a suspending FLIP-21 which is related to this issue. See https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=71012982 > Reusing record object in StreamTaskNetworkInput > --- > > Key: FLINK-15122 > URL: https://issues.apache.org/jira/browse/FLINK-15122 > Project: Flink > Issue Type: Sub-task > Components: Runtime / Network >Reporter: Jingsong Lee >Priority: Major > Fix For: 1.11.0 > > > Now blink's batch is forced to open object reusing, but the data read from > the network is not reused, which will lead to a large GC of the batch job. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot commented on issue #10504: [FLINK-15069][benchmark] Supplement the pipelined shuffle compression case for benchmark
flinkbot commented on issue #10504: [FLINK-15069][benchmark] Supplement the pipelined shuffle compression case for benchmark URL: https://github.com/apache/flink/pull/10504#issuecomment-563761500 ## CI report: * 64f397f5a4fdb5cf37119c1251e7a553d5c4da0f UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #10503: [FLINK-15137][avro] Improve schema derivation for Avro format
flinkbot edited a comment on issue #10503: [FLINK-15137][avro] Improve schema derivation for Avro format URL: https://github.com/apache/flink/pull/10503#issuecomment-563692047 ## CI report: * 0a63ef8576f25cdec9fe106d7f69429fff6a4c7e Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/140348792) Azure: [PENDING](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=3385) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (FLINK-14645) Data types defined in DDL will lose precision and nullability when converting to properties
[
https://issues.apache.org/jira/browse/FLINK-14645?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jark Wu updated FLINK-14645:
Release Note: The {{TableSchema}} derived from properties now bridges to
the default conversion class, i.e. LocalDateTime/LocalDate/LocalTime, instead
of java.sql.Timestamp/Date/Time. All the connectors consume/produce the
java.sql.Timestamp/Date/Time classes should adapt to the new conversion
classes.
> Data types defined in DDL will lose precision and nullability when converting
> to properties
> ---
>
> Key: FLINK-14645
> URL: https://issues.apache.org/jira/browse/FLINK-14645
> Project: Flink
> Issue Type: Sub-task
> Components: Table SQL / API
>Reporter: Jark Wu
>Assignee: Jark Wu
>Priority: Critical
> Labels: pull-request-available
> Fix For: 1.10.0
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Currently, data types defined in DDL will be converted to {{TypeInformation}}
> and use {{TypeStringUtils}} to serialize/deserialize which will lose the
> precision and nullablitiy information.
> We can use {{LogicalType#asSerializableString}} and {{LogicalTypeParser}} to
> serialize/deserialize data types which keeps all the information. But we need
> to figure out how to keep compability with previous versions.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-14645) Data types defined in DDL will lose precision and nullability when converting to properties
[
https://issues.apache.org/jira/browse/FLINK-14645?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jark Wu updated FLINK-14645:
Release Note: The TableSchema derived from properties now bridges to the
default conversion class, i.e. LocalDateTime/LocalDate/LocalTime, instead of
java.sql.Timestamp/Date/Time. All the connectors consume/produce the
java.sql.Timestamp/Date/Time classes should adapt to the new conversion
classes. (was: The {{TableSchema}} derived from properties now bridges to the
default conversion class, i.e. LocalDateTime/LocalDate/LocalTime, instead of
java.sql.Timestamp/Date/Time. All the connectors consume/produce the
java.sql.Timestamp/Date/Time classes should adapt to the new conversion
classes. )
> Data types defined in DDL will lose precision and nullability when converting
> to properties
> ---
>
> Key: FLINK-14645
> URL: https://issues.apache.org/jira/browse/FLINK-14645
> Project: Flink
> Issue Type: Sub-task
> Components: Table SQL / API
>Reporter: Jark Wu
>Assignee: Jark Wu
>Priority: Critical
> Labels: pull-request-available
> Fix For: 1.10.0
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Currently, data types defined in DDL will be converted to {{TypeInformation}}
> and use {{TypeStringUtils}} to serialize/deserialize which will lose the
> precision and nullablitiy information.
> We can use {{LogicalType#asSerializableString}} and {{LogicalTypeParser}} to
> serialize/deserialize data types which keeps all the information. But we need
> to figure out how to keep compability with previous versions.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot commented on issue #10504: [FLINK-15069][benchmark] Supplement the pipelined shuffle compression case for benchmark
flinkbot commented on issue #10504: [FLINK-15069][benchmark] Supplement the pipelined shuffle compression case for benchmark URL: https://github.com/apache/flink/pull/10504#issuecomment-563751655 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit 64f397f5a4fdb5cf37119c1251e7a553d5c4da0f (Tue Dec 10 04:17:22 UTC 2019) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (FLINK-14649) Flatten all the connector properties keys to make it easy to configure in DDL
[
https://issues.apache.org/jira/browse/FLINK-14649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jark Wu updated FLINK-14649:
Release Note: Some indexed connector properties have been refactored to
have a better experience in DDL statement, including Kafka
"connector.properties", "connector.specific-offsets" and Elasticsearch
"connector.hosts". Please see documentation for the description of new
properties. The old properties are deprecated and will be removed in future
versions. (was: {{PlaceHolder}})
> Flatten all the connector properties keys to make it easy to configure in DDL
> -
>
> Key: FLINK-14649
> URL: https://issues.apache.org/jira/browse/FLINK-14649
> Project: Flink
> Issue Type: Sub-task
> Components: Table SQL / API
>Reporter: Jark Wu
>Assignee: Leonard Xu
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.10.0
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> There are some connector properties are lists. For example, Kafka connector
> specific properties have to been set in the following way:
> {code}
> 'connector.properties.0.key' = 'zookeeper.connect',
> 'connector.properties.0.value' = 'localhost:2181',
> 'connector.properties.1.key' = 'bootstrap.servers',
> 'connector.properties.1.value' = 'localhost:9092',
> 'connector.properties.2.key' = 'group.id',
> 'connector.properties.2.value' = 'testGroup',
> {code}
> It is complex and not intuitive to define in this way. In order to cooperate
> with DDL better, we propose to flatten all the property keys.
> It has some disadvantage to define in this way.
> - Users need to keep track of the indices
> - The key space is not constant. Validation of keys would require prefix
> magic and wildcards. Like in TableFactories: `connector.propertie.#.key`.
> - It is complex and not intuitive to define and document.
> See FLIP-86 for the proposed new properties.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-15069) Supplement the pipelined shuffle compression case for benchmark
[ https://issues.apache.org/jira/browse/FLINK-15069?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-15069: --- Labels: pull-request-available (was: ) > Supplement the pipelined shuffle compression case for benchmark > --- > > Key: FLINK-15069 > URL: https://issues.apache.org/jira/browse/FLINK-15069 > Project: Flink > Issue Type: Task > Components: Benchmarks >Reporter: zhijiang >Assignee: zhijiang >Priority: Minor > Labels: pull-request-available > > While reviewing the [PR| > [https://github.com/apache/flink/pull/10375#pullrequestreview-325193504]] of > introducing data compression for persistent storage and network shuffle, we > think it is better to also cover this scenario in the benchmark for tracing > the performance issues future. > This ticket would supplement the compression case for pipelined partition > shuffle, and the compression case for blocking partition would be added in > [FLINK-15070|https://issues.apache.org/jira/browse/FLINK-15070] > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] zhijiangW opened a new pull request #10504: [FLINK-15069][benchmark] Supplement the pipelined shuffle compression case for benchmark
zhijiangW opened a new pull request #10504: [FLINK-15069][benchmark] Supplement the pipelined shuffle compression case for benchmark URL: https://github.com/apache/flink/pull/10504 ## What is the purpose of the change *While reviewing the PR of introducing data compression for persistent storage and network shuffle, we think it is better to also cover this scenario in the benchmark for tracing the performance issues future.* *This PR would supplement the compression case for pipelined partition shuffle, and the compression cases for blocking partition would be added in FLINK-15070.* ## Brief change log - *Introduce `StreamNetworkCompressionThroughputBenchmark` for enabling the pipelined partition shuffle for streaming job.* ## Verifying this change The change is covered by `StreamNetworkCompressionThroughputBenchmarkTest` ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (yes / **no**) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (yes / **no**) - The serializers: (yes / **no** / don't know) - The runtime per-record code paths (performance sensitive): (yes / **no** / don't know) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: (yes / **no** / don't know) - The S3 file system connector: (yes / **no** / don't know) ## Documentation - Does this pull request introduce a new feature? (yes / **no**) - If yes, how is the feature documented? (**not applicable** / docs / JavaDocs / not documented) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Created] (FLINK-15166) Shuffle data compression wrongly decrease the buffer reference count.
Yingjie Cao created FLINK-15166: --- Summary: Shuffle data compression wrongly decrease the buffer reference count. Key: FLINK-15166 URL: https://issues.apache.org/jira/browse/FLINK-15166 Project: Flink Issue Type: Bug Components: Runtime / Network Affects Versions: 1.10.0 Reporter: Yingjie Cao Fix For: 1.10.0 FLINK-15140 report two relevant problems which are both triggered by broadcast partitioner, to make it more clear, I create this Jira to addresses the problems separately. For blocking shuffle compression, we recycle the compressed intermediate buffer each time after we write data out, however when the data is not compressed, the return buffer is the original buffer and should not be recycled, but we wrongly recycled it. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on issue #10458: [FLINK-14815][rest]Expose network metric in IOMetricsInfo
flinkbot edited a comment on issue #10458: [FLINK-14815][rest]Expose network metric in IOMetricsInfo URL: https://github.com/apache/flink/pull/10458#issuecomment-562482569 ## CI report: * 303f921f58ff61aa3e332c36ffdc07d5f9ceb352 Travis: [CANCELED](https://travis-ci.com/flink-ci/flink/builds/139659908) * 0122cc64d2c4a46ee8ba05ca67832a1589b024b1 Travis: [CANCELED](https://travis-ci.com/flink-ci/flink/builds/139667569) * 2b332510e6d2cd2dee836de532139241c20ce00c Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/139677676) * 30f1b29134a585e7d19c0de5e4ef000282a285b2 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/140158853) * 8a5207e384e995fb2f8929e5de3d87e89e801b3e Travis: [CANCELED](https://travis-ci.com/flink-ci/flink/builds/140199482) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=3348) * 54049f914f9ac88570f882317e7f497ccb09587e Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/140203351) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=3350) * 308229677cea17a602f48cce2ce9e289a4ede30a Travis: [CANCELED](https://travis-ci.com/flink-ci/flink/builds/140345632) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=3384) * 7aa5e9c2b36d3fde471ecf7d751c02a17f2b0ce4 Travis: [PENDING](https://travis-ci.com/flink-ci/flink/builds/140348794) Azure: [PENDING](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=3386) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (FLINK-15070) Supplement cases of blocking partition with compression for benchmark
[ https://issues.apache.org/jira/browse/FLINK-15070?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhijiang updated FLINK-15070: - Summary: Supplement cases of blocking partition with compression for benchmark (was: Supplement the case of blocking partition with compression for benchmark) > Supplement cases of blocking partition with compression for benchmark > - > > Key: FLINK-15070 > URL: https://issues.apache.org/jira/browse/FLINK-15070 > Project: Flink > Issue Type: Task > Components: Benchmarks >Reporter: zhijiang >Assignee: Haibo Sun >Priority: Minor > > ATM the benchmark only covers the case of pipelined partition used in > streaming job, so it is better to also cover the case of blocking partition > for batch job. Then we can easily trace the performance concerns for any > changes future. > This ticket would introduce the blocking partition cases for uncompressed > file, uncompressed mmap and compressed file. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-15070) Supplement the case of blocking partition with compression for benchmark
[ https://issues.apache.org/jira/browse/FLINK-15070?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhijiang updated FLINK-15070: - Description: ATM the benchmark only covers the case of pipelined partition used in streaming job, so it is better to also cover the case of blocking partition for batch job. Then we can easily trace the performance concerns for any changes future. This ticket would introduce the blocking partition cases for uncompressed file, uncompressed mmap and compressed file. was:ATM the benchmark only covers the case of pipelined partition used in streaming job, so it is better to also cover the case of blocking partition for batch job. Then we can easily trace the performance concerns for any changes future. > Supplement the case of blocking partition with compression for benchmark > > > Key: FLINK-15070 > URL: https://issues.apache.org/jira/browse/FLINK-15070 > Project: Flink > Issue Type: Task > Components: Benchmarks >Reporter: zhijiang >Assignee: Haibo Sun >Priority: Minor > > ATM the benchmark only covers the case of pipelined partition used in > streaming job, so it is better to also cover the case of blocking partition > for batch job. Then we can easily trace the performance concerns for any > changes future. > This ticket would introduce the blocking partition cases for uncompressed > file, uncompressed mmap and compressed file. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-15070) Supplement the case of blocking partition with compression for benchmark
[ https://issues.apache.org/jira/browse/FLINK-15070?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhijiang updated FLINK-15070: - Summary: Supplement the case of blocking partition with compression for benchmark (was: Supplement the case of bounded blocking partition for benchmark) > Supplement the case of blocking partition with compression for benchmark > > > Key: FLINK-15070 > URL: https://issues.apache.org/jira/browse/FLINK-15070 > Project: Flink > Issue Type: Task > Components: Benchmarks >Reporter: zhijiang >Assignee: Haibo Sun >Priority: Minor > > ATM the benchmark only covers the case of pipelined partition used in > streaming job, so it is better to also cover the case of blocking partition > for batch job. Then we can easily trace the performance concerns for any > changes future. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on issue #10348: [FLINK-14951][tests] Harden the thread safety of State TTL backend tests
flinkbot edited a comment on issue #10348: [FLINK-14951][tests] Harden the thread safety of State TTL backend tests URL: https://github.com/apache/flink/pull/10348#issuecomment-559509076 ## CI report: * 54ecee627e7036f4d150aad330b9772406a19494 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/138580810) * 3e89b2b0d85dd29a76bb10807e2959a7d2ee8295 Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/140343903) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=3383) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (FLINK-15069) Supplement the pipelined shuffle compression case for benchmark
[ https://issues.apache.org/jira/browse/FLINK-15069?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhijiang updated FLINK-15069: - Description: While reviewing the [PR| [https://github.com/apache/flink/pull/10375#pullrequestreview-325193504]] of introducing data compression for persistent storage and network shuffle, we think it is better to also cover this scenario in the benchmark for tracing the performance issues future. This ticket would supplement the compression case for pipelined partition shuffle, and the compression case for blocking partition would be added in [FLINK-15070|https://issues.apache.org/jira/browse/FLINK-15070] was: While reviewing the PR of introducing data compression for persistent storage and network shuffle, we think it is better to also cover this scenario in the benchmark for tracing the performance issues future. Refer to https://github.com/apache/flink/pull/10375#pullrequestreview-325193504 > Supplement the pipelined shuffle compression case for benchmark > --- > > Key: FLINK-15069 > URL: https://issues.apache.org/jira/browse/FLINK-15069 > Project: Flink > Issue Type: Task > Components: Benchmarks >Reporter: zhijiang >Assignee: zhijiang >Priority: Minor > > While reviewing the [PR| > [https://github.com/apache/flink/pull/10375#pullrequestreview-325193504]] of > introducing data compression for persistent storage and network shuffle, we > think it is better to also cover this scenario in the benchmark for tracing > the performance issues future. > This ticket would supplement the compression case for pipelined partition > shuffle, and the compression case for blocking partition would be added in > [FLINK-15070|https://issues.apache.org/jira/browse/FLINK-15070] > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on issue #10503: [FLINK-15137][avro] Improve schema derivation for Avro format
flinkbot edited a comment on issue #10503: [FLINK-15137][avro] Improve schema derivation for Avro format URL: https://github.com/apache/flink/pull/10503#issuecomment-563692047 ## CI report: * 0a63ef8576f25cdec9fe106d7f69429fff6a4c7e Travis: [PENDING](https://travis-ci.com/flink-ci/flink/builds/140348792) Azure: [PENDING](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=3385) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (FLINK-15069) Supplement the pipelined shuffle compression case for benchmark
[ https://issues.apache.org/jira/browse/FLINK-15069?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhijiang updated FLINK-15069: - Summary: Supplement the pipelined shuffle compression case for benchmark (was: Supplement the compression case for benchmark) > Supplement the pipelined shuffle compression case for benchmark > --- > > Key: FLINK-15069 > URL: https://issues.apache.org/jira/browse/FLINK-15069 > Project: Flink > Issue Type: Task > Components: Benchmarks >Reporter: zhijiang >Assignee: zhijiang >Priority: Minor > > While reviewing the PR of introducing data compression for persistent storage > and network shuffle, we think it is better to also cover this scenario in the > benchmark for tracing the performance issues future. > Refer to > https://github.com/apache/flink/pull/10375#pullrequestreview-325193504 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (FLINK-15069) Supplement the compression case for benchmark
[ https://issues.apache.org/jira/browse/FLINK-15069?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhijiang reassigned FLINK-15069: Assignee: zhijiang > Supplement the compression case for benchmark > - > > Key: FLINK-15069 > URL: https://issues.apache.org/jira/browse/FLINK-15069 > Project: Flink > Issue Type: Task > Components: Benchmarks >Reporter: zhijiang >Assignee: zhijiang >Priority: Minor > > While reviewing the PR of introducing data compression for persistent storage > and network shuffle, we think it is better to also cover this scenario in the > benchmark for tracing the performance issues future. > Refer to > https://github.com/apache/flink/pull/10375#pullrequestreview-325193504 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] danny0405 commented on issue #10491: [FLINK-15093][streaming-java] StreamExecutionEnvironment does not cle…
danny0405 commented on issue #10491: [FLINK-15093][streaming-java] StreamExecutionEnvironment does not cle… URL: https://github.com/apache/flink/pull/10491#issuecomment-563718341 @kl0u @aljoscha I have rebase and split this PR into 3 commits, and my travis test passes https://travis-ci.org/danny0405/flink/builds/622967784, can you take a look again, thanks for that in advance ~ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Assigned] (FLINK-15137) Improve schema derivation for Avro format
[
https://issues.apache.org/jira/browse/FLINK-15137?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jark Wu reassigned FLINK-15137:
---
Assignee: Jark Wu
> Improve schema derivation for Avro format
> -
>
> Key: FLINK-15137
> URL: https://issues.apache.org/jira/browse/FLINK-15137
> Project: Flink
> Issue Type: Sub-task
> Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile)
>Reporter: Jark Wu
>Assignee: Jark Wu
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> For JSON, CSV and OldCsv, we already supported {{derive.schema=true}} to get
> the schema from table schema. But for Avro format, a user has to pass an Avro
> schema file or define the format schema explicitly via {{avro.schema}}.
> We can think of if we can drop {{avro.schema}} and make
> {{derive.schema=true}} as the default behavior.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on issue #10458: [FLINK-14815][rest]Expose network metric in IOMetricsInfo
flinkbot edited a comment on issue #10458: [FLINK-14815][rest]Expose network metric in IOMetricsInfo URL: https://github.com/apache/flink/pull/10458#issuecomment-562482569 ## CI report: * 303f921f58ff61aa3e332c36ffdc07d5f9ceb352 Travis: [CANCELED](https://travis-ci.com/flink-ci/flink/builds/139659908) * 0122cc64d2c4a46ee8ba05ca67832a1589b024b1 Travis: [CANCELED](https://travis-ci.com/flink-ci/flink/builds/139667569) * 2b332510e6d2cd2dee836de532139241c20ce00c Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/139677676) * 30f1b29134a585e7d19c0de5e4ef000282a285b2 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/140158853) * 8a5207e384e995fb2f8929e5de3d87e89e801b3e Travis: [CANCELED](https://travis-ci.com/flink-ci/flink/builds/140199482) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=3348) * 54049f914f9ac88570f882317e7f497ccb09587e Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/140203351) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=3350) * 308229677cea17a602f48cce2ce9e289a4ede30a Travis: [PENDING](https://travis-ci.com/flink-ci/flink/builds/140345632) Azure: [PENDING](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=3384) * 7aa5e9c2b36d3fde471ecf7d751c02a17f2b0ce4 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (FLINK-15141) Using decimal type in a sink table, the result returns a not match ValidationException
[ https://issues.apache.org/jira/browse/FLINK-15141?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jark Wu updated FLINK-15141: Fix Version/s: 1.10.0 > Using decimal type in a sink table, the result returns a not match > ValidationException > --- > > Key: FLINK-15141 > URL: https://issues.apache.org/jira/browse/FLINK-15141 > Project: Flink > Issue Type: Bug > Components: Table SQL / Client >Affects Versions: 1.10.0 >Reporter: xiaojin.wy >Assignee: Jark Wu >Priority: Major > Fix For: 1.10.0 > > > The planner what I used is blink. > *The source table is:* > CREATE TABLE `aggtest` ( > a smallint, > b float > ) WITH ( > 'format.field-delimiter'='|', > 'connector.type'='filesystem', > 'format.derive-schema'='true', > > 'connector.path'='hdfs://zthdev/defender_test_data/daily/test_aggregates/sources/aggtest.csv', > 'format.type'='csv' > ); > > *The sink table is:* > CREATE TABLE `agg_decimal_res` ( > avg_107_943 DECIMAL(10, 3) > ) WITH ( > 'format.field-delimiter'='|', > 'connector.type'='filesystem', > 'format.derive-schema'='true', > > 'connector.path'='hdfs://zthdev/defender_test_data/daily/test_aggregates/test_aggregates__test_avg_cast_batch/results/agg_decimal_res.csv', > 'format.type'='csv' > ); > > *The sql is:* > INSERT INTO agg_decimal_res SELECT CAST(avg(b) AS numeric(10,3)) AS > avg_107_943 FROM aggtest; > > After execute the sql, there will be a exception appear, just like this: > [INFO] Submitting SQL update statement to the cluster... > [ERROR] Could not execute SQL statement. Reason: > org.apache.flink.table.api.ValidationException: Field types of query result > and registered TableSink > `default_catalog`.`default_database`.`agg_decimal_res1` do not match. > Query result schema: [avg_107_943: DECIMAL(10, 3)] > TableSink schema: [avg_107_943: DECIMAL(38, 18)] > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (FLINK-15141) Using decimal type in a sink table, the result returns a not match ValidationException
[ https://issues.apache.org/jira/browse/FLINK-15141?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jark Wu reassigned FLINK-15141: --- Assignee: Jark Wu > Using decimal type in a sink table, the result returns a not match > ValidationException > --- > > Key: FLINK-15141 > URL: https://issues.apache.org/jira/browse/FLINK-15141 > Project: Flink > Issue Type: Bug > Components: Table SQL / Client >Affects Versions: 1.10.0 >Reporter: xiaojin.wy >Assignee: Jark Wu >Priority: Major > > The planner what I used is blink. > *The source table is:* > CREATE TABLE `aggtest` ( > a smallint, > b float > ) WITH ( > 'format.field-delimiter'='|', > 'connector.type'='filesystem', > 'format.derive-schema'='true', > > 'connector.path'='hdfs://zthdev/defender_test_data/daily/test_aggregates/sources/aggtest.csv', > 'format.type'='csv' > ); > > *The sink table is:* > CREATE TABLE `agg_decimal_res` ( > avg_107_943 DECIMAL(10, 3) > ) WITH ( > 'format.field-delimiter'='|', > 'connector.type'='filesystem', > 'format.derive-schema'='true', > > 'connector.path'='hdfs://zthdev/defender_test_data/daily/test_aggregates/test_aggregates__test_avg_cast_batch/results/agg_decimal_res.csv', > 'format.type'='csv' > ); > > *The sql is:* > INSERT INTO agg_decimal_res SELECT CAST(avg(b) AS numeric(10,3)) AS > avg_107_943 FROM aggtest; > > After execute the sql, there will be a exception appear, just like this: > [INFO] Submitting SQL update statement to the cluster... > [ERROR] Could not execute SQL statement. Reason: > org.apache.flink.table.api.ValidationException: Field types of query result > and registered TableSink > `default_catalog`.`default_database`.`agg_decimal_res1` do not match. > Query result schema: [avg_107_943: DECIMAL(10, 3)] > TableSink schema: [avg_107_943: DECIMAL(38, 18)] > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] KurtYoung commented on a change in pull request #10494: [FLINK-13438][hive] Support date type in Hive
KurtYoung commented on a change in pull request #10494: [FLINK-13438][hive]
Support date type in Hive
URL: https://github.com/apache/flink/pull/10494#discussion_r355831171
##
File path:
flink-connectors/flink-connector-hive/src/main/java/org/apache/flink/table/catalog/hive/client/HiveShimV100.java
##
@@ -364,9 +366,29 @@ public LocalDateTime toFlinkTimestamp(Object
hiveTimestamp) {
return ((Timestamp) hiveTimestamp).toLocalDateTime();
}
+ @Override
+ public Object toHiveDate(Object flinkDate) {
+ ensureSupportedFlinkDate(flinkDate);
+ return flinkDate instanceof Date ? flinkDate :
Date.valueOf((LocalDate) flinkDate);
+ }
+
+ @Override
+ public LocalDate toFlinkDate(Object hiveDate) {
+ Preconditions.checkArgument(hiveDate instanceof Date,
+ "Expecting Hive timestamp to be an instance of
%s, but actually got %s",
Review comment:
I will check in a hotfix to fix this typo
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] wsry commented on a change in pull request #10492: [FLINK-15140][runtime] Fix shuffle data compression doesn't work with BroadcastRecordWriter.
wsry commented on a change in pull request #10492: [FLINK-15140][runtime] Fix
shuffle data compression doesn't work with BroadcastRecordWriter.
URL: https://github.com/apache/flink/pull/10492#discussion_r355830724
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/PipelinedSubpartition.java
##
@@ -169,7 +169,7 @@ BufferAndBacklog pollBuffer(boolean isLocalChannel) {
BufferConsumer bufferConsumer = buffers.peek();
buffer = bufferConsumer.build();
- if (!isLocalChannel && canBeCompressed(buffer))
{
+ if (!isLocalChannel &&
!bufferConsumer.isCopied() && canBeCompressed(buffer)) {
Review comment:
I can replace the "isCopied" or "isCopyable" with "isBufferShared" which may
be easier to understand.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Assigned] (FLINK-15123) remove uniqueKeys from FlinkStatistic in blink planner
[
https://issues.apache.org/jira/browse/FLINK-15123?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jark Wu reassigned FLINK-15123:
---
Assignee: godfrey he
> remove uniqueKeys from FlinkStatistic in blink planner
> ---
>
> Key: FLINK-15123
> URL: https://issues.apache.org/jira/browse/FLINK-15123
> Project: Flink
> Issue Type: Improvement
> Components: Table SQL / Planner
>Reporter: godfrey he
>Assignee: godfrey he
>Priority: Major
> Fix For: 1.11.0
>
> Attachments: b_5.txt
>
>
> {{uniqueKeys}} is a kind of constraint, it's unreasonable that {{uniqueKeys}}
> is a kind of statistic. so we should remove uniqueKeys from
> {{FlinkStatistic}} in blink planner. Some temporary solutions (e.g.
> {{RichTableSourceQueryOperation}}) should also be resolved after primaryKey
> is introduced in {{TableSchema}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-13410) Csv input format does not support LocalDate
[ https://issues.apache.org/jira/browse/FLINK-13410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16992152#comment-16992152 ] Jark Wu commented on FLINK-13410: - Does this problem still exist? Should we fix it in 1.10.0? > Csv input format does not support LocalDate > --- > > Key: FLINK-13410 > URL: https://issues.apache.org/jira/browse/FLINK-13410 > Project: Flink > Issue Type: Improvement > Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile) >Reporter: Caizhi Weng >Priority: Major > > Csv input format is lacking parsers for LocalDate, LocalTime, etc. As > DataTypes.DATE now defaults to LocalDate, we should add these parsers for > user experience. > A temporal workaround for the users is that, users can call > DataTypes.Timestamp().bridgeTo(java.sql.Timestamp) to use the old > SqlTimestamp converter. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-15123) remove uniqueKeys from FlinkStatistic in blink planner
[
https://issues.apache.org/jira/browse/FLINK-15123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16992153#comment-16992153
]
Jark Wu commented on FLINK-15123:
-
Assigned to you [~godfreyhe].
> remove uniqueKeys from FlinkStatistic in blink planner
> ---
>
> Key: FLINK-15123
> URL: https://issues.apache.org/jira/browse/FLINK-15123
> Project: Flink
> Issue Type: Improvement
> Components: Table SQL / Planner
>Reporter: godfrey he
>Assignee: godfrey he
>Priority: Major
> Fix For: 1.11.0
>
> Attachments: b_5.txt
>
>
> {{uniqueKeys}} is a kind of constraint, it's unreasonable that {{uniqueKeys}}
> is a kind of statistic. so we should remove uniqueKeys from
> {{FlinkStatistic}} in blink planner. Some temporary solutions (e.g.
> {{RichTableSourceQueryOperation}}) should also be resolved after primaryKey
> is introduced in {{TableSchema}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] lirui-apache commented on a change in pull request #10494: [FLINK-13438][hive] Support date type in Hive
lirui-apache commented on a change in pull request #10494: [FLINK-13438][hive]
Support date type in Hive
URL: https://github.com/apache/flink/pull/10494#discussion_r355829871
##
File path:
flink-connectors/flink-connector-hive/src/main/java/org/apache/flink/table/catalog/hive/client/HiveShimV100.java
##
@@ -364,9 +366,29 @@ public LocalDateTime toFlinkTimestamp(Object
hiveTimestamp) {
return ((Timestamp) hiveTimestamp).toLocalDateTime();
}
+ @Override
+ public Object toHiveDate(Object flinkDate) {
+ ensureSupportedFlinkDate(flinkDate);
+ return flinkDate instanceof Date ? flinkDate :
Date.valueOf((LocalDate) flinkDate);
+ }
+
+ @Override
+ public LocalDate toFlinkDate(Object hiveDate) {
+ Preconditions.checkArgument(hiveDate instanceof Date,
+ "Expecting Hive timestamp to be an instance of
%s, but actually got %s",
Review comment:
nice catch :)
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Commented] (FLINK-14647) Improve the exception message when required property is not matched
[
https://issues.apache.org/jira/browse/FLINK-14647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16992150#comment-16992150
]
Jark Wu commented on FLINK-14647:
-
My gut feeling is that we should expose {{String connectorType()}} and
{{List connectorVersions()}} to the factory interface. So that we can
match them in more fine-grained.
> Improve the exception message when required property is not matched
> ---
>
> Key: FLINK-14647
> URL: https://issues.apache.org/jira/browse/FLINK-14647
> Project: Flink
> Issue Type: Sub-task
> Components: Table SQL / API
>Reporter: Jark Wu
>Priority: Major
>
> Currently, all the required properties should exist and match, otherwise,
> {{NoMatchingTableFactoryException}} will be thrown.
> For example, if we have {{connector.type=hbase, connector.versions=1.1.1}},
> the following exception will be thrown.
> {code}
> org.apache.flink.table.api.NoMatchingTableFactoryException: Could not find a
> suitable table factory for 'org.apache.flink.addons.hbase.HBaseTableFactory'
> in
> the classpath.
> Reason: No context matches.
> The following properties are requested:
> connector.type=hbase
> connector.version=1.1.1
> {code}
> It's hard to know the problem is the version is wrong. A quick fixing is move
> version out of {{requiredContext()}} if we only support one version and throw
> a readable exception in {{ConnectorDescriptorValidator#validate}}.
> However, for the multiple-version connectors, e.g. Kafka, maybe we should
> improve the design of {{TableFactory}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot commented on issue #10503: [FLINK-15137][avro] Improve schema derivation for Avro format
flinkbot commented on issue #10503: [FLINK-15137][avro] Improve schema derivation for Avro format URL: https://github.com/apache/flink/pull/10503#issuecomment-563692047 ## CI report: * 0a63ef8576f25cdec9fe106d7f69429fff6a4c7e UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (FLINK-14953) Parquet table source should use schema type to build FilterPredicate
[ https://issues.apache.org/jira/browse/FLINK-14953?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingsong Lee updated FLINK-14953: - Fix Version/s: 1.10.0 > Parquet table source should use schema type to build FilterPredicate > > > Key: FLINK-14953 > URL: https://issues.apache.org/jira/browse/FLINK-14953 > Project: Flink > Issue Type: Bug > Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile) >Affects Versions: 1.8.0, 1.8.2, 1.9.0, 1.9.1 >Reporter: Zhenqiu Huang >Assignee: Zhenqiu Huang >Priority: Major > Fix For: 1.10.0, 1.9.2 > > > The issue happens when the data type of value in predicate inferred from SQL > doesn't match the parquet schema. For example, foo is a long type, foo < 1 is > the predicate. Literal will be recognized as an integration. It causes the > parquet FilterPredicate is mistakenly created for the column of Integer type. > Then, the exception comes. > java.lang.UnsupportedOperationException > at > org.apache.parquet.filter2.recordlevel.IncrementallyUpdatedFilterPredicate$ValueInspector.update(IncrementallyUpdatedFilterPredicate.java:71) > at > org.apache.parquet.filter2.recordlevel.FilteringPrimitiveConverter.addLong(FilteringPrimitiveConverter.java:105) > at > org.apache.parquet.column.impl.ColumnReaderImpl$2$4.writeValue(ColumnReaderImpl.java:268) > at > org.apache.parquet.column.impl.ColumnReaderImpl.writeCurrentValueToConverter(ColumnReaderImpl.java:367) > at > org.apache.parquet.io.RecordReaderImplementation.read(RecordReaderImplementation.java:406) > at > org.apache.flink.formats.parquet.utils.ParquetRecordReader.readNextRecord(ParquetRecordReader.java:235) > at > org.apache.flink.formats.parquet.utils.ParquetRecordReader.reachEnd(ParquetRecordReader.java:207) > at > org.apache.flink.formats.parquet.ParquetInputFormat.reachedEnd(ParquetInputFormat.java:233) > at > org.apache.flink.api.common.operators.GenericDataSourceBase.executeOnCollections(GenericDataSourceBase.java:231) > at > org.apache.flink.api.common.operators.CollectionExecutor.executeDataSource(CollectionExecutor.java:219) > at > org.apache.flink.api.common.operators.CollectionExecutor.execute(CollectionExecutor.java:155) > at > org.apache.flink.api.common.operators.CollectionExecutor.executeUnaryOperator(CollectionExecutor.java:229) > at > org.apache.flink.api.common.operators.CollectionExecutor.execute(CollectionExecutor.java:149) > at > org.apache.flink.api.common.operators.CollectionExecutor.execute(CollectionExecutor.java:131) > at > org.apache.flink.api.common.operators.CollectionExecutor.executeDataSink(CollectionExecutor.java:182) > at > org.apache.flink.api.common.operators.CollectionExecutor.execute(CollectionExecutor.java:158) > at > org.apache.flink.api.common.operators.CollectionExecutor.execute(CollectionExecutor.java:131) > at > org.apache.flink.api.common.operators.CollectionExecutor.execute(CollectionExecutor.java:115) > at > org.apache.flink.api.java.CollectionEnvironment.execute(CollectionEnvironment.java:38) > at > org.apache.flink.test.util.CollectionTestEnvironment.execute(CollectionTestEnvironment.java:52) > at > org.apache.flink.test.util.CollectionTestEnvironment.execute(CollectionTestEnvironment.java:47) > at org.apache.flink.api.java.DataSet.collect(DataSet.java:413) > at > org.apache.flink.formats.parquet.ParquetTableSourceITCase.testScanWithProjectionAndFilter(ParquetTableSourceITCase.java:91) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) > at > org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) > at > org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) > at > org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) > at > org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) > at > org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) > at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55) > at org.junit.rules.RunRules.evaluate(RunRules.java:20) > at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) > at > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) >
[GitHub] [flink] wsry commented on a change in pull request #10492: [FLINK-15140][runtime] Fix shuffle data compression doesn't work with BroadcastRecordWriter.
wsry commented on a change in pull request #10492: [FLINK-15140][runtime] Fix shuffle data compression doesn't work with BroadcastRecordWriter. URL: https://github.com/apache/flink/pull/10492#discussion_r355829457 ## File path: docs/_includes/generated/netty_shuffle_environment_configuration.html ## @@ -30,7 +30,7 @@ taskmanager.network.pipelined-shuffle.compression.enabled Review comment: Thanks for the review and comments, I agree that we can improve or replace the current pipeline compression implement in the future if we have a better solution and the change is simple enough (only need to remove several lines of code). Besides, the added config option is simple enough and we can keep the config option unchanged or map it to new value even when we replace the current implementation in the future. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (FLINK-15152) Job running without periodic checkpoint for stop failed at the beginning
[
https://issues.apache.org/jira/browse/FLINK-15152?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Feng Jiajie updated FLINK-15152:
Description:
I have a streaming job configured with periodically checkpoint, but after one
week running, I found there isn't any checkpoint file.
h2. Reproduce the problem:
1. Job was submitted to YARN:
{code:java}
bin/flink run -m yarn-cluster -p 1 -yjm 1024m -ytm 4096m
flink-example-1.0-SNAPSHOT.jar{code}
2. Then immediately, before all the task switch to RUNNING (about seconds),
I(actually a job control script) send a "stop with savepoint" command by flink
cli:
{code:java}
bin/flink stop -yid application_1575872737452_0019
f75ca6f457828427ed3d413031b92722 -p file:///tmp/some_dir
{code}
log in jobmanager.log:
{code:java}
2019-12-09 17:56:56,512 INFO
org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Checkpoint
triggering task Source: Socket Stream -> Map (1/1) of job
f75ca6f457828427ed3d413031b92722 is not in state RUNNING but SCHEDULED instead.
Aborting checkpoint.
{code}
Then the job task(taskmanager) *continues to run normally without* checkpoint.
h2. The cause of the problem:
1. "stop with savepoint" command call the code
stopCheckpointScheduler(org/apache/flink/runtime/scheduler/LegacyScheduler.java:612)
and then triggerSynchronousSavepoint:
{code:java}
// we stop the checkpoint coordinator so that we are guaranteed
// to have only the data of the synchronous savepoint committed.
// in case of failure, and if the job restarts, the coordinator
// will be restarted by the CheckpointCoordinatorDeActivator.
checkpointCoordinator.stopCheckpointScheduler();{code}
2. but "before all the task switch to RUNNING", triggerSynchronousSavepoint
failed at org/apache/flink/runtime/checkpoint/CheckpointCoordinator.java:509
{code:java}
LOG.info("Checkpoint triggering task {} of job {} is not in state {} but {}
instead. Aborting checkpoint.",
tasksToTrigger[i].getTaskNameWithSubtaskIndex(),
job,
ExecutionState.RUNNING,
ee.getState());
throw new
CheckpointException(CheckpointFailureReason.NOT_ALL_REQUIRED_TASKS_RUNNING);{code}
3. finally, "stop with savepoint" failed, with
"checkpointCoordinator.stopCheckpointScheduler()" but without the termination
of the job. The job is still running without periodically checkpoint.
sample code for reproduce:
{code:java}
public class StreamingJob {
private static StateBackend makeRocksdbBackend() throws IOException {
RocksDBStateBackend rocksdbBackend = new
RocksDBStateBackend("file:///tmp/aaa");
rocksdbBackend.enableTtlCompactionFilter();
rocksdbBackend.setPredefinedOptions(PredefinedOptions.SPINNING_DISK_OPTIMIZED);
return rocksdbBackend;
}
public static void main(String[] args) throws Exception {
// set up the streaming execution environment
final StreamExecutionEnvironment env =
StreamExecutionEnvironment.getExecutionEnvironment();
// 10 sec
env.enableCheckpointing(10_000L, CheckpointingMode.AT_LEAST_ONCE);
env.setStateBackend(makeRocksdbBackend());
env.setRestartStrategy(RestartStrategies.noRestart());
CheckpointConfig checkpointConfig = env.getCheckpointConfig();
checkpointConfig.enableExternalizedCheckpoints(
CheckpointConfig.ExternalizedCheckpointCleanup.RETAIN_ON_CANCELLATION);
checkpointConfig.setFailOnCheckpointingErrors(true);
DataStream text = env.socketTextStream("127.0.0.1", 8912, "\n");
text.map(new MapFunction>() {
@Override
public Tuple2 map(String s) {
String[] s1 = s.split(" ");
return Tuple2.of(Long.parseLong(s1[0]), Long.parseLong(s1[1]));
}
}).keyBy(0).flatMap(new CountWindowAverage()).print();
env.execute("Flink Streaming Java API Skeleton");
}
public static class CountWindowAverage extends
RichFlatMapFunction, Tuple2> {
private transient ValueState> sum;
@Override
public void flatMap(Tuple2 input, Collector>
out) throws Exception {
Tuple2 currentSum = sum.value();
currentSum.f0 += 1;
currentSum.f1 += input.f1;
sum.update(currentSum);
out.collect(new Tuple2<>(input.f0, currentSum.f1));
}
@Override
public void open(Configuration config) {
ValueStateDescriptor> descriptor =
new ValueStateDescriptor<>(
"average", // the state name
TypeInformation.of(new TypeHint>() {
}), // type information
Tuple2.of(0L, 0L)); // default value of the state, if nothing was
set
sum = getRuntimeContext().getState(descriptor);
}
}
}
{code}
was:
I have a streaming job configured with periodically checkpoint, but after one
week running, I found there isn't any checkpoint file.
h2. Reproduce the problem:
1. Job was submitted to YARN:
{code:java}
bin/flink run -m yarn-cluster -p 1 -yjm 1024m -ytm 4096m
flink-example-1.0-SNAPSHOT.jar{code}
2. Then immediately, before all
[jira] [Commented] (FLINK-14604) Bump commons-cli to 1.4
[
https://issues.apache.org/jira/browse/FLINK-14604?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16992145#comment-16992145
]
Wei Zhong commented on FLINK-14604:
---
Due to the adjustment to FLIP-78 is accepted, close this Jira.
> Bump commons-cli to 1.4
> ---
>
> Key: FLINK-14604
> URL: https://issues.apache.org/jira/browse/FLINK-14604
> Project: Flink
> Issue Type: Task
> Components: Build System, Command Line Client
>Reporter: Wei Zhong
>Priority: Major
>
> Currently flink is using commons-cli 1.3.1. There is a
> [bug|https://issues.apache.org/jira/projects/CLI/issues/CLI-265] in it which
> prevent us from using options that accept variable arguments in command line.
> To be precise, it prevents us from accepting a short-name option after a
> varargs option because there is a problem in the implementation of
> DefaultParser#isShortOption() method in commons-cli 1.3.1:
> {code:java}
> /**
> * Tells if the token looks like a short option.
> *
> * @param token
> */
> private boolean isShortOption(String token)
> {
> // short options (-S, -SV, -S=V, -SV1=V2, -S1S2)
> // PROBLEM: It assumes that short option only has single character,
> // but in fact we have many multi-characters short options.
> return token.startsWith("-") && token.length() >= 2 &&
> options.hasShortOption(token.substring(1, 2));
> }
> {code}
> If we bump the version to 1.4, we can solve this problem.
> I request this change because there are 2 varargs options which hit this bug
> in the design of command line options of [Python UDF Dependency
> Management|https://cwiki.apache.org/confluence/display/FLINK/FLIP-78%3A+Flink+Python+UDF+Environment+and+Dependency+Management].
> It will be great helpful if we can bump the commons-cli version to 1.4 :).
> The commons-cli 1.4 is also a stable version which released at Mar, 2017. And
> today its usage statistic is greater than 1.3.1 on [maven central
> repository|https://mvnrepository.com/artifact/commons-cli/commons-cli].
> I have pushed the change to my own travis to check if it breaks something.
> This is the [link|https://travis-ci.org/WeiZhong94/flink/builds/607438208]
> and it seems that everything is fine.
> This is the [link|https://travis-ci.org/WeiZhong94/flink/builds/607481477] of
> travis build which contains e2e tests, it seems fine except the timeout tasks.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Closed] (FLINK-14604) Bump commons-cli to 1.4
[
https://issues.apache.org/jira/browse/FLINK-14604?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wei Zhong closed FLINK-14604.
-
Resolution: Won't Fix
> Bump commons-cli to 1.4
> ---
>
> Key: FLINK-14604
> URL: https://issues.apache.org/jira/browse/FLINK-14604
> Project: Flink
> Issue Type: Task
> Components: Build System, Command Line Client
>Reporter: Wei Zhong
>Priority: Major
>
> Currently flink is using commons-cli 1.3.1. There is a
> [bug|https://issues.apache.org/jira/projects/CLI/issues/CLI-265] in it which
> prevent us from using options that accept variable arguments in command line.
> To be precise, it prevents us from accepting a short-name option after a
> varargs option because there is a problem in the implementation of
> DefaultParser#isShortOption() method in commons-cli 1.3.1:
> {code:java}
> /**
> * Tells if the token looks like a short option.
> *
> * @param token
> */
> private boolean isShortOption(String token)
> {
> // short options (-S, -SV, -S=V, -SV1=V2, -S1S2)
> // PROBLEM: It assumes that short option only has single character,
> // but in fact we have many multi-characters short options.
> return token.startsWith("-") && token.length() >= 2 &&
> options.hasShortOption(token.substring(1, 2));
> }
> {code}
> If we bump the version to 1.4, we can solve this problem.
> I request this change because there are 2 varargs options which hit this bug
> in the design of command line options of [Python UDF Dependency
> Management|https://cwiki.apache.org/confluence/display/FLINK/FLIP-78%3A+Flink+Python+UDF+Environment+and+Dependency+Management].
> It will be great helpful if we can bump the commons-cli version to 1.4 :).
> The commons-cli 1.4 is also a stable version which released at Mar, 2017. And
> today its usage statistic is greater than 1.3.1 on [maven central
> repository|https://mvnrepository.com/artifact/commons-cli/commons-cli].
> I have pushed the change to my own travis to check if it breaks something.
> This is the [link|https://travis-ci.org/WeiZhong94/flink/builds/607438208]
> and it seems that everything is fine.
> This is the [link|https://travis-ci.org/WeiZhong94/flink/builds/607481477] of
> travis build which contains e2e tests, it seems fine except the timeout tasks.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-15152) Job running without periodic checkpoint for stop failed at the beginning
[
https://issues.apache.org/jira/browse/FLINK-15152?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Feng Jiajie updated FLINK-15152:
Description:
I have a streaming job configured with periodically checkpoint, but after one
week running, I found there isn't any checkpoint file.
h2. Reproduce the problem:
1. Job was submitted to YARN:
{code:java}
bin/flink run -m yarn-cluster -p 1 -yjm 1024m -ytm 4096m
flink-example-1.0-SNAPSHOT.jar{code}
2. Then immediately, before all the task switch to RUNNING (about seconds),
I(actually a job control script) send a "stop with savepoint" command by flink
cli:
{code:java}
bin/flink stop -yid application_1575872737452_0019
f75ca6f457828427ed3d413031b92722 -p file:///tmp/some_dir
{code}
log in jobmanager.log:
{code:java}
2019-12-09 17:56:56,512 INFO
org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Checkpoint
triggering task Source: Socket Stream -> Map (1/1) of job
f75ca6f457828427ed3d413031b92722 is not in state RUNNING but SCHEDULED instead.
Aborting checkpoint.
{code}
Then the job task *continues to run normally without* checkpoint.
h2. The cause of the problem:
1. "stop with savepoint" command call the code
stopCheckpointScheduler(org/apache/flink/runtime/scheduler/LegacyScheduler.java:612)
and then triggerSynchronousSavepoint:
{code:java}
// we stop the checkpoint coordinator so that we are guaranteed
// to have only the data of the synchronous savepoint committed.
// in case of failure, and if the job restarts, the coordinator
// will be restarted by the CheckpointCoordinatorDeActivator.
checkpointCoordinator.stopCheckpointScheduler();{code}
2. but "before all the task switch to RUNNING", checkpoint failed at
org/apache/flink/runtime/checkpoint/CheckpointCoordinator.java:509
{code:java}
LOG.info("Checkpoint triggering task {} of job {} is not in state {} but {}
instead. Aborting checkpoint.",
tasksToTrigger[i].getTaskNameWithSubtaskIndex(),
job,
ExecutionState.RUNNING,
ee.getState());
throw new
CheckpointException(CheckpointFailureReason.NOT_ALL_REQUIRED_TASKS_RUNNING);{code}
3. finally, "stop with savepoint" failed, with
"checkpointCoordinator.stopCheckpointScheduler()" but without the termination
of the job.
sample code:
{code:java}
public class StreamingJob {
private static StateBackend makeRocksdbBackend() throws IOException {
RocksDBStateBackend rocksdbBackend = new
RocksDBStateBackend("file:///tmp/aaa");
rocksdbBackend.enableTtlCompactionFilter();
rocksdbBackend.setPredefinedOptions(PredefinedOptions.SPINNING_DISK_OPTIMIZED);
return rocksdbBackend;
}
public static void main(String[] args) throws Exception {
// set up the streaming execution environment
final StreamExecutionEnvironment env =
StreamExecutionEnvironment.getExecutionEnvironment();
// 10 sec
env.enableCheckpointing(10_000L, CheckpointingMode.AT_LEAST_ONCE);
env.setStateBackend(makeRocksdbBackend());
env.setRestartStrategy(RestartStrategies.noRestart());
CheckpointConfig checkpointConfig = env.getCheckpointConfig();
checkpointConfig.enableExternalizedCheckpoints(
CheckpointConfig.ExternalizedCheckpointCleanup.RETAIN_ON_CANCELLATION);
checkpointConfig.setFailOnCheckpointingErrors(true);
DataStream text = env.socketTextStream("127.0.0.1", 8912, "\n");
text.map(new MapFunction>() {
@Override
public Tuple2 map(String s) {
String[] s1 = s.split(" ");
return Tuple2.of(Long.parseLong(s1[0]), Long.parseLong(s1[1]));
}
}).keyBy(0).flatMap(new CountWindowAverage()).print();
env.execute("Flink Streaming Java API Skeleton");
}
public static class CountWindowAverage extends
RichFlatMapFunction, Tuple2> {
private transient ValueState> sum;
@Override
public void flatMap(Tuple2 input, Collector>
out) throws Exception {
Tuple2 currentSum = sum.value();
currentSum.f0 += 1;
currentSum.f1 += input.f1;
sum.update(currentSum);
out.collect(new Tuple2<>(input.f0, currentSum.f1));
}
@Override
public void open(Configuration config) {
ValueStateDescriptor> descriptor =
new ValueStateDescriptor<>(
"average", // the state name
TypeInformation.of(new TypeHint>() {
}), // type information
Tuple2.of(0L, 0L)); // default value of the state, if nothing was
set
sum = getRuntimeContext().getState(descriptor);
}
}
}
{code}
was:
I have a streaming job configured with periodically checkpoint, but after one
week running, I found there isn't any checkpoint file.
h2. Reproduce the problem:
1. Job was submitted to YARN:
{code:java}
bin/flink run -m yarn-cluster -p 1 -yjm 1024m -ytm 4096m
flink-example-1.0-SNAPSHOT.jar{code}
2. Then immediately, before all the task switch to RUNNING (about seconds),
I(actually a job control script) send a "stop with
[GitHub] [flink] flinkbot edited a comment on issue #10491: [FLINK-15093][streaming-java] StreamExecutionEnvironment does not cle…
flinkbot edited a comment on issue #10491: [FLINK-15093][streaming-java] StreamExecutionEnvironment does not cle… URL: https://github.com/apache/flink/pull/10491#issuecomment-563083959 ## CI report: * 2e942aa60f0807a6a41fd0483237d8b571fdca1f Travis: [CANCELED](https://travis-ci.com/flink-ci/flink/builds/140171167) * f9657812903c088739c262b4e418d69683c52989 Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/140180146) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=3341) * d25b4deaa79f06d8a2ba3534d48553bb72e191ea Travis: [CANCELED](https://travis-ci.com/flink-ci/flink/builds/140207958) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=3354) * 0eb37fee2dce639f45d933200b3732c4e444841b Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/140228860) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=3360) * 35b9a4e5e8d4767e431e69a0b8aba1000e9f402f Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/140233479) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=3364) * b9d0dc43b03f25fa366ccbb833693219d45a8ce7 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/140238481) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=3368) * 6812331d348892e6146be0f6a5cddf2401090ecb Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/140340160) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=3382) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #10458: [FLINK-14815][rest]Expose network metric in IOMetricsInfo
flinkbot edited a comment on issue #10458: [FLINK-14815][rest]Expose network metric in IOMetricsInfo URL: https://github.com/apache/flink/pull/10458#issuecomment-562482569 ## CI report: * 303f921f58ff61aa3e332c36ffdc07d5f9ceb352 Travis: [CANCELED](https://travis-ci.com/flink-ci/flink/builds/139659908) * 0122cc64d2c4a46ee8ba05ca67832a1589b024b1 Travis: [CANCELED](https://travis-ci.com/flink-ci/flink/builds/139667569) * 2b332510e6d2cd2dee836de532139241c20ce00c Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/139677676) * 30f1b29134a585e7d19c0de5e4ef000282a285b2 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/140158853) * 8a5207e384e995fb2f8929e5de3d87e89e801b3e Travis: [CANCELED](https://travis-ci.com/flink-ci/flink/builds/140199482) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=3348) * 54049f914f9ac88570f882317e7f497ccb09587e Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/140203351) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=3350) * 308229677cea17a602f48cce2ce9e289a4ede30a Travis: [PENDING](https://travis-ci.com/flink-ci/flink/builds/140345632) Azure: [PENDING](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=3384) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot commented on issue #10503: [FLINK-15137][avro] Improve schema derivation for Avro format
flinkbot commented on issue #10503: [FLINK-15137][avro] Improve schema derivation for Avro format URL: https://github.com/apache/flink/pull/10503#issuecomment-563673149 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit 7462257067eccfec5c3e03021fad8f455d4e0f5c (Tue Dec 10 03:17:15 UTC 2019) **Warnings:** * **This pull request references an unassigned [Jira ticket](https://issues.apache.org/jira/browse/FLINK-15137).** According to the [code contribution guide](https://flink.apache.org/contributing/contribute-code.html), tickets need to be assigned before starting with the implementation work. Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (FLINK-15137) Improve schema derivation for Avro format
[
https://issues.apache.org/jira/browse/FLINK-15137?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-15137:
---
Labels: pull-request-available (was: )
> Improve schema derivation for Avro format
> -
>
> Key: FLINK-15137
> URL: https://issues.apache.org/jira/browse/FLINK-15137
> Project: Flink
> Issue Type: Sub-task
> Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile)
>Reporter: Jark Wu
>Priority: Major
> Labels: pull-request-available
>
> For JSON, CSV and OldCsv, we already supported {{derive.schema=true}} to get
> the schema from table schema. But for Avro format, a user has to pass an Avro
> schema file or define the format schema explicitly via {{avro.schema}}.
> We can think of if we can drop {{avro.schema}} and make
> {{derive.schema=true}} as the default behavior.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] wuchong opened a new pull request #10503: [FLINK-15137][avro] Improve schema derivation for Avro format
wuchong opened a new pull request #10503: [FLINK-15137][avro] Improve schema derivation for Avro format URL: https://github.com/apache/flink/pull/10503 ## What is the purpose of the change Support schema derivation for Avro format and also enables schema derivation as the default behavior for Avro format, so that users doesn't need to specify a format schema explicitly. **NOTE: this PR depends on #10500 (the first commit).** ## Brief change log The commit messages describe the changes. ## Verifying this change - add tests to verify converting a `LogicalType` to `Schema` in `AvroSchemaConverterTest`. - add tests to verify the `(De)SerializationSchema` created from `LogicalType` works well in `AvroRowDeSerializationSchemaTest`. - add tests to verify that the derived schema can work both on serialization and deserialization in `AvroRowFormatFactoryTest`. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (yes / **no**) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (yes / **no**) - The serializers: (yes / **no** / don't know) - The runtime per-record code paths (performance sensitive): (yes / **no** / don't know) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: (yes / **no** / don't know) - The S3 file system connector: (yes / **no** / don't know) ## Documentation - Does this pull request introduce a new feature? (yes / **no**) - If yes, how is the feature documented? (not applicable / **docs** / JavaDocs / not documented) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (FLINK-12160) Support to ship user config files on kubernetes
[
https://issues.apache.org/jira/browse/FLINK-12160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yang Wang updated FLINK-12160:
--
Description:
When deploying flink cluster on Yarn, `yarnShip` cli option could be used to
ship user config files and jars. The files are registered as Yarn local
resource which is saved on hdfs.
However, we do not have a default existing dfs. The `ConfigMap` could be used
to ship small files. The files will be stored in etcd and then mounted to
jobmanager and taskmanager pod so that they could use it as locally.
{code:java}
-kt,--kubernetesship Ship files in the specified directory
(t for transfer). Only small
files(<1MB) could be supported.
{code}
[https://stackoverflow.com/questions/53012798/kubernetes-configmap-size-limitation]
was:When submit a flink job on kubernetes, the jars will be transferred
through docker images or flink blob server. Also we will need a way to transfer
config files, such as hdfs-site.xml/core-site.xml. It could be saved in config
map in etcd and then mounted to jobmanager and taskmanager pod.
> Support to ship user config files on kubernetes
> ---
>
> Key: FLINK-12160
> URL: https://issues.apache.org/jira/browse/FLINK-12160
> Project: Flink
> Issue Type: Sub-task
> Components: Runtime / Coordination
>Reporter: Yang Wang
>Assignee: Yang Wang
>Priority: Major
>
> When deploying flink cluster on Yarn, `yarnShip` cli option could be used to
> ship user config files and jars. The files are registered as Yarn local
> resource which is saved on hdfs.
>
> However, we do not have a default existing dfs. The `ConfigMap` could be used
> to ship small files. The files will be stored in etcd and then mounted to
> jobmanager and taskmanager pod so that they could use it as locally.
> {code:java}
> -kt,--kubernetesship Ship files in the specified
> directory
> (t for transfer). Only small
> files(<1MB) could be supported.
> {code}
> [https://stackoverflow.com/questions/53012798/kubernetes-configmap-size-limitation]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on issue #10458: [FLINK-14815][rest]Expose network metric in IOMetricsInfo
flinkbot edited a comment on issue #10458: [FLINK-14815][rest]Expose network metric in IOMetricsInfo URL: https://github.com/apache/flink/pull/10458#issuecomment-562482569 ## CI report: * 303f921f58ff61aa3e332c36ffdc07d5f9ceb352 Travis: [CANCELED](https://travis-ci.com/flink-ci/flink/builds/139659908) * 0122cc64d2c4a46ee8ba05ca67832a1589b024b1 Travis: [CANCELED](https://travis-ci.com/flink-ci/flink/builds/139667569) * 2b332510e6d2cd2dee836de532139241c20ce00c Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/139677676) * 30f1b29134a585e7d19c0de5e4ef000282a285b2 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/140158853) * 8a5207e384e995fb2f8929e5de3d87e89e801b3e Travis: [CANCELED](https://travis-ci.com/flink-ci/flink/builds/140199482) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=3348) * 54049f914f9ac88570f882317e7f497ccb09587e Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/140203351) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=3350) * 308229677cea17a602f48cce2ce9e289a4ede30a UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Assigned] (FLINK-11899) Introduce vectorized parquet InputFormat for blink runtime
[ https://issues.apache.org/jira/browse/FLINK-11899?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kurt Young reassigned FLINK-11899: -- Assignee: Zhenqiu Huang (was: Jingsong Lee) > Introduce vectorized parquet InputFormat for blink runtime > -- > > Key: FLINK-11899 > URL: https://issues.apache.org/jira/browse/FLINK-11899 > Project: Flink > Issue Type: Sub-task > Components: Table SQL / Runtime >Reporter: Jingsong Lee >Assignee: Zhenqiu Huang >Priority: Major > > VectorizedParquetInputFormat is introduced to read parquet data in batches. > When returning each row of data, instead of actually retrieving each field, > we use BaseRow's abstraction to return a Columnar Row-like view. > This will greatly improve the downstream filtered scenarios, so that there is > no need to access redundant fields on the filtered data. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-11899) Introduce vectorized parquet InputFormat for blink runtime
[ https://issues.apache.org/jira/browse/FLINK-11899?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16992136#comment-16992136 ] Kurt Young commented on FLINK-11899: Thanks [~ZhenqiuHuang], will assign to you. BTW we have already implemented vectorized orc input format, you can take as an example if needed. > Introduce vectorized parquet InputFormat for blink runtime > -- > > Key: FLINK-11899 > URL: https://issues.apache.org/jira/browse/FLINK-11899 > Project: Flink > Issue Type: Sub-task > Components: Table SQL / Runtime >Reporter: Jingsong Lee >Assignee: Jingsong Lee >Priority: Major > > VectorizedParquetInputFormat is introduced to read parquet data in batches. > When returning each row of data, instead of actually retrieving each field, > we use BaseRow's abstraction to return a Columnar Row-like view. > This will greatly improve the downstream filtered scenarios, so that there is > no need to access redundant fields on the filtered data. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-11899) Introduce vectorized parquet InputFormat for blink runtime
[ https://issues.apache.org/jira/browse/FLINK-11899?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16992133#comment-16992133 ] Zhenqiu Huang commented on FLINK-11899: --- [~lzljs3620320][~ykt836] As I am maintaining ParquetInputFormat and ParquetTableSource, I would like to work on this task. Please assign it to me. > Introduce vectorized parquet InputFormat for blink runtime > -- > > Key: FLINK-11899 > URL: https://issues.apache.org/jira/browse/FLINK-11899 > Project: Flink > Issue Type: Sub-task > Components: Table SQL / Runtime >Reporter: Jingsong Lee >Assignee: Jingsong Lee >Priority: Major > > VectorizedParquetInputFormat is introduced to read parquet data in batches. > When returning each row of data, instead of actually retrieving each field, > we use BaseRow's abstraction to return a Columnar Row-like view. > This will greatly improve the downstream filtered scenarios, so that there is > no need to access redundant fields on the filtered data. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-12160) Support to ship user config files on kubernetes
[ https://issues.apache.org/jira/browse/FLINK-12160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Wang updated FLINK-12160: -- Summary: Support to ship user config files on kubernetes (was: Support for transferring user config files on kubernetes) > Support to ship user config files on kubernetes > --- > > Key: FLINK-12160 > URL: https://issues.apache.org/jira/browse/FLINK-12160 > Project: Flink > Issue Type: Sub-task > Components: Runtime / Coordination >Reporter: Yang Wang >Assignee: Yang Wang >Priority: Major > > When submit a flink job on kubernetes, the jars will be transferred through > docker images or flink blob server. Also we will need a way to transfer > config files, such as hdfs-site.xml/core-site.xml. It could be saved in > config map in etcd and then mounted to jobmanager and taskmanager pod. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on issue #10348: [FLINK-14951][tests] Harden the thread safety of State TTL backend tests
flinkbot edited a comment on issue #10348: [FLINK-14951][tests] Harden the thread safety of State TTL backend tests URL: https://github.com/apache/flink/pull/10348#issuecomment-559509076 ## CI report: * 54ecee627e7036f4d150aad330b9772406a19494 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/138580810) * 3e89b2b0d85dd29a76bb10807e2959a7d2ee8295 Travis: [PENDING](https://travis-ci.com/flink-ci/flink/builds/140343903) Azure: [PENDING](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=3383) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-15103) Performance regression on 3.12.2019 in various benchmarks
[ https://issues.apache.org/jira/browse/FLINK-15103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16992128#comment-16992128 ] Xintong Song commented on FLINK-15103: -- True, I was also discussing this issue with [~yunta], who is looking into the regression with rocksdb state backend. I did not know that that the benchmarks are using LocalExecutor until I looked into their source codes yesterday. For LocalExecutor, JVM heap size should not be affected by taskmanager.memory.off-heap. The operators with regression also not use managed memory. Then the only thing came to my mind is number / size of network buffers. Although we do not start separate JVM process for TaskExecutors in LocalExecutor, the number of network buffers used by NettyShuffleEnvironment is calculated from the configuration on launching the TaskExecutors. Another thing to notice is that, at the commit that introduces the regression (4b8ed643a4d85c9440a8adbc0798b8a4bbd9520b), FLIP-49 is not yet activated and TaskExecutors are still using the legacy logic to decide memory sizes. When FLIP-49 is activated in later commit (9d1256ccbf8eb1556016b6805c3a91e2787d298a), the regression still exist. To fully understand the problem, we need to analyze the regression in both configuration logics. I'm still running tests and looking into the codes, looking for proves that supports my hypothesis about the network buffer size. Will post updates here if I find anything. > Performance regression on 3.12.2019 in various benchmarks > - > > Key: FLINK-15103 > URL: https://issues.apache.org/jira/browse/FLINK-15103 > Project: Flink > Issue Type: Bug > Components: Benchmarks >Reporter: Piotr Nowojski >Priority: Blocker > Fix For: 1.10.0 > > > Various benchmarks show a performance regression that happened on December > 3rd: > [arrayKeyBy (probably the most easily > visible)|http://codespeed.dak8s.net:8000/timeline/#/?exe=1=arrayKeyBy=2=200=off=on=on] > > [tupleKeyBy|http://codespeed.dak8s.net:8000/timeline/#/?exe=1=tupleKeyBy=2=200=off=on=on] > > [twoInputMapSink|http://codespeed.dak8s.net:8000/timeline/#/?exe=1=twoInputMapSink=2=200=off=on=on] > [globalWindow (small > one)|http://codespeed.dak8s.net:8000/timeline/#/?exe=1=globalWindow=2=200=off=on=on] > and possible others. > Probably somewhere between those commits: -8403fd4- 2d67ee0..60b3f2f -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (FLINK-15017) add a thrift server for Flink
[ https://issues.apache.org/jira/browse/FLINK-15017?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16992126#comment-16992126 ] Kurt Young edited comment on FLINK-15017 at 12/10/19 2:31 AM: -- I think this is a good addition to Flink SQL. We have to take care about the relationship with SQL CLi gateway mode before we start doing this. was (Author: ykt836): I think this is a good addition to Flink SQL. We have to take care about the relationship with SQL CLi gateway mode before we starting to do this. > add a thrift server for Flink > - > > Key: FLINK-15017 > URL: https://issues.apache.org/jira/browse/FLINK-15017 > Project: Flink > Issue Type: New Feature > Components: Client / Job Submission >Reporter: Bowen Li >Assignee: Kurt Young >Priority: Major > Fix For: 1.11.0 > > > allow jdbc/odbc clients to execute SQL queries over jdbc/odbc on Flink. > can refer to Spark thrift server, a port of Hive's HiveServer2, as example. > https://github.com/apache/spark/tree/master/sql/hive-thriftserver > https://cwiki.apache.org/confluence/display/Hive/HiveServer2+Overview -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-15017) add a thrift server for Flink
[ https://issues.apache.org/jira/browse/FLINK-15017?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16992126#comment-16992126 ] Kurt Young commented on FLINK-15017: I think this is a good addition to Flink SQL. We have to take care about the relationship with SQL CLi gateway mode before we starting to do this. > add a thrift server for Flink > - > > Key: FLINK-15017 > URL: https://issues.apache.org/jira/browse/FLINK-15017 > Project: Flink > Issue Type: New Feature > Components: Client / Job Submission >Reporter: Bowen Li >Assignee: Kurt Young >Priority: Major > Fix For: 1.11.0 > > > allow jdbc/odbc clients to execute SQL queries over jdbc/odbc on Flink. > can refer to Spark thrift server, a port of Hive's HiveServer2, as example. > https://github.com/apache/spark/tree/master/sql/hive-thriftserver > https://cwiki.apache.org/confluence/display/Hive/HiveServer2+Overview -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (FLINK-15164) Introduce ParquetColumnarRowSplitReader to parquet format
[ https://issues.apache.org/jira/browse/FLINK-15164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingsong Lee closed FLINK-15164. Resolution: Duplicate > Introduce ParquetColumnarRowSplitReader to parquet format > - > > Key: FLINK-15164 > URL: https://issues.apache.org/jira/browse/FLINK-15164 > Project: Flink > Issue Type: New Feature > Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile) >Reporter: Zhenqiu Huang >Assignee: Zhenqiu Huang >Priority: Major > Fix For: 1.11.0 > > > With ParquetColumnarRowSplitReader support, the hive connector can directly > read data stored as parquet format. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-15164) Introduce ParquetColumnarRowSplitReader to parquet format
[ https://issues.apache.org/jira/browse/FLINK-15164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16992123#comment-16992123 ] Jingsong Lee commented on FLINK-15164: -- Hi [~ZhenqiuHuang], this is duplicated with https://issues.apache.org/jira/browse/FLINK-11899 If you are interested in this, fell free to ask committers assigning it to you. > Introduce ParquetColumnarRowSplitReader to parquet format > - > > Key: FLINK-15164 > URL: https://issues.apache.org/jira/browse/FLINK-15164 > Project: Flink > Issue Type: New Feature > Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile) >Reporter: Zhenqiu Huang >Assignee: Zhenqiu Huang >Priority: Major > Fix For: 1.11.0 > > > With ParquetColumnarRowSplitReader support, the hive connector can directly > read data stored as parquet format. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-14815) Expose network metric in IOMetricsInfo
[ https://issues.apache.org/jira/browse/FLINK-14815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16992120#comment-16992120 ] lining commented on FLINK-14815: [~pnowojski] could you review it? > Expose network metric in IOMetricsInfo > -- > > Key: FLINK-14815 > URL: https://issues.apache.org/jira/browse/FLINK-14815 > Project: Flink > Issue Type: Sub-task > Components: Runtime / Metrics, Runtime / Network, Runtime / REST >Reporter: lining >Assignee: lining >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > * SubTask > ** pool usage: outPoolUsage, inputExclusiveBuffersUsage, > inputFloatingBuffersUsage. > *** If the subtask is not back pressured, but it is causing backpressure > (full input, empty output) > *** By comparing exclusive/floating buffers usage, whether all channels are > back-pressure or only some of them > ** back-pressured for show whether it is back pressured. > * Vertex > ** pool usage: outPoolUsageAvg, inputExclusiveBuffersUsageAvg, > inputFloatingBuffersUsageAvg > ** back-pressured for show whether it is back pressured(merge all iths > subtasks) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] walterddr commented on a change in pull request #9404: [FLINK-13667][ml] Add the utility class for the Table
walterddr commented on a change in pull request #9404: [FLINK-13667][ml] Add
the utility class for the Table
URL: https://github.com/apache/flink/pull/9404#discussion_r355814299
##
File path:
flink-ml-parent/flink-ml-lib/src/main/java/org/apache/flink/ml/common/utils/TableUtil.java
##
@@ -0,0 +1,439 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.flink.ml.common.utils;
+
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.table.api.TableSchema;
+import org.apache.flink.types.Row;
+import org.apache.flink.util.Preconditions;
+
+import org.apache.flink.shaded.guava18.com.google.common.base.Joiner;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import java.util.UUID;
+
+/**
+ * Utility to operator to interact with Table contents, such as rows and
columns.
+ */
+public class TableUtil {
+ /**
+* Return a temp table named with prefix `temp_`, follow by a random
UUID.
+*
+* UUID hyphens ("-") will be replaced by underscores ("_").
+*
+* @return tableName
+*/
+ public static synchronized String getTempTableName() {
+ return ("temp_" + UUID.randomUUID().toString().replaceAll("-",
"_"))
+ .toLowerCase();
+ }
+
+ /**
+* Find the index of targetCol in string array
tableCols. It will ignore the case of the
+* tableCols.
+*
+* @param tableCols a string array among which to find the targetCol.
+* @param targetCol the targetCol to find.
+* @return the index of the targetCol, if not found, returns -1.
+*/
+ public static int findColIndex(String[] tableCols, String targetCol) {
+ Preconditions.checkNotNull(targetCol, "targetCol is null!");
+ for (int i = 0; i < tableCols.length; i++) {
+ if (targetCol.equalsIgnoreCase(tableCols[i])) {
+ return i;
+ }
+ }
+ return -1;
+ }
+
+ /**
+* Find the index of targetCol from the
tableSchema.
+*
+* @param tableSchema the TableSchema among which to find the targetCol.
+* @param targetCol the targetCols to find.
+* @return the index of the targetCol.
+*/
+ public static int findColIndex(TableSchema tableSchema, String
targetCol) {
+ return findColIndex(tableSchema.getFieldNames(), targetCol);
+ }
+
+ /**
+* Find the indices of targetCols in string array
tableCols. If
+* targetCols is
+* null, it will be replaced by the tableCols
+*
+* @param tableCols a string array among which to find the targetCols.
+* @param targetCols the targetCols to find.
+* @return the indices of the targetCols.
+*/
+ public static int[] findColIndices(String[] tableCols, String[]
targetCols) {
+ if (targetCols == null) {
+ int[] indices = new int[tableCols.length];
+ for (int i = 0; i < tableCols.length; i++) {
+ indices[i] = i;
+ }
+ return indices;
+ }
+ int[] indices = new int[targetCols.length];
+ for (int i = 0; i < indices.length; i++) {
+ indices[i] = findColIndex(tableCols, targetCols[i]);
+ }
+ return indices;
+ }
+
+ /**
+* Find the indices of targetCols from the
tableSchema.
+*
+* @param tableSchema the TableSchema among which to find the
targetCols.
+* @param targetCols the targetCols to find.
+* @return the indices of the targetCols.
+*/
+ public static int[] findColIndices(TableSchema tableSchema, String[]
targetCols) {
+ return findColIndices(tableSchema.getFieldNames(), targetCols);
+ }
+
+ /**
+* Find the types of the targetCols. If the targetCol not
exist, return null.
+
[GitHub] [flink] walterddr commented on a change in pull request #9404: [FLINK-13667][ml] Add the utility class for the Table
walterddr commented on a change in pull request #9404: [FLINK-13667][ml] Add
the utility class for the Table
URL: https://github.com/apache/flink/pull/9404#discussion_r355814399
##
File path:
flink-ml-parent/flink-ml-lib/src/main/java/org/apache/flink/ml/common/utils/TableUtil.java
##
@@ -0,0 +1,439 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.flink.ml.common.utils;
+
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.table.api.TableSchema;
+import org.apache.flink.types.Row;
+import org.apache.flink.util.Preconditions;
+
+import org.apache.flink.shaded.guava18.com.google.common.base.Joiner;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import java.util.UUID;
+
+/**
+ * Utility to operator to interact with Table contents, such as rows and
columns.
+ */
+public class TableUtil {
+ /**
+* Return a temp table named with prefix `temp_`, follow by a random
UUID.
+*
+* UUID hyphens ("-") will be replaced by underscores ("_").
+*
+* @return tableName
+*/
+ public static synchronized String getTempTableName() {
+ return ("temp_" + UUID.randomUUID().toString().replaceAll("-",
"_"))
+ .toLowerCase();
+ }
+
+ /**
+* Find the index of targetCol in string array
tableCols. It will ignore the case of the
+* tableCols.
+*
+* @param tableCols a string array among which to find the targetCol.
+* @param targetCol the targetCol to find.
+* @return the index of the targetCol, if not found, returns -1.
+*/
+ public static int findColIndex(String[] tableCols, String targetCol) {
+ Preconditions.checkNotNull(targetCol, "targetCol is null!");
+ for (int i = 0; i < tableCols.length; i++) {
+ if (targetCol.equalsIgnoreCase(tableCols[i])) {
+ return i;
+ }
+ }
+ return -1;
+ }
+
+ /**
+* Find the index of targetCol from the
tableSchema.
+*
+* @param tableSchema the TableSchema among which to find the targetCol.
+* @param targetCol the targetCols to find.
+* @return the index of the targetCol.
+*/
+ public static int findColIndex(TableSchema tableSchema, String
targetCol) {
+ return findColIndex(tableSchema.getFieldNames(), targetCol);
+ }
+
+ /**
+* Find the indices of targetCols in string array
tableCols. If
+* targetCols is
+* null, it will be replaced by the tableCols
+*
+* @param tableCols a string array among which to find the targetCols.
+* @param targetCols the targetCols to find.
+* @return the indices of the targetCols.
+*/
+ public static int[] findColIndices(String[] tableCols, String[]
targetCols) {
+ if (targetCols == null) {
+ int[] indices = new int[tableCols.length];
+ for (int i = 0; i < tableCols.length; i++) {
+ indices[i] = i;
+ }
+ return indices;
+ }
+ int[] indices = new int[targetCols.length];
+ for (int i = 0; i < indices.length; i++) {
+ indices[i] = findColIndex(tableCols, targetCols[i]);
+ }
+ return indices;
+ }
+
+ /**
+* Find the indices of targetCols from the
tableSchema.
+*
+* @param tableSchema the TableSchema among which to find the
targetCols.
+* @param targetCols the targetCols to find.
+* @return the indices of the targetCols.
+*/
+ public static int[] findColIndices(TableSchema tableSchema, String[]
targetCols) {
+ return findColIndices(tableSchema.getFieldNames(), targetCols);
+ }
+
+ /**
+* Find the types of the targetCols. If the targetCol not
exist, return null.
+
[GitHub] [flink] walterddr commented on a change in pull request #9404: [FLINK-13667][ml] Add the utility class for the Table
walterddr commented on a change in pull request #9404: [FLINK-13667][ml] Add
the utility class for the Table
URL: https://github.com/apache/flink/pull/9404#discussion_r355814412
##
File path:
flink-ml-parent/flink-ml-lib/src/main/java/org/apache/flink/ml/common/utils/TableUtil.java
##
@@ -0,0 +1,439 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.flink.ml.common.utils;
+
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.table.api.TableSchema;
+import org.apache.flink.types.Row;
+import org.apache.flink.util.Preconditions;
+
+import org.apache.flink.shaded.guava18.com.google.common.base.Joiner;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import java.util.UUID;
+
+/**
+ * Utility to operator to interact with Table contents, such as rows and
columns.
+ */
+public class TableUtil {
+ /**
+* Return a temp table named with prefix `temp_`, follow by a random
UUID.
+*
+* UUID hyphens ("-") will be replaced by underscores ("_").
+*
+* @return tableName
+*/
+ public static synchronized String getTempTableName() {
+ return ("temp_" + UUID.randomUUID().toString().replaceAll("-",
"_"))
+ .toLowerCase();
+ }
+
+ /**
+* Find the index of targetCol in string array
tableCols. It will ignore the case of the
+* tableCols.
+*
+* @param tableCols a string array among which to find the targetCol.
+* @param targetCol the targetCol to find.
+* @return the index of the targetCol, if not found, returns -1.
+*/
+ public static int findColIndex(String[] tableCols, String targetCol) {
+ Preconditions.checkNotNull(targetCol, "targetCol is null!");
+ for (int i = 0; i < tableCols.length; i++) {
+ if (targetCol.equalsIgnoreCase(tableCols[i])) {
+ return i;
+ }
+ }
+ return -1;
+ }
+
+ /**
+* Find the index of targetCol from the
tableSchema.
+*
+* @param tableSchema the TableSchema among which to find the targetCol.
+* @param targetCol the targetCols to find.
+* @return the index of the targetCol.
+*/
+ public static int findColIndex(TableSchema tableSchema, String
targetCol) {
+ return findColIndex(tableSchema.getFieldNames(), targetCol);
+ }
+
+ /**
+* Find the indices of targetCols in string array
tableCols. If
+* targetCols is
+* null, it will be replaced by the tableCols
+*
+* @param tableCols a string array among which to find the targetCols.
+* @param targetCols the targetCols to find.
+* @return the indices of the targetCols.
+*/
+ public static int[] findColIndices(String[] tableCols, String[]
targetCols) {
+ if (targetCols == null) {
+ int[] indices = new int[tableCols.length];
+ for (int i = 0; i < tableCols.length; i++) {
+ indices[i] = i;
+ }
+ return indices;
+ }
+ int[] indices = new int[targetCols.length];
+ for (int i = 0; i < indices.length; i++) {
+ indices[i] = findColIndex(tableCols, targetCols[i]);
+ }
+ return indices;
+ }
+
+ /**
+* Find the indices of targetCols from the
tableSchema.
+*
+* @param tableSchema the TableSchema among which to find the
targetCols.
+* @param targetCols the targetCols to find.
+* @return the indices of the targetCols.
+*/
+ public static int[] findColIndices(TableSchema tableSchema, String[]
targetCols) {
+ return findColIndices(tableSchema.getFieldNames(), targetCols);
+ }
+
+ /**
+* Find the types of the targetCols. If the targetCol not
exist, return null.
+
[GitHub] [flink] walterddr commented on a change in pull request #9404: [FLINK-13667][ml] Add the utility class for the Table
walterddr commented on a change in pull request #9404: [FLINK-13667][ml] Add
the utility class for the Table
URL: https://github.com/apache/flink/pull/9404#discussion_r355814236
##
File path:
flink-ml-parent/flink-ml-lib/src/main/java/org/apache/flink/ml/common/utils/TableUtil.java
##
@@ -0,0 +1,439 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.flink.ml.common.utils;
+
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.table.api.TableSchema;
+import org.apache.flink.types.Row;
+import org.apache.flink.util.Preconditions;
+
+import org.apache.flink.shaded.guava18.com.google.common.base.Joiner;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import java.util.UUID;
+
+/**
+ * Utility to operator to interact with Table contents, such as rows and
columns.
+ */
+public class TableUtil {
+ /**
+* Return a temp table named with prefix `temp_`, follow by a random
UUID.
+*
+* UUID hyphens ("-") will be replaced by underscores ("_").
+*
+* @return tableName
+*/
+ public static synchronized String getTempTableName() {
+ return ("temp_" + UUID.randomUUID().toString().replaceAll("-",
"_"))
+ .toLowerCase();
+ }
+
+ /**
+* Find the index of targetCol in string array
tableCols. It will ignore the case of the
+* tableCols.
+*
+* @param tableCols a string array among which to find the targetCol.
+* @param targetCol the targetCol to find.
+* @return the index of the targetCol, if not found, returns -1.
+*/
+ public static int findColIndex(String[] tableCols, String targetCol) {
+ Preconditions.checkNotNull(targetCol, "targetCol is null!");
+ for (int i = 0; i < tableCols.length; i++) {
+ if (targetCol.equalsIgnoreCase(tableCols[i])) {
+ return i;
+ }
+ }
+ return -1;
+ }
+
+ /**
+* Find the index of targetCol from the
tableSchema.
+*
+* @param tableSchema the TableSchema among which to find the targetCol.
+* @param targetCol the targetCols to find.
+* @return the index of the targetCol.
+*/
+ public static int findColIndex(TableSchema tableSchema, String
targetCol) {
+ return findColIndex(tableSchema.getFieldNames(), targetCol);
+ }
+
+ /**
+* Find the indices of targetCols in string array
tableCols. If
+* targetCols is
+* null, it will be replaced by the tableCols
+*
+* @param tableCols a string array among which to find the targetCols.
+* @param targetCols the targetCols to find.
+* @return the indices of the targetCols.
+*/
+ public static int[] findColIndices(String[] tableCols, String[]
targetCols) {
+ if (targetCols == null) {
+ int[] indices = new int[tableCols.length];
+ for (int i = 0; i < tableCols.length; i++) {
+ indices[i] = i;
+ }
+ return indices;
+ }
+ int[] indices = new int[targetCols.length];
+ for (int i = 0; i < indices.length; i++) {
+ indices[i] = findColIndex(tableCols, targetCols[i]);
+ }
+ return indices;
+ }
+
+ /**
+* Find the indices of targetCols from the
tableSchema.
+*
+* @param tableSchema the TableSchema among which to find the
targetCols.
+* @param targetCols the targetCols to find.
+* @return the indices of the targetCols.
+*/
+ public static int[] findColIndices(TableSchema tableSchema, String[]
targetCols) {
+ return findColIndices(tableSchema.getFieldNames(), targetCols);
+ }
+
+ /**
+* Find the types of the targetCols. If the targetCol not
exist, return null.
+
[GitHub] [flink] walterddr commented on a change in pull request #9404: [FLINK-13667][ml] Add the utility class for the Table
walterddr commented on a change in pull request #9404: [FLINK-13667][ml] Add
the utility class for the Table
URL: https://github.com/apache/flink/pull/9404#discussion_r355814170
##
File path:
flink-ml-parent/flink-ml-lib/src/main/java/org/apache/flink/ml/common/utils/TableUtil.java
##
@@ -0,0 +1,439 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.flink.ml.common.utils;
+
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.table.api.TableSchema;
+import org.apache.flink.types.Row;
+import org.apache.flink.util.Preconditions;
+
+import org.apache.flink.shaded.guava18.com.google.common.base.Joiner;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import java.util.UUID;
+
+/**
+ * Utility to operator to interact with Table contents, such as rows and
columns.
+ */
+public class TableUtil {
+ /**
+* Return a temp table named with prefix `temp_`, follow by a random
UUID.
+*
+* UUID hyphens ("-") will be replaced by underscores ("_").
+*
+* @return tableName
+*/
+ public static synchronized String getTempTableName() {
Review comment:
missing test for this API.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #10348: [FLINK-14951][tests] Harden the thread safety of State TTL backend tests
flinkbot edited a comment on issue #10348: [FLINK-14951][tests] Harden the thread safety of State TTL backend tests URL: https://github.com/apache/flink/pull/10348#issuecomment-559509076 ## CI report: * 54ecee627e7036f4d150aad330b9772406a19494 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/138580810) * 3e89b2b0d85dd29a76bb10807e2959a7d2ee8295 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] walterddr commented on a change in pull request #9355: [FLINK-13577][ml] Add an util class to build result row and generate …
walterddr commented on a change in pull request #9355: [FLINK-13577][ml] Add an
util class to build result row and generate …
URL: https://github.com/apache/flink/pull/9355#discussion_r355813464
##
File path:
flink-ml-parent/flink-ml-lib/src/test/java/org/apache/flink/ml/common/utils/OutputColsHelperTest.java
##
@@ -0,0 +1,261 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.flink.ml.common.utils;
+
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.table.api.TableSchema;
+import org.apache.flink.types.Row;
+
+import org.junit.Assert;
+import org.junit.Test;
+
+/**
+ * Unit test for OutputColsHelper.
+ */
+public class OutputColsHelperTest {
+
+ private TableSchema tableSchema = new TableSchema(
+ new String[]{"f0", "f1", "f2"},
+ new TypeInformation[]{
+ TypeInformation.of(String.class),
+ TypeInformation.of(Long.class),
+ TypeInformation.of(Integer.class)
+ }
+ );
+ private String[] reservedColNames = new String[]{"f0"};
+ private Row row = Row.of("a", 1L, 1);
+
+ @Test
+ public void testResultSchema() {
+ TableSchema expectSchema = new TableSchema(
+ new String[]{"f0", "f1", "f2", "res"},
+ new TypeInformation[]{
+ TypeInformation.of(String.class),
+ TypeInformation.of(Long.class),
+ TypeInformation.of(Integer.class),
+ TypeInformation.of(String.class)
+ }
+ );
+ OutputColsHelper helper = new OutputColsHelper(
+ tableSchema, "res",
+ TypeInformation.of(String.class)
+ );
+ Assert.assertEquals(helper.getResultSchema(), expectSchema);
Review comment:
@xuyang1706 according to the tradition of `junit`, the expected value should
be the first in the `assertXXX` calls. could you please fix this?
See: http://junit.sourceforge.net/javadoc/org/junit/Assert.html
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Commented] (FLINK-15129) Return JobClient instead of JobClient Future from executeAsync()
[
https://issues.apache.org/jira/browse/FLINK-15129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16992116#comment-16992116
]
Jeff Zhang commented on FLINK-15129:
Return JobClient directly make more sense for me.
> Return JobClient instead of JobClient Future from executeAsync()
>
>
> Key: FLINK-15129
> URL: https://issues.apache.org/jira/browse/FLINK-15129
> Project: Flink
> Issue Type: Sub-task
> Components: API / DataSet, API / DataStream
>Reporter: Aljoscha Krettek
>Priority: Major
>
> Currently, users have to write this when they want to use the {{JobClient}}:
> {code}
> CompletableFuture jobClientFuture = env.executeAsync();
> JobClient jobClient = jobClientFuture.get();
> // or use thenApply/thenCompose etc.
> {code}
> instead we could always return a {{JobClient}} right away and therefore
> remove one step for the user.
> I don't know if it's always the right choice, but currently we always return
> an already completed future that contains the {{JobClient}}. In the future we
> might want to return a future that actually completes at some later point, we
> would not be able to do this if we directly return a {{JobClient}} and would
> have to block in {{executeAsync()}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] KarmaGYZ commented on issue #10348: [FLINK-14951][tests] Harden the thread safety of State TTL backend tests
KarmaGYZ commented on issue #10348: [FLINK-14951][tests] Harden the thread safety of State TTL backend tests URL: https://github.com/apache/flink/pull/10348#issuecomment-563578121 Thanks for the review @azagrebin . Addressed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #10491: [FLINK-15093][streaming-java] StreamExecutionEnvironment does not cle…
flinkbot edited a comment on issue #10491: [FLINK-15093][streaming-java] StreamExecutionEnvironment does not cle… URL: https://github.com/apache/flink/pull/10491#issuecomment-563083959 ## CI report: * 2e942aa60f0807a6a41fd0483237d8b571fdca1f Travis: [CANCELED](https://travis-ci.com/flink-ci/flink/builds/140171167) * f9657812903c088739c262b4e418d69683c52989 Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/140180146) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=3341) * d25b4deaa79f06d8a2ba3534d48553bb72e191ea Travis: [CANCELED](https://travis-ci.com/flink-ci/flink/builds/140207958) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=3354) * 0eb37fee2dce639f45d933200b3732c4e444841b Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/140228860) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=3360) * 35b9a4e5e8d4767e431e69a0b8aba1000e9f402f Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/140233479) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=3364) * b9d0dc43b03f25fa366ccbb833693219d45a8ce7 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/140238481) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=3368) * 6812331d348892e6146be0f6a5cddf2401090ecb Travis: [PENDING](https://travis-ci.com/flink-ci/flink/builds/140340160) Azure: [PENDING](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=3382) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #10491: [FLINK-15093][streaming-java] StreamExecutionEnvironment does not cle…
flinkbot edited a comment on issue #10491: [FLINK-15093][streaming-java] StreamExecutionEnvironment does not cle… URL: https://github.com/apache/flink/pull/10491#issuecomment-563083959 ## CI report: * 2e942aa60f0807a6a41fd0483237d8b571fdca1f Travis: [CANCELED](https://travis-ci.com/flink-ci/flink/builds/140171167) * f9657812903c088739c262b4e418d69683c52989 Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/140180146) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=3341) * d25b4deaa79f06d8a2ba3534d48553bb72e191ea Travis: [CANCELED](https://travis-ci.com/flink-ci/flink/builds/140207958) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=3354) * 0eb37fee2dce639f45d933200b3732c4e444841b Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/140228860) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=3360) * 35b9a4e5e8d4767e431e69a0b8aba1000e9f402f Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/140233479) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=3364) * b9d0dc43b03f25fa366ccbb833693219d45a8ce7 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/140238481) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=3368) * 6812331d348892e6146be0f6a5cddf2401090ecb UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (FLINK-15052) Test transformation.clear() in sqlClient
[ https://issues.apache.org/jira/browse/FLINK-15052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Danny Chen updated FLINK-15052: --- Summary: Test transformation.clear() in sqlClient (was: sql client doesn't clear previous job graph ) > Test transformation.clear() in sqlClient > > > Key: FLINK-15052 > URL: https://issues.apache.org/jira/browse/FLINK-15052 > Project: Flink > Issue Type: Bug > Components: Table SQL / Client >Reporter: Kurt Young >Assignee: Danny Chen >Priority: Blocker > Fix For: 1.10.0 > > > when executing multiple commands from sql client, the later job graph will > include all job graphs which already executed. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on issue #10502: [FLINK-14825][state-processor-api][docs] Rework state processor api documentation
flinkbot edited a comment on issue #10502: [FLINK-14825][state-processor-api][docs] Rework state processor api documentation URL: https://github.com/apache/flink/pull/10502#issuecomment-563465099 ## CI report: * 0904024ed3b34e27bc8e98c92b162024d5f40d95 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/140322110) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=3380) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #10502: [FLINK-14825][state-processor-api][docs] Rework state processor api documentation
flinkbot edited a comment on issue #10502: [FLINK-14825][state-processor-api][docs] Rework state processor api documentation URL: https://github.com/apache/flink/pull/10502#issuecomment-563465099 ## CI report: * 0904024ed3b34e27bc8e98c92b162024d5f40d95 Travis: [PENDING](https://travis-ci.com/flink-ci/flink/builds/140322110) Azure: [PENDING](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=3380) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] tweise commented on a change in pull request #10285: [FLINK-14896] [flink-kinesis-connector] Set jackson and guava dependency to flink-shaded
tweise commented on a change in pull request #10285: [FLINK-14896] [flink-kinesis-connector] Set jackson and guava dependency to flink-shaded URL: https://github.com/apache/flink/pull/10285#discussion_r355724390 ## File path: flink-connectors/flink-connector-kinesis/pom.xml ## @@ -193,25 +193,30 @@ under the License. com.amazonaws:* + com.fasterxml:* com.google.protobuf:* org.apache.httpcomponents:* javax.xml.bind:jaxb-api - - com.google.protobuf - org.apache.flink.kinesis.shaded.com.google.protobuf + com.amazonaws + org.apache.flink.streaming.kinesis.shaded.com.amazonaws - com.amazonaws - org.apache.flink.kinesis.shaded.com.amazonaws + com.fasterxml + org.apache.flink.streaming.kinesis.shaded.com.fasterxml + + Review comment: I see it used in `KinesisProducer`, but not on the API surface. We should be able to use flink shaded in this case, since we already depend on it via `flink-streaming-java`: ``` [INFO] --- maven-dependency-plugin:3.1.1:tree (default-cli) @ flink-connector-kinesis_2.11 --- [INFO] org.apache.flink:flink-connector-kinesis_2.11:jar:1.11-SNAPSHOT [INFO] +- org.apache.flink:flink-streaming-java_2.11:jar:1.11-SNAPSHOT:provided [INFO] | +- org.apache.flink:flink-core:jar:1.11-SNAPSHOT:provided [INFO] | +- org.apache.flink:flink-runtime_2.11:jar:1.11-SNAPSHOT:provided [INFO] | +- org.apache.flink:flink-clients_2.11:jar:1.11-SNAPSHOT:provided [INFO] | | \- org.apache.flink:flink-optimizer_2.11:jar:1.11-SNAPSHOT:provided [INFO] | +- org.apache.flink:flink-shaded-guava:jar:18.0-9.0:provided [INFO] | \- org.apache.commons:commons-math3:jar:3.5:provided [INFO] +- com.google.guava:guava:jar:18.0:compile ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot commented on issue #10502: [FLINK-14825][state-processor-api][docs] Rework state processor api documentation
flinkbot commented on issue #10502: [FLINK-14825][state-processor-api][docs] Rework state processor api documentation URL: https://github.com/apache/flink/pull/10502#issuecomment-563465099 ## CI report: * 0904024ed3b34e27bc8e98c92b162024d5f40d95 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] tweise commented on issue #10285: [FLINK-14896] [flink-kinesis-connector] Set jackson and guava dependency to flink-shaded
tweise commented on issue #10285: [FLINK-14896] [flink-kinesis-connector] Set jackson and guava dependency to flink-shaded URL: https://github.com/apache/flink/pull/10285#issuecomment-563465284 Unfortunately there is a high probability of jackson libraries conflicting with older versions in user space. I would prefer it remains shaded as is. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot commented on issue #10502: [FLINK-14825][state-processor-api][docs] Rework state processor api documentation
flinkbot commented on issue #10502: [FLINK-14825][state-processor-api][docs] Rework state processor api documentation URL: https://github.com/apache/flink/pull/10502#issuecomment-563457964 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit 0904024ed3b34e27bc8e98c92b162024d5f40d95 (Mon Dec 09 21:55:15 UTC 2019) ✅no warnings Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
