[jira] [Assigned] (FLINK-10531) State TTL RocksDb backend end-to-end test failed on Travis
[
https://issues.apache.org/jira/browse/FLINK-10531?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Renjie Liu reassigned FLINK-10531:
--
Assignee: Renjie Liu
> State TTL RocksDb backend end-to-end test failed on Travis
> --
>
> Key: FLINK-10531
> URL: https://issues.apache.org/jira/browse/FLINK-10531
> Project: Flink
> Issue Type: Bug
> Components: Tests
>Affects Versions: 1.6.1
>Reporter: Till Rohrmann
>Assignee: Renjie Liu
>Priority: Critical
> Labels: test-stability
>
> The {{State TTL RocksDb backend end-to-end test}} end-to-end test failed on
> Travis.
> https://travis-ci.org/apache/flink/jobs/438226190
> https://api.travis-ci.org/v3/job/438226190/log.txt
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (FLINK-8953) Resolve unresolved field references in FieldComputer expressions
[
https://issues.apache.org/jira/browse/FLINK-8953?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Renjie Liu reassigned FLINK-8953:
-
Assignee: (was: Renjie Liu)
> Resolve unresolved field references in FieldComputer expressions
>
>
> Key: FLINK-8953
> URL: https://issues.apache.org/jira/browse/FLINK-8953
> Project: Flink
> Issue Type: Improvement
> Components: Table API & SQL
>Reporter: Timo Walther
>Priority: Major
>
> When implementing the {{FieldComputer.getExpression}} method, it is not
> possible to use API classes but only internal expression case classes.
> It would be great to also define timestamp extractors like:
> {code}
> def getExpression(fieldAccesses: Array[ResolvedFieldReference]): Expression
> = {
> // 'x.cast(Types.LONG)
> // ExpressionParser.parseExpression("x.cast(LONG)")
> }
> {code}
> An even better solution would be to provide different `getExpression()`
> methods that an implementor can override. The general goal should be to
> define this as natural as possible. In the future we should also support SQL:
> {code}
> def getJavaExpression(fieldAccesses: Array[ResolvedFieldReference]): String
> = {
> "x.cast(LONG)"
> }
> def getSQLExpression(fieldAccesses: Array[ResolvedFieldReference]): String
> = {
> "CAST(x AS LONG)"
> }
> {code}
> The final design is still up for discussion. These are just ideas.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (FLINK-10101) Mesos web ui url is missing.
Renjie Liu created FLINK-10101: -- Summary: Mesos web ui url is missing. Key: FLINK-10101 URL: https://issues.apache.org/jira/browse/FLINK-10101 Project: Flink Issue Type: Bug Components: Mesos Affects Versions: 1.5.2, 1.5.1, 1.5.0 Reporter: Renjie Liu Assignee: Renjie Liu Mesos web ui url is missing in new deploy mode. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (FLINK-9720) Introduce ResourceTag class for tag support in scheduling
[ https://issues.apache.org/jira/browse/FLINK-9720?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Renjie Liu updated FLINK-9720: -- Fix Version/s: (was: 1.6.0) 1.7.0 > Introduce ResourceTag class for tag support in scheduling > - > > Key: FLINK-9720 > URL: https://issues.apache.org/jira/browse/FLINK-9720 > Project: Flink > Issue Type: New Feature > Components: Scheduler >Affects Versions: 1.5.0 >Reporter: Renjie Liu >Assignee: Renjie Liu >Priority: Major > Labels: pull-request-available > Fix For: 1.7.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (FLINK-9662) Task manager isolation for jobs
[ https://issues.apache.org/jira/browse/FLINK-9662?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Renjie Liu updated FLINK-9662: -- Fix Version/s: (was: 1.6.0) 1.7.0 > Task manager isolation for jobs > --- > > Key: FLINK-9662 > URL: https://issues.apache.org/jira/browse/FLINK-9662 > Project: Flink > Issue Type: New Feature > Components: Distributed Coordination >Affects Versions: 1.5.0, 1.6.0 >Reporter: Renjie Liu >Assignee: Renjie Liu >Priority: Major > Fix For: 1.7.0 > > Attachments: job isolation sequence.jpg > > > Disable task manager sharing for different jobs. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (FLINK-9948) JSON format does not convert timestamps correctly
[
https://issues.apache.org/jira/browse/FLINK-9948?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Renjie Liu reassigned FLINK-9948:
-
Assignee: (was: Renjie Liu)
> JSON format does not convert timestamps correctly
> -
>
> Key: FLINK-9948
> URL: https://issues.apache.org/jira/browse/FLINK-9948
> Project: Flink
> Issue Type: Sub-task
> Components: Table API & SQL
>Reporter: Timo Walther
>Priority: Major
>
> Currently, the JSON format supports timestamps in the UTC timezone
> {{"1990-10-14T12:12:43Z"}}. However, the conversion in the current
> implementation is not correct since timestamps in SQL do not contain a
> timezone and depend on the local timezone of the JVM. Casting a rowtime into
> a string results in an undesired offset at the moment. Fixing this problem
> might depend on the bigger issue of proper timezone support.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (FLINK-9948) JSON format does not convert timestamps correctly
[
https://issues.apache.org/jira/browse/FLINK-9948?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Renjie Liu reassigned FLINK-9948:
-
Assignee: Renjie Liu
> JSON format does not convert timestamps correctly
> -
>
> Key: FLINK-9948
> URL: https://issues.apache.org/jira/browse/FLINK-9948
> Project: Flink
> Issue Type: Sub-task
> Components: Table API & SQL
>Reporter: Timo Walther
>Assignee: Renjie Liu
>Priority: Major
>
> Currently, the JSON format supports timestamps in the UTC timezone
> {{"1990-10-14T12:12:43Z"}}. However, the conversion in the current
> implementation is not correct since timestamps in SQL do not contain a
> timezone and depend on the local timezone of the JVM. Casting a rowtime into
> a string results in an undesired offset at the moment. Fixing this problem
> might depend on the bigger issue of proper timezone support.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (FLINK-9740) Support group windows over intervals of months
[
https://issues.apache.org/jira/browse/FLINK-9740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Renjie Liu reassigned FLINK-9740:
-
Assignee: Renjie Liu
> Support group windows over intervals of months
> ---
>
> Key: FLINK-9740
> URL: https://issues.apache.org/jira/browse/FLINK-9740
> Project: Flink
> Issue Type: New Feature
> Components: Table API & SQL
>Affects Versions: 1.5.0
>Reporter: Timo Walther
>Assignee: Renjie Liu
>Priority: Major
>
> Currently, time-based group windows can be defined using intervals of
> milliseconds such as {{.window(Tumble over 10.minutes on 'rowtime as 'w)}}.
> For some use cases it might useful to define windows of months (esp. in
> event-time) that work even with leap years and other special time cases.
> The following should be supported in Table API & SQL:
> {{.window(Tumble over 1.month on 'rowtime as 'w)}}
> {{.window(Tumble over 1.quarter on 'rowtime as 'w)}}
> {{.window(Tumble over 1.year on 'rowtime as 'w)}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (FLINK-8865) Add CLI query code completion in SQL Client
[ https://issues.apache.org/jira/browse/FLINK-8865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Renjie Liu reassigned FLINK-8865: - Assignee: (was: Renjie Liu) > Add CLI query code completion in SQL Client > --- > > Key: FLINK-8865 > URL: https://issues.apache.org/jira/browse/FLINK-8865 > Project: Flink > Issue Type: Sub-task > Components: Table API & SQL >Reporter: Timo Walther >Priority: Major > > This issue is a subtask of part two "Full Embedded SQL Client" of the > implementation plan mentioned in > [FLIP-24|https://cwiki.apache.org/confluence/display/FLINK/FLIP-24+-+SQL+Client]. > Calcite already offers a code completion functionality. It would be great if > we could expose this feature also through the SQL CLI Client. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-9936) Mesos resource manager unable to connect to master after failover
[ https://issues.apache.org/jira/browse/FLINK-9936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16561101#comment-16561101 ] Renjie Liu commented on FLINK-9936: --- [~gjy] I'm working on this and I've already made an internal version testing in our own deployment which works well. I'm going to publish our patch and add some tests to it. > Mesos resource manager unable to connect to master after failover > - > > Key: FLINK-9936 > URL: https://issues.apache.org/jira/browse/FLINK-9936 > Project: Flink > Issue Type: Bug > Components: Mesos, Scheduler >Affects Versions: 1.5.0, 1.5.1, 1.6.0 >Reporter: Renjie Liu >Assignee: Renjie Liu >Priority: Blocker > Fix For: 1.5.2, 1.6.0 > > > When deployed in mesos session cluster mode, the connector monitor keeps > reporting unable to connect to mesos after restart. In fact, scheduler driver > already connected to mesos master, but when the connected message is lost. > This is because leadership is not granted yet and fence id is not set, the > rpc service ignores the connected message. So we should connect to mesos > master after leadership is granted. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (FLINK-7470) Acquire RM leadership before registering with Mesos
[ https://issues.apache.org/jira/browse/FLINK-7470?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Renjie Liu closed FLINK-7470. - Resolution: Duplicate > Acquire RM leadership before registering with Mesos > --- > > Key: FLINK-7470 > URL: https://issues.apache.org/jira/browse/FLINK-7470 > Project: Flink > Issue Type: Bug > Components: Mesos >Reporter: Eron Wright >Priority: Major > Fix For: 1.7.0 > > > Mesos doesn't support fencing tokens in the scheduler protocol; it assumes > external leader election among scheduler instances. The last connection > wins; prior connections for a given framework ID are closed. > The Mesos RM should not register as a framework until it has acquired RM > leadership. Evolve the ResourceManager as necessary. One option is to > introduce an ResourceManagerRunner that acquires leadership before starting > the RM. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (FLINK-9936) Mesos resource manager unable to connect to master after failover
Renjie Liu created FLINK-9936: - Summary: Mesos resource manager unable to connect to master after failover Key: FLINK-9936 URL: https://issues.apache.org/jira/browse/FLINK-9936 Project: Flink Issue Type: Bug Components: Mesos, Scheduler Affects Versions: 1.5.1, 1.5.0 Reporter: Renjie Liu Assignee: Renjie Liu Fix For: 1.6.0 When deployed in mesos session cluster mode, the connector monitor keeps reporting unable to connect to mesos after restart. In fact, scheduler driver already connected to mesos master, but when the connected message is lost. This is because leadership is not granted yet and fence id is not set, the rpc service ignores the connected message. So we should connect to mesos master after leadership is granted. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (FLINK-9726) Enable job task manager isolation use tag support
[ https://issues.apache.org/jira/browse/FLINK-9726?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Renjie Liu closed FLINK-9726. - Resolution: Duplicate Duplicated > Enable job task manager isolation use tag support > - > > Key: FLINK-9726 > URL: https://issues.apache.org/jira/browse/FLINK-9726 > Project: Flink > Issue Type: New Feature > Components: Scheduler >Affects Versions: 1.5.0 >Reporter: Renjie Liu >Assignee: Renjie Liu >Priority: Major > Fix For: 1.6.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (FLINK-9096) flip 6 allocates different resources for different jobs
[ https://issues.apache.org/jira/browse/FLINK-9096?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Renjie Liu closed FLINK-9096. - Resolution: Duplicate Duplication of 5131 > flip 6 allocates different resources for different jobs > --- > > Key: FLINK-9096 > URL: https://issues.apache.org/jira/browse/FLINK-9096 > Project: Flink > Issue Type: Improvement > Components: Cluster Management >Affects Versions: 1.6.0 >Reporter: Renjie Liu >Assignee: Renjie Liu >Priority: Major > > Currently in flip6 session mode, the resource manager will allocate > containers for diffrent jobs. However, these containers are all the same > size, e.g. with same memory and cpu cores. Since different jobs have > different resource requirements, this may waste a lot of waste. The goal of > this issue is to allow resource manager to be able to allocate different > resources for different jobs, as per job cluster can do. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (FLINK-8865) Add CLI query code completion in SQL Client
[ https://issues.apache.org/jira/browse/FLINK-8865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Renjie Liu reassigned FLINK-8865: - Assignee: Renjie Liu > Add CLI query code completion in SQL Client > --- > > Key: FLINK-8865 > URL: https://issues.apache.org/jira/browse/FLINK-8865 > Project: Flink > Issue Type: Sub-task > Components: Table API & SQL >Reporter: Timo Walther >Assignee: Renjie Liu >Priority: Major > > This issue is a subtask of part two "Full Embedded SQL Client" of the > implementation plan mentioned in > [FLIP-24|https://cwiki.apache.org/confluence/display/FLINK/FLIP-24+-+SQL+Client]. > Calcite already offers a code completion functionality. It would be great if > we could expose this feature also through the SQL CLI Client. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-8858) Add support for INSERT INTO in SQL Client
[ https://issues.apache.org/jira/browse/FLINK-8858?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16533453#comment-16533453 ] Renjie Liu commented on FLINK-8858: --- Hi [~twalthr] That's ok since currently I have no time for that. > Add support for INSERT INTO in SQL Client > - > > Key: FLINK-8858 > URL: https://issues.apache.org/jira/browse/FLINK-8858 > Project: Flink > Issue Type: Sub-task > Components: Table API & SQL >Affects Versions: 1.6.0 >Reporter: Renjie Liu >Assignee: Renjie Liu >Priority: Major > > The current design of SQL Client embedded mode doesn't support long running > queries. It would be useful for simple jobs that can be expressed in a single > sql statement if we can submit sql statements stored in files as long running > queries. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (FLINK-9671) Add configuration to enable task manager isolation.
[ https://issues.apache.org/jira/browse/FLINK-9671?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Renjie Liu closed FLINK-9671. - Resolution: Won't Fix > Add configuration to enable task manager isolation. > --- > > Key: FLINK-9671 > URL: https://issues.apache.org/jira/browse/FLINK-9671 > Project: Flink > Issue Type: New Feature > Components: Distributed Coordination, Scheduler >Affects Versions: 1.5.0 >Reporter: Renjie Liu >Assignee: Renjie Liu >Priority: Major > Fix For: 1.5.1 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (FLINK-9727) Allow user to pass resource tag expression in command line
Renjie Liu created FLINK-9727: - Summary: Allow user to pass resource tag expression in command line Key: FLINK-9727 URL: https://issues.apache.org/jira/browse/FLINK-9727 Project: Flink Issue Type: New Feature Components: Scheduler Affects Versions: 1.5.0 Reporter: Renjie Liu Assignee: Renjie Liu Fix For: 1.6.0, 1.5.1 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (FLINK-9726) Enable job task manager isolation use tag support
Renjie Liu created FLINK-9726: - Summary: Enable job task manager isolation use tag support Key: FLINK-9726 URL: https://issues.apache.org/jira/browse/FLINK-9726 Project: Flink Issue Type: New Feature Components: Scheduler Affects Versions: 1.5.0 Reporter: Renjie Liu Assignee: Renjie Liu Fix For: 1.6.0, 1.5.1 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (FLINK-9725) Add tag support to standalone cluster manager.
Renjie Liu created FLINK-9725: - Summary: Add tag support to standalone cluster manager. Key: FLINK-9725 URL: https://issues.apache.org/jira/browse/FLINK-9725 Project: Flink Issue Type: New Feature Components: Scheduler Affects Versions: 1.5.0 Reporter: Renjie Liu Assignee: Renjie Liu Fix For: 1.6.0, 1.5.1 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (FLINK-9724) Add tag support to yarn resource manager
Renjie Liu created FLINK-9724: - Summary: Add tag support to yarn resource manager Key: FLINK-9724 URL: https://issues.apache.org/jira/browse/FLINK-9724 Project: Flink Issue Type: New Feature Components: Scheduler Affects Versions: 1.5.0 Reporter: Renjie Liu Assignee: Renjie Liu Fix For: 1.6.0, 1.5.1 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (FLINK-9723) Add tags support to mesos resource manager
Renjie Liu created FLINK-9723: - Summary: Add tags support to mesos resource manager Key: FLINK-9723 URL: https://issues.apache.org/jira/browse/FLINK-9723 Project: Flink Issue Type: New Feature Components: Scheduler Affects Versions: 1.5.0 Reporter: Renjie Liu Assignee: Renjie Liu Fix For: 1.6.0, 1.5.1 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (FLINK-9722) Add resource tag predicate into slot manager
Renjie Liu created FLINK-9722: - Summary: Add resource tag predicate into slot manager Key: FLINK-9722 URL: https://issues.apache.org/jira/browse/FLINK-9722 Project: Flink Issue Type: New Feature Components: Scheduler Affects Versions: 1.5.0 Reporter: Renjie Liu Assignee: Renjie Liu Fix For: 1.6.0, 1.5.1 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (FLINK-9721) Introduce ResourceTagExpression class
Renjie Liu created FLINK-9721: - Summary: Introduce ResourceTagExpression class Key: FLINK-9721 URL: https://issues.apache.org/jira/browse/FLINK-9721 Project: Flink Issue Type: New Feature Components: Scheduler Affects Versions: 1.5.0 Reporter: Renjie Liu Assignee: Renjie Liu Fix For: 1.6.0, 1.5.1 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (FLINK-9720) Introduce ResourceTag class for tag support in scheduling
Renjie Liu created FLINK-9720: - Summary: Introduce ResourceTag class for tag support in scheduling Key: FLINK-9720 URL: https://issues.apache.org/jira/browse/FLINK-9720 Project: Flink Issue Type: New Feature Components: Scheduler Affects Versions: 1.5.0 Reporter: Renjie Liu Assignee: Renjie Liu Fix For: 1.6.0, 1.5.1 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (FLINK-9719) Mesos should use hostname as default jobmanager.rpc.address
Renjie Liu created FLINK-9719: - Summary: Mesos should use hostname as default jobmanager.rpc.address Key: FLINK-9719 URL: https://issues.apache.org/jira/browse/FLINK-9719 Project: Flink Issue Type: Bug Components: Mesos Affects Versions: 1.5.0 Reporter: Renjie Liu Assignee: Renjie Liu Fix For: 1.5.1 When running session cluster, flink uses jobmanager.rpc.address from config file or command line arguments. However, when deploying session cluster on mesos in high availability mode, we need to run entry point in scheduling systems such as marathon, in which case we can't assign hostname to jobmanager.rpc.address or rest.address, so we need to use hostname as default value. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (FLINK-9670) Introduce slot manager factory
[ https://issues.apache.org/jira/browse/FLINK-9670?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Renjie Liu closed FLINK-9670. - Resolution: Invalid > Introduce slot manager factory > -- > > Key: FLINK-9670 > URL: https://issues.apache.org/jira/browse/FLINK-9670 > Project: Flink > Issue Type: New Feature > Components: Distributed Coordination, Scheduler >Affects Versions: 1.5.0 >Reporter: Renjie Liu >Assignee: Renjie Liu >Priority: Major > Fix For: 1.5.1 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (FLINK-9669) Introduce task manager assignment store
[ https://issues.apache.org/jira/browse/FLINK-9669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Renjie Liu closed FLINK-9669. - Resolution: Invalid > Introduce task manager assignment store > --- > > Key: FLINK-9669 > URL: https://issues.apache.org/jira/browse/FLINK-9669 > Project: Flink > Issue Type: New Feature > Components: Distributed Coordination, Scheduler >Affects Versions: 1.5.0 >Reporter: Renjie Liu >Assignee: Renjie Liu >Priority: Major > Labels: pull-request-available > Fix For: 1.5.1 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-9662) Task manager isolation for jobs
[ https://issues.apache.org/jira/browse/FLINK-9662?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16527474#comment-16527474 ] Renjie Liu commented on FLINK-9662: --- [~till.rohrmann] [~elevy] According to our discussion above, I've updated the design doc, please help to review this. > Task manager isolation for jobs > --- > > Key: FLINK-9662 > URL: https://issues.apache.org/jira/browse/FLINK-9662 > Project: Flink > Issue Type: New Feature > Components: Distributed Coordination >Affects Versions: 1.5.0, 1.6.0 >Reporter: Renjie Liu >Assignee: Renjie Liu >Priority: Major > Fix For: 1.6.0 > > Attachments: job isolation sequence.jpg > > > Disable task manager sharing for different jobs. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-9662) Task manager isolation for jobs
[ https://issues.apache.org/jira/browse/FLINK-9662?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16526247#comment-16526247 ] Renjie Liu commented on FLINK-9662: --- Would it be better if we reuse the existing job id field in SlotStatus class? Till Rohrmann (JIRA) 于 2018年6月28日周四 下午7:29写道： -- Liu, Renjie Software Engineer, MVAD > Task manager isolation for jobs > --- > > Key: FLINK-9662 > URL: https://issues.apache.org/jira/browse/FLINK-9662 > Project: Flink > Issue Type: New Feature > Components: Distributed Coordination >Affects Versions: 1.5.0, 1.6.0 >Reporter: Renjie Liu >Assignee: Renjie Liu >Priority: Major > Fix For: 1.6.0 > > Attachments: job isolation sequence.jpg > > > Disable task manager sharing for different jobs. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-9662) Task manager isolation for jobs
[ https://issues.apache.org/jira/browse/FLINK-9662?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16525864#comment-16525864 ] Renjie Liu commented on FLINK-9662: --- [~till.rohrmann] [~elevy] [~gzm55] I think it's quite tricky to use job id as tags to achieve task manager isolation. In my opinion, task manage isolation and resource tags are two problems and should be adressed separately. To share resources with same tag requirements, we just disable job isolation. But I agree to store job id assignment in task managers. And I've drawn a sequence diagram for the new design. !job isolation sequence.jpg! As with [~gzm55] 's problem, I think it's better to use expressions to define tag requirements. > Task manager isolation for jobs > --- > > Key: FLINK-9662 > URL: https://issues.apache.org/jira/browse/FLINK-9662 > Project: Flink > Issue Type: New Feature > Components: Distributed Coordination >Affects Versions: 1.5.0, 1.6.0 >Reporter: Renjie Liu >Assignee: Renjie Liu >Priority: Major > Fix For: 1.6.0 > > Attachments: job isolation sequence.jpg > > > Disable task manager sharing for different jobs. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (FLINK-9662) Task manager isolation for jobs
[ https://issues.apache.org/jira/browse/FLINK-9662?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Renjie Liu updated FLINK-9662: -- Attachment: job isolation sequence.jpg > Task manager isolation for jobs > --- > > Key: FLINK-9662 > URL: https://issues.apache.org/jira/browse/FLINK-9662 > Project: Flink > Issue Type: New Feature > Components: Distributed Coordination >Affects Versions: 1.5.0, 1.6.0 >Reporter: Renjie Liu >Assignee: Renjie Liu >Priority: Major > Fix For: 1.6.0 > > Attachments: job isolation sequence.jpg > > > Disable task manager sharing for different jobs. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (FLINK-9671) Add configuration to enable task manager isolation.
Renjie Liu created FLINK-9671: - Summary: Add configuration to enable task manager isolation. Key: FLINK-9671 URL: https://issues.apache.org/jira/browse/FLINK-9671 Project: Flink Issue Type: New Feature Components: Distributed Coordination, Scheduler Affects Versions: 1.5.0 Reporter: Renjie Liu Assignee: Renjie Liu Fix For: 1.5.1 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (FLINK-9670) Introduce slot manager factory
Renjie Liu created FLINK-9670: - Summary: Introduce slot manager factory Key: FLINK-9670 URL: https://issues.apache.org/jira/browse/FLINK-9670 Project: Flink Issue Type: New Feature Components: Distributed Coordination, Scheduler Affects Versions: 1.5.0 Reporter: Renjie Liu Assignee: Renjie Liu Fix For: 1.5.1 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (FLINK-9669) Introduce task manager assignment store
Renjie Liu created FLINK-9669: - Summary: Introduce task manager assignment store Key: FLINK-9669 URL: https://issues.apache.org/jira/browse/FLINK-9669 Project: Flink Issue Type: New Feature Components: Distributed Coordination, Scheduler Affects Versions: 1.5.0 Reporter: Renjie Liu Assignee: Renjie Liu Fix For: 1.5.1 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-8886) Job isolation via scheduling in shared cluster
[ https://issues.apache.org/jira/browse/FLINK-8886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16523444#comment-16523444 ] Renjie Liu commented on FLINK-8886: --- [~till.rohrmann], [~elevy], [~mingleizhang] I've created an [issue|https://issues.apache.org/jira/browse/FLINK-9662] for the first part of problem and wrote a design doc for it, please help to review. > Job isolation via scheduling in shared cluster > -- > > Key: FLINK-8886 > URL: https://issues.apache.org/jira/browse/FLINK-8886 > Project: Flink > Issue Type: Improvement > Components: Distributed Coordination, Local Runtime, Scheduler >Affects Versions: 1.5.0 >Reporter: Elias Levy >Assignee: Renjie Liu >Priority: Major > > Flink's TaskManager executes tasks from different jobs within the same JVM as > threads. We prefer to isolate different jobs on their own JVM. Thus, we > must use different TMs for different jobs. As currently the scheduler will > allocate task slots within a TM to tasks from different jobs, that means we > must stand up one cluster per job. This is wasteful, as it requires at least > two JobManagers per cluster for high-availability, and the JMs have low > utilization. > Additionally, different jobs may require different resources. Some jobs are > compute heavy. Some are IO heavy (lots of state in RocksDB). At the moment > the scheduler threats all TMs are equivalent, except possibly in their number > of available task slots. Thus, one is required to stand up multiple cluster > if there is a need for different types of TMs. > It would be useful if one could specify requirements on job, such that they > are only scheduled on a subset of TMs. Properly configured, that would > permit isolation of jobs in a shared cluster and scheduling of jobs with > specific resource needs. > One possible implementation is to specify a set of tags on the TM config file > which the TMs used when registering with the JM, and another set of tags > configured within the job or supplied when submitting the job. The scheduler > could then match the tags in the job with the tags in the TMs. In a > restrictive mode the scheduler would assign a job task to a TM only if all > tags match. In a relaxed mode the scheduler could assign a job task to a TM > if there is a partial match, while giving preference to a more accurate match. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (FLINK-9663) Allow jobs to be scheduled to hosts that meet requirements.
[ https://issues.apache.org/jira/browse/FLINK-9663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Renjie Liu updated FLINK-9663: -- Summary: Allow jobs to be scheduled to hosts that meet requirements. (was: Allow jobs to be scheduled to task managers with special tags) > Allow jobs to be scheduled to hosts that meet requirements. > --- > > Key: FLINK-9663 > URL: https://issues.apache.org/jira/browse/FLINK-9663 > Project: Flink > Issue Type: New Feature > Components: Distributed Coordination, Scheduler >Affects Versions: 1.5.0 >Reporter: Renjie Liu >Assignee: Renjie Liu >Priority: Major > > Different jobs may have different requirements for resources, e.g. large > network bindwidth. We need to allow jobs to be scheduled to machines that > meets these requirements. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (FLINK-9663) Allow jobs to be scheduled to task managers with special tags
Renjie Liu created FLINK-9663: - Summary: Allow jobs to be scheduled to task managers with special tags Key: FLINK-9663 URL: https://issues.apache.org/jira/browse/FLINK-9663 Project: Flink Issue Type: New Feature Components: Distributed Coordination, Scheduler Affects Versions: 1.5.0 Reporter: Renjie Liu Assignee: Renjie Liu Different jobs may have different requirements for resources, e.g. large network bindwidth. We need to allow jobs to be scheduled to machines that meets these requirements. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (FLINK-9662) Task manager isolation for jobs
Renjie Liu created FLINK-9662: - Summary: Task manager isolation for jobs Key: FLINK-9662 URL: https://issues.apache.org/jira/browse/FLINK-9662 Project: Flink Issue Type: New Feature Components: Distributed Coordination Affects Versions: 1.5.0 Reporter: Renjie Liu Assignee: Renjie Liu Fix For: 1.5.1 Disable task manager sharing for different jobs. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (FLINK-8886) Job isolation via scheduling in shared cluster
[ https://issues.apache.org/jira/browse/FLINK-8886?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Renjie Liu reassigned FLINK-8886: - Assignee: Renjie Liu > Job isolation via scheduling in shared cluster > -- > > Key: FLINK-8886 > URL: https://issues.apache.org/jira/browse/FLINK-8886 > Project: Flink > Issue Type: Improvement > Components: Distributed Coordination, Local Runtime, Scheduler >Affects Versions: 1.5.0 >Reporter: Elias Levy >Assignee: Renjie Liu >Priority: Major > > Flink's TaskManager executes tasks from different jobs within the same JVM as > threads. We prefer to isolate different jobs on their own JVM. Thus, we > must use different TMs for different jobs. As currently the scheduler will > allocate task slots within a TM to tasks from different jobs, that means we > must stand up one cluster per job. This is wasteful, as it requires at least > two JobManagers per cluster for high-availability, and the JMs have low > utilization. > Additionally, different jobs may require different resources. Some jobs are > compute heavy. Some are IO heavy (lots of state in RocksDB). At the moment > the scheduler threats all TMs are equivalent, except possibly in their number > of available task slots. Thus, one is required to stand up multiple cluster > if there is a need for different types of TMs. > It would be useful if one could specify requirements on job, such that they > are only scheduled on a subset of TMs. Properly configured, that would > permit isolation of jobs in a shared cluster and scheduling of jobs with > specific resource needs. > One possible implementation is to specify a set of tags on the TM config file > which the TMs used when registering with the JM, and another set of tags > configured within the job or supplied when submitting the job. The scheduler > could then match the tags in the job with the tags in the TMs. In a > restrictive mode the scheduler would assign a job task to a TM only if all > tags match. In a relaxed mode the scheduler could assign a job task to a TM > if there is a partial match, while giving preference to a more accurate match. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-8886) Job isolation via scheduling in shared cluster
[ https://issues.apache.org/jira/browse/FLINK-8886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16502997#comment-16502997 ] Renjie Liu commented on FLINK-8886: --- I think there are two parts of this issue: # Task manager isolation. Jobs should be isolated on different jvms. # Resource isolation. Different jobs assigned to different tms. How about we open two sub issues for this? > Job isolation via scheduling in shared cluster > -- > > Key: FLINK-8886 > URL: https://issues.apache.org/jira/browse/FLINK-8886 > Project: Flink > Issue Type: Improvement > Components: Distributed Coordination, Local Runtime, Scheduler >Affects Versions: 1.5.0 >Reporter: Elias Levy >Priority: Major > > Flink's TaskManager executes tasks from different jobs within the same JVM as > threads. We prefer to isolate different jobs on their own JVM. Thus, we > must use different TMs for different jobs. As currently the scheduler will > allocate task slots within a TM to tasks from different jobs, that means we > must stand up one cluster per job. This is wasteful, as it requires at least > two JobManagers per cluster for high-availability, and the JMs have low > utilization. > Additionally, different jobs may require different resources. Some jobs are > compute heavy. Some are IO heavy (lots of state in RocksDB). At the moment > the scheduler threats all TMs are equivalent, except possibly in their number > of available task slots. Thus, one is required to stand up multiple cluster > if there is a need for different types of TMs. > It would be useful if one could specify requirements on job, such that they > are only scheduled on a subset of TMs. Properly configured, that would > permit isolation of jobs in a shared cluster and scheduling of jobs with > specific resource needs. > One possible implementation is to specify a set of tags on the TM config file > which the TMs used when registering with the JM, and another set of tags > configured within the job or supplied when submitting the job. The scheduler > could then match the tags in the job with the tags in the TMs. In a > restrictive mode the scheduler would assign a job task to a TM only if all > tags match. In a relaxed mode the scheduler could assign a job task to a TM > if there is a partial match, while giving preference to a more accurate match. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (FLINK-8953) Resolve unresolved field references in FieldComputer expressions
[
https://issues.apache.org/jira/browse/FLINK-8953?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Renjie Liu reassigned FLINK-8953:
-
Assignee: Renjie Liu
> Resolve unresolved field references in FieldComputer expressions
>
>
> Key: FLINK-8953
> URL: https://issues.apache.org/jira/browse/FLINK-8953
> Project: Flink
> Issue Type: Improvement
> Components: Table API & SQL
>Reporter: Timo Walther
>Assignee: Renjie Liu
>Priority: Major
>
> When implementing the {{FieldComputer.getExpression}} method, it is not
> possible to use API classes but only internal expression case classes.
> It would be great to also define timestamp extractors like:
> {code}
> def getExpression(fieldAccesses: Array[ResolvedFieldReference]): Expression
> = {
> // 'x.cast(Types.LONG)
> // ExpressionParser.parseExpression("x.cast(LONG)")
> }
> {code}
> An even better solution would be to provide different `getExpression()`
> methods that an implementor can override. The general goal should be to
> define this as natural as possible. In the future we should also support SQL:
> {code}
> def getJavaExpression(fieldAccesses: Array[ResolvedFieldReference]): String
> = {
> "x.cast(LONG)"
> }
> def getSQLExpression(fieldAccesses: Array[ResolvedFieldReference]): String
> = {
> "CAST(x AS LONG)"
> }
> {code}
> The final design is still up for discussion. These are just ideas.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (FLINK-9117) Disable artifact download in mesos container environment.
Renjie Liu created FLINK-9117: - Summary: Disable artifact download in mesos container environment. Key: FLINK-9117 URL: https://issues.apache.org/jira/browse/FLINK-9117 Project: Flink Issue Type: Bug Components: Cluster Management Affects Versions: 1.5.0 Reporter: Renjie Liu Assignee: Renjie Liu Fix For: 1.6.0 In the current implementation, the mesos fetcher needs to download all artifacts from artifact server. However this is not necessary when mesos.resourcemanager.tasks.container.type is set to docker since we can include docker artifacts in the docker image. So the issue will add a configuration so that we can disable downloading of artifacts. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (FLINK-9077) flip6 cluster framework doesn't work on mesos
[
https://issues.apache.org/jira/browse/FLINK-9077?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Renjie Liu closed FLINK-9077.
-
Resolution: Not A Problem
It has been proven that this is caused by my mistake. When starting
mesos-appmaster.sh, we also need to put the following configurations in the
command line so that task manager can find jobmanager.
*-Djobmanager.rpc.address=xxx -Djobmanager.rpc.port=6123
-Dtaskmanager.numberOfTaskSlots=5*
> flip6 cluster framework doesn't work on mesos
> -
>
> Key: FLINK-9077
> URL: https://issues.apache.org/jira/browse/FLINK-9077
> Project: Flink
> Issue Type: Bug
> Components: Cluster Management
>Affects Versions: 1.5.0
>Reporter: Renjie Liu
>Assignee: Renjie Liu
>Priority: Blocker
> Fix For: 1.5.0
>
> Attachments: flink-liurj-mesos-appmaster-qt9ss.prod.mediav.com.log
>
>
> I'm testing flip6 on mesos, and the following is my config:
>
> _{color:#a64d79}jobmanager.rpc.address:
> [qt9ss.prod.mediav.com|http://qt9ss.prod.mediav.com/]{color}_
> _{color:#a64d79}jobmanager.rpc.port: 6123{color}_
> _{color:#a64d79}jobmanager.heap.mb: 1024{color}_
> _{color:#a64d79}taskmanager.heap.mb: 1024{color}_
> _{color:#a64d79}taskmanager.numberOfTaskSlots: 5{color}_
> _{color:#a64d79}parallelism.default: 1{color}_
> _{color:#a64d79}web.port: 8081{color}_
> _{color:#a64d79}mesos.master:
> zk://[dk71ss.jx.shbt2.qihoo.net:2191|http://dk71ss.jx.shbt2.qihoo.net:2191/],[dk72ss.jx.shbt2.qihoo.net:2191|http://dk72ss.jx.shbt2.qihoo.net:2191/],[dk5ss.jx.shbt2.qihoo.net:2191/mesos|http://dk5ss.jx.shbt2.qihoo.net:2191/mesos]{color}_
> _{color:#a64d79}mesos.resourcemanager.tasks.container.type: docker{color}_
> _{color:#a64d79}[mesos.resourcemanager.tasks.container.image.name|http://mesos.resourcemanager.tasks.container.image.name/]:
>
> [dk1ss.prod.mediav.com:5000/adq/flink:1.6.0-SNAPSHOT|http://dk1ss.prod.mediav.com:5000/adq/flink:1.6.0-SNAPSHOT]{color}_
> _{color:#a64d79}mesos.resourcemanager.framework.user: mediav{color}_
> _{color:#a64d79}mesos.resourcemanager.tasks.cpus: 5{color}_
> _{color:#a64d79}mesos.resourcemanager.tasks.mem: 10240{color}_
> _{color:#a64d79}[mesos.resourcemanager.framework.name|http://mesos.resourcemanager.framework.name/]:
> Flink{color}_
> _{color:#a64d79}mesos.failover-timeout: 60{color}_
>
> {color:#a64d79}{color:#33}When I submit a job with parallism 1, the job
> manager keeps reporting that slot can't be rallocated. Attached is my log of
> jobmanager.{color}{color}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (FLINK-9077) flip6 cluster framework doesn't work on mesos
[
https://issues.apache.org/jira/browse/FLINK-9077?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Renjie Liu updated FLINK-9077:
--
Priority: Blocker (was: Major)
> flip6 cluster framework doesn't work on mesos
> -
>
> Key: FLINK-9077
> URL: https://issues.apache.org/jira/browse/FLINK-9077
> Project: Flink
> Issue Type: Bug
> Components: Cluster Management
>Affects Versions: 1.5.0
>Reporter: Renjie Liu
>Assignee: Renjie Liu
>Priority: Blocker
> Fix For: 1.5.0
>
> Attachments: flink-liurj-mesos-appmaster-qt9ss.prod.mediav.com.log
>
>
> I'm testing flip6 on mesos, and the following is my config:
>
> _{color:#a64d79}jobmanager.rpc.address:
> [qt9ss.prod.mediav.com|http://qt9ss.prod.mediav.com/]{color}_
> _{color:#a64d79}jobmanager.rpc.port: 6123{color}_
> _{color:#a64d79}jobmanager.heap.mb: 1024{color}_
> _{color:#a64d79}taskmanager.heap.mb: 1024{color}_
> _{color:#a64d79}taskmanager.numberOfTaskSlots: 5{color}_
> _{color:#a64d79}parallelism.default: 1{color}_
> _{color:#a64d79}web.port: 8081{color}_
> _{color:#a64d79}mesos.master:
> zk://[dk71ss.jx.shbt2.qihoo.net:2191|http://dk71ss.jx.shbt2.qihoo.net:2191/],[dk72ss.jx.shbt2.qihoo.net:2191|http://dk72ss.jx.shbt2.qihoo.net:2191/],[dk5ss.jx.shbt2.qihoo.net:2191/mesos|http://dk5ss.jx.shbt2.qihoo.net:2191/mesos]{color}_
> _{color:#a64d79}mesos.resourcemanager.tasks.container.type: docker{color}_
> _{color:#a64d79}[mesos.resourcemanager.tasks.container.image.name|http://mesos.resourcemanager.tasks.container.image.name/]:
>
> [dk1ss.prod.mediav.com:5000/adq/flink:1.6.0-SNAPSHOT|http://dk1ss.prod.mediav.com:5000/adq/flink:1.6.0-SNAPSHOT]{color}_
> _{color:#a64d79}mesos.resourcemanager.framework.user: mediav{color}_
> _{color:#a64d79}mesos.resourcemanager.tasks.cpus: 5{color}_
> _{color:#a64d79}mesos.resourcemanager.tasks.mem: 10240{color}_
> _{color:#a64d79}[mesos.resourcemanager.framework.name|http://mesos.resourcemanager.framework.name/]:
> Flink{color}_
> _{color:#a64d79}mesos.failover-timeout: 60{color}_
>
> {color:#a64d79}{color:#33}When I submit a job with parallism 1, the job
> manager keeps reporting that slot can't be rallocated. Attached is my log of
> jobmanager.{color}{color}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (FLINK-9096) flip 6 allocates different resources for different jobs
Renjie Liu created FLINK-9096: - Summary: flip 6 allocates different resources for different jobs Key: FLINK-9096 URL: https://issues.apache.org/jira/browse/FLINK-9096 Project: Flink Issue Type: Improvement Components: Cluster Management Affects Versions: 1.6.0 Reporter: Renjie Liu Assignee: Renjie Liu Currently in flip6 session mode, the resource manager will allocate containers for diffrent jobs. However, these containers are all the same size, e.g. with same memory and cpu cores. Since different jobs have different resource requirements, this may waste a lot of waste. The goal of this issue is to allow resource manager to be able to allocate different resources for different jobs, as per job cluster can do. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (FLINK-9077) flip6 cluster framework doesn't work on mesos
[
https://issues.apache.org/jira/browse/FLINK-9077?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Renjie Liu updated FLINK-9077:
--
Description:
I'm testing flip6 on mesos, and the following is my config:
_{color:#a64d79}jobmanager.rpc.address:
[qt9ss.prod.mediav.com|http://qt9ss.prod.mediav.com/]{color}_
_{color:#a64d79}jobmanager.rpc.port: 6123{color}_
_{color:#a64d79}jobmanager.heap.mb: 1024{color}_
_{color:#a64d79}taskmanager.heap.mb: 1024{color}_
_{color:#a64d79}taskmanager.numberOfTaskSlots: 5{color}_
_{color:#a64d79}parallelism.default: 1{color}_
_{color:#a64d79}web.port: 8081{color}_
_{color:#a64d79}mesos.master:
zk://[dk71ss.jx.shbt2.qihoo.net:2191|http://dk71ss.jx.shbt2.qihoo.net:2191/],[dk72ss.jx.shbt2.qihoo.net:2191|http://dk72ss.jx.shbt2.qihoo.net:2191/],[dk5ss.jx.shbt2.qihoo.net:2191/mesos|http://dk5ss.jx.shbt2.qihoo.net:2191/mesos]{color}_
_{color:#a64d79}mesos.resourcemanager.tasks.container.type: docker{color}_
_{color:#a64d79}[mesos.resourcemanager.tasks.container.image.name|http://mesos.resourcemanager.tasks.container.image.name/]:
[dk1ss.prod.mediav.com:5000/adq/flink:1.6.0-SNAPSHOT|http://dk1ss.prod.mediav.com:5000/adq/flink:1.6.0-SNAPSHOT]{color}_
_{color:#a64d79}mesos.resourcemanager.framework.user: mediav{color}_
_{color:#a64d79}mesos.resourcemanager.tasks.cpus: 5{color}_
_{color:#a64d79}mesos.resourcemanager.tasks.mem: 10240{color}_
_{color:#a64d79}[mesos.resourcemanager.framework.name|http://mesos.resourcemanager.framework.name/]:
Flink{color}_
_{color:#a64d79}mesos.failover-timeout: 60{color}_
{color:#a64d79}{color:#33}When I submit a job with parallism 1, the job
manager keeps reporting that slot can't be rallocated. Attached is my log of
jobmanager.{color}{color}
> flip6 cluster framework doesn't work on mesos
> -
>
> Key: FLINK-9077
> URL: https://issues.apache.org/jira/browse/FLINK-9077
> Project: Flink
> Issue Type: Bug
> Components: Cluster Management
>Affects Versions: 1.5.0
>Reporter: Renjie Liu
>Assignee: Renjie Liu
>Priority: Major
> Fix For: 1.5.0
>
>
> I'm testing flip6 on mesos, and the following is my config:
>
> _{color:#a64d79}jobmanager.rpc.address:
> [qt9ss.prod.mediav.com|http://qt9ss.prod.mediav.com/]{color}_
> _{color:#a64d79}jobmanager.rpc.port: 6123{color}_
> _{color:#a64d79}jobmanager.heap.mb: 1024{color}_
> _{color:#a64d79}taskmanager.heap.mb: 1024{color}_
> _{color:#a64d79}taskmanager.numberOfTaskSlots: 5{color}_
> _{color:#a64d79}parallelism.default: 1{color}_
> _{color:#a64d79}web.port: 8081{color}_
> _{color:#a64d79}mesos.master:
> zk://[dk71ss.jx.shbt2.qihoo.net:2191|http://dk71ss.jx.shbt2.qihoo.net:2191/],[dk72ss.jx.shbt2.qihoo.net:2191|http://dk72ss.jx.shbt2.qihoo.net:2191/],[dk5ss.jx.shbt2.qihoo.net:2191/mesos|http://dk5ss.jx.shbt2.qihoo.net:2191/mesos]{color}_
> _{color:#a64d79}mesos.resourcemanager.tasks.container.type: docker{color}_
> _{color:#a64d79}[mesos.resourcemanager.tasks.container.image.name|http://mesos.resourcemanager.tasks.container.image.name/]:
>
> [dk1ss.prod.mediav.com:5000/adq/flink:1.6.0-SNAPSHOT|http://dk1ss.prod.mediav.com:5000/adq/flink:1.6.0-SNAPSHOT]{color}_
> _{color:#a64d79}mesos.resourcemanager.framework.user: mediav{color}_
> _{color:#a64d79}mesos.resourcemanager.tasks.cpus: 5{color}_
> _{color:#a64d79}mesos.resourcemanager.tasks.mem: 10240{color}_
> _{color:#a64d79}[mesos.resourcemanager.framework.name|http://mesos.resourcemanager.framework.name/]:
> Flink{color}_
> _{color:#a64d79}mesos.failover-timeout: 60{color}_
>
> {color:#a64d79}{color:#33}When I submit a job with parallism 1, the job
> manager keeps reporting that slot can't be rallocated. Attached is my log of
> jobmanager.{color}{color}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (FLINK-9077) flip6 cluster framework doesn't work on mesos
Renjie Liu created FLINK-9077: - Summary: flip6 cluster framework doesn't work on mesos Key: FLINK-9077 URL: https://issues.apache.org/jira/browse/FLINK-9077 Project: Flink Issue Type: Bug Components: Cluster Management Affects Versions: 1.5.0 Reporter: Renjie Liu Assignee: Renjie Liu Fix For: 1.5.0 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (FLINK-9077) flip6 cluster framework doesn't work on mesos
[
https://issues.apache.org/jira/browse/FLINK-9077?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Renjie Liu updated FLINK-9077:
--
Attachment: flink-liurj-mesos-appmaster-qt9ss.prod.mediav.com.log
> flip6 cluster framework doesn't work on mesos
> -
>
> Key: FLINK-9077
> URL: https://issues.apache.org/jira/browse/FLINK-9077
> Project: Flink
> Issue Type: Bug
> Components: Cluster Management
>Affects Versions: 1.5.0
>Reporter: Renjie Liu
>Assignee: Renjie Liu
>Priority: Major
> Fix For: 1.5.0
>
> Attachments: flink-liurj-mesos-appmaster-qt9ss.prod.mediav.com.log
>
>
> I'm testing flip6 on mesos, and the following is my config:
>
> _{color:#a64d79}jobmanager.rpc.address:
> [qt9ss.prod.mediav.com|http://qt9ss.prod.mediav.com/]{color}_
> _{color:#a64d79}jobmanager.rpc.port: 6123{color}_
> _{color:#a64d79}jobmanager.heap.mb: 1024{color}_
> _{color:#a64d79}taskmanager.heap.mb: 1024{color}_
> _{color:#a64d79}taskmanager.numberOfTaskSlots: 5{color}_
> _{color:#a64d79}parallelism.default: 1{color}_
> _{color:#a64d79}web.port: 8081{color}_
> _{color:#a64d79}mesos.master:
> zk://[dk71ss.jx.shbt2.qihoo.net:2191|http://dk71ss.jx.shbt2.qihoo.net:2191/],[dk72ss.jx.shbt2.qihoo.net:2191|http://dk72ss.jx.shbt2.qihoo.net:2191/],[dk5ss.jx.shbt2.qihoo.net:2191/mesos|http://dk5ss.jx.shbt2.qihoo.net:2191/mesos]{color}_
> _{color:#a64d79}mesos.resourcemanager.tasks.container.type: docker{color}_
> _{color:#a64d79}[mesos.resourcemanager.tasks.container.image.name|http://mesos.resourcemanager.tasks.container.image.name/]:
>
> [dk1ss.prod.mediav.com:5000/adq/flink:1.6.0-SNAPSHOT|http://dk1ss.prod.mediav.com:5000/adq/flink:1.6.0-SNAPSHOT]{color}_
> _{color:#a64d79}mesos.resourcemanager.framework.user: mediav{color}_
> _{color:#a64d79}mesos.resourcemanager.tasks.cpus: 5{color}_
> _{color:#a64d79}mesos.resourcemanager.tasks.mem: 10240{color}_
> _{color:#a64d79}[mesos.resourcemanager.framework.name|http://mesos.resourcemanager.framework.name/]:
> Flink{color}_
> _{color:#a64d79}mesos.failover-timeout: 60{color}_
>
> {color:#a64d79}{color:#33}When I submit a job with parallism 1, the job
> manager keeps reporting that slot can't be rallocated. Attached is my log of
> jobmanager.{color}{color}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-8953) Resolve unresolved field references in FieldComputer expressions
[
https://issues.apache.org/jira/browse/FLINK-8953?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16401391#comment-16401391
]
Renjie Liu commented on FLINK-8953:
---
Hi, Tim:
I want to look into this, could you assign this to me?
> Resolve unresolved field references in FieldComputer expressions
>
>
> Key: FLINK-8953
> URL: https://issues.apache.org/jira/browse/FLINK-8953
> Project: Flink
> Issue Type: Improvement
> Components: Table API & SQL
>Reporter: Timo Walther
>Priority: Major
>
> When implementing the {{FieldComputer.getExpression}} method, it is not
> possible to use API classes but only internal expression case classes.
> It would be great to also define timestamp extractors like:
> {code}
> def getExpression(fieldAccesses: Array[ResolvedFieldReference]): Expression
> = {
> // 'x.cast(Types.LONG)
> // ExpressionParser.parseExpression("x.cast(LONG)")
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-8852) SQL Client does not work with new FLIP-6 mode
[ https://issues.apache.org/jira/browse/FLINK-8852?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16399891#comment-16399891 ] Renjie Liu commented on FLINK-8852: --- Hi, [~twalthr]: I would like to look into this, could you assign this task to me? > SQL Client does not work with new FLIP-6 mode > - > > Key: FLINK-8852 > URL: https://issues.apache.org/jira/browse/FLINK-8852 > Project: Flink > Issue Type: Sub-task > Components: Table API & SQL >Affects Versions: 1.5.0 >Reporter: Fabian Hueske >Assignee: Timo Walther >Priority: Blocker > Fix For: 1.5.0 > > > The SQL client does not submit queries to local Flink cluster that runs in > FLIP-6 mode. It doesn't throw an exception either. > Job submission works if the legacy Flink cluster mode is used (`mode: old`) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (FLINK-8919) Add KeyedProcessFunctionWIthCleanupState
Renjie Liu created FLINK-8919: - Summary: Add KeyedProcessFunctionWIthCleanupState Key: FLINK-8919 URL: https://issues.apache.org/jira/browse/FLINK-8919 Project: Flink Issue Type: Improvement Components: Core Affects Versions: 1.6.0 Reporter: Renjie Liu Assignee: Renjie Liu Fix For: 1.6.0 ProcessFunctionWithCleanupState is a useful tool and I think we also need one for the new KeyedProcessFunction api. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (FLINK-8858) SQL Client to submit long running query in file
Renjie Liu created FLINK-8858: - Summary: SQL Client to submit long running query in file Key: FLINK-8858 URL: https://issues.apache.org/jira/browse/FLINK-8858 Project: Flink Issue Type: New Feature Components: Table API & SQL Affects Versions: 1.6.0 Reporter: Renjie Liu Assignee: Renjie Liu The current design of SQL Client embedded mode doesn't support long running queries. It would be useful for simple jobs that can be expressed in a single sql statement if we can submit sql statements stored in files as long running queries. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (FLINK-6968) Store streaming, updating tables with unique key in queryable state
[ https://issues.apache.org/jira/browse/FLINK-6968?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Renjie Liu reassigned FLINK-6968: - Assignee: Renjie Liu > Store streaming, updating tables with unique key in queryable state > --- > > Key: FLINK-6968 > URL: https://issues.apache.org/jira/browse/FLINK-6968 > Project: Flink > Issue Type: New Feature > Components: Table API & SQL >Reporter: Fabian Hueske >Assignee: Renjie Liu >Priority: Major > > Streaming tables with unique key are continuously updated. For example > queries with a non-windowed aggregation generate such tables. Commonly, such > updating tables are emitted via an upsert table sink to an external datastore > (k-v store, database) to make it accessible to applications. > This issue is about adding a feature to store and maintain such a table as > queryable state in Flink. By storing the table in Flnk's queryable state, we > do not need an external data store to access the results of the query but can > query the results directly from Flink. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-6131) Add side inputs for DataStream API
[ https://issues.apache.org/jira/browse/FLINK-6131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16038420#comment-16038420 ] Renjie Liu commented on FLINK-6131: --- Hi, we are facing the same issues and want to solve it with community together. Could anyone assign this task to me? > Add side inputs for DataStream API > -- > > Key: FLINK-6131 > URL: https://issues.apache.org/jira/browse/FLINK-6131 > Project: Flink > Issue Type: Improvement > Components: DataStream API >Reporter: Aljoscha Krettek >Assignee: Aljoscha Krettek > > This is an umbrella issue for tracking the implementation of FLIP-17: > https://cwiki.apache.org/confluence/display/FLINK/FLIP-17+Side+Inputs+for+DataStream+API. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (FLINK-5445) flink kafka consumer should support rate control
Renjie Liu created FLINK-5445: - Summary: flink kafka consumer should support rate control Key: FLINK-5445 URL: https://issues.apache.org/jira/browse/FLINK-5445 Project: Flink Issue Type: Bug Components: Kafka Connector Affects Versions: 1.1.3 Reporter: Renjie Liu Priority: Critical Now kafka connector does not support rate control at all, this may lead checkpoints to fail and expire if downstream processor is slow. Though this will not compromise the guarantee of flink's checkpoint, this will stop kafka consumer from comitting offsets since kafka consumer commits offsets in the callback of checkpoint complete. The committed offsets is important for monitoring purpose. So I think we need to support rate limit control in kafka consumer. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
