[jira] [Created] (FLINK-9052) Flink complains when scala.Option is used inside POJO
Truong Duc Kien created FLINK-9052:
--
Summary: Flink complains when scala.Option is used inside POJO
Key: FLINK-9052
URL: https://issues.apache.org/jira/browse/FLINK-9052
Project: Flink
Issue Type: Bug
Components: Type Serialization System
Affects Versions: 1.4.2
Reporter: Truong Duc Kien
According to the documentation, Flink has a specialized serializer for Option
type. However, when an Option field is used inside a POJO, the following
WARNING appears in TaskManagers' log.
{{No fields detected for class scala.Option. Cannot be used as a PojoType. Will
be handled as GenericType}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (FLINK-9051) Invert flip6 profile
Chesnay Schepler created FLINK-9051:
---
Summary: Invert flip6 profile
Key: FLINK-9051
URL: https://issues.apache.org/jira/browse/FLINK-9051
Project: Flink
Issue Type: Improvement
Components: Build System, Tests
Reporter: Chesnay Schepler
Assignee: Chesnay Schepler
Fix For: 1.5.0
The {{flip6}} profile enables flip6 during tests and currently has to be
explicitly enabled.
Given that the flip6 mode is the new default, we should invert the profile
semantics to _disable_ flip6 mode instead.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (FLINK-9050) Expose operator IO counter metrics
Chesnay Schepler created FLINK-9050: --- Summary: Expose operator IO counter metrics Key: FLINK-9050 URL: https://issues.apache.org/jira/browse/FLINK-9050 Project: Flink Issue Type: New Feature Components: Metrics Reporter: Chesnay Schepler Assignee: Chesnay Schepler Fix For: 1.6.0 To properly expose the number of records read by sources / emitted by sink we have to expose the operator counter metrics. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (FLINK-9049) Create unified interfaces to configure and instatiate TableSink
Shuyi Chen created FLINK-9049:
-
Summary: Create unified interfaces to configure and instatiate
TableSink
Key: FLINK-9049
URL: https://issues.apache.org/jira/browse/FLINK-9049
Project: Flink
Issue Type: Task
Components: Table API & SQL
Reporter: Shuyi Chen
Assignee: Shuyi Chen
This is a similar effort to
[FLINK-8240|https://issues.apache.org/jira/browse/FLINK-8240], we want to
create a unified interface for discovery and instantiation of TableSink, and
later support table DDL in flink. The proposed solution would use similar
approach in [FLINK-8240|https://issues.apache.org/jira/browse/FLINK-8240], and
can re-use most of the implementations already done in
[FLINK-8240|https://issues.apache.org/jira/browse/FLINK-8240].
1) Add TableSinkFactory{Service} similar to TableSourceFactory{Service]
2) Add a common property called "tableType" with values (source, sink and both)
for both TableSource and TableSink.
3) in yaml file, replace "sources" with "tables", and use tableType to identify
whether it's source or sink.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (FLINK-9048) LocalFlinkMiniClusterITCase#testLocalFlinkMiniClusterWithMultipleTaskManagers sometimes fails
Ted Yu created FLINK-9048:
-
Summary:
LocalFlinkMiniClusterITCase#testLocalFlinkMiniClusterWithMultipleTaskManagers
sometimes fails
Key: FLINK-9048
URL: https://issues.apache.org/jira/browse/FLINK-9048
Project: Flink
Issue Type: Test
Reporter: Ted Yu
As of commit e0bc37bef69f5376d03214578e9b95816add661b, I got the following :
{code}
testLocalFlinkMiniClusterWithMultipleTaskManagers(org.apache.flink.test.runtime.minicluster.LocalFlinkMiniClusterITCase)
Time elapsed: 41.681 sec <<< FAILURE!
java.lang.AssertionError: Thread
Thread[ForkJoinPool.commonPool-worker-25,5,main] was started by the mini
cluster, but not shut down
at org.junit.Assert.fail(Assert.java:88)
at
org.apache.flink.test.runtime.minicluster.LocalFlinkMiniClusterITCase.testLocalFlinkMiniClusterWithMultipleTaskManagers(LocalFlinkMiniClusterITCase.java:174)
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (FLINK-9047) SlotPool can fail to release slots
Till Rohrmann created FLINK-9047:
Summary: SlotPool can fail to release slots
Key: FLINK-9047
URL: https://issues.apache.org/jira/browse/FLINK-9047
Project: Flink
Issue Type: Bug
Components: Distributed Coordination
Affects Versions: 1.5.0
Reporter: Till Rohrmann
Fix For: 1.5.0
The {{SlotPool}} releases idling slots. If the release operation fails (e.g.
timeout), then it simply continues using the slot. This is problematic if the
owning {{TaskExecutor}} failed before and was unregistered in the meantime from
the {{SlotPool}}. As a result, the {{SlotPool}} will reuse the slot and
whenever it tries to return because it is idling it will fail again. This,
effectively, renders the scheduling of a job impossible.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
Re: [DISCUSS] Flink security improvements
Please accept my apologies also. On Mon, Mar 19, 2018 at 2:46 AM, Till Rohrmann wrote: > Hi Shuyi, > > sorry for the unresponsiveness on your proposal. Currently, the community > is strongly focused on fixing and testing Flink 1.5 so that we can release > it soon. My gut feeling is that the community will pick up the security > improvements thread once most of the blocking issues are resolved. Please > bear with us until then. > > Cheers, > Till > > On Fri, Mar 9, 2018 at 1:05 AM, Shuyi Chen wrote: > > > Ping :) > > > > On Wed, Feb 21, 2018 at 7:16 PM, Shuyi Chen wrote: > > > > > Hi Eron, thanks a lot for taking a look at the proposal, the comments > are > > > very useful. I've updated the document to address your concerns. Could > > you > > > please help take another look, and suggest what the next step is? > Highly > > > appreciated. > > > > > > Shuyi > > > > > > On Thu, Feb 15, 2018 at 4:19 AM, Shuyi Chen > wrote: > > > > > >> Hi community, > > >> > > >> I would like to propose a few improvements in Flink security regarding > > >> scalability and extensibility. Here is the proposal: > > >> > > >> https://docs.google.com/document/d/10V7LiNlUJKeKZ58mkR7oVv1t > > >> 6BrC6TZi3FGf2Dm6-i8/edit?usp=sharing > > >> > > >> Comments are highly appreciated. Please let me know what the next step > > >> will be. > > >> > > >> Thanks a lot > > >> Shuyi > > >> > > >> -- > > >> "So you have to trust that the dots will somehow connect in your > > future." > > >> > > > > > > > > > > > > -- > > > "So you have to trust that the dots will somehow connect in your > future." > > > > > > > > > > > -- > > "So you have to trust that the dots will somehow connect in your future." > > >
Re: FLIP-6 / Kubernetes
Thanks Stephan for the information, it's exciting. This is an aspect for which the FLIP-6 design doc is lacking detail; somewhere we should work through how this will work (e.g. from a job recovery and upgrade perspective). If there's a doc or an open issue, please share, thanks! On Wed, Mar 21, 2018 at 5:41 AM, Stephan Ewen wrote: > Hi Christophe! > > I could imagine 1.5.1 or 1.6. > > It is important to point out that you can use Flink with Kubernetes quite > well already. That addition is mainly intended to make it more idiomatic > for Kubernetes users. > > Best, > Stephan > > > On Wed, Mar 21, 2018 at 12:03 PM, Christophe Jolif > wrote: > > > Hi Stephan, > > > > > This feature did not make it for 1.5, but should come very > > soon after. > > > > Are you talking about a 1.5.1 here? What is the envisioned timeline? > > > > Thanks! > > > > > > On Tue, Mar 20, 2018 at 9:30 PM, Stephan Ewen wrote: > > > > > @Eron That is definitely the way we want to suggest as the way to use > k8s > > > in the future. This feature did not make it for 1.5, but should come > very > > > soon after. > > > > > > @Thomas An implementation of a ResourceManager for k8s should come in > the > > > near future. Would be happy to jump on a joint FLIP, after the 1.5 > > release > > > and Flink Forward (in three weeks or so). > > > > > > On Tue, Mar 20, 2018 at 4:25 PM, Eron Wright > > wrote: > > > > > > > Till, is it possible to package a Flink application as a > > self-contained > > > > deployment on Kubernetes? I mean, run a Flink application using > 'flink > > > > run' such that it launches its own RM/JM and waits for a sufficient # > > of > > > > TMs to join? > > > > > > > > Thanks! > > > > > > > > On Mon, Mar 19, 2018 at 2:57 AM, Till Rohrmann > > > > > wrote: > > > > > > > > > I forgot to add Flink's K8 documentation [1] which might also be > > > helpful > > > > > with getting started. > > > > > > > > > > [1] > > > > > https://ci.apache.org/projects/flink/flink-docs- > > master/ops/deployment/ > > > > > kubernetes.html > > > > > > > > > > Cheers, > > > > > Till > > > > > > > > > > On Mon, Mar 19, 2018 at 10:54 AM, Till Rohrmann < > > trohrm...@apache.org> > > > > > wrote: > > > > > > > > > > > Hi Thomas, > > > > > > > > > > > > I think the one way to get started would be to adapt the Flink > > docker > > > > > > images [1,2] to run with Flink 1.5. Per default, they will use > the > > > > Flip-6 > > > > > > components. > > > > > > > > > > > > Flink 1.5 won't come with a dedicated integration with Kubernetes > > > which > > > > > is > > > > > > able to start new pods. However, it should work that you manually > > or > > > by > > > > > the > > > > > > virtues of an external system start new pods which can be used. > > Once > > > > the > > > > > > new pods have been started and the TaskExecutors have registered > > one > > > > > would > > > > > > have to rescale the job manually to the new parallelism. > > > > > > > > > > > > [1] https://flink.apache.org/news/2017/05/16/official-docker- > > > > image.html > > > > > > [2] https://hub.docker.com/r/_/flink/ > > > > > > > > > > > > Cheers, > > > > > > Till > > > > > > > > > > > > On Sun, Mar 18, 2018 at 9:05 PM, Thomas Weise > > > wrote: > > > > > > > > > > > >> Hi, > > > > > >> > > > > > >> What would be a good starting point to try out Flink on > Kubernetes > > > > (any > > > > > >> examples / tutorials)? > > > > > >> > > > > > >> Also, will the FLIP-6 work in 1.5 enable dynamic scaling on > > > > Kubernetes? > > > > > >> > > > > > >> Thanks, > > > > > >> Thomas > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > Christophe > > >
Re: Flip 6 mesos support
The resources consumed by the JobMaster can be specified by `jobmanager.heap.mb`. Cheers, Till On Wed, Mar 21, 2018 at 3:20 PM, Renjie Liu wrote: > Hi, Till: > > In fact, I want to ask the resources consume by job manager > > Till Rohrmann 于 2018年3月21日周三 下午8:17写道： > > > As many as the application needs to run. If you start a job with > > parallelism 10 then it will ask for 10 slots (assuming slot sharing). > > > > On Wed, Mar 21, 2018 at 12:04 PM, Renjie Liu > > wrote: > > > > > So how many slots a job manager may consume? > > > > > > On Wed, Mar 21, 2018 at 6:50 PM Till Rohrmann > > > wrote: > > > > > > > At the moment this is not possible. In order to do this, you will > have > > to > > > > use the per-job mode and run each job on a dedicated Flink cluster. > > > > > > > > On Wed, Mar 21, 2018 at 11:33 AM, Renjie Liu < > liurenjie2...@gmail.com> > > > > wrote: > > > > > > > > > For example, we have 2 jobs. > > > > > For job 1, I want to start job manger with 1 CPU and 100M memory. > > Job 1 > > > > > need s10 slots, and I want to deploy these 10 slots in 2 task > > managers, > > > > > each with 5 cores and 1G memory. > > > > > > > > > > For job 2, I want to start job manager with 2 CPU and 200M memory. > > Job > > > 2 > > > > > needs 100 slots and I want to deploy these 100 slot in 10 task > > > managers, > > > > > each with 10 cores and 2G memory. > > > > > > > > > > Is this possible? > > > > > > > > > > On Wed, Mar 21, 2018 at 6:19 PM Till Rohrmann < > trohrm...@apache.org> > > > > > wrote: > > > > > > > > > > > Hi Renjie, > > > > > > > > > > > > what do you mean with specifying different JM and TM resources > for > > > > > > different jobs exactly? > > > > > > > > > > > > Cheers, > > > > > > Till > > > > > > > > > > > > On Wed, Mar 21, 2018 at 10:55 AM, Renjie Liu < > > > liurenjie2...@gmail.com> > > > > > > wrote: > > > > > > > > > > > > > Hi, Till: > > > > > > > > > > > > > > How to specify job manager and task manager resources for > > different > > > > > jobs > > > > > > in > > > > > > > session mode? > > > > > > > > > > > > > > On Sun, Mar 18, 2018 at 1:10 AM Till Rohrmann < > > > trohrm...@apache.org> > > > > > > > wrote: > > > > > > > > > > > > > > > Hi Shuyi, > > > > > > > > > > > > > > > > best if you look at the other e2e tests in the > > > > flink-end-to-end-tests > > > > > > > > module. For example the Kafka e2e test under > > > > > > > > flink/flink-end-to-end-tests/test-scripts/test_streaming_ > > > > > kafka010.sh. > > > > > > > > > > > > > > > > Cheers, > > > > > > > > Till > > > > > > > > > > > > > > > > On Fri, Mar 16, 2018 at 10:20 PM, Shuyi Chen < > > suez1...@gmail.com > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > Hi Till, > > > > > > > > > > > > > > > > > > For FLINK-8562, the test is passing now because it's not > > really > > > > > > > > > checking the right thing. > > > > > > > > > > > > > > > > > > Yes, I can help with the Kerberos integration ticket. > > > > > > > > > > > > > > > > > > Is there an example on how the e2e test should be > structured > > > and > > > > > > > invoked? > > > > > > > > > > > > > > > > > > Thanks > > > > > > > > > Shuyi > > > > > > > > > > > > > > > > > > On Fri, Mar 16, 2018 at 6:51 AM, Till Rohrmann < > > > > > trohrm...@apache.org > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > Hi Shuyi, > > > > > > > > > > > > > > > > > > > > thanks for the working on FLINK-8562. Once this issue is > > > fixed, > > > > > it > > > > > > > will > > > > > > > > > > automatically be executed on the Flip-6 components. In > fact > > > it > > > > is > > > > > > > > already > > > > > > > > > > being executed on Flip-6. > > > > > > > > > > > > > > > > > > > > But what you could help the community with is setting up > an > > > > > > automated > > > > > > > > > > end-to-end test for the Kerberos integration if you want: > > > > > > > > > > https://issues.apache.org/jira/browse/FLINK-8981. > > > > > > > > > > > > > > > > > > > > The Flink community is currently working on automating > more > > > and > > > > > > more > > > > > > > > > tests > > > > > > > > > > in order to facilitate faster releases and improve the > test > > > > > > coverage. > > > > > > > > You > > > > > > > > > > can find more about this effort here: > > > > > > > > > > https://issues.apache.org/jira/browse/FLINK-8970. > > > > > > > > > > > > > > > > > > > > Cheers, > > > > > > > > > > Till > > > > > > > > > > > > > > > > > > > > On Thu, Mar 15, 2018 at 8:45 PM, Shuyi Chen < > > > > suez1...@gmail.com> > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > Hi Till, > > > > > > > > > > > > > > > > > > > > > > This is Shuyi :) Thanks a lot. In FLINK-8562, I already > > > sent > > > > a > > > > > PR > > > > > > > to > > > > > > > > > > > resolve the issue, your help to take a look will be > > great. > > > > > > > > > > > > > > > > > > > > > > Please let me know what I can help to test the Kerberos > > > > > > > > > authentication, I > > > > > > > >
[jira] [Created] (FLINK-9046) Improve error messages when receiving unexpected messages.
Kostas Kloudas created FLINK-9046:
-
Summary: Improve error messages when receiving unexpected messages.
Key: FLINK-9046
URL: https://issues.apache.org/jira/browse/FLINK-9046
Project: Flink
Issue Type: Bug
Components: REST, Webfrontend
Affects Versions: 1.5.0
Reporter: Kostas Kloudas
Currently, in many cases, when a Rest Handler received an unexpected messages,
e.g. for a job that it does not exist, it logs the full stack trace often with
misguiding messages. This can happen for example if we launch a cluster,
connect to the WebUI and monitor a job, then kill the cluster while not
shutting down the WebUI and start a new cluster on the same port. In this case
the WebUI will keep asking for the previous job and the JobDetailsHandler will
log the following:
{code:java}
2018-03-21 14:27:24,319 ERROR
org.apache.flink.runtime.rest.handler.job.JobDetailsHandler - Exception
occurred in REST handler.
org.apache.flink.runtime.rest.NotFoundException: Job
548afad8217f4a18db7f50e60d48885a not found
at
org.apache.flink.runtime.rest.handler.job.AbstractExecutionGraphHandler.lambda$handleRequest$1(AbstractExecutionGraphHandler.java:90)
at
java.util.concurrent.CompletableFuture.uniExceptionally(CompletableFuture.java:870)
at
java.util.concurrent.CompletableFuture$UniExceptionally.tryFire(CompletableFuture.java:852)
at
java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
at
java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977)
at
org.apache.flink.runtime.rest.handler.legacy.ExecutionGraphCache.lambda$getExecutionGraph$0(ExecutionGraphCache.java:133)
at
java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760)
at
java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736)
at
java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
at
java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977)
at
org.apache.flink.runtime.concurrent.FutureUtils$1.onComplete(FutureUtils.java:755)
at akka.dispatch.OnComplete.internal(Future.scala:258)
at akka.dispatch.OnComplete.internal(Future.scala:256)
at akka.dispatch.japi$CallbackBridge.apply(Future.scala:186)
at akka.dispatch.japi$CallbackBridge.apply(Future.scala:183)
at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:36)
at
org.apache.flink.runtime.concurrent.Executors$DirectExecutionContext.execute(Executors.java:83)
at scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:44)
at scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:252)
at akka.pattern.PromiseActorRef.$bang(AskSupport.scala:534)
at
akka.pattern.PipeToSupport$PipeableFuture$$anonfun$pipeTo$1.applyOrElse(PipeToSupport.scala:20)
at
akka.pattern.PipeToSupport$PipeableFuture$$anonfun$pipeTo$1.applyOrElse(PipeToSupport.scala:18)
at scala.concurrent.Future$$anonfun$andThen$1.apply(Future.scala:436)
at scala.concurrent.Future$$anonfun$andThen$1.apply(Future.scala:435)
at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:36)
at
akka.dispatch.BatchingExecutor$AbstractBatch.processBatch(BatchingExecutor.scala:55)
at
akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply$mcV$sp(BatchingExecutor.scala:91)
at
akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply(BatchingExecutor.scala:91)
at
akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply(BatchingExecutor.scala:91)
at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:72)
at akka.dispatch.BatchingExecutor$BlockableBatch.run(BatchingExecutor.scala:90)
at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:39)
at
akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:415)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at
scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at
scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107){code}
The same holds when asking through the WebUI for the logs of TM. In this case,
the logs will contain:
{code:java}
2018-03-21 12:26:05,096 ERROR
org.apache.flink.runtime.rest.handler.taskmanager.TaskManagerDetailsHandler -
Implementation error: Unhandled exception.
org.apache.flink.runtime.resourcemanager.exceptions.UnknownTaskExecutorException:
No TaskExecutor registered under b4c468fc0fe729b09e35d319cbc45c39.
at
org.apache.flink.runtime.resourcemanager.ResourceManager.requestTaskManagerInfo(ResourceManager.java:538)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(D
Re: Flip 6 mesos support
Hi, Till: In fact, I want to ask the resources consume by job manager Till Rohrmann 于 2018年3月21日周三 下午8:17写道： > As many as the application needs to run. If you start a job with > parallelism 10 then it will ask for 10 slots (assuming slot sharing). > > On Wed, Mar 21, 2018 at 12:04 PM, Renjie Liu > wrote: > > > So how many slots a job manager may consume? > > > > On Wed, Mar 21, 2018 at 6:50 PM Till Rohrmann > > wrote: > > > > > At the moment this is not possible. In order to do this, you will have > to > > > use the per-job mode and run each job on a dedicated Flink cluster. > > > > > > On Wed, Mar 21, 2018 at 11:33 AM, Renjie Liu > > > wrote: > > > > > > > For example, we have 2 jobs. > > > > For job 1, I want to start job manger with 1 CPU and 100M memory. > Job 1 > > > > need s10 slots, and I want to deploy these 10 slots in 2 task > managers, > > > > each with 5 cores and 1G memory. > > > > > > > > For job 2, I want to start job manager with 2 CPU and 200M memory. > Job > > 2 > > > > needs 100 slots and I want to deploy these 100 slot in 10 task > > managers, > > > > each with 10 cores and 2G memory. > > > > > > > > Is this possible? > > > > > > > > On Wed, Mar 21, 2018 at 6:19 PM Till Rohrmann > > > > wrote: > > > > > > > > > Hi Renjie, > > > > > > > > > > what do you mean with specifying different JM and TM resources for > > > > > different jobs exactly? > > > > > > > > > > Cheers, > > > > > Till > > > > > > > > > > On Wed, Mar 21, 2018 at 10:55 AM, Renjie Liu < > > liurenjie2...@gmail.com> > > > > > wrote: > > > > > > > > > > > Hi, Till: > > > > > > > > > > > > How to specify job manager and task manager resources for > different > > > > jobs > > > > > in > > > > > > session mode? > > > > > > > > > > > > On Sun, Mar 18, 2018 at 1:10 AM Till Rohrmann < > > trohrm...@apache.org> > > > > > > wrote: > > > > > > > > > > > > > Hi Shuyi, > > > > > > > > > > > > > > best if you look at the other e2e tests in the > > > flink-end-to-end-tests > > > > > > > module. For example the Kafka e2e test under > > > > > > > flink/flink-end-to-end-tests/test-scripts/test_streaming_ > > > > kafka010.sh. > > > > > > > > > > > > > > Cheers, > > > > > > > Till > > > > > > > > > > > > > > On Fri, Mar 16, 2018 at 10:20 PM, Shuyi Chen < > suez1...@gmail.com > > > > > > > > wrote: > > > > > > > > > > > > > > > Hi Till, > > > > > > > > > > > > > > > > For FLINK-8562, the test is passing now because it's not > really > > > > > > > > checking the right thing. > > > > > > > > > > > > > > > > Yes, I can help with the Kerberos integration ticket. > > > > > > > > > > > > > > > > Is there an example on how the e2e test should be structured > > and > > > > > > invoked? > > > > > > > > > > > > > > > > Thanks > > > > > > > > Shuyi > > > > > > > > > > > > > > > > On Fri, Mar 16, 2018 at 6:51 AM, Till Rohrmann < > > > > trohrm...@apache.org > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > Hi Shuyi, > > > > > > > > > > > > > > > > > > thanks for the working on FLINK-8562. Once this issue is > > fixed, > > > > it > > > > > > will > > > > > > > > > automatically be executed on the Flip-6 components. In fact > > it > > > is > > > > > > > already > > > > > > > > > being executed on Flip-6. > > > > > > > > > > > > > > > > > > But what you could help the community with is setting up an > > > > > automated > > > > > > > > > end-to-end test for the Kerberos integration if you want: > > > > > > > > > https://issues.apache.org/jira/browse/FLINK-8981. > > > > > > > > > > > > > > > > > > The Flink community is currently working on automating more > > and > > > > > more > > > > > > > > tests > > > > > > > > > in order to facilitate faster releases and improve the test > > > > > coverage. > > > > > > > You > > > > > > > > > can find more about this effort here: > > > > > > > > > https://issues.apache.org/jira/browse/FLINK-8970. > > > > > > > > > > > > > > > > > > Cheers, > > > > > > > > > Till > > > > > > > > > > > > > > > > > > On Thu, Mar 15, 2018 at 8:45 PM, Shuyi Chen < > > > suez1...@gmail.com> > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > Hi Till, > > > > > > > > > > > > > > > > > > > > This is Shuyi :) Thanks a lot. In FLINK-8562, I already > > sent > > > a > > > > PR > > > > > > to > > > > > > > > > > resolve the issue, your help to take a look will be > great. > > > > > > > > > > > > > > > > > > > > Please let me know what I can help to test the Kerberos > > > > > > > > authentication, I > > > > > > > > > > am decently familiar with the Kerberos and YARN security > > part > > > > in > > > > > > > Flink. > > > > > > > > > > > > > > > > > > > > As a starting point, I'd suggest to add an integration > test > > > > > similar > > > > > > > to > > > > > > > > > > YARNSessionFIFOSecuredITCase > > > > > > > > > > for flip6. > > > > > > > > > > > > > > > > > > > > Shuyi > > > > > > > > > > > > > > > > > > > > On Thu, Mar 15, 2018 at 5:44 AM, Till Rohrmann < > > > > > > trohrm...@apache.org > > > > > > > > > > > > >
[jira] [Created] (FLINK-9045) LocalEnvironment with web UI does not work with flip-6
Stephan Ewen created FLINK-9045:
---
Summary: LocalEnvironment with web UI does not work with flip-6
Key: FLINK-9045
URL: https://issues.apache.org/jira/browse/FLINK-9045
Project: Flink
Issue Type: Bug
Components: Webfrontend
Reporter: Stephan Ewen
Fix For: 1.5.0
The following code is supposed to start a web UI when executing in-IDE. Does
not work with flip-6, as far as I can see.
{code}
final ExecutionEnvironment env =
ExecutionEnvironment.createLocalEnvironmentWithWebUI(new Configuration());
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (FLINK-9044) Fork a new stream in a stream
Herbert Koelman created FLINK-9044: -- Summary: Fork a new stream in a stream Key: FLINK-9044 URL: https://issues.apache.org/jira/browse/FLINK-9044 Project: Flink Issue Type: Wish Reporter: Herbert Koelman Make it possible to fork a new stream in a running stream. This can be usefull when computational stream's state is affected (while running) by an external element/event. When such an event is detected, it would then be possible to fire up a new stream that would be in charge to recompute data that were handled before the state change event was detected. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
Re: Request for contributor permissions
Welcome to the community Qi Yu. I've given you contributor permissions. Cheers, Till On Wed, Mar 21, 2018 at 12:13 PM, Qi Yu wrote: > Hi devs, > I'm very interested in Flink and willing to contribute to it. Cloud > anybody give me the contributor permissions? My jira username is “yuqi”. > Thanks~ > Regards, > Qi Yu
[jira] [Created] (FLINK-9043) Flink recover from checkpoint like Spark Streaming
godfrey johnson created FLINK-9043: -- Summary: Flink recover from checkpoint like Spark Streaming Key: FLINK-9043 URL: https://issues.apache.org/jira/browse/FLINK-9043 Project: Flink Issue Type: New Feature Reporter: godfrey johnson I know a flink job can reovery from checkpoint with restart strategy, but can not recovery as spark streaming jobs when job is starting. Every time, the submitted flink job is regarded as a new job, while , in the spark streaming job, which can detect the checkpoint directory first, and then recovery from the latest succeed one. However, Flink only can recovery until the job failed first, then retry with strategy. So, would flink support to recover from the checkpoint directly in a new job? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
Re: FLIP-6 / Kubernetes
Hi Christophe! I could imagine 1.5.1 or 1.6. It is important to point out that you can use Flink with Kubernetes quite well already. That addition is mainly intended to make it more idiomatic for Kubernetes users. Best, Stephan On Wed, Mar 21, 2018 at 12:03 PM, Christophe Jolif wrote: > Hi Stephan, > > > This feature did not make it for 1.5, but should come very > soon after. > > Are you talking about a 1.5.1 here? What is the envisioned timeline? > > Thanks! > > > On Tue, Mar 20, 2018 at 9:30 PM, Stephan Ewen wrote: > > > @Eron That is definitely the way we want to suggest as the way to use k8s > > in the future. This feature did not make it for 1.5, but should come very > > soon after. > > > > @Thomas An implementation of a ResourceManager for k8s should come in the > > near future. Would be happy to jump on a joint FLIP, after the 1.5 > release > > and Flink Forward (in three weeks or so). > > > > On Tue, Mar 20, 2018 at 4:25 PM, Eron Wright > wrote: > > > > > Till, is it possible to package a Flink application as a > self-contained > > > deployment on Kubernetes? I mean, run a Flink application using 'flink > > > run' such that it launches its own RM/JM and waits for a sufficient # > of > > > TMs to join? > > > > > > Thanks! > > > > > > On Mon, Mar 19, 2018 at 2:57 AM, Till Rohrmann > > > wrote: > > > > > > > I forgot to add Flink's K8 documentation [1] which might also be > > helpful > > > > with getting started. > > > > > > > > [1] > > > > https://ci.apache.org/projects/flink/flink-docs- > master/ops/deployment/ > > > > kubernetes.html > > > > > > > > Cheers, > > > > Till > > > > > > > > On Mon, Mar 19, 2018 at 10:54 AM, Till Rohrmann < > trohrm...@apache.org> > > > > wrote: > > > > > > > > > Hi Thomas, > > > > > > > > > > I think the one way to get started would be to adapt the Flink > docker > > > > > images [1,2] to run with Flink 1.5. Per default, they will use the > > > Flip-6 > > > > > components. > > > > > > > > > > Flink 1.5 won't come with a dedicated integration with Kubernetes > > which > > > > is > > > > > able to start new pods. However, it should work that you manually > or > > by > > > > the > > > > > virtues of an external system start new pods which can be used. > Once > > > the > > > > > new pods have been started and the TaskExecutors have registered > one > > > > would > > > > > have to rescale the job manually to the new parallelism. > > > > > > > > > > [1] https://flink.apache.org/news/2017/05/16/official-docker- > > > image.html > > > > > [2] https://hub.docker.com/r/_/flink/ > > > > > > > > > > Cheers, > > > > > Till > > > > > > > > > > On Sun, Mar 18, 2018 at 9:05 PM, Thomas Weise > > wrote: > > > > > > > > > >> Hi, > > > > >> > > > > >> What would be a good starting point to try out Flink on Kubernetes > > > (any > > > > >> examples / tutorials)? > > > > >> > > > > >> Also, will the FLIP-6 work in 1.5 enable dynamic scaling on > > > Kubernetes? > > > > >> > > > > >> Thanks, > > > > >> Thomas > > > > >> > > > > > > > > > > > > > > > > > > > > > > > -- > Christophe >
Re: Flip 6 mesos support
As many as the application needs to run. If you start a job with parallelism 10 then it will ask for 10 slots (assuming slot sharing). On Wed, Mar 21, 2018 at 12:04 PM, Renjie Liu wrote: > So how many slots a job manager may consume? > > On Wed, Mar 21, 2018 at 6:50 PM Till Rohrmann > wrote: > > > At the moment this is not possible. In order to do this, you will have to > > use the per-job mode and run each job on a dedicated Flink cluster. > > > > On Wed, Mar 21, 2018 at 11:33 AM, Renjie Liu > > wrote: > > > > > For example, we have 2 jobs. > > > For job 1, I want to start job manger with 1 CPU and 100M memory. Job 1 > > > need s10 slots, and I want to deploy these 10 slots in 2 task managers, > > > each with 5 cores and 1G memory. > > > > > > For job 2, I want to start job manager with 2 CPU and 200M memory. Job > 2 > > > needs 100 slots and I want to deploy these 100 slot in 10 task > managers, > > > each with 10 cores and 2G memory. > > > > > > Is this possible? > > > > > > On Wed, Mar 21, 2018 at 6:19 PM Till Rohrmann > > > wrote: > > > > > > > Hi Renjie, > > > > > > > > what do you mean with specifying different JM and TM resources for > > > > different jobs exactly? > > > > > > > > Cheers, > > > > Till > > > > > > > > On Wed, Mar 21, 2018 at 10:55 AM, Renjie Liu < > liurenjie2...@gmail.com> > > > > wrote: > > > > > > > > > Hi, Till: > > > > > > > > > > How to specify job manager and task manager resources for different > > > jobs > > > > in > > > > > session mode? > > > > > > > > > > On Sun, Mar 18, 2018 at 1:10 AM Till Rohrmann < > trohrm...@apache.org> > > > > > wrote: > > > > > > > > > > > Hi Shuyi, > > > > > > > > > > > > best if you look at the other e2e tests in the > > flink-end-to-end-tests > > > > > > module. For example the Kafka e2e test under > > > > > > flink/flink-end-to-end-tests/test-scripts/test_streaming_ > > > kafka010.sh. > > > > > > > > > > > > Cheers, > > > > > > Till > > > > > > > > > > > > On Fri, Mar 16, 2018 at 10:20 PM, Shuyi Chen > > > > > wrote: > > > > > > > > > > > > > Hi Till, > > > > > > > > > > > > > > For FLINK-8562, the test is passing now because it's not really > > > > > > > checking the right thing. > > > > > > > > > > > > > > Yes, I can help with the Kerberos integration ticket. > > > > > > > > > > > > > > Is there an example on how the e2e test should be structured > and > > > > > invoked? > > > > > > > > > > > > > > Thanks > > > > > > > Shuyi > > > > > > > > > > > > > > On Fri, Mar 16, 2018 at 6:51 AM, Till Rohrmann < > > > trohrm...@apache.org > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > Hi Shuyi, > > > > > > > > > > > > > > > > thanks for the working on FLINK-8562. Once this issue is > fixed, > > > it > > > > > will > > > > > > > > automatically be executed on the Flip-6 components. In fact > it > > is > > > > > > already > > > > > > > > being executed on Flip-6. > > > > > > > > > > > > > > > > But what you could help the community with is setting up an > > > > automated > > > > > > > > end-to-end test for the Kerberos integration if you want: > > > > > > > > https://issues.apache.org/jira/browse/FLINK-8981. > > > > > > > > > > > > > > > > The Flink community is currently working on automating more > and > > > > more > > > > > > > tests > > > > > > > > in order to facilitate faster releases and improve the test > > > > coverage. > > > > > > You > > > > > > > > can find more about this effort here: > > > > > > > > https://issues.apache.org/jira/browse/FLINK-8970. > > > > > > > > > > > > > > > > Cheers, > > > > > > > > Till > > > > > > > > > > > > > > > > On Thu, Mar 15, 2018 at 8:45 PM, Shuyi Chen < > > suez1...@gmail.com> > > > > > > wrote: > > > > > > > > > > > > > > > > > Hi Till, > > > > > > > > > > > > > > > > > > This is Shuyi :) Thanks a lot. In FLINK-8562, I already > sent > > a > > > PR > > > > > to > > > > > > > > > resolve the issue, your help to take a look will be great. > > > > > > > > > > > > > > > > > > Please let me know what I can help to test the Kerberos > > > > > > > authentication, I > > > > > > > > > am decently familiar with the Kerberos and YARN security > part > > > in > > > > > > Flink. > > > > > > > > > > > > > > > > > > As a starting point, I'd suggest to add an integration test > > > > similar > > > > > > to > > > > > > > > > YARNSessionFIFOSecuredITCase > > > > > > > > > for flip6. > > > > > > > > > > > > > > > > > > Shuyi > > > > > > > > > > > > > > > > > > On Thu, Mar 15, 2018 at 5:44 AM, Till Rohrmann < > > > > > trohrm...@apache.org > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > Hi Renjie, > > > > > > > > > > > > > > > > > > > > thanks for the pointer with the > > YARNSessionFIFOSecuredITCase. > > > > > > You're > > > > > > > > > right > > > > > > > > > > that we should fix this test. There is FLINK-8562 which > > seems > > > > to > > > > > > > > address > > > > > > > > > > the problem. Will take a look. > > > > > > > > > > > > > > > > > > > > Additionally, we want to te
Request for contributor permissions
Hi devs, I'm very interested in Flink and willing to contribute to it. Cloud anybody give me the contributor permissions? My jira username is “yuqi”. Thanks~ Regards, Qi Yu
Re: Flip 6 mesos support
So how many slots a job manager may consume? On Wed, Mar 21, 2018 at 6:50 PM Till Rohrmann wrote: > At the moment this is not possible. In order to do this, you will have to > use the per-job mode and run each job on a dedicated Flink cluster. > > On Wed, Mar 21, 2018 at 11:33 AM, Renjie Liu > wrote: > > > For example, we have 2 jobs. > > For job 1, I want to start job manger with 1 CPU and 100M memory. Job 1 > > need s10 slots, and I want to deploy these 10 slots in 2 task managers, > > each with 5 cores and 1G memory. > > > > For job 2, I want to start job manager with 2 CPU and 200M memory. Job 2 > > needs 100 slots and I want to deploy these 100 slot in 10 task managers, > > each with 10 cores and 2G memory. > > > > Is this possible? > > > > On Wed, Mar 21, 2018 at 6:19 PM Till Rohrmann > > wrote: > > > > > Hi Renjie, > > > > > > what do you mean with specifying different JM and TM resources for > > > different jobs exactly? > > > > > > Cheers, > > > Till > > > > > > On Wed, Mar 21, 2018 at 10:55 AM, Renjie Liu > > > wrote: > > > > > > > Hi, Till: > > > > > > > > How to specify job manager and task manager resources for different > > jobs > > > in > > > > session mode? > > > > > > > > On Sun, Mar 18, 2018 at 1:10 AM Till Rohrmann > > > > wrote: > > > > > > > > > Hi Shuyi, > > > > > > > > > > best if you look at the other e2e tests in the > flink-end-to-end-tests > > > > > module. For example the Kafka e2e test under > > > > > flink/flink-end-to-end-tests/test-scripts/test_streaming_ > > kafka010.sh. > > > > > > > > > > Cheers, > > > > > Till > > > > > > > > > > On Fri, Mar 16, 2018 at 10:20 PM, Shuyi Chen > > > wrote: > > > > > > > > > > > Hi Till, > > > > > > > > > > > > For FLINK-8562, the test is passing now because it's not really > > > > > > checking the right thing. > > > > > > > > > > > > Yes, I can help with the Kerberos integration ticket. > > > > > > > > > > > > Is there an example on how the e2e test should be structured and > > > > invoked? > > > > > > > > > > > > Thanks > > > > > > Shuyi > > > > > > > > > > > > On Fri, Mar 16, 2018 at 6:51 AM, Till Rohrmann < > > trohrm...@apache.org > > > > > > > > > > wrote: > > > > > > > > > > > > > Hi Shuyi, > > > > > > > > > > > > > > thanks for the working on FLINK-8562. Once this issue is fixed, > > it > > > > will > > > > > > > automatically be executed on the Flip-6 components. In fact it > is > > > > > already > > > > > > > being executed on Flip-6. > > > > > > > > > > > > > > But what you could help the community with is setting up an > > > automated > > > > > > > end-to-end test for the Kerberos integration if you want: > > > > > > > https://issues.apache.org/jira/browse/FLINK-8981. > > > > > > > > > > > > > > The Flink community is currently working on automating more and > > > more > > > > > > tests > > > > > > > in order to facilitate faster releases and improve the test > > > coverage. > > > > > You > > > > > > > can find more about this effort here: > > > > > > > https://issues.apache.org/jira/browse/FLINK-8970. > > > > > > > > > > > > > > Cheers, > > > > > > > Till > > > > > > > > > > > > > > On Thu, Mar 15, 2018 at 8:45 PM, Shuyi Chen < > suez1...@gmail.com> > > > > > wrote: > > > > > > > > > > > > > > > Hi Till, > > > > > > > > > > > > > > > > This is Shuyi :) Thanks a lot. In FLINK-8562, I already sent > a > > PR > > > > to > > > > > > > > resolve the issue, your help to take a look will be great. > > > > > > > > > > > > > > > > Please let me know what I can help to test the Kerberos > > > > > > authentication, I > > > > > > > > am decently familiar with the Kerberos and YARN security part > > in > > > > > Flink. > > > > > > > > > > > > > > > > As a starting point, I'd suggest to add an integration test > > > similar > > > > > to > > > > > > > > YARNSessionFIFOSecuredITCase > > > > > > > > for flip6. > > > > > > > > > > > > > > > > Shuyi > > > > > > > > > > > > > > > > On Thu, Mar 15, 2018 at 5:44 AM, Till Rohrmann < > > > > trohrm...@apache.org > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > Hi Renjie, > > > > > > > > > > > > > > > > > > thanks for the pointer with the > YARNSessionFIFOSecuredITCase. > > > > > You're > > > > > > > > right > > > > > > > > > that we should fix this test. There is FLINK-8562 which > seems > > > to > > > > > > > address > > > > > > > > > the problem. Will take a look. > > > > > > > > > > > > > > > > > > Additionally, we want to test Kerberos authentication > > > explicitly > > > > as > > > > > > > part > > > > > > > > of > > > > > > > > > the release testing for Flink 1.5. I will shortly send > > around a > > > > > mail > > > > > > > > where > > > > > > > > > I will lay out the ongoing testing efforts and where more > is > > > > > needed. > > > > > > > > > > > > > > > > > > Cheers, > > > > > > > > > Till > > > > > > > > > > > > > > > > > > On Thu, Mar 15, 2018 at 7:37 AM, Renjie Liu < > > > > > liurenjie2...@gmail.com > > > > > > > > > > > > > > > > wrote: > > > > > > > > > >
Re: FLIP-6 / Kubernetes
Hi Stephan, > This feature did not make it for 1.5, but should come very soon after. Are you talking about a 1.5.1 here? What is the envisioned timeline? Thanks! On Tue, Mar 20, 2018 at 9:30 PM, Stephan Ewen wrote: > @Eron That is definitely the way we want to suggest as the way to use k8s > in the future. This feature did not make it for 1.5, but should come very > soon after. > > @Thomas An implementation of a ResourceManager for k8s should come in the > near future. Would be happy to jump on a joint FLIP, after the 1.5 release > and Flink Forward (in three weeks or so). > > On Tue, Mar 20, 2018 at 4:25 PM, Eron Wright wrote: > > > Till, is it possible to package a Flink application as a self-contained > > deployment on Kubernetes? I mean, run a Flink application using 'flink > > run' such that it launches its own RM/JM and waits for a sufficient # of > > TMs to join? > > > > Thanks! > > > > On Mon, Mar 19, 2018 at 2:57 AM, Till Rohrmann > > wrote: > > > > > I forgot to add Flink's K8 documentation [1] which might also be > helpful > > > with getting started. > > > > > > [1] > > > https://ci.apache.org/projects/flink/flink-docs-master/ops/deployment/ > > > kubernetes.html > > > > > > Cheers, > > > Till > > > > > > On Mon, Mar 19, 2018 at 10:54 AM, Till Rohrmann > > > wrote: > > > > > > > Hi Thomas, > > > > > > > > I think the one way to get started would be to adapt the Flink docker > > > > images [1,2] to run with Flink 1.5. Per default, they will use the > > Flip-6 > > > > components. > > > > > > > > Flink 1.5 won't come with a dedicated integration with Kubernetes > which > > > is > > > > able to start new pods. However, it should work that you manually or > by > > > the > > > > virtues of an external system start new pods which can be used. Once > > the > > > > new pods have been started and the TaskExecutors have registered one > > > would > > > > have to rescale the job manually to the new parallelism. > > > > > > > > [1] https://flink.apache.org/news/2017/05/16/official-docker- > > image.html > > > > [2] https://hub.docker.com/r/_/flink/ > > > > > > > > Cheers, > > > > Till > > > > > > > > On Sun, Mar 18, 2018 at 9:05 PM, Thomas Weise > wrote: > > > > > > > >> Hi, > > > >> > > > >> What would be a good starting point to try out Flink on Kubernetes > > (any > > > >> examples / tutorials)? > > > >> > > > >> Also, will the FLIP-6 work in 1.5 enable dynamic scaling on > > Kubernetes? > > > >> > > > >> Thanks, > > > >> Thomas > > > >> > > > > > > > > > > > > > > -- Christophe
[jira] [Created] (FLINK-9042) Port ResumeCheckpointManuallyITCase to flip6
Chesnay Schepler created FLINK-9042: --- Summary: Port ResumeCheckpointManuallyITCase to flip6 Key: FLINK-9042 URL: https://issues.apache.org/jira/browse/FLINK-9042 Project: Flink Issue Type: Improvement Components: State Backends, Checkpointing, Tests Affects Versions: 1.5.0 Reporter: Chesnay Schepler Assignee: Chesnay Schepler Fix For: 1.5.0 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
Re: Flip 6 mesos support
At the moment this is not possible. In order to do this, you will have to use the per-job mode and run each job on a dedicated Flink cluster. On Wed, Mar 21, 2018 at 11:33 AM, Renjie Liu wrote: > For example, we have 2 jobs. > For job 1, I want to start job manger with 1 CPU and 100M memory. Job 1 > need s10 slots, and I want to deploy these 10 slots in 2 task managers, > each with 5 cores and 1G memory. > > For job 2, I want to start job manager with 2 CPU and 200M memory. Job 2 > needs 100 slots and I want to deploy these 100 slot in 10 task managers, > each with 10 cores and 2G memory. > > Is this possible? > > On Wed, Mar 21, 2018 at 6:19 PM Till Rohrmann > wrote: > > > Hi Renjie, > > > > what do you mean with specifying different JM and TM resources for > > different jobs exactly? > > > > Cheers, > > Till > > > > On Wed, Mar 21, 2018 at 10:55 AM, Renjie Liu > > wrote: > > > > > Hi, Till: > > > > > > How to specify job manager and task manager resources for different > jobs > > in > > > session mode? > > > > > > On Sun, Mar 18, 2018 at 1:10 AM Till Rohrmann > > > wrote: > > > > > > > Hi Shuyi, > > > > > > > > best if you look at the other e2e tests in the flink-end-to-end-tests > > > > module. For example the Kafka e2e test under > > > > flink/flink-end-to-end-tests/test-scripts/test_streaming_ > kafka010.sh. > > > > > > > > Cheers, > > > > Till > > > > > > > > On Fri, Mar 16, 2018 at 10:20 PM, Shuyi Chen > > wrote: > > > > > > > > > Hi Till, > > > > > > > > > > For FLINK-8562, the test is passing now because it's not really > > > > > checking the right thing. > > > > > > > > > > Yes, I can help with the Kerberos integration ticket. > > > > > > > > > > Is there an example on how the e2e test should be structured and > > > invoked? > > > > > > > > > > Thanks > > > > > Shuyi > > > > > > > > > > On Fri, Mar 16, 2018 at 6:51 AM, Till Rohrmann < > trohrm...@apache.org > > > > > > > > wrote: > > > > > > > > > > > Hi Shuyi, > > > > > > > > > > > > thanks for the working on FLINK-8562. Once this issue is fixed, > it > > > will > > > > > > automatically be executed on the Flip-6 components. In fact it is > > > > already > > > > > > being executed on Flip-6. > > > > > > > > > > > > But what you could help the community with is setting up an > > automated > > > > > > end-to-end test for the Kerberos integration if you want: > > > > > > https://issues.apache.org/jira/browse/FLINK-8981. > > > > > > > > > > > > The Flink community is currently working on automating more and > > more > > > > > tests > > > > > > in order to facilitate faster releases and improve the test > > coverage. > > > > You > > > > > > can find more about this effort here: > > > > > > https://issues.apache.org/jira/browse/FLINK-8970. > > > > > > > > > > > > Cheers, > > > > > > Till > > > > > > > > > > > > On Thu, Mar 15, 2018 at 8:45 PM, Shuyi Chen > > > > wrote: > > > > > > > > > > > > > Hi Till, > > > > > > > > > > > > > > This is Shuyi :) Thanks a lot. In FLINK-8562, I already sent a > PR > > > to > > > > > > > resolve the issue, your help to take a look will be great. > > > > > > > > > > > > > > Please let me know what I can help to test the Kerberos > > > > > authentication, I > > > > > > > am decently familiar with the Kerberos and YARN security part > in > > > > Flink. > > > > > > > > > > > > > > As a starting point, I'd suggest to add an integration test > > similar > > > > to > > > > > > > YARNSessionFIFOSecuredITCase > > > > > > > for flip6. > > > > > > > > > > > > > > Shuyi > > > > > > > > > > > > > > On Thu, Mar 15, 2018 at 5:44 AM, Till Rohrmann < > > > trohrm...@apache.org > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > Hi Renjie, > > > > > > > > > > > > > > > > thanks for the pointer with the YARNSessionFIFOSecuredITCase. > > > > You're > > > > > > > right > > > > > > > > that we should fix this test. There is FLINK-8562 which seems > > to > > > > > > address > > > > > > > > the problem. Will take a look. > > > > > > > > > > > > > > > > Additionally, we want to test Kerberos authentication > > explicitly > > > as > > > > > > part > > > > > > > of > > > > > > > > the release testing for Flink 1.5. I will shortly send > around a > > > > mail > > > > > > > where > > > > > > > > I will lay out the ongoing testing efforts and where more is > > > > needed. > > > > > > > > > > > > > > > > Cheers, > > > > > > > > Till > > > > > > > > > > > > > > > > On Thu, Mar 15, 2018 at 7:37 AM, Renjie Liu < > > > > liurenjie2...@gmail.com > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > Thanks for the clarification > > > > > > > > > > > > > > > > > > On Thu, Mar 15, 2018 at 2:30 PM 周思华 > > > wrote: > > > > > > > > > > > > > > > > > > > Hi Renjie, > > > > > > > > > > if I am not misunderstand, you just need to start the > > cluster > > > > as > > > > > > > normal > > > > > > > > > as > > > > > > > > > > before. The dispatcher and resourcemanager are spawned by > > > > > > > > > ClusterEntryPoint > > > > > > > >
[jira] [Created] (FLINK-9041) Refactor StreamTaskTest to not use scala and akka
Chesnay Schepler created FLINK-9041: --- Summary: Refactor StreamTaskTest to not use scala and akka Key: FLINK-9041 URL: https://issues.apache.org/jira/browse/FLINK-9041 Project: Flink Issue Type: Improvement Components: Streaming, Tests Affects Versions: 1.5.0 Reporter: Chesnay Schepler -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (FLINK-9040) JobVertex#setMaxParallelism does not valid argument
Chesnay Schepler created FLINK-9040:
---
Summary: JobVertex#setMaxParallelism does not valid argument
Key: FLINK-9040
URL: https://issues.apache.org/jira/browse/FLINK-9040
Project: Flink
Issue Type: Bug
Components: Local Runtime
Affects Versions: 1.5.0
Reporter: Chesnay Schepler
{code}
/**
* Sets the maximum parallelism for the task.
*
* @param maxParallelism The maximum parallelism to be set. must be between 1
and Short.MAX_VALUE.
*/
public void setMaxParallelism(int maxParallelism) {
this.maxParallelism = maxParallelism;
}
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
Re: Flip 6 mesos support
For example, we have 2 jobs. For job 1, I want to start job manger with 1 CPU and 100M memory. Job 1 need s10 slots, and I want to deploy these 10 slots in 2 task managers, each with 5 cores and 1G memory. For job 2, I want to start job manager with 2 CPU and 200M memory. Job 2 needs 100 slots and I want to deploy these 100 slot in 10 task managers, each with 10 cores and 2G memory. Is this possible? On Wed, Mar 21, 2018 at 6:19 PM Till Rohrmann wrote: > Hi Renjie, > > what do you mean with specifying different JM and TM resources for > different jobs exactly? > > Cheers, > Till > > On Wed, Mar 21, 2018 at 10:55 AM, Renjie Liu > wrote: > > > Hi, Till: > > > > How to specify job manager and task manager resources for different jobs > in > > session mode? > > > > On Sun, Mar 18, 2018 at 1:10 AM Till Rohrmann > > wrote: > > > > > Hi Shuyi, > > > > > > best if you look at the other e2e tests in the flink-end-to-end-tests > > > module. For example the Kafka e2e test under > > > flink/flink-end-to-end-tests/test-scripts/test_streaming_kafka010.sh. > > > > > > Cheers, > > > Till > > > > > > On Fri, Mar 16, 2018 at 10:20 PM, Shuyi Chen > wrote: > > > > > > > Hi Till, > > > > > > > > For FLINK-8562, the test is passing now because it's not really > > > > checking the right thing. > > > > > > > > Yes, I can help with the Kerberos integration ticket. > > > > > > > > Is there an example on how the e2e test should be structured and > > invoked? > > > > > > > > Thanks > > > > Shuyi > > > > > > > > On Fri, Mar 16, 2018 at 6:51 AM, Till Rohrmann > > > > > wrote: > > > > > > > > > Hi Shuyi, > > > > > > > > > > thanks for the working on FLINK-8562. Once this issue is fixed, it > > will > > > > > automatically be executed on the Flip-6 components. In fact it is > > > already > > > > > being executed on Flip-6. > > > > > > > > > > But what you could help the community with is setting up an > automated > > > > > end-to-end test for the Kerberos integration if you want: > > > > > https://issues.apache.org/jira/browse/FLINK-8981. > > > > > > > > > > The Flink community is currently working on automating more and > more > > > > tests > > > > > in order to facilitate faster releases and improve the test > coverage. > > > You > > > > > can find more about this effort here: > > > > > https://issues.apache.org/jira/browse/FLINK-8970. > > > > > > > > > > Cheers, > > > > > Till > > > > > > > > > > On Thu, Mar 15, 2018 at 8:45 PM, Shuyi Chen > > > wrote: > > > > > > > > > > > Hi Till, > > > > > > > > > > > > This is Shuyi :) Thanks a lot. In FLINK-8562, I already sent a PR > > to > > > > > > resolve the issue, your help to take a look will be great. > > > > > > > > > > > > Please let me know what I can help to test the Kerberos > > > > authentication, I > > > > > > am decently familiar with the Kerberos and YARN security part in > > > Flink. > > > > > > > > > > > > As a starting point, I'd suggest to add an integration test > similar > > > to > > > > > > YARNSessionFIFOSecuredITCase > > > > > > for flip6. > > > > > > > > > > > > Shuyi > > > > > > > > > > > > On Thu, Mar 15, 2018 at 5:44 AM, Till Rohrmann < > > trohrm...@apache.org > > > > > > > > > > wrote: > > > > > > > > > > > > > Hi Renjie, > > > > > > > > > > > > > > thanks for the pointer with the YARNSessionFIFOSecuredITCase. > > > You're > > > > > > right > > > > > > > that we should fix this test. There is FLINK-8562 which seems > to > > > > > address > > > > > > > the problem. Will take a look. > > > > > > > > > > > > > > Additionally, we want to test Kerberos authentication > explicitly > > as > > > > > part > > > > > > of > > > > > > > the release testing for Flink 1.5. I will shortly send around a > > > mail > > > > > > where > > > > > > > I will lay out the ongoing testing efforts and where more is > > > needed. > > > > > > > > > > > > > > Cheers, > > > > > > > Till > > > > > > > > > > > > > > On Thu, Mar 15, 2018 at 7:37 AM, Renjie Liu < > > > liurenjie2...@gmail.com > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > Thanks for the clarification > > > > > > > > > > > > > > > > On Thu, Mar 15, 2018 at 2:30 PM 周思华 > > wrote: > > > > > > > > > > > > > > > > > Hi Renjie, > > > > > > > > > if I am not misunderstand, you just need to start the > cluster > > > as > > > > > > normal > > > > > > > > as > > > > > > > > > before. The dispatcher and resourcemanager are spawned by > > > > > > > > ClusterEntryPoint > > > > > > > > > (you can have a look at yarn-session.sh & > > FlinkYarnSessionCli & > > > > > > > > > YarnSessionClusterEntrypoint), and the TM are spawned by > > > > > > > ResourceManager > > > > > > > > > lazily (ResourceManager will setup TM according to the > > > submitted > > > > > job) > > > > > > > or > > > > > > > > > spawned by the setup script (you can have a look at > > > > > > start-cluster.sh). > > > > > > > > > > > > > > > > > > > > > > > > > > > Best Regards, > > > > > > > > > Sihua Zhou > > > > > > > > > > > > > > > > > > > > > > > > > >
Re: Flip 6 mesos support
Hi Renjie, what do you mean with specifying different JM and TM resources for different jobs exactly? Cheers, Till On Wed, Mar 21, 2018 at 10:55 AM, Renjie Liu wrote: > Hi, Till: > > How to specify job manager and task manager resources for different jobs in > session mode? > > On Sun, Mar 18, 2018 at 1:10 AM Till Rohrmann > wrote: > > > Hi Shuyi, > > > > best if you look at the other e2e tests in the flink-end-to-end-tests > > module. For example the Kafka e2e test under > > flink/flink-end-to-end-tests/test-scripts/test_streaming_kafka010.sh. > > > > Cheers, > > Till > > > > On Fri, Mar 16, 2018 at 10:20 PM, Shuyi Chen wrote: > > > > > Hi Till, > > > > > > For FLINK-8562, the test is passing now because it's not really > > > checking the right thing. > > > > > > Yes, I can help with the Kerberos integration ticket. > > > > > > Is there an example on how the e2e test should be structured and > invoked? > > > > > > Thanks > > > Shuyi > > > > > > On Fri, Mar 16, 2018 at 6:51 AM, Till Rohrmann > > > wrote: > > > > > > > Hi Shuyi, > > > > > > > > thanks for the working on FLINK-8562. Once this issue is fixed, it > will > > > > automatically be executed on the Flip-6 components. In fact it is > > already > > > > being executed on Flip-6. > > > > > > > > But what you could help the community with is setting up an automated > > > > end-to-end test for the Kerberos integration if you want: > > > > https://issues.apache.org/jira/browse/FLINK-8981. > > > > > > > > The Flink community is currently working on automating more and more > > > tests > > > > in order to facilitate faster releases and improve the test coverage. > > You > > > > can find more about this effort here: > > > > https://issues.apache.org/jira/browse/FLINK-8970. > > > > > > > > Cheers, > > > > Till > > > > > > > > On Thu, Mar 15, 2018 at 8:45 PM, Shuyi Chen > > wrote: > > > > > > > > > Hi Till, > > > > > > > > > > This is Shuyi :) Thanks a lot. In FLINK-8562, I already sent a PR > to > > > > > resolve the issue, your help to take a look will be great. > > > > > > > > > > Please let me know what I can help to test the Kerberos > > > authentication, I > > > > > am decently familiar with the Kerberos and YARN security part in > > Flink. > > > > > > > > > > As a starting point, I'd suggest to add an integration test similar > > to > > > > > YARNSessionFIFOSecuredITCase > > > > > for flip6. > > > > > > > > > > Shuyi > > > > > > > > > > On Thu, Mar 15, 2018 at 5:44 AM, Till Rohrmann < > trohrm...@apache.org > > > > > > > > wrote: > > > > > > > > > > > Hi Renjie, > > > > > > > > > > > > thanks for the pointer with the YARNSessionFIFOSecuredITCase. > > You're > > > > > right > > > > > > that we should fix this test. There is FLINK-8562 which seems to > > > > address > > > > > > the problem. Will take a look. > > > > > > > > > > > > Additionally, we want to test Kerberos authentication explicitly > as > > > > part > > > > > of > > > > > > the release testing for Flink 1.5. I will shortly send around a > > mail > > > > > where > > > > > > I will lay out the ongoing testing efforts and where more is > > needed. > > > > > > > > > > > > Cheers, > > > > > > Till > > > > > > > > > > > > On Thu, Mar 15, 2018 at 7:37 AM, Renjie Liu < > > liurenjie2...@gmail.com > > > > > > > > > > wrote: > > > > > > > > > > > > > Thanks for the clarification > > > > > > > > > > > > > > On Thu, Mar 15, 2018 at 2:30 PM 周思华 > wrote: > > > > > > > > > > > > > > > Hi Renjie, > > > > > > > > if I am not misunderstand, you just need to start the cluster > > as > > > > > normal > > > > > > > as > > > > > > > > before. The dispatcher and resourcemanager are spawned by > > > > > > > ClusterEntryPoint > > > > > > > > (you can have a look at yarn-session.sh & > FlinkYarnSessionCli & > > > > > > > > YarnSessionClusterEntrypoint), and the TM are spawned by > > > > > > ResourceManager > > > > > > > > lazily (ResourceManager will setup TM according to the > > submitted > > > > job) > > > > > > or > > > > > > > > spawned by the setup script (you can have a look at > > > > > start-cluster.sh). > > > > > > > > > > > > > > > > > > > > > > > > Best Regards, > > > > > > > > Sihua Zhou > > > > > > > > > > > > > > > > > > > > > > > > 发自网易邮箱大师 > > > > > > > > > > > > > > > > > > > > > > > > On 03/15/2018 10:14，Renjie Liu > wrote： > > > > > > > > Hi, Till: > > > > > > > > In fact I'm asking how to deploy other components such as > > > > dispatcher, > > > > > > > etc. > > > > > > > > > > > > > > > > Till Rohrmann 于 2018年3月15日周四 > 上午12:17写道： > > > > > > > > > > > > > > > > Hi Renjie, > > > > > > > > > > > > > > > > in the current master and release-1.5 branch flip-6 is > > activated > > > by > > > > > > > > default. If you want to turn it off you have to add `mode: > old` > > > to > > > > > your > > > > > > > > flink-conf.yaml. I'm really happy that you want to test it > out > > > :-) > > > > > > > > > > > > > > > > Cheers, > > > > > > > > Till > > > > > > > > > > > > > > > > On Wed, Mar 14, 2018
[jira] [Created] (FLINK-9039) Broken link to Hadoop Setup Guide in docs
Florian Schmidt created FLINK-9039: -- Summary: Broken link to Hadoop Setup Guide in docs Key: FLINK-9039 URL: https://issues.apache.org/jira/browse/FLINK-9039 Project: Flink Issue Type: Bug Affects Versions: 1.4.2 Reporter: Florian Schmidt On [https://ci.apache.org/projects/flink/flink-docs-release-1.4/start/dependencies.html] under the section Hadoop Dependencies there is a link to "Hadoop Setup Guide" which links to [https://ci.apache.org/projects/flink/flink-docs-release-1.4/ops/deployment/hadoop.html,] which in turn does not exist -- This message was sent by Atlassian JIRA (v7.6.3#76005)
Re: Flip 6 mesos support
Hi, Till: How to specify job manager and task manager resources for different jobs in session mode? On Sun, Mar 18, 2018 at 1:10 AM Till Rohrmann wrote: > Hi Shuyi, > > best if you look at the other e2e tests in the flink-end-to-end-tests > module. For example the Kafka e2e test under > flink/flink-end-to-end-tests/test-scripts/test_streaming_kafka010.sh. > > Cheers, > Till > > On Fri, Mar 16, 2018 at 10:20 PM, Shuyi Chen wrote: > > > Hi Till, > > > > For FLINK-8562, the test is passing now because it's not really > > checking the right thing. > > > > Yes, I can help with the Kerberos integration ticket. > > > > Is there an example on how the e2e test should be structured and invoked? > > > > Thanks > > Shuyi > > > > On Fri, Mar 16, 2018 at 6:51 AM, Till Rohrmann > > wrote: > > > > > Hi Shuyi, > > > > > > thanks for the working on FLINK-8562. Once this issue is fixed, it will > > > automatically be executed on the Flip-6 components. In fact it is > already > > > being executed on Flip-6. > > > > > > But what you could help the community with is setting up an automated > > > end-to-end test for the Kerberos integration if you want: > > > https://issues.apache.org/jira/browse/FLINK-8981. > > > > > > The Flink community is currently working on automating more and more > > tests > > > in order to facilitate faster releases and improve the test coverage. > You > > > can find more about this effort here: > > > https://issues.apache.org/jira/browse/FLINK-8970. > > > > > > Cheers, > > > Till > > > > > > On Thu, Mar 15, 2018 at 8:45 PM, Shuyi Chen > wrote: > > > > > > > Hi Till, > > > > > > > > This is Shuyi :) Thanks a lot. In FLINK-8562, I already sent a PR to > > > > resolve the issue, your help to take a look will be great. > > > > > > > > Please let me know what I can help to test the Kerberos > > authentication, I > > > > am decently familiar with the Kerberos and YARN security part in > Flink. > > > > > > > > As a starting point, I'd suggest to add an integration test similar > to > > > > YARNSessionFIFOSecuredITCase > > > > for flip6. > > > > > > > > Shuyi > > > > > > > > On Thu, Mar 15, 2018 at 5:44 AM, Till Rohrmann > > > > > wrote: > > > > > > > > > Hi Renjie, > > > > > > > > > > thanks for the pointer with the YARNSessionFIFOSecuredITCase. > You're > > > > right > > > > > that we should fix this test. There is FLINK-8562 which seems to > > > address > > > > > the problem. Will take a look. > > > > > > > > > > Additionally, we want to test Kerberos authentication explicitly as > > > part > > > > of > > > > > the release testing for Flink 1.5. I will shortly send around a > mail > > > > where > > > > > I will lay out the ongoing testing efforts and where more is > needed. > > > > > > > > > > Cheers, > > > > > Till > > > > > > > > > > On Thu, Mar 15, 2018 at 7:37 AM, Renjie Liu < > liurenjie2...@gmail.com > > > > > > > > wrote: > > > > > > > > > > > Thanks for the clarification > > > > > > > > > > > > On Thu, Mar 15, 2018 at 2:30 PM 周思华 wrote: > > > > > > > > > > > > > Hi Renjie, > > > > > > > if I am not misunderstand, you just need to start the cluster > as > > > > normal > > > > > > as > > > > > > > before. The dispatcher and resourcemanager are spawned by > > > > > > ClusterEntryPoint > > > > > > > (you can have a look at yarn-session.sh & FlinkYarnSessionCli & > > > > > > > YarnSessionClusterEntrypoint), and the TM are spawned by > > > > > ResourceManager > > > > > > > lazily (ResourceManager will setup TM according to the > submitted > > > job) > > > > > or > > > > > > > spawned by the setup script (you can have a look at > > > > start-cluster.sh). > > > > > > > > > > > > > > > > > > > > > Best Regards, > > > > > > > Sihua Zhou > > > > > > > > > > > > > > > > > > > > > 发自网易邮箱大师 > > > > > > > > > > > > > > > > > > > > > On 03/15/2018 10:14，Renjie Liu wrote： > > > > > > > Hi, Till: > > > > > > > In fact I'm asking how to deploy other components such as > > > dispatcher, > > > > > > etc. > > > > > > > > > > > > > > Till Rohrmann 于 2018年3月15日周四 上午12:17写道： > > > > > > > > > > > > > > Hi Renjie, > > > > > > > > > > > > > > in the current master and release-1.5 branch flip-6 is > activated > > by > > > > > > > default. If you want to turn it off you have to add `mode: old` > > to > > > > your > > > > > > > flink-conf.yaml. I'm really happy that you want to test it out > > :-) > > > > > > > > > > > > > > Cheers, > > > > > > > Till > > > > > > > > > > > > > > On Wed, Mar 14, 2018 at 3:03 PM, Renjie Liu < > > > liurenjie2...@gmail.com > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > Hi Till: > > > > > > > Is there any doc on deploying flink in flip6 mode? We want to > > help > > > > > > > testing > > > > > > > it. > > > > > > > > > > > > > > Till Rohrmann 于 2018年3月14日周三 下午7:08写道： > > > > > > > > > > > > > > Hi Renjie, > > > > > > > > > > > > > > in order to make Mesos work, we only needed to implement a > Mesos > > > > > > > specific > > > > > > > ResourceManager. Look at MesosResourceMana
[jira] [Created] (FLINK-9038) YARNSessionCapacitySchedulerITCase fails on travis
Chesnay Schepler created FLINK-9038: --- Summary: YARNSessionCapacitySchedulerITCase fails on travis Key: FLINK-9038 URL: https://issues.apache.org/jira/browse/FLINK-9038 Project: Flink Issue Type: Bug Components: Tests, YARN Affects Versions: 1.5.0 Reporter: Chesnay Schepler Fix For: 1.5.0 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
Re: Is Hadoop 3.0 integration planned?
That is definitely a good thing to have, would like to have a discussion about how to approach that after 1.5 is released. On Wed, Mar 21, 2018 at 5:39 AM, Jayant Ameta wrote: > > Jayant Ameta >
[jira] [Created] (FLINK-9037) Test flake Kafka09ITCase#testCancelingEmptyTopic
Stephan Ewen created FLINK-9037:
---
Summary: Test flake Kafka09ITCase#testCancelingEmptyTopic
Key: FLINK-9037
URL: https://issues.apache.org/jira/browse/FLINK-9037
Project: Flink
Issue Type: Bug
Components: Tests
Affects Versions: 1.5.0
Reporter: Stephan Ewen
{code}
Test
testCancelingEmptyTopic(org.apache.flink.streaming.connectors.kafka.Kafka09ITCase)
failed with:
org.junit.runners.model.TestTimedOutException: test timed out after 6
milliseconds
{code}
Full log: https://api.travis-ci.org/v3/job/356044885/log.txt
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
