Re: Running continuously on yarn with kerberos

I created https://issues.apache.org/jira/browse/FLINK-2977 On Thu, Nov 5, 2015 at 12:25 PM, Robert Metzger wrote: > Hi Niels, > thank you for analyzing the issue so properly. I agree with you. It seems > that HDFS and HBase are using their own tokes which we need to

Re: How best to deal with wide, structured tuples?

Hi, these are some interesting Ideas. I have some thoughts, though, about the current implementation. 1. With Schema and Field you are basically re-implementing RowTypeInfo, so it should not be required. Maybe just an easier way to create a RowTypeInfo. 2. Right now, in Flink the

Re: IT's with POJO's

Hi Nick, both JIRA and the mailing list are good. In this case I'd say JIRA would be better because then everybody has the full context of the discussion. The issue is fixed in 0.10, which is not yet released. You can work around the issue by implementing a custom SourceFunction which returns

Re: Running continuously on yarn with kerberos

Hi Niels, thank you for analyzing the issue so properly. I agree with you. It seems that HDFS and HBase are using their own tokes which we need to transfer from the client to the YARN containers. We should be able to port the fix from Spark (which they got from Storm) into our YARN client. I think

Re: IT's with POJO's

Thank you Robert, I'll give this a spin -- obvious now that you point it out. I'll go ahead and continue the line on inquiry on the JIRA. -n On Thu, Nov 5, 2015 at 4:18 AM, Robert Metzger wrote: > Hi Nick, > > both JIRA and the mailing list are good. In this case I'd say

Re: Published test artifacts for flink streaming

Hi Robert, It seems "type" was what I needed. This it also looks like the test jar has an undeclared dependency. In the end, the following allowed me to use TestStreamEnvironment for my integration test. Thanks a lot! -n org.apache.flink flink-streaming-core

Re: Published test artifacts for flink streaming

Thanks Stephan, I'll check that out in the morning. Generally speaking, it would be great to have some single-jvm example tests for those of us getting started. Following the example of WindowingIntegrationTest is mostly working, though reusing my single sink instance with its static collection

Re: Running on a firewalled Yarn cluster?

While discussing with my colleagues about the issue today, we came up with another approach to resolve the issue: d) Upload the job jar to HDFS (or another FS) and trigger the execution of the jar using an HTTP request to the web interface. We could add some tooling into the /bin/flink client to

Re: Running continuously on yarn with kerberos

Thank you for looking into the problem, Niels. Let us know if you need anything. We would be happy to merge a pull request once you have verified the fix. On Thu, Nov 5, 2015 at 1:38 PM, Niels Basjes wrote: > I created https://issues.apache.org/jira/browse/FLINK-2977 > > On

Re: Running on a firewalled Yarn cluster?

Hi, I checked and this setting has been set to a limited port range of only 100 port numbers. I tried to find the actual port an AM is running on and couldn't find it (I'm not the admin on that cluster) The url to the AM that I use to access it always looks like this: