About the blob client and blob server authentication

Hi: I find the the flink issue 2425. https://github.com/apache/flink/pull/2425 This issue will do the authentication by using security cookie between the blob client and blob server! In my opinion, to use the SASL digest-md5 is much more authority. what do you think? BTW, when this issue is me

Re: Cross operation on two huge datasets

Hi all, @Gwen From the database's point of view, the only way to avoid Cartesian product in join is to use index, which exhibits as key grouping in Flink. However, it only supports many-to-one mapping now, i.e., a shape or a point can only be distributed to a single group. Only points and shapes b

回复：TaskManager failure detection

Hi Dominik,      As I know, the JobManager would detect the failure of TaskManager by akka watch mechanism. It is similar with heartbeat or ping way in network stack.You can refer to this link "https://cwiki.apache.org/confluence/display/FLINK/Akka+and+Actors";.   Futhermore, the upstream and do

Re: Flink jdbc

Hi, I should also mention that the JdbcOutputFormat batches writes to the database. Since it is not integrated with the Flink's checkpointing mechanism, data might get lost in case of a failure. I would recommend to implement a JdbcSinkFunction based on the code of the JdbcOutputFormat. If you use

Re: Cross operation on two huge datasets

Hi Gwen, Flink usually performs a block nested loop join to cross two data sets. This algorithm spills one input to disk and streams the other input. For each input it fills a memory buffer and to perform the cross. Then the buffer of the spilled input is refilled with spilled records and records

Re: Compiling Flink on macOS.

Problem solved, thanks. I did the following steps. git fetch and rebased my master on the origin/master then rebased my local branch on master... so now it looks like: * 9ab0faa - (HEAD -> pm/lior/py_tests) python: adding the --list-tests and --mvn-mode flags. (2 minutes ago) * d2cab93 - python

Re: Compiling Flink on macOS.

commit d6aed38b3a15946d383d762030b5f5c1418388de Author: wenlong.lwl Date: Fri Jan 6 16:32:08 2017 +0800 Can you update your local workspace and try again ? On Wed, Feb 22, 2017 at 10:53 AM, Lior Amar wrote: > Here is my commit path > * 59ba32e - (HEAD -> pm/lior/py_tests, pm/pm/lior/py_tests

Re: Compiling Flink on macOS.

Here is my commit path * 59ba32e - (HEAD -> pm/lior/py_tests, pm/pm/lior/py_tests) python: adding the --list-tests and --mvn-mode flags. (2 hours ago) * e640894 - python: clearing the arguments StringBuilder so it can be called multiple times with different args (2 hours ago) * 2619fb9 - python:

Re: Compiling Flink on macOS.

Which commit is your local repo ? As of commit b0f0f3722fac4726fba879736c7ee85993b392db , I don't see IRichSpout in flink-contrib/flink-storm-examples/src/main/java/org/apache/flink/storm/util/FiniteInMemorySpout.java Don't know where the backtype came from On Wed, Feb 22, 2017 at 10:36 AM, Lio

Re: Compiling Flink on macOS.

lior@MacBook-Pro-2 ~/src/flink/flink/flink-contrib/flink-storm-examples (pm/lior/py_tests) $ mvn install [INFO] Scanning for projects... [INFO] [INFO] [INFO] Building flink-storm-examples 1.3-SNAPSHOT [INFO] --

Re: Compiling Flink on macOS.

Can you show the error(s) you got ? What is the Java version you're using ? Thanks On Wed, Feb 22, 2017 at 10:24 AM, Lior Amar wrote: > Hi all, > > My name is Lior and I an new to this community. > I am working on enhancing the python API of Flink (batch). > > I am having some issues compiling

Compiling Flink on macOS.

Hi all, My name is Lior and I an new to this community. I am working on enhancing the python API of Flink (batch). I am having some issues compiling Flink on macOS with the latest master, it seems that flink-storm-examples module is failing. A quick question, how can I skip this module when I am

Re: Unknown I/O error while extracting contained jar files

Hi, which JVM variant and version are you using? What's your operating system? This is a pretty generic issue. If you search for "ZipException: error in opening zip file" on google, you'll find plenty of people who have this issue as well. I think its unrelated to Flink and more of a general JVM

Re: Checkpointing with RocksDB as statebackend

Hey Vinay! Do you have more of the stack trace? It seems like the root exception is missing... Stephan On Wed, Feb 22, 2017 at 9:55 AM, vinay patil wrote: > Hi Stephan, > > You can see the snaphots in the earlier mail > > When the size of the record increases task managers are getting killed

Unknown I/O error while extracting contained jar files

when i executor ./bin/flink run -c myMainClass myjar.jar i got the following exception. and it's work fine when i run unzip myjar.jar why i get this exception? Thanks. org.apache.flink.client.program.ProgramInvocationException: Unknown I/O error while extracting contained jar files. at

Re: Deployment Architecture for Flink Applications

Hi CVP, On how people use Flink, you can check this blogpost to see how Alibaba does it: http://data-artisans.com/blink-flink-alibaba-search/ In addition, you can also find some more information on the matter on the talks from the last Fli

Re: 回复：Transfer information from one window to the next

I solved the state you were talking about. The solution would like like this (similar to what you wrote): stream.keyBy(...).timeWindow(...) .apply(new WindowFunction() { public void apply(K key, W window, Iterable elements, Collector out) { out.collect(new Tuple3<>(key, wi

Deployment Architecture for Flink Applications

Hi Team, We are analysing different deployment options for managing Flink Jobs on AWS EC2 instances. Basically, the options (Resource Manangers) in front of us are using: -> Standalone cluster -> On YARN -> Deploy using Mesos/Marthon -> Deploy using Kubernetes/Docker

TaskManager failure detection

Hi, As I’m investigating onto Flink’s fault tolerance capabilities, I would like to know what component and class is in charge of TaskManager failure detection and checkpoint restoring? In addition, how does Flink actually determine that a TaskManager has failed due to e.g. hardware failures?

Re: Apache Flink and Elasticsearch send Json Object instead of string

Hi, The Flink Elasticsearch Sink uses the Elasticsearch Java client to send the indexing requests, so whatever the client supports, it will be achievable through the `ElasticsearchSinkFunction` also. From a quick check at the Elasticsearch Javadocs, I think you can also just set the document j

RE: Flink checkpointing gets stuck

I changed the checkpoint interval to 30 minutes, and also switched RocksDB predefined options to FLASH_SSD_OPTIMIZED, as suggested by Vinay. The problem hasn't exactly occurred since yesterday, but perhaps it just takes it longer to happen again because the checkpoints are much less frequent now

Re: How important is 'registerType'?

Hi Dmitry, do you maybe have a more complete stack trace? I have a suspicion but I would like to confirm that. Best, Aljoscha On Fri, 17 Feb 2017 at 20:24 Dmitry Golubets wrote: > Hi Till, > > It happened during deserialization of a savepoint. > > Best regards, > Dmitry > > On Fri, Feb 17, 2017

Re: Checkpointing with RocksDB as statebackend

Hi Stephan, You can see the snaphots in the earlier mail When the size of the record increases task managers are getting killed with the default FLASH_SSD_OPTIMIZED option. When I tried to set backgroundflushes to 4 and backgroundCompactions to 8 it ran for more time but then I got the following