Re: Combining streams with static data and using REST API as a sink
Hi Josh, I have a use-case similar to yours. I need to join a stream with data from a database to which I have access via a REST API. Since the Side inputs API continues begin and ongoing work. I am wondering how did you approached it, Did you use the rich function updating it periodically? Thank you in advance! Nancy -- View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Combining-streams-with-static-data-and-using-REST-API-as-a-sink-tp7083p13902.html Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.
Flink Docker Kubernetes Gitlab CI CDeployment
Hi all, Has someone successfully run Flink jobs with this type of setup (Gitlab CI CD and Kubernetes)? Since Flink jobs can’t be dockerized and deployed in a natural way as part of the container (according to Flip-6) I am not very sure of how is the best way of doing this. We are thinking of having 3 Docker files: One for the Job manager, one for the Task Manager, and one for a container that will submit the jars and properties File to the Job Manager. With this setup we can still use the official Flink Docker images, and we can automatically generate the third Docker file with the project specific tasks. This setup allows to have Continuous deployment. However, it implies that we will have a new Flink cluster every time there is a new code version. What do you think of this set up? Has someone implemented something similar? Is there any documentation about Flink-Gitlab-CI-CD-Kubernetes? Thank you very much, Nancy -- View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Flink-Docker-Kubernetes-Gitlab-CI-CDeployment-tp13371.html Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.
Re: flink-1.2 and unit testing / flinkspector
Hi Tarandeep and Ted, I am in this route now. I am trying to use Flinkspector with Flink 1.2 using your instructions but failing miserably. After applying the changes, when I try to run "mvn clean install", some Tests fail and therefore I am not able to build successfully. I am wondering if there is a pull request, where I can access to a version of Flinkspector working with Flink 1.2. Thank you, Nancy -- View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/flink-1-2-and-unit-testing-flinkspector-tp12281p12377.html Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.
Re: Cassandra Sink version
The documentation you mentioned says: "The Java client driver 3.0.7 (branch 3.0.x) is compatible with Apache Cassandra 1.2, 2.0, 2.1, 2.2 and 3.0". Thank you Kostas! -- View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Cassandra-Sink-version-tp12329p12340.html Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.
Cassandra Sink version
Hi all, Which Cassandra version is currently supported by the Flink 1.2 connector? and I am wondering which version Flink 1.3 will be supporting? Thank you! -- View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Cassandra-Sink-version-tp12329.html Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.
readFile - Continuous file processing
Hi guys, I have the following use case. Every day a new file is created and periodically some log records are appended to it. I am reading the file in the following way: executionEnvironment.readFile(format, directoryPath, PROCESS_CONTINUOUSLY, period.toMilliseconds(),filePathFilter); However, Flink takes modified files as new files and consequently all the content of the modified file gets processed again. I know that a solution is to process the file until it contains all the records of the day but I will like to process the file continuously. Therefore, I am wondering if there is a way of processing just the new records in a file? Thank you in advance! :) Nancy -- View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/readFile-Continuous-file-processing-tp11384.html Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.
Flink configuration
Hi all, I have been reading about how to configure Flink when we have a set up consisting on a couple of VMs with more than 1 vCore. I am a bit confused about how to set the degree of parallelism in the taskmanager.numberOfTaskSlots parameter: * According to the Flink documentation[1], this value is typically proportional to the number of/ physical CPU cores/ that the TaskManager’s machine has. * However the YARN documentation[2], makes reference to the number of /Virtual CPU cores/ per TaskManager. My question is, If my Flink Jobs will be running on VMs (without using YARN), the "taskmanager.numberOfTaskSlots" will depend on the number of vCPU that mi VM has? or must be related to the physical cores? Thanks in advance for your help! Nancy [1]https://ci.apache.org/projects/flink/flink-docs-release-0.8/config.html [2]https://ci.apache.org/projects/flink/flink-docs-release-0.8/yarn_setup.html -- View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Flink-configuration-tp11210.html Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.
Consistency guarantees on multiple sinks
Hi, If in a Job there is more than one sink declared, what happens when a failure occurs? all the sink operations get aborted? (atomically as in a transactional environment), or the exactly-once-processing consistency guarantees are provided just when one sink is declared per job? Is it recommended to have more than one sink per job? Thank you! Nancy Estrada -- View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Consistency-guarantees-on-multiple-sinks-tp10877.html Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.
