Re: Class loading issues when using Remote Execution Environment

I couldn't spot any error in what you tried to do. Does the job-submission succeed if you submit the jar through the command-line client? Can you share the project, or a minimal reproducing version? On 25.04.2018 00:41, kedar mhaswade wrote: I am trying to get gradoop_demo

Re: Run programs w/ params including comma via REST api

other characters as delimiter for program arguments. Best, - Dongwon 2018. 4. 25. 오후 5:28, Chesnay Schepler <ches...@apache.org <mailto:ches...@apache.org>> 작성: Currently I don't see a way to circumvent the splitting. You will have to use a different delimiter, I guess a semi

Re: Keyby connect for a one to many relationship - DataStream API - Ride Enrichment (CoProcessFunction)

You can still connect the streams but it will be more complex than the reference solution. You will have to store the events from B in a ListState instead. If an A arrives, store it in the value state, emit a tuple (A, B_x) for every stored B, and clear B. From that point on, emit a new tuple

Re: Table/SQL Kafka Sink Question

Hello, as far as i can this is not possible. I'm including Timo, maybe he can explain why this isn't supported. On 26.03.2018 21:56, Pavel Ciorba wrote: Hi everyone! Can I specify a *message key* using the Kafka sink in the Table/SQL API ? The goal is to sink each row as JSON along side

Re: Programmatic creation of YARN sessions and deployment (running) Flink jobs on it.

Hello, I think the flink-conf.yaml should only be required on the node on which you call yarn-session.sh. For starting the session cluster programmatically you would have to look into the YarnClusterDescriptor (for starting the session cluster) and the YarnClusterClient for submitting jobs

Re: java.lang.IllegalArgumentException: The implementation of the provided ElasticsearchSinkFunction is not serializable. The object probably contains or references non-serializable fields.

Your anonymous ElasticsearchSinkFunction accesses the client variable that is defined outside of the function. For the function to be serializable, said Client must be as well. I suggest to turn your function into a named class with a constructor that accepts the indexName. On 27.03.2018

Re: regarding the use of colocation groups

Hello, your first use-case should be achievable by using a custom partitioner , probably with a KeySelector that returns the word. As for the second use-case, typically this would be achieved

Re: Flink Kafka producer: Object of class is not serializable

As answered on SO, the problem is the KeyedDeserializationSchema which is an anonymous inner class. On 31.03.2018 20:25, Soheil Pourbafrani wrote: I got an error in using Flink Kafka connector for producing data. I describe the problem here

Re: pyflink not working

I've commented in the linked JIRA, let's move this discussion there. On 20.03.2018 10:00, Ganesh Manal wrote: Hi, Not able to execute the pyflink job using the pyflink script. Similar to already logged issue – https://issues.apache.org/jira/browse/FLINK-8909

Re: Metric Registry Warnings

FLINK-7100 is about taskmanager metrics being registered twice, whereas here we're dealing with job metrics. Do you have multiple sources? If so, do they have unique names? On 20.03.2018 15:06, Fabian Hueske wrote: Hi Pedro, Can you reopen FLINK-7100 and post a comment with your error

Re: Error while reporting metrics - ConcorrentModificationException

A wrapped Kafka metric was accessing state of the consumer while said state was modified. As far as I can tell this is a Kafka issue and there's nothing we can do. Unless this happens frequently it should be safe to ignore it. On 20.03.2018 15:02, PedroMrChaves wrote: Hello, I have the

Re: cannot see monitor metrics

This is currently expected behavior. We do not measure incoming records for sources, and outgoing records for sinks as we can't do this in a generalized fashion. See FLINK-7286 . On 19.03.2018 02:37, ?? wrote: I run flink on yarn , Im

Re: Securing metrics reporters (Flink <-> Prometheus)

Hello, this is currently not supported. Regards, Chesnay On 02.03.2018 18:38, Fritz Budiyanto wrote: Hi All, How can I configure encryption for metrics.reports ? Particularly Prometheus ? I do not see any mention of encryption in the metrics.reports traffic in flink documentation. Is

Re: ClassNotFoundException with Flink 1.4.1 and Kafka connector

f org.apache.kafka.common.serialization.Serializer at org.apache.kafka.common.config.AbstractConfig.getConfiguredInstance(AbstractConfig.java:248) at org.apache.kafka.clients.producer.KafkaProducer.(KafkaProducer.java:342) ... 17 more regards. On Wed, Feb 28, 2018 at 2:31 PM, Chesnay Schepler <ches...@apache

Re: ClassNotFoundException with Flink 1.4.1 and Kafka connector

Hello, this is probably caused by a known issue in 1.4.1: https://issues.apache.org/jira/browse/FLINK-8741 This bug is not present in 1.4.0, and it will fixed in 1.4.2 which should be released within the next days. As a temporary workaround you can copy your app-assembly-1.0.jar into the

Re: Web history limit in flink 1.5

https://ci.apache.org/projects/flink/flink-docs-release-1.5/ops/config.html#jobstore-expiration-time On 28.06.2018 11:15, eSKa wrote: Hello, we were playing around with flink 1.5 - so far so good. Only thing that we are missing is web history setup. In flink 1.4 and before we were using

Re: Debug job execution from savepoint

Similarly, the AbstractOperatorRestoreTestBase shows how to start en embedded cluster and submit a job from a savepoint. see * #restoreJob * #createJobGraph * MiniClusterResource On 22.06.2018 13:53, Tzu-Li (Gordon) Tai wrote: Hi, The tests in Flink uses a `AbstractStreamOperatorTestHarness`

Re: # of active session windows of a streaming job

Checkpointing of metrics is a manual process. The operator must write the current value into state, retrieve it on restore and restore the counter's count. On 20.06.2018 12:10, Fabian Hueske wrote: Hi Dongwon, You are of course right! We need to decrement the counter when the window is

Re: [DISCUSS] Dropping flink-storm?

I've created https://issues.apache.org/jira/browse/FLINK-10509 for removing flink-storm. On 28.09.2018 15:22, Till Rohrmann wrote: Hi everyone, I would like to discuss how to proceed with Flink's storm compatibility layer flink-strom. While working on removing Flink's legacy mode, I noticed

Re: Flink Python streaming

Hello, to use libraries you have to supply them when submitting the job as described below. Additional directories/files will be placed in the same directory as your script on each TM. See https://ci.apache.org/projects/flink/flink-docs-master/dev/stream/python.html#executing-plans Note

Re: what's the meaning of latency indicator reported by flink metrics through prometheus?

1) correct 2) is the number of measurements; due to the random distribution of latency markers this value can be surprisingly low depending on the latency marker interval 3) I don't know, but it isn't exposed by Flink. On 08.10.2018 10:17, varuy322 wrote: Hi there, I have integrated kafka

Re: [DISCUSS] Dropping flink-storm?

. On 08.10.2018 11:04, Till Rohrmann wrote: Thanks for opening the issue Chesnay. I think the overall consensus is to drop flink-storm and only keep the Bolt and Spout wrappers. Thanks for your feedback! Cheers, Till On Mon, Oct 8, 2018 at 9:37 AM Chesnay Schepler <mailto:ches...@apache.org>&

Re: [DISCUSS] Breaking the Scala API for Scala 2.12 Support

I'd rather not maintain 2 master branches. Beyond the maintenance overhead I'm wondering about the benefit, as the API break still has to happen at some point. @Aljoscha how much work for supporting scala 2.12 can be merged without breaking the API? If this is the only blocker I suggest to

Re: [DISCUSS] Breaking the Scala API for Scala 2.12 Support

, Chesnay Schepler wrote: I'd rather not maintain 2 master branches. Beyond the maintenance overhead I'm wondering about the benefit, as the API break still has to happen at some point. @Aljoscha how much work for supporting scala 2.12 can be merged without breaking the API? If this is the only

Re: [DISCUSS] Breaking the Scala API for Scala 2.12 Support

a 2.12 release. On 8. Oct 2018, at 13:00, Chesnay Schepler wrote: And the remaining parts would only be about breaking the API? On 08.10.2018 12:24, Aljoscha Krettek wrote: I have an open PR that does everything we can do for preparing the code base for Scala 2.12 without breaking the API

Re: Getting NoMethod found error while running job on flink 1.6.1

The cause cannot be that flink-metrics-core is not on the classpath as in that case you'd get a ClassNotFoundError. This is a version conflict, either caused by your fat jar bundling an older version of flink-metrics-core but a newer version of the kafka connector, or you upgrade your

Re: Flink does not checkpoint if operator is in Finished state

Known issue: https://issues.apache.org/jira/browse/FLINK-2491 On 15.10.2018 12:23, Kien Truong wrote: Hi, As mentioned in the title, my Flink job will not check point if there are any finished source operator. Is this a bug or is it working as intended ? Regards, Kien

Re: Job Records In/Out metrics clarification

There is a known issue in 1.5.0 with the numRecordsIn/Out metrics for chained tasks: https://issues.apache.org/jira/browse/FLINK-9530 This has been fixed in 1.5.1. On 15.10.2018 14:37, Rafi Aroch wrote: Hi, Below is the In/Out metrics as they appear in the Flink UI. I was wondering what are

Re: Monitoring/rest_api,The "/jobs" how to use?

As the documentation states this is an effectively internal REST call used by the CLI. To submit jobs via REST I would suggest to check out the jar submission routine. https://ci.apache.org/projects/flink/flink-docs-master/monitoring/rest_api.html#jars-upload

Re: Monitoring/rest_api,The "/jobs" how to use?

/session that allows to monitor its sub-jobs. I think that this thing would be extremely useful..an alternative would be to use Oozie, Airflow or simliar but this would be more straightforward IMHO. Best, Flavio On Fri, Oct 19, 2018 at 10:29 AM Chesnay Schepler <mailto:ches...@apache.org>&

Re: Either bug in the flink, or out of date documentation ( flink 1.6 , cancel API rest endpoint )

The section you're looking at is the legacy documentation which only applies if the cluster is running in legacy mode. You want to look at the "Dispatcher" section (https://ci.apache.org/projects/flink/flink-docs-release-1.6/monitoring/rest_api.html#dispatcher), which documents the PATCH

[ANNOUNCE] Apache Flink 1.6.2 released

The Apache Flink community is very happy to announce the release of Apache Flink 1.6.2, which is the second bugfix release for the Apache Flink 1.6 series. Apache Flink® is an open-source stream processing framework for distributed, high-performing, always-available, and accurate data

[ANNOUNCE] Apache Flink 1.5.5 released

The Apache Flink community is very happy to announce the release of Apache Flink 1.5.5, which is the fifth bugfix release for the Apache Flink 1.5 series. Apache Flink® is an open-source stream processing framework for distributed, high-performing, always-available, and accurate data

Re: HighAvailability :: FsNegativeRunningJobsRegistry

The release process for 1.6.2 is currently ongoing and will hopefully be finished within the next days. In the mean-time you could use 1.6.2-rc1 artifacts: binaries: https://dist.apache.org/repos/dist/dev/flink/flink-1.6.2/ maven:

Re: Are flink connectors included in the binary release ?

Connectors are never contained in binary releases as they are supposed t be packaged into the user-jar. On 14.11.2018 10:12, Jeff Zhang wrote: I don't see the jars of flink connectors in the binary release of flink 1.6.1, so just want to confirm whether flink binary release include these

Re: Flink 1.7 RC missing flink-scala-shell jar

This is intended. Increasing the scala version basically broke the scala-shell and we haven't had the time to fix it. It is thus only available with scala 2.11. I agree that the error message could be better though. On 14.11.2018 03:44, Hao Sun wrote: I do not see flink-scala-shell jar under

Re: Job xxx not found exception when starting Flink program in Local

Did you have the WebUI open from a previous execution? If so then the UI might still be requesting jobs from the previous job. On 13.11.2018 08:01, 徐涛 wrote: Hi Experts, When I start Flink program in local, I found that the following exception throws out, I do not know why it happens because

Re: Get savepoint status fails - Flink 1.6.2

The documentation you linked only applies if legacy mode is enabled. The URL you used initially (i.e. "/jobs/:jobid/savepoints/:triggerid:triggerid") is correct. My guess is that either the JobID or triggerID is not correct. On 13.11.2018 17:24, PedroMrChaves wrote: Hello, I am trying to

Re: Field could not be resolved by the field mapping when using kafka connector

This issue was already resolved in another thread by the same author. On 15.11.2018 10:52, Dominik Wosiński wrote: Hey, Could You please show a sample data that You want to process? This would help in verifying the issue. Best Regards, Dom. wt., 13 lis 2018 o 13:58 Jeff Zhang

Re: Rescaling Flink job from an External Checkpoint

The docs are worded that way since not all backends support it. I believe rescaling does work for RocksDB checkpoints, but we do not provide any /guarantee /that this remains to be the case. Basically, use at your own risk. On 13.11.2018 13:24, suraj7 wrote: Hi, I'm using Flink 1.5 with

Re: Could not find previous entry with key.

Can you provide us with the implementation of your Event and IoTEvent classes? On 15.11.2018 06:10, Steve Bistline wrote: Any thoughts on where to start with this error would be appreciated. Caused by: java.lang.IllegalStateException: Could not find previous entry with key: first event,

Re: Partitioning by composite key, But type and number of keys are dynamic

Why don't you calculate the hashCode for each field, combine them and use that as the key? You won't get around calculating /something /for each field and combining the result. On 15.11.2018 11:02, Gaurav Luthra wrote: Hi Chesnay, My End user will be aware about the fields of "input records"

Re: Run Time Exception

I'm highly encouraging you to read through the examples and Batch API documentation: https://ci.apache.org/projects/flink/flink-docs-release-1.6/dev/batch/#example-program https://ci.apache.org/projects/flink/flink-docs-release-1.6/dev/batch/dataset_transformations.html#dataset-transformations

Re: [DISCUSS] Dropping flink-storm?

:13 AM Chesnay Schepler <mailto:ches...@apache.org>> wrote: I don't believe that to be the consensus. For starters it is contradictory; we can't /drop /flink-storm yet still /keep //some parts/. From my understanding we drop flink-storm completely, and put a note in

Re: getRuntimeContext(): The runtime context has not been initialized.

In which method are you calling getRuntimeContext()? This method can only be used after open() has been called. On 09.10.2018 17:09, Ahmad Hassan wrote: Hi, We want to use MapState inside fold function to keep the map of all products that we see in 24 hour window to store huge state in

Re: [DISCUSS] Dropping flink-storm?

I'm very much in favor of dropping it. Flink has been continually growing in terms of features, and IMO we've reached the point where we should cull some of the more obscure ones. flink-storm, while interesting from a theoretical standpoint, offers too little value. Note that the bolt/spout

Re: Upgrade Flink with newer Java version

Please see https://issues.apache.org/jira/browse/FLINK-8033. On 01.10.2018 13:39, Georgi Stoyanov wrote: Hi, Oracle will stop support for Java 8 on Jan 2019. Do you guys plans to upgrade the version? If so, do you have ticket which we can watch for updates? Regards, G. Stoyanov

Re: Rolling File Sink Exception

You're closing the stream and then call super.close() which calls flush, which fails since you already closed the stream. If you don't close the stream the problem should disappear. On 03.09.2018 09:30, clay wrote: When I want to write compressed string data to hdfs, I found that flink

Re: Flink 1.5.2 query

Cannot be avoided. The CLI eagerly loads client classes for yarn, which as see fails since the hadoop classes aren't available. If you don't use YARN you can safely ignore this. On 03.09.2018 14:37, Parth Sarathy wrote: Hi, When using flink 1.5.2, “Apache Flink Only” binary

Re: Can't access Flink Dashboard at 8081, running Flink program using Eclipse

you can setup a specific port using https://ci.apache.org/projects/flink/flink-docs-master/ops/config.html#rest-port. On 03.09.2018 12:12, Mar_zieh wrote: Hello I added these dependencies to "pom.xml"; also, I added configuration to my code like these: Configuration config = new

Re: Failed to trigger savepoint

You will have to take a look at the JobManager/TaskManager logs. On 04.09.2018 12:02, Paul Lam wrote: Hi, I’m using Flink 1.5.3 and failed to trigger savepoint for a Flink on YARN job. The stack traces shows that an exception occurred while triggering the checkpoint, but the normal

Re: LocalEnvironment and Python streaming

No, this isn't really possible. You need a java process to kick off the processing. The only thing i can come up with is to open the flink-streaming-python module in the IDE and manually call the PythonStreamBinder class with the same arguments that you pass in the CLI as a test. On

Re: [External] Re: Setting a custom Kryo serializer in Flink-Python

I can't really help you here. Digging into the backing java internals isn't supported, and neither is registering a kryo serializer (which is why it isn't exposed in the python environment). The jython-related serialization logic doesn't care about Flink's usual type serialization mechanism,

Re: Help: how to get latency value comfortable in Flink1.5?

So there's 2 issues here: 1) your reporter configuration is wrong, configuration values for a specific reporter are prefixed with "metrics.reporter", not "metrics.reporters" (note the "s"). See below for a correct config. metrics.reporters: varuy metrics.reporter.varuy.host: tracing115

Re: Information required regarding SSL algorithms for Flink 1.5.x

Please see https://ci.apache.org/projects/flink/flink-docs-release-1.5/ops/config.html#security-ssl-algorithms for the SSL algorithms that are available by default for 1.5 . On 27.09.2018 13:24, V N, Suchithra (Nokia - IN/Bangalore) wrote: Gentle reminder on this question. *From:*V N,

Re: Regarding implementation of aggregate function using a ProcessFunction

Please see: https://issues.apache.org/jira/browse/FLINK-10250 On 28.09.2018 11:27, vino yang wrote: Hi Gaurav, Yes, you are right. It is really not allowed to use RichFunction. I will Ping Timo, he may give you a more professional answer. Thanks, vino. Gaurav Luthra

Re: When could flink 1.7.1 be downloaded from maven repository

There was a problem during the release process that delayed the maven artifacts, but it appears they are available now: https://mvnrepository.com/artifact/org.apache.flink/flink-table_2.12/1.7.1 On 29.12.2018 02:33, 徐涛 wrote: Hi Experts, From github I saw flink 1.7.1 is released about

Re: Change Window Size during runtime

You can't change the window size at runtime. On 03.01.2019 00:54, Rad Rad wrote: Hi All, I have one stream is consumed by FlinkKafkaConsumer which will be joined with another stream for defined window size such as Time.milliseconds(1). How can I change window size during runtime to

Re: Problem building 1.7.1 with scala-2.12

When building Flink for scala 2.12 you have to pass "-Dscala-2.12" to maven; see the flink-connector-kafka-0.9 pom for details. (look for the scala-2.11 profile) On 02.01.2019 17:48, Cliff Resnick wrote: The build fails at flink-connector-kafka-0.9 because _2.12 libraries apparently do not

Re: Flink error reading file over network (Windows)

Yes, you'll need to create your own InputFormat that understands SMB. On 03.01.2019 08:26, miki haiat wrote: Hi, Im trying to read a csv file from windows shard drive. I tried numbers option but i failed. I cant find an option to use SMB format, so im assuming that create my own input format

[DISCUSS] Towards a leaner flink-dist

Hello, the binary distribution that we release by now contains quite a lot of optional components, including various filesystems, metric reporters and libraries. Most users will only use a fraction of these, and as such pretty much only increase the size of flink-dist. With Flink growing

Re: Unable to override metric format for Prometheus Reporter

The prometheus reporter ignores scope formats. In fact all reporters that work with tags (i.e., key-value-pairs) ignore them, the idea being that you would search specific metrics based on their tags. I'm not aware of any intermediate workarounds. There is a JIRA for this issue:

Re: Connection leak with flink elastic Sink

Specifically which connector are you using, and which Flink version? On 12.12.2018 13:31, Vijay Bhaskar wrote: Hi We are using flink elastic sink which streams at the rate of 1000 events/sec, as described in

Re: problem submitting job, it hangs there

The directory is automatically created when Flink is started; maybe it was deleted by some cleanup process? In any case we can make a small adjustment to the code to create all required directories when they don't exist. On 13.12.2018 14:46, Chang Liu wrote: Dear All, I did a workaround

Re: Flink metrics in kubernetes deployment

If you're working with 1.7/master you're probably running into https://issues.apache.org/jira/browse/FLINK-11127 . On 17.12.2018 18:12, eric hoffmann wrote: Hi, In a Kubernetes delpoyment, im not able to display metrics in the dashboard, I try to expose and fix the

Re: How to test window

Since you define a 15 second window you have to ensure that your source generates at least 15 seconds worth of data; otherwise the window will never fire. Since you do not use event-time your source has to actually run for at least 15 seconds; for this case collection sources will simply not

[ANNOUNCE] Apache Flink 1.7.1 released

The Apache Flink community is very happy to announce the release of Apache Flink 1.7.1, which is the first bugfix release for the Apache Flink 1.7 series. Apache Flink® is an open-source stream processing framework for distributed, high-performing, always-available, and accurate data

Re: Flink 1.7 jobmanager tries to lookup taskmanager by its hostname in k8s environment

This is a known issue, see https://issues.apache.org/jira/browse/FLINK-11127. I'm not aware of a workaround. On 12.12.2018 14:07, Sergei Poganshev wrote: When I to deploy Flink 1.7 job to Kubernetes, the job itself runs, but upon visiting Flink UI I can see no metrics and there are WARN

Re: flink list and flink run commands timeout

018-09-05 15:20:24,330 ERROR org.apache.flink.client.cli.CliFrontend- Error while running the command. On Wednesday, September 5, 2018, 3:41:29 a.m. EDT, Chesnay Schepler mailto:ches...@apache.org>> wrote: Please enable DEBUG logging for the client and

Re: NoClassDefFoundError javax.xml.bind.DatatypeConverterImpl

Small correction: Flink 1.7 does not support jdk9; we only fixed some of the issues, not all of them. On 06.12.2018 07:13, Mike Mintz wrote: Hi Flink developers, We're running some new DataStream jobs on Flink 1.7.0 using the shaded Hadoop S3 file system, and running into frequent errors

Re: Backpressure metrics as JMX Bean

No this is not possible. On 06.12.2018 16:04, sayat wrote: Dear Flink community, Does anyone know if it is possible to expose Flink BackPressure number via JMX MBean? The one that shows in Flink UI?

Re: flink-s3-fs-presto:1.7.0 is missing shaded com/facebook/presto/hadoop

I've filed a JIRA: https://issues.apache.org/jira/browse/FLINK-11085 On 06.12.2018 14:03, Sergei Poganshev wrote: When I try to configure checkpointing using Presto in 1.7.0 the following exception occurs: java.lang.NoClassDefFoundError:

Re: Check-pointing error

Would it be possible for you to try this with 1.6-SNAPSHOT? This issue may have been fixed with https://issues.apache.org/jira/browse/FLINK-10839. On 29.11.2018 12:11, Felipe Quirce wrote: Hi I'm using the flink 1.6.2, and full stack trace is java.io.IOException: Exception while applying

Re: Checkpointing to gcs taking too long

Please provide the full Exception stack trace and the configuration of your job (parallelism, number of stateful operators). Have you tried using the gcs-connector in isolation? This may not be an issue with Flink. On 28.11.2018 07:01, prakhar_mathur wrote: I am trying to run flink on

Re: SQL Query named operator exceeds 80 characters

This is a safeguard in the metric system to prevent extremely long names (as these could cause the reporting to fail); so long as the prefix is unique you can safely ignore this warning. On 29.11.2018 10:40, Timo Walther wrote: Unfortunetely, renaming of operators is not supported so far. We

Re: Check-pointing error

Please provide the full exception stack trace and version of Flink that you are using. On 29.11.2018 11:55, Felipe Quirce wrote: Hi, I have found a problem during the checkpoint. Could anyone help me or help me to debug it? Exception: 1804 2018-11-29 11:31:00,448 INFO

Re: Flink - Metric are not reported

Please enable WARN logging and check for warnings by the SLF4JReporter and/or MetricQueryService. On 27.11.2018 17:00, bastien dine wrote: Hello everyone, Once again I require your help ! I am trying to report custom metric (see my code below) Yet, I do not see them anywhere.. nor in the

Re: Could not find previous entry with key.

imeServiceManager.java:108) at org.apache.flink.streaming.api.operators.AbstractStreamOperator.processWatermark(AbstractStreamOperator.java:734) at org.apache.flink.streaming.runtime.io.StreamInputProcessor$ForwardingValveOutputHandler.handleWatermark(StreamInputProcessor.java:262)

Re: Counter Implementation in Flink

The default implementation is the SimpleCounter, however I believe that the implementation isn't of interest compared to how it is used. Please expand on the graph ( which metric is represented by what color ) and include all of your code that interacts with the your counter. On 24.11.2018

Re: Tentative release date for 1.6.3

The 1.7.0 release is currently ongoing and expected to be finished within the week. On 26.11.2018 18:36, Vishal Santoshi wrote: +1. Do not see 1.7 release details anywhere. On Mon, Nov 26, 2018, 8:02 AM galantaa wrote: Hi vino, We want to upgrade flink

Re: Flink - Metric are not reported

ddedMetric to report ?) -- Bastien DINE Data Architect / Software Engineer / Sysadmin bastiendine.io <http://bastiendine.io> Le mer. 28 nov. 2018 à 15:54, Chesnay Schepler <mailto:ches...@apache.org>> a écrit : How quick does job batch terminate? Metrics are unre

Re: Flink - Metric are not reported

uous) Regards, Bastien -- Bastien DINE Data Architect / Software Engineer / Sysadmin bastiendine.io <http://bastiendine.io> Le mar. 27 nov. 2018 à 17:07, Chesnay Schepler <mailto:ches...@apache.org>> a écrit : Please enable WARN logging and check for warnings by th

Re: flink hadoop 3 integration plan

We certainly want to look into hadoop3 support for 1.8, but we'll have to take a look at the changes to hadoop2 first before we can give any definitive answer. On 28.11.2018 07:41, Ming Zhang wrote: Hi All， now we plan to move CDH6 which is based on hadoop3, anyone knows the plan of flink

Re: Custom scheduler in Flink

There's no /reasonable /way to implement a custom Scheduler, i.e., something where can just plug in your scheduler in a nice way. For this you'll have to directly modify the source of Flink. The work in https://issues.apache.org/jira/browse/FLINK-8886 may also be of interest, but is still in

Re: The way to write a UDF with generic type

I believe you have to extend "org.apache.flink.table.functions.AggregateFunction" instead for it to work with SQL (or more generally speaking, any sub-class of "org.apache.flink.table.functions.UserDefinedFunction". On 04.01.2019 05:18, yinhua.dai wrote: Hi Community, I tried to write a UDF

Re: Can't list logs or stdout through web console on Flink 1.7 Kubernetes

@Steven: Do you happen do know whether a JIRA exists for this issue? @Joshua: Does this also happen if you use log4j? On 26.12.2018 11:33, Joshua Fan wrote: wow, I met similar situation using flink 1.7 on yarn. there was no jobmanager.log on the node but jobmanager.out and jobmanager.error,

Re: Access Flink configuration in user functions

The configuration is not accessible to user-functions or the main method. The could either override ConfigurableStatebackend#configure, or configure the statebackend globally (see

Re: Counter Metrics not getting removed from Flink GUI after close()

What you're trying to do is not possible. Even if you close the group /it still exists/, and is returned by subsequent calls to addGroup("mygroup"). However since it is closed all registration calls will be ignored, hence why the value isn't updating. You can only update a metric by storing a

Re: Counter Metrics not getting removed from Flink GUI after close()

Metrics for a given job will be available in the GUI until the Job has finished. On 08.01.2019 17:08, Gaurav Luthra wrote: Hi, I am using ProcessWindowFunction, and in process() function I am adding user scoped Group as mentioned below. MetricGroup myMetricGroup=

Re: Is there an example of flink cluster "as a job" deployment on k8s ?

I believe we're targeting late February. On 03.01.2019 20:10, Vishal Santoshi wrote: This is interesting https://issues.apache.org/jira/browse/FLINK-9953. When would 1.8 be out ? On Wed, Dec 12, 2018 at 12:45 PM Vishal Santoshi mailto:vishal.santo...@gmail.com>> wrote: thanks On

Re: getting an error when configuring state backend to hdfs

Are you including the filesystems in your jar? Filesystem jars must be placed in the /lib directory of the flink distribution. On 19.12.2018 15:03, Avi Levi wrote: Hi, I am trying to set the backend state to hdfs /val stateUri = "hdfs/path_to_dir"/ /val backend: RocksDBStateBackend = new

Re: getting an error when configuring state backend to hdfs

flink-hadoop-fs should be in /lib On 19.12.2018 16:44, Avi Levi wrote: Hi Chesnay, What do you mean? I am creating a fat jar with all dependencies (using sbt assembly). which jar I should place in the /lib directory ? On Wed, Dec 19, 2018 at 4:44 PM Chesnay Schepler <mailto:c

Re: How can I visualize the metrics Counter and Meter at VisualVM?

Did you explicitly connector to the specific host/port, or just to a local process? (If you specify a port, you must connect to the specific port) On 29/03/2019 15:25, Felipe Gutierrez wrote: Hi, I have configured JMXReporter on Flink. I have a RichMapFunction class emitting a counter and a

Re: InvalidProgramException when trying to sort a group within a dataset

Your user-defined functions are referencing the class "com.iri.aa.etl.rgm.service.ThresholdAcvCalcServiceImpl" which isn't serializable. My guess is that "ThresholdAcvFact" is a non-static inner class, however I would need to see the entire class to give an accurate analysis. On 02/04/2019

Re: BucketAssigner - Confusion

BucketID is a variable type, and conceptually you can use any type so long as you can provide a serializer for it (BucketAssigner#getSerializer). The documentation is wrong in this instance. The convenience Flink APIs (StreamingFileSink#forRowFormat/StreamingFileSink#forBulkFormat) default

[DISCUSS] Drop Elasticssearch 1 connector

Hello everyone, I'm proposing to remove the connector for elasticsearch 1. The connector is used significantly less than more recent versions (2&5 are downloaded 4-5x more), and hasn't seen any development for over a hear, yet still incurred maintenance overhead due to licensing and testing.

Re: Flink - Type Erasure Exception trying to use Tuple6 instead of Tuple

> I tried using [ keyBy(KeySelector, TypeInformation) ] What was the result of this approach? On 03/04/2019 17:36, Vijay Balakrishnan wrote: Hi Tim, Thanks for your reply. I am not seeing an option to specify a .returns(new TypeHintString,String,String,String,String>>(){}) with KeyedStream

Re: print() method does not always print on the taskmanager.out file

This kind of sounds like a Outputstream flushing issue. Try calling "System.out.flush()" now and then in your sink and report back. On 04/04/2019 18:04, Felipe Gutierrez wrote: Hello, I am studying the parallelism of tasks on DataStream. So, I have configured Flink to execute on my machine

Re: How to submit Flink program to Yarn without upload the fat jar?

Which Flink version are you using? The DISABLED value has not been working since 1.5, so you may be stuck with uploading the app jar every time. On 04/04/2019 11:35, 徐涛 wrote: Hi Experts, When submitting a Flink program to Yarn, the app jar( a fat jar about 200M with Flink

Re: okio and okhttp not shaded in the Flink Uber Jar on EMR

Short-term I'd try relocating the okio/okhttp dependencies in your jar. I'm not too keen on adding more relocations to the hadoop jar; I can't gauge the possible side-effects. On 27.02.2019 14:54, Austin Cawley-Edwards wrote: Following up to add more info, I am building my app with maven

<    1   2   3   4   5   6   7   8   9   10   >