Re: Running Flink on java 11

In regards to what we test: We run our tests against Java 8 *and *Java 11, with the compilation and testing being done with the same JDK. In other words, we don't check whether Flink compiled with JDK 8 runs on JDK 11, but we currently have no reason to believe that there is a problem (and som

Re: Elasticsink sometimes gives NoClassDefFoundError

Could you clarify under what circumstances you see this issue? You say "sometimes"; is the job running normally and then failing due to this error? Does it happen when submitting/canceling/restarting a job etc. On 08/01/2020 12:20, Jayant Ameta wrote: Hi, I see the following error sometimes on

Re: Flink task node shut it self off.

The logs show 2 interesting pieces of information: ... 2019-12-19 18:33:23,278 INFO org.apache.kafka.clients.FetchSessionHandler  - [Consumer clientId=consumer-4, groupId=ccdb-prod-import] Error sending fetch request (sessionId=INVALID, epoch=INITIAL) to node 0: org.apach

Re: Change Akka Ask Timeout for Job Submission Only

There are 3 communication layers involved here: 1) client <=> server (REST API) This goes through REST and does not use timeouts AFAIK. We wait until a response comes or the connection terminates. 2) server (REST API) <=> processes (JM, Dispatcher) This goes through akka, with "web.timeout"

Re: kafka: how to stop consumption temporarily

Are you asking how to detect from within the job whether the dump is complete, or how to combine these 2 jobs? If you had a way to notice whether the dump is complete, then I would suggest to create a custom source that wraps 2 kafka sources, and switch between them at will based on your condi

Re: Migrate custom partitioner from Flink 1.7 to Flink 1.9

You should be able to implement this on the DataStream API level using DataStream#broadcast and #union like this: input = ... singleChannel = input.filter(x -> !x.isBroadCastPartitioning); broadcastChannel = input.filter(x -> x.isBroadCastPartitioning); result = broadcastChannel.broadcast().u

Re: Yarn Kerberos issue

From what I understand from the documentation, if you want to use delegation tokens you always first have to issue a ticket using kinit; so you did everything correctly? On 02/01/2020 13:00, Juan Gentile wrote: Hello, Im trying to submit a job (batch worcount) to a Yarn cluster. I’m trying

Re: Flink Prometheus metric doubt

In practice the documentation is incorrect. While technically the metric _would_ emit -1 if the job is in a failed/finished state, the reality is that at this point the metric is unregistered and no longer updated, since the owning component (the jobmanager) is shutting down. I can't think of

Re: Flink 'Job Cluster' mode Ui Access

al release. Regards, Chesnay On 13/12/2019 09:58, Jatin Banger wrote: Sure, here it is. Job Manager Logs with logging level as DEBUG On Wed, Dec 11, 2019 at 3:14 PM Chesnay Schepler <mailto:ches...@apache.org>> wrote: Would it be possible for you to provide us with full debug log

Re: Sample Code for querying Flink's default metrics

Yes, when a cluster was started it takes a few seconds for (any) metrics to be available. On 12/12/2019 11:36, Pankaj Chand wrote: Hi Vino, Thank you for the links regarding backpressure! I am currently using code to get metrics by calling REST API via curl. However, many times the REST API

Re: Flink 'Job Cluster' mode Ui Access

Can i raise this as bug ? Best Regards, Jatin On Thu, Dec 5, 2019 at 8:59 PM Chesnay Schepler mailto:ches...@apache.org>> wrote: Ok, it's good to know that the WebUI files are there. Please enable DEBUG

Re: Change Flink binding address in local mode

At this point I would suggest to file a ticket; these are the options that _should_ control the behavior but apparently aren't in all cases. On 08/12/2019 12:23, Andrea Cardaci wrote: Hi, Flink (or some of its services) listens on three random TCP ports during the local[1] execution, e.g., 399

Re: Flink 1.9.1 allocating more containers than needed

I would expect January. With 1.8.3 release being underway, 1.10 feature freeze coming close and, of course, Christmas, it seems unlikely that we'll manage to pump out another bugfix release in December. On 06/12/2019 15:18, eSKa wrote: Thank you for quick reply. Will wait for 1.9.2 then. I bel

Re: Flink 1.9.1 allocating more containers than needed

Note that FLINK-10848 is included in 1.9.X, but it didn't fix the issue completely. On 06/12/2019 15:10, Chesnay Schepler wrote: There are some follow-up issues that are fixed for 1.9.2; release date for that is TBD. https://issues.apache.org/jira/browse/FLINK-12342 https://issues.apach

Re: Flink 1.9.1 allocating more containers than needed

There are some follow-up issues that are fixed for 1.9.2; release date for that is TBD. https://issues.apache.org/jira/browse/FLINK-12342 https://issues.apache.org/jira/browse/FLINK-13184 On 06/12/2019 15:08, eSKa wrote: Hello, recently we have upgraded our environment to from 1.6.4 to 1.9.1.

Re: Flink 'Job Cluster' mode Ui Access

s actually being requested. On 05/12/2019 05:57, Jatin Banger wrote: I have tried that already using '$FLINK_HOME/bin/jobmanager.sh" start-foreground Ui comes fine with this one. Which means web/index.html is present. On Wed, Dec 4, 2019 at 9:01 PM Chesnay Schepler <mailto:che

Re: Flink 'Job Cluster' mode Ui Access

eded by the job. Best regards, Jatin On Fri, Nov 29, 2019 at 4:18 PM Chesnay Schepler <mailto:ches...@apache.org>> wrote: To clarify, you ran "mvn package -pl flink-dist -am" to build Fink? If so, could you run that again and provide us with the maven output?

Re: Building with Hadoop 3

There's no JIRA and no one actively working on it. I'm not aware of any investigations on the matter; hence the first step would be to just try it out. A flink-shaded artifact isn't a hard requirement; Flink will work with any 2.X hadoop distribution (provided that there aren't any dependency

Re: Read multiline JSON/XML

I know that at least the Table API can read json, but I don't know how well this translates into other APIs. On 29/11/2019 12:09, Flavio Pompermaier wrote: Hi to all, is there any out-of-the-box opt

Re: Read multiline JSON/XML

Why vino? He's specifically asking whether Flink offers something _like_ spark. On 29/11/2019 14:39, vino yang wrote: Hi Flavio, IMO, it would take more effect to ask this question in the Spark user mailing list. WDYT? Best, Vino Flavio Pompermaier > 于2019年11月

Re: Apache Flink - High Available K8 Setup Wrongly Marked Dead TaskManager

art cause not enough task manager on cluster. Thanks Chesnay Schepler mailto:ches...@apache.org>>, 28 Kas 2019 Per, 18:55 tarihinde şunu yazdı: The akka.watch configuration options haven't been used for a while irrespective of FLINK-13883 (but I can't quite tell atm sin

Re: Flink 'Job Cluster' mode Ui Access

ng procedure for the deployment: https://github.com/apache/flink/blob/master/flink-container/docker/README.md And i tried accessing the path you mentioned: # curl :4081/#/overview {"errors":["Not found."]} Best Regards, Jatin On Thu, Nov 28, 2019 at 10:21 PM Chesnay Scheple

Re: ProcessFunction collect and close, when to use?

1) You should never call close() on the collector; Flink will do that automatically. 2) No, it shouldn't block anything. Flink will look at the next record to process, notice it's a barrier and pass it on immediately. On 29/11/2019 05:29, shuwen zhou wrote: Hi Community, In ProcessFunction cla

Re: Flink 'Job Cluster' mode Ui Access

Could you try accessing :/#/overview ? The REST API is obviously accessible, and hence the WebUI should be too. How did you setup the session cluster? Are you using some custom Flink build or something, which potentially excluded flink-runtime-web from the classpath? On 28/11/2019 10:02, Jat

Re: Apache Flink - High Available K8 Setup Wrongly Marked Dead TaskManager

The akka.watch configuration options haven't been used for a while irrespective of FLINK-13883 (but I can't quite tell atm since when). Let's start with what version of Flink you are using, and what the taskmanager/jobmanager logs say. On 25/11/2019 12:05, Eray Arslan wrote: Hi, I have some

Re: ***UNCHECKED*** Error while confirming Checkpoint

This looks to me like the TwoPhaseCommitSinkFunction is a bit too strict. The notification for complete checkpoints is not reliable; it may be late, not come at all, possibly even in different order than expected. As such, if you a simple case of snapshot -> snapshot -> notify -> notify the s

Re: Completed job wasn't saved to archive

ations (like checkpoints saving) work fine. But I don't see archived job or message about archiving failure at all. It looks like Flink just didn't try to save job to archive. 21.11.2019, 17:17, "Chesnay Schepler" : If the archiving fails there should be some log mess

Re: Completed job wasn't saved to archive

If the archiving fails there should be some log message, like "Failed to archive job" or "Could not archive completed job..." . If nothing of the sort is logged my first instinct would be that the operation is being slowed down, _a lot_. Where are you archiving them to? Could it be the write op

Re: IOException when using Prometheus Monitor

From what I found so far this appears to be an incompatibility between the pushgateway and client version. So you can either a) use an older version of the pushgateway b) bump the version of the prometheus reporter. Unfortunately I cannot tell you which version you would need. On 20/11/2019 10

Re: How can I get the backpressure signals inside my function or operator?

I don't think there is a truly sane way to do this. I could envision a separate application triggering samples via the REST API, writing the results into kafka which your operator can read. This is probably the most reasonable solution I can come up with. Any attempt at accessing the TaskExec

Re: Sending custom statsd tags

Not possible, you'll have to extend the StatsDReporter yourself to add arbitrary tags. On 30/10/2019 12:52, Prakhar Mathur wrote: Hi, We are running Flink 1.6.2. We are using flink-metrics-statsd jar in order to send metrics to telegraf. In order to send custom metrics, we are using MetricGr

Re: Can flink 1.9.1 use flink-shaded 2.8.3-1.8.2

, otherwise it may make user confused. Chesnay Schepler mailto:ches...@apache.org>> 于2019年10月25日周五 下午4:21写道： If you need hadoop, but the approach outlined here <https://ci.apache.org/projects/flink/flink-docs-release-1.9/dev/projectsetup/dependencies.html> doesn't work

Re: Can flink 1.9.1 use flink-shaded 2.8.3-1.8.2

If you need hadoop, but the approach outlined here doesn't work for you, then you still need a flink-shaded-hadoop-jar that you can download here

Re: flink 1.9

We will not release Flink version bundling Hadoop. The versioning for flink-shaded-hadoop-uber is entirely decoupled from Flink version. You can just use the flink-shaded-hadoop-uber jar linked on the downloads page with any Flink version. On 18/10/2019 13:25, GezimSejdiu wrote: Hi Flink com

Re: Fwd: Is it possible to get Flink job name in an operator?

If you have control over the job you can modify it to use ExEnv#execute(String jobName), and üass this explicitly to your functions in some form (like the global job parameters). Beyond that there is no way to access the job name from within a function/operator. On 15/10/2019 08:53, 马阳阳 wrot

Re: flink 1.9

Java 11 support will be part of Flink 1.10 (FLINK-10725). You can take the current master and compile&run it on Java 11. We have not investigated later Java versions yet. On 09/10/2019 14:14, Vishal Santoshi wrote: Thank you. A related question, has flink been tested with jdk11 or above. ? O

Re: Hiding the user Configurations from the flink dashboard

No, parameters stored in the global job parameters cannot be hidden. Only options configured in flink-conf.yaml are hidden, iff their key contains "password" or "secret". On 09/10/2019 08:26, vivekanand yaram wrote: Hello All, I m just wondering , is there a way to hide the user configuratio

Re: [QUESTION] How to parallelize with explicit punctuation in Flink?

In other words, you need a way to partition the stream such that a series of items followed by a barrier are never interrupted. I'm wondering whether you could just apply DataStream#partitionCustom to your source: public static class BarrierPartitionerimplements Partitioner { private int

Re: Computing two aggregate functions on the same window

There doesn't seem to be a built-in way to apply multiple aggregations to a window. You could use an aggregate function that combines other aggregate functions, but admittedly this will get unwieldy as the number of functions increase: public static class MultiAggregateFunction, ACC2, OUT2,

Re: Passing parameters to filter function (in DataStreams)

You can compute the threshold ahead of time and reference it directly in the filter function. (Below are 2 examples, depending on whether you like lambdas or not) final int threshold =computeThreshold(); temperatureStream.filter(new FilterFunction() { @Override public boolean filter(Integer

Re: Implementing CheckpointableInputFormat

You have to use StreamExecutionEnvironment#createFileInput for implementing CheckpointableInputFormat to have any effect. This internally results in it being used by the MonitoringFileSource. If you use StreamExecutionEnvironment#createInput nothing will be checkpointed for the source; and yes t

Re: Building Flink 1.6.4 fails with "object scala in compiler mirror not found"

The only cause I know for errors such as this are corrupt jars. Try cleaning the maven cache and see if the issue persists. On 07/10/2019 17:36, Aikio, Torste wrote: Hi, I'm trying to build Flink 1.6.4 from source and some of the tests for flink-scala module are failing for me. Are there some

Re: POJO serialization vs immutability

one might correct me if I'm wrong). Jan [1] http://web.mit.edu/6.031/www/sp17/classes/15-equality/ On 10/7/19 1:37 PM, Chesnay Schepler wrote: This question should only be relevant for cases where POJOs are used as keys, in which case they /must not/ return a class-constant nor effectiv

Re: POJO serialization vs immutability

This question should only be relevant for cases where POJOs are used as keys, in which case they /must not/ return a class-constant nor effectively-random value, as this would break the hash partitioning. This is somewhat alluded to in the keyBy() documentation

Re: Flink KPL based on a custom class object

You have to implement a SerializationSchema and pass that into the FlinkKinesisProducer. The error message you received is caused by the compiler attempting to determine the generic type of the producer, but not being able to do so since your myObject class does not implement the correct inter

Re: Difficult to debug reason for checkpoint decline

There does indeed appear to be a code path in the StreamTask where an exception might not be logger on the TaskExecutor. (StreamTask#handleExecutionException) In FLINK-10753 the CheckpointCoordinator was adjusted to log the full stacktrace, and is part of 1.5.6. On 07/10/2019 09:51, Daniel Ha

Re: kinesis consumer metrics user variables

or with boolean option to control it. On Mon, Oct 7, 2019 at 11:05 AM Chesnay Schepler <mailto:ches...@apache.org>> wrote: What exactly would you prefer? Without the stream name and shard id you'd end up with name clashes all over the place. Why can you not aggregate

Re: containThrowable missing in ExceptionUtils

The listed method no longer exists and was subsumed by ExceptionUtils#findThrowable, which also gives access to the Throwable if it could be found. I have filed FLINK-14334 for updating the documentation. On 02/10/2019 15:48, Nicholas Walton wrote: Hi, I’m trying to implement a failure handl

Re: kinesis consumer metrics user variables

What exactly would you prefer? Without the stream name and shard id you'd end up with name clashes all over the place. Why can you not aggregate them? Surely Datadog supports some way to define a wildcard when definying the tags to aggregate. On 03/10/2019 09:09, Yitzchak Lieberman wrote: H

Re: Questions about how to use State Processor API

1. Only the Java API is supported. 2. As far as I can tell you are correct, the given checkpoint path isn't really used. On 04/10/2019 10:39, Tony Wei wrote: Hi, I'm recently trying to use State Processor API, but I have some questions during the development. 1. Does `OperatorTransformati

Re: HBase Connectors(sink)

Where did you see an HBase sink? On the current master, and all releases that I can remember, flink-hbase only contains input formats / sources. On 06/09/2019 03:23, Ni Yanchun wrote: Hi all, I have found that flink could use hbase as sink in flink source code, but it does not appear

Re: Build Flink against a vendor specific Hadoop version

Yes, please open a JIRA ticket. On 30/08/2019 18:46, Elise RAMÉ wrote: Thank you all ! Classpath option works for me and is easier so I’ll do this way. About flink-shaded and vendor-repos, would it be helpful if I describe this issue in a Jira ticket ? Le 30 août 2019 à 11:43, Chesnay

Re: Configure Prometheus Exporter

I’m trying to do. I think Flink by default exports all metrics. Is there anything else I can do to achieve this goal? On Aug 7, 2019, at 03:58, Chesnay Schepler <mailto:ches...@apache.org>> wrote: This is not possible. Are you trying to limit which metrics are exposed? On 07/08/2

Re: Configure Prometheus Exporter

This is not possible. Are you trying to limit which metrics are exposed? On 07/08/2019 06:52, Chaoran Yu wrote: Hello guys,    Does anyone know if the Prometheus metrics exported via the JMX reporter or the Prometheus reporter can be configured using a YAML file similar to this one

Re: Extending REST API with new endpoints

There's no built-in way to extend the REST API. You will have to create a fork and either extend the DIspatcherRestEndpoint (or parent classes), or implement another WebMonitorExtension and modify the DispatcherRestEndpoint to load that one as well. On 23/07/2019 15:51, Oytun Tez wrote: Ping,

Re: How to handle JDBC connections in a topology

Note that in order for the static class approach to work you have to ensure that the class is loaded by the parent classloader, either by placing the class in /lib or configuring `classloader.parent-first-patterns-additional` to pick up this particular class. On 24/07/2019 10:24, Haibo Sun wr

Re: Accessing variables build within a map/flatmap function within a DS

Note that this will only work when running the the application in the IDE; specifically it will not work when running on an actual cluster, since your function isn't executed on the same machine as your (presumably) main[] function. We can give you better advice if you tell us what exactly you

Re: Does Flink support raw generic types in a merged stream?

Have you looked at org.apache.flink.types.Either? If you'd wrap all elements in both streams before the union you should be able to join them properly. On 17/07/2019 14:18, John Tipper wrote: Hi All, Can I union/join 2 streams containing generic classes, where each stream has a different pa

Re: Converting Metrics from a Reporter to a Custom Events mapping

You can configure multiple reporters, so just configured both the reporter that the app users want and your own which does the transformation and sending to kinesis. On 16/07/2019 09:37, Haibo Sun wrote: Hi, Vijay Or can you implement a Reporter that transforms the metrics and sends them dir

Re: Apache Flink - Gauge implementation

This can't be implemented as there's no guarantee that getValue is only called once. Why do you want to reset the count? If you're interested in rates, why aren't you using a meter? On 10/07/2019 09:37, Xintong Song wrote: Hi Singh, Could your problem be solved by simply record the previous

Re: Apache Flink - How to find the number of window instances in an application

This is unfortunately not possible. On 04/07/2019 19:40, M Singh wrote: Hi: I wanted to find out if there is a metric to find out the the number of global or non-global window instances in a Flink application. Thanks Mans

Re: UnsupportedOperationException from org.apache.flink.shaded.asm6.org.objectweb.asm.ClassVisitor.visitNestHostExperimental using Java 11

Flink only supports Java 8. On 04/07/2019 15:34, Rauch, Jochen wrote: Hi all, I have implemented following code snippet with Apache Flink 1.8: flinkConfiguration.getEnvironment().readTextFile(outputFile.getAbsolutePath(), "ISO-8859-1") .flatMap(new FlatMapFunctionObject>, Integer>>() {  

Re: Apache Flink - Are counters reliable and accurate ?

e-run your test. On 27/06/2019 22:41, M Singh wrote: Hi Chesnay: Thanks for your response. My job runs for a few minutes and i've tried setting the reporter interval to 1 second. I will try the counter on a longer running job. Thanks again. On Thursday, June 27, 2019, 11:46:17 AM ED

Re: Apache Flink - Are counters reliable and accurate ?

1) None that I'm aware of. 2) You should use counters. 3) No, counters are not checkpointed, but you could store the value in state yourself. 4) None that I'm aware of that doesn't require modifications to the application logic. How long does your job run for, and how do you access metrics? O

Re: Maybe a flink bug. Job keeps in FAILING state

-- From:Joshua Fan Send Time:2019年6月20日(星期四) 11:55 To:zhijiang Cc:user ; Till Rohrmann ; Chesnay Schepler Subject:Re: Maybe a flink bug. Job keeps in FAILING state zhijiang I did not capture the job ui, the

Re: Apache Flink - Question about metric registry and reporter and context information

You cannot access context information in the metric reporter itself /in a nice way/. You can wait for the first metric to be registered, and then extract arbitrary metric variables from the metric group. On 15/06/2019 19:31, Yun Tang wrote: Hi 1) Yes, the metrics reporter was instantiated per

Re: [DISCUSS] Deprecate previous Python APIs

I have created a JIRA and PR for removing the Python APIs. On 11/06/2019 15:30, Stephan Ewen wrote: Hi all! I would suggest to deprecating the existing python APIs for DataSet and DataStream API w

Re: Updated Flink Documentation and tutorials

ly tied to that single job, much like Hadoop's Application Master for each job. Please let me know which one is correct. Pankaj On Thu, Jun 20, 2019, 4:54 AM Chesnay Schepler <mailto:ches...@apache.org>> wrote: What makes you believe that t

Re: Updated Flink Documentation and tutorials

What makes you believe that they are out-dated? On 19/06/2019 19:17, Pankaj Chand wrote: Hello, Please let me know how to get the updated documentation and tutorials of Apache Flink. The stable v1.8 and v1.9-snapshot release of the documentation seems to be outdated. Thanks! Pankaj

Re: [External] checkpoint metadata not in configured directory state.checkpoints.dir

The _metadata is always stored in the same directory as the checkpoint data. As outlined here "state.checkpoints.dir" serves as a cluster-wide configuration that _can_ be overwritten with a

Re: Simple stateful polling source

een a RichParallelSourceFunction and a RichSourceFunction. Is this last (RichSourceFunction) implicitly using parallelism = 1? On Wed, Jun 19, 2019 at 2:25 PM Chesnay Schepler <mailto:ches...@apache.org>> wrote: It returns a list of states so that state can be re-distributed if t

Re: Simple stateful polling source

n one entry for redistributable unit of state" but this is not very clear to me.. Best, Flavio On Wed, Jun 19, 2019 at 12:40 PM Chesnay Schepler <mailto:ches...@apache.org>> wrote: This looks fine to me. What exactly were you worried about? On 19/06/2019 12:33, Flavio Pompe

Re: Simple stateful polling source

This looks fine to me. What exactly were you worried about? On 19/06/2019 12:33, Flavio Pompermaier wrote: Hi to all, in my use case I have to ingest data from a rest service, where I periodically poll the data (of course a queue would be a better choice but this doesn't depend on me). So I

Re: Side output in ProcessFunction.onTimer

ProcessFunction#onTimer provides an OnTimerContext parameter which allows you to use side-outputs. On 18/06/2019 17:41, Frank Wilson wrote: Hi, Is there a way to make side outputs in an onTimer callback in ProcessFunction? I want to side output events that belong to a session that was below

Re: Maybe a flink bug. Job keeps in FAILING state

@Till have you see something like this before? Despite all source tasks reaching a terminal state on a TM (FAILED) it does not send updates to the JM for all of them, but only a single one. On 18/06/2019 12:14, Joshua Fan wrote: Hi All, There is a topology of 3 operator, such as, source, parse

Re: Flink sent/received bytes related metrics: how are they calculated?

:08 Chesnay Schepler mailto:ches...@apache.org>> ha scritto: Which version of Flink are you using? There were some issues at some point about double-counting. On 14/06/2019 09:49, Andrea Spina wrote: Dear Community, I'd like to ask for some details about bytes

Re: Flink sent/received bytes related metrics: how are they calculated?

Which version of Flink are you using? There were some issues at some point about double-counting. On 14/06/2019 09:49, Andrea Spina wrote: Dear Community, I'd like to ask for some details about bytes related metrics in Flink. Precisely, I'm looking at *bytes sent* and *bytes received *metrics:

Re: [DISCUSS] Deprecate previous Python APIs

I would just remove them. As you said, there are very limited as to what features they support, and haven't been under active development for several releases. Existing users (if there even are any) could continue to use older version against newer releases. It's is slightly more involved than

Re: Building Flink distribution with Scala2.12

https://ci.apache.org/projects/flink/flink-docs-master/flinkDev/building.html#scala-versions On 29/05/2019 15:29, Boris Lublinsky wrote: Thanks Martijn, this was it. It would be nice to have this in documentation. Boris Lublinsky FDP Architect boris.lublin...@lightbend.com

Re: Job recovers from an old dangling CheckPoint in case of Job Cluster based Flink pipeline

Can you provide us the jobmanager logs? After the first restart the JM should have started deleting older checkpoints as new ones were created. After the second restart the JM should have recovered all 10 checkpoints, start from the latest, and start pruning old ones as new ones were created.

Re: Minicluster missing in scala runtime in version 1.8 source code

wrote: MiniCluster is there in java code but not in Scala. Zeppelin flink interpreter looks for LocalFlinkMiniCluster class located under scala code org.apache.flink.runtime.minicluster.LocalFlinkMiniCluster *From:*Chesnay Schepler mailto:ches...@apache.org>> *Sent:* Monday, 27 May 2019 6:58 PM

Re: Minicluster missing in scala runtime in version 1.8 source code

It's still there: https://github.com/apache/flink/tree/release-1.8/flink-runtime/src/main/java/org/apache/flink/runtime/minicluster Please provide more details on what you believe to be missing. On 27/05/2019 01:02, Ravi Pullareddy wrote: Hi There I notice mini cluster folder missing under s

Re: [Discuss] Add JobListener (hook) in flink job lifecycle

This issue is another case where we have problems figuring out the boundaries and responsibilities between the ExecutionEnvironments and the ClusterClient. I believe we should figure this out first, and decide whether the ClusterClient (or anything based on it) should be made public to accomo

Re: Flink not giving full reason as to why job submission failed

Please open a new JIRA. FLINK-11902 modified the REST API to no longer hide the exception, but the WebUI isn't handling the error response properly (it only reads and displays part of it). On 20/05/2019 16:24, Wouter Zorgdrager wrote: Hi Harshith, This is indeed an issue not resolved in 1.8.

Re: [SURVEY] Usage of flink-ml and [DISCUSS] Delete flink-ml

I believe we can remove it regardless since users could just use the 1.8 version against future releases. Generally speaking, any library/connector that is no longer actively developed can be removed from the project as existing users can always rely on previous versions, which should continue

Re: Passing a custom SourceContext to a SourceFunction

You cannot control what kind of SourceContext is passed into your function. What are you trying to achieve? On 15/05/2019 09:30, Debasish Ghosh wrote: Hi - I have a custom SourceFunction .. class MySourceFunction[T](data: Seq[T]) extends SourceFunction[T] { def run(ctx: SourceContext[T]): U

Re: Flink with HBase

Unless I'm blind Flink does not provide an HBase sink, so it depends on the implementation you're using. On 15/05/2019 20:17, Nikhil Goyal wrote: Hi, Does flink support exactly once processing using HBase as sink? I am not able to find any documentation supporting the same. Thanks Nikhil

Re: User Interface not showing the actual count received and produced

Flink currently does not measure incoming records for sources and outgoing records for sink, see https://issues.apache.org/jira/browse/FLINK-7286. On 15/05/2019 19:43, PoolakkalMukkath, Shakir wrote: Hi Flink team, I am developing a flow which uses ·FlinkKafkaConsumer010 to consume message

Re: writeAsFormattedText sets only Unix/Linux line endings

The line-ending is hard-coded into the TextOutputFormat. You will have to either extend this class and override #writeRecord(), or convert your POJOs to a Tuple and use the CsvOutputFormat, which supports setting the line delimiter (called recordDelimiter). On 09/05/2019 08:32, Papadopoulos, K

Re: Unable to build flink from source

You're likely using Java9+, but 1.3.3 only supports Java 8 (and maybe still 7). On 06/05/2019 03:20, syed wrote: Hi I am trying to build flink 1.3.3 from source using IntelliJ IDEA Ultimate 2019.1 IDE. When I build the project, I am receiving the following error *java package sun.misc does not

Re: ClassNotFoundException on remote cluster

2 May 2019 at 17:32, Abhishek Jain <mailto:abhijai...@gmail.com>> wrote: This is a spring boot app that I've packaged using maven (Apache Maven 3.3.9). I've verified the class is present in the jar as well. On Thu, 2 May 2019 at 17:25, Chesnay Schepler mailto:ches...@a

Re: Can't build Flink for Scala 2.12

You can monitor https://issues.apache.org/jira/browse/FLINK-12392 for the compile issue. On 01/05/2019 22:05, Chesnay Schepler wrote: You are correct, that is a typo. Very well done for spotting it, will fix it right away. We can conclude that the current SNAPSHOT version does not build

Re: configuration of standalone cluster

OpenJDK 64-Bit Server VM (build 11.0.2+9-Ubuntu-3ubuntu118.04.3, mixed mode, sharing) swissbib@sb-ust1:~$ Is version 8 more appropriate? Günter On 02.05.19 13:48, Chesnay Schepler wrote: Which java version are you using? On 01/05/2019 21:31, Günter Hipler wrote: Hi, For the first time I&#

Re: ClassNotFoundException on remote cluster

How are you packaging the jar that you submit? Specifically, are you ensuring that all your classes are actually contained within? On 02/05/2019 13:38, Abhishek Jain wrote: Hi, I'm running into ClassNotFoundException only when I run my application on a standalone cluster (using flink cli). If

Re: Type Erasure - Usage of class Tuple as a type is not allowed. Use a concrete subclass (e.g. Tuple1, Tuple2, etc.

I'm not sure what you're asking. If you have a Deserialization schema that convert the data into a Map you're done as I understand it, what do you believe to be missing? If, for a given job, the number/types of fields are fixed you could look into using Row. On 01/05/2019 22:40, Vijay Balak

Re: configuration of standalone cluster

Which java version are you using? On 01/05/2019 21:31, Günter Hipler wrote: Hi, For the first time I'm trying to set up a standalone cluster. My current configuration 4 server (1 jobmanger and 3 taskmanager) a) starting the cluster swissbib@sb-ust1:/swissbib_index/apps/flink/bin$ ./start-clu

Re: Can't build Flink for Scala 2.12

You are correct, that is a typo. Very well done for spotting it, will fix it right away. We can conclude that the current SNAPSHOT version does not build with scala 2.12 at the moment; hopefully there aren't too many issues. On 01/05/2019 08:05, Visser, M.J.H. (Martijn) wrote: In the meant

Re: Job Startup Arguments

The passed job arguments can not be queried via the REST API. When submitting jobs through the CLI these parameters never arrive at the cluster; in case of REST API submission they are immediately discarded after the submission has finished. On 25/04/2019 12:25, Dawid Wysakowicz wrote: Hi Ste

Re: metric does not display on web

"No metrics available" indicates that either a) metrics have not been queried yet or b) metrics can not be transferred from the TaskManagers to the JobManager. Regarding the first option, how long have you waited for metrics to show up? It may take a bit for metrics to be available (around 10 se

Re: get custom gauge metric from WebMonitorEndpoint

You should be able to get the value of your custom metric. You'll have to set the log level to DEBUG and scan the logs for metric-related errors. On 23/04/2019 22:25, Georgi Stoyanov wrote: I've got custom metric -> || And I'm using them as suggested in the documentation -> || | |getRuntime

<    2   3   4   5   6   7   8   9   10   11   >