Hi,
As we are aware, Currently we cannot use RichAggregateFunction in
aggregate() method upon windowed stream. So, To access the state in your
customAggregateFunction, you can implement it using a ProcessFuntion.
This issue is faced by many developers.
So, someone must have implemented or tried
Hi Dawid,
Thanks for your time on this. The diff should have pointed out only the top
3 commits, but since it did not, it is possible I did not rebase my branch
against 1.4.2 correctly. I'll check this out and get back to you if I hit
the same issue again.
Thanks again,
Shailesh
On Thu, Sep 27,
Hi.
We have created our own database source that pools the data with a configured
interval. We then use a co processed function. It takes to input one from our
database and one from our data input. I require that you keyby with the
attributes you use lookup in your map function.
To delay
Hi,
I saw Apache Flink User Mailing List archive. - static/dynamic lookups in flink
streaming being discussed, and then I saw this FLIP
https://cwiki.apache.org/confluence/display/FLINK/FLIP-17+Side+Inputs+for+DataStream+API.
I know we havent made much progress on this topic. I still wanted to
Hi everyone,
Today I get this error again in another job, and I find some logs indicating
it’s probably related to the delegation token.
```
2018-09-28 09:49:09,668 INFO org.apache.flink.yarn.YarnResourceManager
- Closing TaskExecutor connection
Aah got it.
https://ci.apache.org/projects/flink/flink-docs-stable/monitoring/debugging_classloading.html
On Thu, Sep 27, 2018 at 11:04 AM Vishal Santoshi
wrote:
> Makes sense. An additional query.. How does flink handle class loading.
> Is there a separate class loader per job ? In essence
Hi Julio,
If you are using a HA mode depending on other services like ZooKeeper, you
can also check whether that service is OK when the JM lost leadership.
In our experience, network partitioning of JM to ZK and ZK object exceeding
max size limit can also lead to JM leadership lost.
vino yang
Hi Hequn,
If limit n is not supported in streaming, how to solve top n problem in
stream scenario?
Best
Henry
> 在 2018年9月28日，上午12:03，Hequn Cheng 写道：
>
> Hi Henry,
>
> Currently, Order By is supported in Streaming while Limit is only
> supported in Batch. Another thing to be noted
Hi Julio,
Which version of Flink are you using? If it is 1.5+, then you can try to
increase the heartbeat timeout by configuring it[1].
In addition, the possible cause is that the load of tm is too heavy, for
example, because the Full GC causes JVM stalls,
or deadlocks and other issues may cause
Hi Julio,
If the single JobManager lost temporarily and reconnected later, it could
be regranted leadership. And if you use Flink on Yarn, the Yarn RM
(according to configuration) would start a new ApplicationMaster to act as
a take-over JobManager.
Best,
tison.
Julio Biason 于2018年9月28日周五
Hi,
We are new to Flink and would like to get some ideas of achieving parallelism
and for maintaining state in Flink for few interesting problems.
We receive files that contain multiple types of events. We need to process them
in the following way:
1. Multiple event sources send event files to
Yun,
> Then I would share some experience about tuning RocksDB performance. Since
> you did not cache index and filter in block cache, it's no worry about the
> competition between data blocks and index blocks[1]. And to improve
> the read performance, you should increase your block cache size
Hey guys,
I'm seeing a weird error happening here: We have our JobManager configured
in HA mode, but with a single JobManager in the cluster (the second one was
in another machine that start showing flaky network, so we removed it).
Everything is running in Standalone mode.
Sometimes, the jobs
Hi again,
We managed at the end to get data into Kinesalite using the
FlinkKinesisProducer, but to do so, we had to use different configuration,
such as ignoring the 'aws.endpoint' setting and going for the ones that the
Kinesis configuration will expect. So, to our FlinkKinesisProducer we pass
Hi Henry,
Currently, Order By is supported in Streaming while Limit is only
supported in Batch. Another thing to be noted is, for Order by, the result
of streaming queries must be primarily sorted on an ascending time
attribute[1].
[1]
Makes sense. An additional query.. How does flink handle class loading. Is
there a separate class loader per job ? In essence if I have a static
member in a class in a job, it would be highly inapprpraite that that
static member is available to another job.
On Thu, Sep 27, 2018, 8:15 AM Kostas
Hi Averell,
> On Sep 27, 2018, at 3:09 PM, Averell wrote:
>
> Hi Kostas,
>
> Yes, I want them as metrics, as they are purely for monitoring purpose.
> There's no need of fault tolerance.
>
> If I use side-output, for example for that metric no.1, I would need a
> tumbling AllWindowFunction,
Hi Kostas,
Yes, I want them as metrics, as they are purely for monitoring purpose.
There's no need of fault tolerance.
If I use side-output, for example for that metric no.1, I would need a
tumbling AllWindowFunction, which, as I understand, would introduce some
delay to both the normal
Hi,
I want a top n result on each hop window result, but some error throws
out when I add the order by sentence or the limit sentence, so how do I
implement such case ?
Thanks a lot.
SELECT
trackId as id,track_title as description, count(*) as cnt
FROM
play
WHERE
Hi Kostas,
Sorry, I forget that. I'm using Flink 1.5.3.
Best,
Paul Lam
Kostas Kloudas 于2018年9月27日周四 下午8:22写道：
> Hi Paul,
>
> I am also cc’ing Till and Gary who may be able to help, but to give them
> more information,
> it would help if you told us which Flink version you are using.
>
>
Hi Paul,
I am also cc’ing Till and Gary who may be able to help, but to give them more
information,
it would help if you told us which Flink version you are using.
Cheers,
Kostas
> On Sep 27, 2018, at 1:24 PM, Paul Lam wrote:
>
> Hi,
>
> One of my Flink on YARN jobs got into a weird
Hi Vishal,
Currently there is no way to share (user-defined) resources between tasks on
the same TM.
So I suppose that a singleton is the best way to go for now.
Cheers,
Kostas
> On Sep 27, 2018, at 3:43 AM, Hequn Cheng wrote:
>
> Hi vishal,
>
> Yes, we can define a static connection to
Hi Averell,
From what I understand for your use case, it is possible to do what you want
with Flink.
If you are implementing a function, then you have access to the metric system
through
the runtime context (see [1] for more information).
Some things to take into consideration:
1) Metrics
Please see
https://ci.apache.org/projects/flink/flink-docs-release-1.5/ops/config.html#security-ssl-algorithms
for the SSL algorithms that are available by default for 1.5 .
On 27.09.2018 13:24, V N, Suchithra (Nokia - IN/Bangalore) wrote:
Gentle reminder on this question.
*From:*V N,
So there's 2 issues here:
1) your reporter configuration is wrong, configuration values for a
specific reporter are prefixed with "metrics.reporter", not
"metrics.reporters" (note the "s"). See below for a correct config.
metrics.reporters: varuy
metrics.reporter.varuy.host: tracing115
Hi,
One of my Flink on YARN jobs got into a weird situation after a fail-fast
restart. The restart was triggered by loss of a TaskManager, but when the job
was recovering, one of its subtask (1/24) couldn’t be deployed, and finally
failed with NoResourceAvailableException.
Looking into the
Gentle reminder on this question.
From: V N, Suchithra (Nokia - IN/Bangalore)
Sent: Monday, September 24, 2018 3:56 PM
To: user@flink.apache.org
Subject: Information required regarding SSL algorithms for Flink 1.5.x
Hello,
We have a query regarding SSL algorithms available for Flink versions.
Hi,
I recently tried to update a flink job from 1.3.2 to 1.6.0. It deploys
successfully as usual, but logs the following exception shortly after
starting:
Caused by: org.apache.avro.AvroRuntimeException:
avro.shaded.com.google.common.util.concurrent.UncheckedExecutionException:
Hi Shailesh,
I am afraid it is gonna be hard to help you, as this branch differs
significantly from 1.4.2 release (I've done diff across your branch and
tag/release-1.4.2). Moreover the code in the branch you've provided
still does not correspond to the lines in the exception you've posted
Good day everyone,
I have a stream with two timestamps (ts1 and ts2) inside each record. My
event time is ts1. This ts1 has value truncated to a quarter (like 23:30,
23:45, 00:00,...)
I want to report two metrics:
1. A meter which counts number of records per value of ts1. (fig.1)
Hi Dawid,
Yes, it is version 1.4.2. We are running vanilla flink, but have added a
couple of changes in the CEP operator specifically (top 3 commits here:
https://github.com/jainshailesh/flink/commits/poc_on_1.4.2). Changes I've
made to CEP operators do not touch the checkpointing path, just
Hi Till,
I have copy the flink-metrics-graphite-1.5.2.jar to lib/, and restart
flink.It seems make no difference.
When I checked the jobmanager log, the metrics configured in
flink-conf.yaml, could be load since it printed as below:
"2018-09-27 09:50:23,953 INFO
32 matches
Mail list logo
