e array to get each of them, same for two integers, and nearly any
> other types).
>
> I feel there should be a more general way of doing this regardless of the
> fields on the class you're de/serializing.
>
> What do you do in these cases? It should be a pretty comm
String code;
public double price;
public String description;
public long created;
}
---
Regards,
Matt
[1] http://data-artisans.com/kafka-flink-a-practical-how-to/
/java/eu/bde/sc4pilot/fcd/FcdTaxiEvent.java#L60
I would like to see a more "generic" approach for the class Product in my
last message. I believe a more general purpose de/serializer for POJOs
should be possible to achieve using reflection.
On Wed, Dec 7, 2016 at 1:16 PM, Luigi Selmi wrote:
> Hi Ma
,
Matt
[1] https://github.com/Dromit/StreamTest/
[2]
https://github.com/Dromit/StreamTest/blob/master/src/main/java/com/stream/Serde.java
[3]
https://github.com/Dromit/StreamTest/blob/master/src/main/java/com/stream/MainProducer.java#L19
[4]
https://github.com/Dromit/StreamTest/blob/master/src/main/java
ou provide a working example of a fold function
with both a FoldFunction and a WindowFunction?
Regards,
Matt
[1]
https://ci.apache.org/projects/flink/flink-docs-master/dev/windows.html#windowfunction-with-incremental-aggregation
[2] https://gist.github.com/cc7ed5570e4ce30c3a482ab835e3983d
In case this is important, if I remove the WindowFunction, and only use the
FoldFunction it works fine.
I don't see what is wrong...
On Mon, Dec 12, 2016 at 10:53 AM, Matt wrote:
> Hi,
>
> I'm following the documentation [1] of window functions with incremental
> aggrega
I'm using 1.2-SNAPSHOT, should it work in that version?
On Mon, Dec 12, 2016 at 12:18 PM, Yassine MARZOUGUI <
y.marzou...@mindlytix.com> wrote:
> Hi Matt,
>
> What version of Flink are you using?
> The incremental agregation with fold(ACC, FoldFunction, WindowFunction)
I just checked with version 1.1.3 and it works fine, the problem is that in
that version we can't use Kafka 0.10 if I'm not work. Thank you for the
workaround!
Best,
Matt
On Mon, Dec 12, 2016 at 1:52 PM, Yassine MARZOUGUI <
y.marzou...@mindlytix.com> wrote:
> Yes, it was
Err, I meant if I'm not wrong *
On Mon, Dec 12, 2016 at 2:02 PM, Matt wrote:
> I just checked with version 1.1.3 and it works fine, the problem is that
> in that version we can't use Kafka 0.10 if I'm not work. Thank you for the
> workaround!
>
> Best,
> Matt
&g
hieve this with
predefined triggers or a custom trigger is the only way to go here?
Best regards,
Matt
ng #4, after doing some more tests I think it's more complex than I
first thought. I'll probably create another thread explaining more that
specific question.
Thanks,
Matt
On Wed, Dec 14, 2016 at 2:52 PM, Jamie Grier
wrote:
> For #1 there are a couple of ways to do this. The e
lements of B that would have gone into the same tumbling window, not with
the last 3 consecutive elements?
I hope the problem is clear, don't hesitate to ask for further
clarification!
Regards,
Matt
ments in C (triangles), I have to
process n *independent* elements of B (n=2 in the example).
Maybe there's a better or simpler way to do this. Any idea is appreciated!
Regards,
Matt
[1] http://i.imgur.com/dG5AkJy.png
On Thu, Dec 15, 2016 at 3:22 AM, Matt wrote:
> Hello,
>
> I h
jects
for stream C.
Anyway, I've already solved this problem a few days back.
Regards,
Matt
On Mon, Dec 19, 2016 at 5:57 AM, Fabian Hueske wrote:
> Hi Matt,
>
> the combination of a tumbling time window and a count window is one way to
> define a sliding window.
> In your exam
ed by: java.lang.NullPointerException*
The field mentioned is a HashMap, and some keys are mapped
to null values.
Why isn't it possible to forward/serialize those elements with null values?
What do you do when your elements may contain nulls?
Regards,
Matt
f you need any other information let me know.
Regards,
Matt
On Tue, Dec 20, 2016 at 6:46 AM, Stephan Ewen wrote:
> The "null" support in some types is not fully developed. However in that
> case I am wondering why it does not work. Can you share the stack trace, so
> we can
for sharing the stack trace.
>
> This seems not really Flink related, it is part of the specific Avro
> encoding logic.
> The Avro Generic Record Type apparently does not allow the map value to be
> null.
>
>
>
> On Tue, Dec 20, 2016 at 4:55 PM, Matt wrote:
>
>&
nt is this?
Regards,
Matt
Just to be clear, the stream is of String elements. The first part of the
pipeline (up to the first .apply) receives those strings, and returns
objects of another class ("A" let's say).
On Thu, Dec 22, 2016 at 6:04 PM, Matt wrote:
> Hello,
>
> I have a window process
Anyone has any experience mining a Flink+Kafka stream?
I'm looking for an online analysis framework to apply some classifiers on a
time serie.
Any example of how to integrate Flink with MOA, Samoa, ADAMS, DataSketches
or any other framework is appreciated.
Regards,
Matt
I'm still looking for an answer to this question. Hope you can give me some
insight!
On Thu, Dec 22, 2016 at 6:17 PM, Matt wrote:
> Just to be clear, the stream is of String elements. The first part of the
> pipeline (up to the first .apply) receives those strings, and returns
&
ained with
enough instances. Is it possible to do this? If I'm not wrong env.execute
(line 24) can be used only once.*
Regards,
Matt
g it to one node and then another
makes the whole data flow unpractical. It's better to move all created
instances to one single node where only one instance of the classifier
is maintained.
I'm not sure if this is possible or how to do this.
On Thu, Jan 12, 2017 at 11:11 PM, Matt wrote:
Hi all,
What's the purpose of .keyBy() on ConnectedStream? How does it affect
.map() and .flatMap()?
I'm not finding a way to group stream elements based on a key, something
like a Window on a normal Stream, but for a ConnectedStream.
Regards,
Matt
a and read from that
topic on predictionStream instead of initializing it with a reference of
statsStream. I would rather avoid writing unnecessarily into kafka.
Is there any other way to achieve this?
Thanks,
Matt
On Fri, Jan 27, 2017 at 6:35 AM, Timo Walther wrote:
> Hi Matt,
>
I'll create a new thread with my last message since it's not completely
related with the original question here.
On Sat, Jan 28, 2017 at 11:55 AM, Matt wrote:
> Aha, ok, got it!
>
> I just realized that this ConnectedStream I was talking about (A) depends
> on another Conn
But I would rather avoid writing unnecessarily into kafka.
Is there any other way to achieve this?
Thanks,
Matt
ect from *Input2*)
and finally to *predictionStream* (flatMap2).
The same operator is never applied twice to the object, thus I would say
this dataflow is cyclic only in the dependencies of the stream
(predictionStream depends on statsStream, but it depends on
predictionStream in the first place).
;
*green*.keyBy(...).flatMap(...);
---
Any idea is welcome.
Matt
On Sat, Jan 28, 2017 at 5:31 PM, Matt wrote:
> I'm aware of IterativeStream but I don't think it's useful in this case.
>
> As shown in the example above, my use case is "cyclic" in that the same
> obje
I really don't know what you mean, I've been reading the documentation and
examples showing iterations. but it just won't work for me I believe. Maybe
you can write a quick example? It doesn't matter the details, only the
topology.
If anyone else has an idea it's very we
gt;.
Does this still exist? Am I missing something obvious?
Thanks in advance for any help,
Matt
Hi,
>Am I missing something obvious?
So it was that!
Thanks very much for the help, sure I'll be able to figure that out.
Matt
From: Tzu-Li (Gordon) Tai
Sent: 27 February 2017 12:17
To: user@flink.apache.org
Subject: Re: Fw: Flink Kinesis Connector
interface in the current version.
If anyone has any code or a working project to use as a reference that
would be awesome for me and for the rest of us looking for a time-series
database solution!
Best regards,
Matt
[1] https://github.com/druid-io/tranquility/blob/master/docs/flink.md
some ideas from the search engine industry).
The other promising alternative is Prometheus, though I haven't had a look
at it yet, I plan to do so in the near future.
If anyone is using a time-series database and wants to tell us about it
that would be helpful!
Best regards,
Matt
_2016-08-27_at_00.32.42.png?t=1491606817725
On Sat, Apr 8, 2017 at 9:40 PM, Matt wrote:
> I compared them some days ago.
>
> I found a useful article about many of the tsdb available out there [1],
> check the big table on the article, it's really helpful. The thing that
> b
are executed where
the data resides? In case there's no way to guarantee that unless you
enable local environment, what do you think of that approach (in terms of
performance)?
Any additional insight regarding stream processing on Ignite or any other
distributed storage is very welcome!
Best
cated computation
system. Running a collocated Ignite closure and executing a Flink job in a
local environments should be enough.
Why do you recommend against custom collocation? I may be missing something.
Matt
On Mon, Apr 24, 2017 at 9:47 AM, Ufuk Celebi wrote:
> Hey Matt,
>
> in
to the chunk
of code that creates the job. Is this currently possible?
Any fix or workaround is appreciated!
Best,
Matt
[1] https://gist.github.com/f248187b9638023b95ba8bd9d7f06215
[2] https://gist.github.com/796ee05425535ece1736df7b1e884cce
ction in
Test$Foo with the same result: it says "Cannot load user class:
com.test.Test$Foo"
Looks like Flink is not using the correct ClassLoader. Any idea?
Regards,
Matt
On Tue, Apr 25, 2017 at 7:00 AM, Stefan Richter wrote:
> Hi,
>
> I would expect that the local environment
the class.
Not sure what is wrong.
On Tue, Apr 25, 2017 at 5:38 PM, Matt wrote:
> Hi Stefan,
>
> Check the code here: https://gist.github.com/
> 17d82ee7dd921a0d649574a361cc017d , the output is at the bottom of the
> page.
>
> Here are the results of the additional tests you
ome a general fix.
>
> I have heard that Till is about to change some things about local
> execution, so I included him in CC. Maybe he can provide additional hints
> how your use case might be better supported in the upcoming Flink 1.3.
>
> Best,
> Stefan
>
> Am 25.04.2
ssage on the list
[1].
The idea is to collocate Flink jobs on Ignite nodes, so each dataflow only
processes the elements stored on the local in-memory database. I get the
impression this should be much faster than randomly picking a Flink node
and sending all the data over the network.
Any insight on
se in [1].
Best,
Matt
[1] https://gist.github.com/17d82ee7dd921a0d649574a361cc017d
On Thu, Apr 27, 2017 at 10:09 AM, Matt wrote:
> Hi Till,
>
> Great! Do you know if it's planned to be included in v1.2.x or should we
> wait for v1.3? I'll give it a try as soon as it'
Thanks for your help Till.
I will create a self contained test case in a moment and send you the link,
wait for it.
Cheers,
Matt
On Wed, May 17, 2017 at 4:38 AM, Till Rohrmann wrote:
> Hi Matt,
>
> alright, then we have to look into it again. I tried to run your example,
> howe
should be enabled (as in ignite.xml),
because it must match the config on the client node.
If you follow the Readme file it's everything there, if you have any
problem let me know!
Cheers,
Matt
[1] https://github.com/Dromit/FlinkTest
On Wed, May 17, 2017 at 3:49 PM, Matt wrote:
> Th
where the data is stored.
[1] https://apacheignite-mix.readme.io/docs/flink-streamer
Best,
Matt
Matt
On Mon, May 29, 2017 at 9:37 AM, Till Rohrmann wrote:
> Hi Matt,
>
> I looked into it and it seems that the Task does not respect the context
> class loader. The problem is that
Suppose I'm using state stored in-memory that has a TTL of 7 days max. Should I
run into any issues with state this long other than potential OOM?
Let's suppose I extend this such that we add rocksdb...any concerns with this
with respect to maintenance?
Most of the examples that I've been seein
and be purged from the checkpoint?
Flink Version 1.7.1
Thanks!
-Matt
hing is set in terms of watermarks – do they apply for Process
Time?
The set of keys processed in the stream is stable over time
The checkpoint size actually looks pretty stable now that the interval was
increased. Is it possible that the short checkpoint interval prevented
compaction?
Thanks!
consistent (follows the
same pattern).
Changing the checkpoint interval seemed to fix the problem of the large and
growing checkpoint size but I’m not sure why.
Thanks!
-Matt
From: Till Rohrmann
Date: Thursday, May 28, 2020 at 10:48 AM
To: "Wissman, Matt"
Cc: Guowei Ma , "user@f
: {{ .Values.flink.checkpointUrl }}/checkpoints
state.savepoints.dir: {{ .Values.flink.checkpointUrl }}/savepoints
state.backend.incremental: true
state.backend.rocksdb.localdir: /tmp/taskmanager
Thanks!
-Matt
From: Guowei Ma
Date: Monday, June 1, 2020 at 1:01 AM
To: "Wissman, Matt"
We're currently using this template:
https://github.com/docker-flink/examples/tree/master/helm/flink for running
kubernetes flink for running a job specific cluster ( with a nit of specifying
the class as the main runner for the cluster ).
How would I go about setting up adding savepoints, so
I'm not the original poster, but I'm running into this same issue. What you
just described is exactly what I want. I presume you guys are using some
variant of this helm
https://github.com/docker-flink/examples/tree/master/helm/flink to configure
your k8s cluster? I'm also assuming that this cl
think I know what to do internally
On 2020/06/15 15:11:32, Robert Metzger wrote:
> Hi Matt,
>
> sorry for the late reply. Why are you using the "flink-docker" helm example
> instead of
> https://ci.apache.org/projects/flink/flink-docs-master/ops/deployment
checkpoints.dir=$(S3_CHECKPOINT)
62 - -Dstate.savepoints.dir=$(S3_SAVEPOINT)
63 - --allowNonRestoredState
64 - -s $(S3_SAVEPOINT)
I originally didn't have the last 2 args, I added them based upon various
emails I saw on this list and other google search results, to no avail.
Thanks
-Matt
I'm wondering what the best practice is for using secrets in a Flink program,
and I can't find any info in the docs or posted anywhere else.
I need to store an access token to one of my APIs for flink to use to dump
results into, and right now I'm passing it through as a configuration
parameter
tion that would allow the
fallback checkpoint to be something more recent?
Thanks,
Matt
2021/10/11 12:22:28.137 INFO c.i.strmprocess.ArgsPreprocessor -
latestSavepointPrefix:desanalytics-7216-doc-comprehension-analytics-7216-prd/4/savepoints/savepoint-00-abb450590ca7/_metadata
LastModifie
t: Tuesday, October 19, 2021 3:03 AM
To: LeVeck, Matt
Cc: user@flink.apache.org
Subject: Re: Flink ignoring latest checkpoint on restart?
This email is from an external sender.
Hi Matt,
this seems interesting, I'm aware of some possible inconsistency issues with
unstable connections [1], but I h
anks in advance!
All the best,
Matt (mcasters, Apache Hop PMC)
fication to that effect? Just
brainstorming ;-) (and forking apache/flink-kubernetes-operator)
All the best,
Matt
On Tue, Jun 21, 2022 at 2:52 PM Őrhidi Mátyás
wrote:
> Hi Matt,
>
> - In FlinkDeployments you can utilize an init container to download your
> artifact onto a shared
want to figure out a way to do this with Flink as well since I believe,
especially on AWS (even with Spark centric options on EMR, EMR serverless),
that running a pipeline is just too complicated. Your work really helps!
All the best,
Matt
On Tue, Jun 21, 2022 at 4:53 PM Őrhidi Mátyás
wrote:
>
hings easy. Following page after page of complicated instructions
just to get a few files into a pod container... I feel it's just a bit
much.
But again, this is my frustration with k8s, not with Flink ;-)
Cheers,
Matt
On Wed, Jun 22, 2022 at 5:32 AM Yang Wang wrote:
> Matyas and Gyula have
Hi Mátyás & all,
Thanks again for the advice so far. On a related note I noticed Java 8
being used, indicated in the log.
org.apache.flink.runtime.entrypoint.ClusterEntrypoint[] -
JAVA_HOME: /usr/local/openjdk-8
Is there a way to use Java 11 to start Flink with?
Kind regards,
sks in my
first Beam pipeline so it should be simple enough but it just times out.
Next step for me is to document the result which will end up on
hop.apache.org. I'll probably also want to demo this in Austin at the
upcoming Beam summit.
Thanks a lot for your time and help so far!
Cheers,
Matt
e out!
Cheers,
Matt
On Mon, Jun 27, 2022 at 4:59 AM Yang Wang wrote:
> Could you please share the JobManager logs of failed deployment? It will
> also help a lot if you could show the pending pod status via "kubectl
> describe ".
>
> Given that the current Flink Kubernet
possible to overcome these issues and have event
fan-out working on AWS?
Thanks,
Matt
sink, plus not having large data size
requirements, I suspect this is due to a bug.
I'm running v1.13.2 and have created a docker-based reproduction repository
here: https://github.com/mattfysh/pyflink-oom
Please take a look and let me know what you think
Thanks!
Matt
.
Please let me know which sections of the docs, or which areas of Python, I
should read to learn how to find a solution to this problem
Thanks
On Mon, 31 Oct 2022 at 18:49, Leonard Xu wrote:
> Hi, Matt
>
> I’ve checked your job is pretty simple, I've CC Xingbo who is a PyFlink
&g
Hi, I'm following the kinesis connector instructions as documented here:
https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/connectors/datastream/kinesis/
I'm also running Flink in standalone session mode using docker compose and
the Python images, as described in the Flink docs (Depl
image, but bundled in your user-jar (along with the connector).
>
> On 08/11/2022 02:14, Matt Fysh wrote:
> > Hi, I'm following the kinesis connector instructions as documented
> > here:
> >
> https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/connecto
e in this area for some
pointers on what else I could start reading
Thanks
Matt
/blob/main/flink-connector-dynamodb/src/test/java/org/apache/flink/connector/dynamodb/sink/examples/SinkIntoDynamoDb.java
Thank you again for your help and sharing those resources.
Cheers,
Matt.
On Wed, 9 Nov 2022 at 03:51, Teoh, Hong wrote:
> Hi Matt,
>
>
>
> First of all, aweso
see if there is any error information
about the metrics;
[1]
https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/config/#metrics-reporter-%3Cname%3E-filter-excludes
--
Best,
Matt Wang
Replied Message
| From | patricia lee |
| Date | 09/18/2023 16:58 |
| To
73 matches
Mail list logo
