Why don’t you let your flink job move them once it’s done?
> On 9. Mar 2018, at 03:12, flinkuser101 wrote:
> I am reading files from a folder suppose
> Files are pushed into that folder.
Kostas is right, unfortunately I had to stop the work, cause we were missing
BroadcastState. I hope I will get back to this feature soon and finish it for
> On 8 Mar 2018, at 17:28, Vishal Santoshi wrote:
> Perfect. Thanks.
> On Thu, Mar 8, 2018 at
there’s no need to implement the window logic with the low-level
`ProcessFunction` yourself. Flink has provided built-in window operators and
you just need to implement the `WindowFunction` for that .
I was trying to emulate tumbling window in event time space. Here is the link
to my code.
I am using the process function to do the custom processing which I want to do
within every window. I am having an issue of how to emit results at the end of
every window since my watermark only gets
I am reading files from a folder suppose
Files are pushed into that folder.
Flink is reading files from the folder fine but as the no of files grows how
do I move the files into another folder? Currently I am using cronjob to
Hello, from what I understand in the documentation it appears there is no
way to assign UUIDs to operators added to the DAG by Flink SQL. Is my
I'd very much like to be able to assign UUIDs to those operators. I want to
run a program using some Flink SQL, create a save
Hi Xingcan, Timo,
Thanks for the information.
I am going to convert the result table to DataStream and follow the logic of
TimeBoundedStreamInnerJoin to do the timed-window join. Should I do this? Is
there any concern from performance or stability perspective?
Yep. I think this leads to this general question and may be not pertinent
to https://github.com/apache/flink/pull/5342. How do we throttle a source
if the held back data gets unreasonably large ? I know that that is in
itself a broader question but delayed watermarks of slow stream accentuates
The join would not cause backpressure but rather put all events that cannot
be processed yet into state to process them later.
So this works well if the data that is provided by the streams is roughly
aligned by event time.
2018-03-08 9:04 GMT-08:00 Vishal Santoshi :
Aah we have it here
On Thu, Mar 8, 2018 at 11:45 AM, Vishal Santoshi
> This is very interesting. I would imagine that there will be high back
This is very interesting. I would imagine that there will be high back
pressure on the LEFT source effectively throttling it but as is the current
state that is likely effect other pipelines as the free o/p buffer on the
source side and and i/p buffers on the consumer side start blocking and get
Flink does currently not support holding back individual streams, for
example it is not possible to align streams on (offset) event-time.
However, the Flink community is working on a windowed join for the
DataStream API, that only holds the relevant tail of the stream as state.
On Thu, Mar 8, 2018 at 10:41 AM, Kostas Kloudas wrote:
> Hi Vishal,
> Dawid (cc’ed) who was working on that stopped because in the past Flink
> did not support broadcast state.
> This is now added (in the master) and the implementation of
Hi Yan & Timo,
this is confirmed to be a bug and I’ve created an issue  for it.
I’ll explain more about this query. In Flink SQL/Table API, the DISTINCT
keyword will be implemented with an aggregation, which outputs a retract stream
. In that situation, all the time-related fields will
This is a bug in flink. As a workaround, you can cast eventTime to other
basic sql types(for example, cast eventTime as varchar).
@Timo and @Xingcan, I think we have to materialize time indicators in
conditions of LogicalFilter. I created an issue and we can have
Dawid (cc’ed) who was working on that stopped because in the past Flink
did not support broadcast state.
This is now added (in the master) and the implementation of FLINK-7129
will continue hopefully soon.
> On Mar 8, 2018, at 4:09 PM, Vishal Santoshi
What about https://issues.apache.org/jira/browse/FLINK-7129 ? Do
you folks intend to conclude this ticket too ?
On Thu, Mar 8, 2018 at 1:08 AM, Fabian Hueske wrote:
> We hope to pick up FLIP-20 after Flink 1.5.0 has been released.
> 2018-03-07 22:05
I got it. That’s really a big problem.
Thank you very much
> On 8 Mar 2018, at 21:03, kedar mhaswade wrote:
> Also, in addition to what Gary said, if you take Flink completely out of
> picture and wrote a simple Java class with a main method and the static block
Also, in addition to what Gary said, if you take Flink completely out of
picture and wrote a simple Java class with a main method and the static
block (!) which does some long running task like getLiveInfo(), then
chances are that your class will make the JVM hang!
Basically what you are doing is
thanks for looking into this. This definitely seems to be a bug. Maybe
in the org.apache.flink.table.calcite.RelTimeIndicatorConverter. In any
case we should create an issue for it.
Am 3/8/18 um 7:27 AM schrieb Yan Zhou [FDS Science]:
the CsvTableSource and its builder are currently not able to specify
event-time or processing-time. I'm sure this will change in the near
future. Until then I would recommend to either extend it yourself or use
the DataStream API first to do the parsing and watermarking and then
Can you share the JobManager and TaskManager logs returned by:
yarn logs -applicationId ?
Is your browser rendering a blank page, or does the HTTP request not finish?
Can you show the output of one of the following commands:
curl -v http://host:port
Double checking: The "deserialize(byte message)" already receives an
additional byte with too many bytes?
I wonder if this might be an issue in Kafka then, or in the specific way
Kafka is configured.
On Wed, Mar 7, 2018 at 5:40 PM, Philip Doctor
I think, I found a code path (race between threads) that may lead to two
markers being in the list.
I created https://issues.apache.org/jira/browse/FLINK-8896 to track this
and will have a pull request ready (probably) today.
On 07/03/18 10:09, Mu Kong wrote:
> Hi Gordon,
> Thanks for
You are not shutting down the ScheduledExecutorService , which means that
after job cancelation the thread will continue running getLiveInfo(). The
code class loader, and your classes won't be garbage collected. You should
the RichFunction#close callback to shutdown your thread
we're considering flink for a couple of our projects. I'm doing a
trial implementation for one of them. So far, I like a lot of things,
however there are a couple of issues that I can't figure out how to
resolve. Not sure if it's me misunderstanding the tool, or flink just
doesn't have a
Mail list logo