The operators for stream jobs will not use memory management which is only for
batch jobs as you said.
I guess the initial feedback is for batch jobs from the description?
--
发件人:Paul Lam
发送时间:2018年10月17日(星期三) 14:35
收件人:Zhijiang(wang
Hi Zhijiang,
Does the memory management apply to streaming jobs as well? A previous post[1]
said that it can only be used in batch API, but I might miss some updates on
that. Thank you!
[1] https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=53741525
Best,
Paul Lam
> 在 2018年10月17
Hi Julien,
Flink would manage the default 70% fraction of free memory in TaskManager for
caching data efficiently, just as you mentioned in this article
"https://flink.apache.org/news/2015/05/11/Juggling-with-Bits-and-Bytes.html";.
These managed memories are persistent resident and referenced b
Hi Julien,
AFAIK, streaming jobs put data objects on heap, so the it depends on the JVM GC
to release the memory.
Best,
Paul Lam
> 在 2018年10月12日,14:29,jpreis...@free.fr 写道:
>
> Hi,
>
> My use case is :
> - I use Flink 1.4.1 in standalone cluster with 5 VM (1 VM = 1 JobManager + 1
> TaskM
Hi Hequn,
I tried with following :
Configuration conf = new Configuration();
conf.setString("state.checkpoints.dir","file:///home/sohanvir/Desktop/flink/checkpoints2");
final StreamExecutionEnvironment env =
StreamExecutionEnvironment.createLocalEnvironment(1,conf);
CheckpointConf
Hi sohimankotia,
Have you ever enableExternalizedCheckpoints[1]?
> // enable externalized checkpoints which are retained after job
> cancellation
>
> env.getCheckpointConfig().enableExternalizedCheckpoints(ExternalizedCheckpointCleanup.RETAIN_ON_CANCELLATION);
Best, Hequn
[1]
https://ci.apache
Hi Shahar,
The table function takes a single row but can output multi rows. You can
split the row based on the "last" event. The code looks like:
val sessionResult =
> "SELECT " +
> " lastUDAF(line) AS lastEvents "
> "FROM MyTable " +
> "GROUP BY SESSION(rowtime
Hi Andrew,
You should call it manually, as the global window does not have a natural
end.
Best, Hequn
On Wed, Oct 17, 2018 at 2:47 AM Andrew Danks wrote:
> Hi Fabian & Hequn,
>
> Thank you for your responses. I am just responding now as I was out of
> office for the last few days
>
> You menti
Hi Fabian & Hequn,
Thank you for your responses. I am just responding now as I was out of office
for the last few days
You mentioned that clear() is called when the time exceeds the window’s end
timestamp. For my application I am using a GlobalWindow on a keyed stream --
would clear() get call
Im wondering how does that work, it seems that a table function still takes
a single row's values as an input, am i wrong (or at least that is how the
examples show)?
How would the SQL look like?
On Fri, Oct 12, 2018 at 9:15 PM Hequn Cheng wrote:
> Hi shkob1,
>
> > while one is time(session inac
Thanks!
On Fri, Oct 12, 2018 at 9:29 PM Hequn Cheng wrote:
> Hi shkob1,
>
> Currently, the idle state retention time is only used for unbounded
> operators in sql/table-api. The unbounded operators include non-window
> group by, non-window join, unbounded over, etc. The retention time affects
>
Hi,
I am using following in code :
1. flink 1.4
2. running example on IDE
3. Enabled Exactly once semantics
4. Window Aggregation
5. Checkpoint is enabled at 20 Sec
6/ RocksDB as state backend
Workflow :
Kafka Source -> map -> keyBy -> Window(60 Sec) -> ApplyFunction ->
Aggregated Record to K
Hi to all,
which is the correct wat to report back to the user a failure from a job
submission in FLink?
If everything is OK the job run API returns the job id but what if there
are error in parameter validation or some other problem?
Which is the right way to report back to the user the job error
Done: https://issues.apache.org/jira/browse/FLINK-10562
On Tue, Oct 16, 2018 at 11:12 AM Timo Walther wrote:
> Hi Flavio,
>
> yes you are right, I don't see a reason why we should not support such
> table names. Feel free to open an issue for it.
>
> Regards,
> Timo
>
>
> Am 16.10.18 um 10:56 sc
Hi Flavio,
yes you are right, I don't see a reason why we should not support such
table names. Feel free to open an issue for it.
Regards,
Timo
Am 16.10.18 um 10:56 schrieb miki haiat:
Im not sure if it will solve this issue but can you try to register
the your catalog [1]
1.https://ci.ap
Im not sure if it will solve this issue but can you try to register the
your catalog [1]
1.
https://ci.apache.org/projects/flink/flink-docs-release-1.6/dev/table/common.html#register-an-external-catalog
On Tue, Oct 16, 2018 at 11:40 AM Flavio Pompermaier
wrote:
> Hi to all,
> in my job I'm try
Hi to all,
in my job I'm trying to read a dataset whose name/id starts with a number.
It seems that when using the Table API to read that dataset, if the name
starts with a number it is a problem..am I wrong? I can't find anything
about table id constraints on the documentation and it seems that i
Did you execute this command ?
Note: If using MiniKube please make sure to execute minikube ssh 'sudo ip
> link set docker0 promisc on' before deploying a Flink cluster. Otherwise
> Flink components are not able to self reference themselves through a
> Kubernetes service.
On Tue, Oct 16, 2018 at
I followed the Doc(
https://ci.apache.org/projects/flink/flink-docs-release-1.6/ops/deployment/kubernetes.html#session-cluster-resource-definitions)
to run flink on kubernetes,
but there is an exception(java.net.UnknownHostException: flink-jobmanager:
Temporary failure in name resolution).
I use
19 matches
Mail list logo