We could successfully run the job without issues. Thanks a lot everyone for
the support.
FYI: with Flink we completed in 3h28m the job that was planned to run for
15 days 24/7 relying on our legacy customer approach. :-)
saluti,
Stefano
2016-04-28 14:50 GMT+02:00 Fabian Hueske :
> Yes, assignin
Yes, assigning more than 0.5GB to a JM is a good idea. 3GB is maybe a bit
too much, 2GB should be enough.
Increasing the timeout should not hurt either.
2016-04-28 14:14 GMT+02:00 Flavio Pompermaier :
> So what do you suggest to try for the next run?
> I was going to increase the Job Manager heap
So what do you suggest to try for the next run?
I was going to increase the Job Manager heap to 3 GB and maybe change some
gc setting.
Do you think I should increase also the akka timeout or other things?
On Thu, Apr 28, 2016 at 2:06 PM, Fabian Hueske wrote:
> Hmm, 113k splits is quite a lot.
>
Hmm, 113k splits is quite a lot.
However, the IF uses the DefaultInputSplitAssigner which is very
lightweight and should handle a large number of splits well.
2016-04-28 13:50 GMT+02:00 Flavio Pompermaier :
> We generate 113k splits because we can't query more than 100k or records
> per split (
We generate 113k splits because we can't query more than 100k or records
per split (and we have to manage 11 billions of records). We tried to run
the job only once, before running it the 2nd time we would like to
understand which parameter to tune in order to (try to at least to) avoid
such an err
Is the problem reproducible?
Maybe the SplitAssigner gets stuck somehow, but I've never observed
something like that.
How many splits do you generate?
I guess it is not related, but 512MB for a TM is not a lot on machines with
16GB RAM.
2016-04-28 12:12 GMT+02:00 Flavio Pompermaier :
> When doe
When does this usually happens? Is it because the JobManager has too few
resources (of some type)?
Our current configuration of the cluster has 4 machines (with 4 CPUs and 16
GB of RAM) and one machine has both a JobManager and a TaskManger (the
other 3 just a TM).
Our flink-conf.yml on every mac
I checked the input format from your PR, but didn't see anything
suspicious.
It is definitely OK if the processing of an input split tasks more than 10
seconds. That should not be the cause.
It rather looks like the DataSourceTask fails to request a new split from
the JobManager.
2016-04-28 9:37
Digging the logs, we found this:
WARN Remoting - Tried to associate with unreachable remote address
[akka.tcp://flink@127.0.0.1:34984]. Address is now gated for 5000 ms, all
messages to this address will be delivered to dead letters. Reason:
Connessione rifiutata: /127.0.0.1:34984
however, it is
I had this type of exception when trying to build and test Flink on a
"small machine". I worked around the test increasing the timeout for Akka.
https://github.com/stefanobortoli/flink/blob/FLINK-1827/flink-tests/src/test/java/org/apache/flink/test/checkpointing/EventTimeAllWindowCheckpointingITCa
A precursor of the modified connector (since we started a long time ago).
However the idea is the same, I compute the inputSplits and then I get the
data split by split (similarly to what it happens in FLINK-3750 -
https://github.com/apache/flink/pull/1941 )
Best,
Flavio
On Wed, Apr 27, 2016 at 5
Are you using your modified connector or the currently available one?
On 27.04.2016 17:35, Flavio Pompermaier wrote:
Hi to all,
I'm running a Flink Job on a JDBC datasource and I obtain the
following exception:
java.lang.RuntimeException: Requesting the next InputSplit failed.
at
org.apache.
12 matches
Mail list logo
