Hi Till,
Sorry for the delay, you were right, I was not restarting the yarn cluster…
Many thanks for your help!
Ana
On 11 Jan 2016, at 14:39, Till Rohrmann
> wrote:
You have to restart the yarn cluster to let your changes take effect. You can
Hi Till,
Thanks for that! I can see the "Logger in LineSplitter.flatMap” output if I
retrieve the task manager logs manually (under
/var/log/hadoop-yarn/containers/application_X/…). However that solution is not
ideal when for instance I am using 32 machines for my mapReduce operations.
I
Hi Ana,
good to hear that you found the logging statements. You can check in Yarn’s
web interface whether there are still occupied containers. Alternatively
you can go to the different machines and run jps which lists you the
running Java processes. If you see an ApplicationMaster or
Hi Till,
Thanks for your help. I have checked both in Yarn’s web interface and through
command line and it seems that there are not occupied containers.
Additionally, I have checked the configuration values in the web interface and
even though I have changed the log.aggregation property in the
Thanks for the tip Robert! It was a good idea to rule out other possible
causes, but I am afraid that is not the problem. If we stick to the
WordCountExample (for simplicity), the Exception is thrown if placed into the
flatMap function.
I am going to try to re-write my problem and all the
You’re right that the log statements of the LineSplitter are in the logs of
the cluster nodes, because that’s where the LineSplitter code is executed.
In contrast, you create a TestClass on the client when you submit the
program. Therefore, you see the logging statement “Logger in TestClass” on
Hi Till,
I am afraid it does not work in any case.
I am following the steps you indicate on your websites (for yarn configuration
and loggers with slf4j):
1) Enable log aggregation in yarn-site:
https://ci.apache.org/projects/flink/flink-docs-release-0.7/yarn_setup.html#log-files
2) Include
Maybe the isConverged() method is never called? For making that sure, just
throw a RuntimeException inside the method to see whats happening.
On Wed, Jan 6, 2016 at 3:58 PM, Ana M. Martinez wrote:
> Hi Till,
>
> I am afraid it does not work in any case.
>
> I am following the
Hi Till,
Sorry for the delay (Xmas break). I have activated log aggregation on
flink-conf.yaml with yarn.log-aggregation-enable: true (as I can’t find a
yarn-site.xml).
But the command yarn logs -applicationId application_1451903796996_0008 gives
me the following output:
INFO client.RMProxy:
In which log file are you exactly looking for the logging statements? And
on what machine? You have to look on the machines on which the yarn
container were started. Alternatively if you have log aggregation
activated, then you can simply retrieve the log files via yarn logs.
Cheers,
Till
On
Hi Till,
Many thanks for your quick response.
I have modified the WordCountExample to re-reproduce my problem in a simple
example.
I run the code below with the following command:
./bin/flink run -m yarn-cluster -yn 1 -ys 4 -yjm 1024 -ytm 1024 -c
mypackage.WordCountExample ../flinklink.jar
Hi Ana,
you can simply modify the `log4j.properties` file in the `conf` directory.
It should be automatically included in the Yarn application.
Concerning your logging problem, it might be that you have set the logging
level too high. Could you share the code with us?
Cheers,
Till
On Thu, Dec
12 matches
Mail list logo