In the default worker.xml implementation, file names for Storm logs are set by
${sys:workers.artifacts}/${sys:storm.id}/${sys:worker.port}/${sys:logfile.name}.
Storm.id is what is used to distinguish between workers. Instead of using
storm.id, we would like to use our own per topology id to determine the
directory that the logs are written to. How would I go about setting this at
the worker level? My initial thought was to use log4j2's provided system
lookup, but I don't believe Storm provides a way to set unique system variables
per worker. The config property worker.childopts can be used to set system
variables, but this is defined once per cluster in the storm.yaml file, I don't
think there is a way to pass in a unique id for every worker that is started.
Any sort of programmatic log4j2 property substitution would not work because
the earliest place to set this for a worker would be in a WorkerHook, and Storm
logs before the WorkerHook is called. So have any others came up with a
solution to a similar issue (custom log file location set by user per
topology)? Is there a way to set unique system variables per worker/topology
provided by Storm?
Customizing Storm file locations at the worker/topology level
Mitchell Rathbun (BLOOMBERG/ 731 LEX) Sun, 15 Apr 2018 16:44:30 -0700
