In the default worker.xml implementation, file names for Storm logs are set by ${sys:workers.artifacts}/${sys:storm.id}/${sys:worker.port}/${sys:logfile.name}. Storm.id is what is used to distinguish between workers. Instead of using storm.id, we would like to use our own per topology id to determine the directory that the logs are written to. How would I go about setting this at the worker level? My initial thought was to use log4j2's provided system lookup, but I don't believe Storm provides a way to set unique system variables per worker. The config property worker.childopts can be used to set system variables, but this is defined once per cluster in the storm.yaml file, I don't think there is a way to pass in a unique id for every worker that is started. Any sort of programmatic log4j2 property substitution would not work because the earliest place to set this for a worker would be in a WorkerHook, and Storm logs before the WorkerHook is called. So have any others came up with a solution to a similar issue (custom log file location set by user per topology)? Is there a way to set unique system variables per worker/topology provided by Storm?
Customizing Storm file locations at the worker/topology level
Mitchell Rathbun (BLOOMBERG/ 731 LEX) Sun, 15 Apr 2018 16:44:30 -0700