[jira] [Updated] (ATLAS-3610) Enable logging to multiple targets (logdir, stdout)
[ https://issues.apache.org/jira/browse/ATLAS-3610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Madhan Neethiraj updated ATLAS-3610: Fix Version/s: (was: 2.3.0) > Enable logging to multiple targets (logdir, stdout) > --- > > Key: ATLAS-3610 > URL: https://issues.apache.org/jira/browse/ATLAS-3610 > Project: Atlas > Issue Type: Improvement > Components: atlas-core >Affects Versions: 3.0.0 >Reporter: Mariusz Górski >Priority: Minor > Fix For: 3.0.0 > > Attachments: ATLAS-3610-1.patch, ATLAS-3610-2.patch, > ATLAS-3610-3.patch, ATLAS-3610.patch > > Time Spent: 50m > Remaining Estimate: 0h > > *Problem description* > When starting Atlas, Python scripts are invoked to spawn all necessary > processes. This is achieved with the use of _subprocess.Popen()_ in > _runProcess()_ function of _atlas_config.py_ script. While doing that, stdout > is being redirected to log files, which (for java processes) can override > log4j configuration (if configured with stdout handler). This creates > situation when log4j configuration cannot be relied upon and prevents logging > to stdout. > *Proposed solution* > * Add possibility to log to stdout by using multiple stdout/stderr handlers > * Adjust _runProcess()_ function from _atlas_config.py_ script (and all > functions relying on it) to provide _logconsole_ kwarg which will add > additional handler to Popen stdout/stderr redirection > * Add _ENABLE_LOGGING_TO_CONSOLE_ env variable (defaulting to _False_) for > backwards compatibility. > * Adjust _atlas_start.py_ script to make use of _ENABLE_LOGGING_TO_CONSOLE_ > and provide it in functions invoking atlas/solr/elasticsearch/zookeeper > *Predicted Benefits* > This improvement would bring the benefit of enabling logging to stdout (which > is supressed for now even if configured in spawned components), which could > be very useful when running Atlas in docker container or in Kubernetes/OC, > where some logging scraping is present. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (ATLAS-3610) Enable logging to multiple targets (logdir, stdout)
[ https://issues.apache.org/jira/browse/ATLAS-3610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Madhan Neethiraj updated ATLAS-3610: Fix Version/s: (was: 2.2.0) 2.3.0 > Enable logging to multiple targets (logdir, stdout) > --- > > Key: ATLAS-3610 > URL: https://issues.apache.org/jira/browse/ATLAS-3610 > Project: Atlas > Issue Type: Improvement > Components: atlas-core >Affects Versions: 3.0.0 >Reporter: Mariusz Górski >Priority: Minor > Fix For: 3.0.0, 2.3.0 > > Attachments: ATLAS-3610-1.patch, ATLAS-3610-2.patch, > ATLAS-3610-3.patch, ATLAS-3610.patch > > Time Spent: 50m > Remaining Estimate: 0h > > *Problem description* > When starting Atlas, Python scripts are invoked to spawn all necessary > processes. This is achieved with the use of _subprocess.Popen()_ in > _runProcess()_ function of _atlas_config.py_ script. While doing that, stdout > is being redirected to log files, which (for java processes) can override > log4j configuration (if configured with stdout handler). This creates > situation when log4j configuration cannot be relied upon and prevents logging > to stdout. > *Proposed solution* > * Add possibility to log to stdout by using multiple stdout/stderr handlers > * Adjust _runProcess()_ function from _atlas_config.py_ script (and all > functions relying on it) to provide _logconsole_ kwarg which will add > additional handler to Popen stdout/stderr redirection > * Add _ENABLE_LOGGING_TO_CONSOLE_ env variable (defaulting to _False_) for > backwards compatibility. > * Adjust _atlas_start.py_ script to make use of _ENABLE_LOGGING_TO_CONSOLE_ > and provide it in functions invoking atlas/solr/elasticsearch/zookeeper > *Predicted Benefits* > This improvement would bring the benefit of enabling logging to stdout (which > is supressed for now even if configured in spawned components), which could > be very useful when running Atlas in docker container or in Kubernetes/OC, > where some logging scraping is present. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (ATLAS-3610) Enable logging to multiple targets (logdir, stdout)
[ https://issues.apache.org/jira/browse/ATLAS-3610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Madhan Neethiraj updated ATLAS-3610: Fix Version/s: (was: 2.1.0) 2.2.0 > Enable logging to multiple targets (logdir, stdout) > --- > > Key: ATLAS-3610 > URL: https://issues.apache.org/jira/browse/ATLAS-3610 > Project: Atlas > Issue Type: Improvement > Components: atlas-core >Affects Versions: 3.0.0 >Reporter: Mariusz Górski >Priority: Minor > Fix For: 3.0.0, 2.2.0 > > Attachments: ATLAS-3610-1.patch, ATLAS-3610-2.patch, > ATLAS-3610-3.patch, ATLAS-3610.patch > > Time Spent: 50m > Remaining Estimate: 0h > > *Problem description* > When starting Atlas, Python scripts are invoked to spawn all necessary > processes. This is achieved with the use of _subprocess.Popen()_ in > _runProcess()_ function of _atlas_config.py_ script. While doing that, stdout > is being redirected to log files, which (for java processes) can override > log4j configuration (if configured with stdout handler). This creates > situation when log4j configuration cannot be relied upon and prevents logging > to stdout. > *Proposed solution* > * Add possibility to log to stdout by using multiple stdout/stderr handlers > * Adjust _runProcess()_ function from _atlas_config.py_ script (and all > functions relying on it) to provide _logconsole_ kwarg which will add > additional handler to Popen stdout/stderr redirection > * Add _ENABLE_LOGGING_TO_CONSOLE_ env variable (defaulting to _False_) for > backwards compatibility. > * Adjust _atlas_start.py_ script to make use of _ENABLE_LOGGING_TO_CONSOLE_ > and provide it in functions invoking atlas/solr/elasticsearch/zookeeper > *Predicted Benefits* > This improvement would bring the benefit of enabling logging to stdout (which > is supressed for now even if configured in spawned components), which could > be very useful when running Atlas in docker container or in Kubernetes/OC, > where some logging scraping is present. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (ATLAS-3610) Enable logging to multiple targets (logdir, stdout)
[ https://issues.apache.org/jira/browse/ATLAS-3610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Keval Bhatt updated ATLAS-3610: --- Fix Version/s: 2.1.0 > Enable logging to multiple targets (logdir, stdout) > --- > > Key: ATLAS-3610 > URL: https://issues.apache.org/jira/browse/ATLAS-3610 > Project: Atlas > Issue Type: Improvement > Components: atlas-core >Affects Versions: 3.0.0 >Reporter: Mariusz Górski >Priority: Minor > Fix For: 2.1.0, 3.0.0 > > Attachments: ATLAS-3610-1.patch, ATLAS-3610-2.patch, ATLAS-3610.patch > > Time Spent: 50m > Remaining Estimate: 0h > > *Problem description* > When starting Atlas, Python scripts are invoked to spawn all necessary > processes. This is achieved with the use of _subprocess.Popen()_ in > _runProcess()_ function of _atlas_config.py_ script. While doing that, stdout > is being redirected to log files, which (for java processes) can override > log4j configuration (if configured with stdout handler). This creates > situation when log4j configuration cannot be relied upon and prevents logging > to stdout. > *Proposed solution* > * Add possibility to log to stdout by using multiple stdout/stderr handlers > * Adjust _runProcess()_ function from _atlas_config.py_ script (and all > functions relying on it) to provide _logconsole_ kwarg which will add > additional handler to Popen stdout/stderr redirection > * Add _ENABLE_LOGGING_TO_CONSOLE_ env variable (defaulting to _False_) for > backwards compatibility. > * Adjust _atlas_start.py_ script to make use of _ENABLE_LOGGING_TO_CONSOLE_ > and provide it in functions invoking atlas/solr/elasticsearch/zookeeper > *Predicted Benefits* > This improvement would bring the benefit of enabling logging to stdout (which > is supressed for now even if configured in spawned components), which could > be very useful when running Atlas in docker container or in Kubernetes/OC, > where some logging scraping is present. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (ATLAS-3610) Enable logging to multiple targets (logdir, stdout)
[ https://issues.apache.org/jira/browse/ATLAS-3610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mariusz Górski updated ATLAS-3610: -- Attachment: (was: ATLAS-3610-2.patch) > Enable logging to multiple targets (logdir, stdout) > --- > > Key: ATLAS-3610 > URL: https://issues.apache.org/jira/browse/ATLAS-3610 > Project: Atlas > Issue Type: Improvement > Components: atlas-core >Affects Versions: 3.0.0 >Reporter: Mariusz Górski >Priority: Minor > Fix For: 3.0.0 > > Attachments: ATLAS-3610-1.patch, ATLAS-3610.patch > > Time Spent: 50m > Remaining Estimate: 0h > > *Problem description* > When starting Atlas, Python scripts are invoked to spawn all necessary > processes. This is achieved with the use of _subprocess.Popen()_ in > _runProcess()_ function of _atlas_config.py_ script. While doing that, stdout > is being redirected to log files, which (for java processes) can override > log4j configuration (if configured with stdout handler). This creates > situation when log4j configuration cannot be relied upon and prevents logging > to stdout. > *Proposed solution* > * Add possibility to log to stdout by using multiple stdout/stderr handlers > * Adjust _runProcess()_ function from _atlas_config.py_ script (and all > functions relying on it) to provide _logconsole_ kwarg which will add > additional handler to Popen stdout/stderr redirection > * Add _ENABLE_LOGGING_TO_CONSOLE_ env variable (defaulting to _False_) for > backwards compatibility. > * Adjust _atlas_start.py_ script to make use of _ENABLE_LOGGING_TO_CONSOLE_ > and provide it in functions invoking atlas/solr/elasticsearch/zookeeper > *Predicted Benefits* > This improvement would bring the benefit of enabling logging to stdout (which > is supressed for now even if configured in spawned components), which could > be very useful when running Atlas in docker container or in Kubernetes/OC, > where some logging scraping is present. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (ATLAS-3610) Enable logging to multiple targets (logdir, stdout)
[ https://issues.apache.org/jira/browse/ATLAS-3610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mariusz Górski updated ATLAS-3610: -- Attachment: (was: ATLAS-3610-2.patch) > Enable logging to multiple targets (logdir, stdout) > --- > > Key: ATLAS-3610 > URL: https://issues.apache.org/jira/browse/ATLAS-3610 > Project: Atlas > Issue Type: Improvement > Components: atlas-core >Affects Versions: 3.0.0 >Reporter: Mariusz Górski >Priority: Minor > Fix For: 3.0.0 > > Attachments: ATLAS-3610-1.patch, ATLAS-3610-2.patch, ATLAS-3610.patch > > Time Spent: 50m > Remaining Estimate: 0h > > *Problem description* > When starting Atlas, Python scripts are invoked to spawn all necessary > processes. This is achieved with the use of _subprocess.Popen()_ in > _runProcess()_ function of _atlas_config.py_ script. While doing that, stdout > is being redirected to log files, which (for java processes) can override > log4j configuration (if configured with stdout handler). This creates > situation when log4j configuration cannot be relied upon and prevents logging > to stdout. > *Proposed solution* > * Add possibility to log to stdout by using multiple stdout/stderr handlers > * Adjust _runProcess()_ function from _atlas_config.py_ script (and all > functions relying on it) to provide _logconsole_ kwarg which will add > additional handler to Popen stdout/stderr redirection > * Add _ENABLE_LOGGING_TO_CONSOLE_ env variable (defaulting to _False_) for > backwards compatibility. > * Adjust _atlas_start.py_ script to make use of _ENABLE_LOGGING_TO_CONSOLE_ > and provide it in functions invoking atlas/solr/elasticsearch/zookeeper > *Predicted Benefits* > This improvement would bring the benefit of enabling logging to stdout (which > is supressed for now even if configured in spawned components), which could > be very useful when running Atlas in docker container or in Kubernetes/OC, > where some logging scraping is present. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (ATLAS-3610) Enable logging to multiple targets (logdir, stdout)
[ https://issues.apache.org/jira/browse/ATLAS-3610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mariusz Górski updated ATLAS-3610: -- Attachment: ATLAS-3610-1.patch > Enable logging to multiple targets (logdir, stdout) > --- > > Key: ATLAS-3610 > URL: https://issues.apache.org/jira/browse/ATLAS-3610 > Project: Atlas > Issue Type: Improvement > Components: atlas-core >Affects Versions: 3.0.0 >Reporter: Mariusz Górski >Priority: Minor > Fix For: 3.0.0 > > Attachments: ATLAS-3610-1.patch, ATLAS-3610.patch > > Time Spent: 50m > Remaining Estimate: 0h > > *Problem description* > When starting Atlas, Python scripts are invoked to spawn all necessary > processes. This is achieved with the use of _subprocess.Popen()_ in > _runProcess()_ function of _atlas_config.py_ script. While doing that, stdout > is being redirected to log files, which (for java processes) can override > log4j configuration (if configured with stdout handler). This creates > situation when log4j configuration cannot be relied upon and prevents logging > to stdout. > *Proposed solution* > * Add possibility to log to stdout by using multiple stdout/stderr handlers > * Adjust _runProcess()_ function from _atlas_config.py_ script (and all > functions relying on it) to provide _logconsole_ kwarg which will add > additional handler to Popen stdout/stderr redirection > * Add _ENABLE_LOGGING_TO_CONSOLE_ env variable (defaulting to _False_) for > backwards compatibility. > * Adjust _atlas_start.py_ script to make use of _ENABLE_LOGGING_TO_CONSOLE_ > and provide it in functions invoking atlas/solr/elasticsearch/zookeeper > *Predicted Benefits* > This improvement would bring the benefit of enabling logging to stdout (which > is supressed for now even if configured in spawned components), which could > be very useful when running Atlas in docker container or in Kubernetes/OC, > where some logging scraping is present. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (ATLAS-3610) Enable logging to multiple targets (logdir, stdout)
[ https://issues.apache.org/jira/browse/ATLAS-3610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mariusz Górski updated ATLAS-3610: -- Description: *Problem description* When starting Atlas, Python scripts are invoked to spawn all necessary processes. This is achieved with the use of _subprocess.Popen()_ in _runProcess()_ function of _atlas_config.py_ script. While doing that, stdout is being redirected to log files, which (for java processes) can override log4j configuration (if configured with stdout handler). This creates situation when log4j configuration cannot be relied upon and prevents logging to stdout. *Proposed solution* * Add possibility to log to stdout by using multiple stdout/stderr handlers * Adjust _runProcess()_ function from _atlas_config.py_ script (and all functions relying on it) to provide _logconsole_ kwarg which will add additional handler to Popen stdout/stderr redirection * Add _ENABLE_LOGGING_TO_CONSOLE_ env variable (defaulting to _False_) for backwards compatibility. * Adjust _atlas_start.py_ script to make use of _ENABLE_LOGGING_TO_CONSOLE_ and provide it in functions invoking atlas/solr/elasticsearch/zookeeper *Predicted Benefits* This improvement would bring the benefit of enabling logging to stdout (which is supressed for now even if configured in spawned components), which could be very useful when running Atlas in docker container or in Kubernetes/OC, where some logging scraping is present. was: *Problem description* When starting Atlas, Python scripts are invoked to spawn all necessary processes. This is achieved with the use of _subprocess.Popen()_ in _runProcess()_ function of _atlas_config.py_ script. While doing that, stdout is being redirected to log files, which (for java processes) can override log4j configuration (if configured with stdout handler). This creates situation when log4j configuration cannot be relied upon and prevents logging to stdout. *Proposed solution* * Add possibility to log to stdout by using multiple stdout/stderr handlers * Adjust _runProcess()_ function from _atlas_config.py_ script (and all functions relying on it) to provide _logconsole_ kwarg which will add additional handler to Popen stdout/stderr redirection * Add _ENABLE_LOGGING_TO_CONSOLE_ env variable (defaulting to _False_) for backwards compatibility. * Adjust _atlas_start.py_ script to make use of _ENABLE_LOGGING_TO_CONSOLE_ and provide it in functions invoking atlas/solr/elasticsearch/zookeeper *Predicted Benefits* This improvement would bring the benefit of having single place where logging logic is defined (log4j config file) and enable logging to stdout, which could be very useful when running Atlas in docker container or in Kubernetes/OC, where some logging scraping is present. > Enable logging to multiple targets (logdir, stdout) > --- > > Key: ATLAS-3610 > URL: https://issues.apache.org/jira/browse/ATLAS-3610 > Project: Atlas > Issue Type: Improvement > Components: atlas-core >Affects Versions: 3.0.0 >Reporter: Mariusz Górski >Priority: Minor > Fix For: 3.0.0 > > Attachments: ATLAS-3610.patch > > Time Spent: 50m > Remaining Estimate: 0h > > *Problem description* > When starting Atlas, Python scripts are invoked to spawn all necessary > processes. This is achieved with the use of _subprocess.Popen()_ in > _runProcess()_ function of _atlas_config.py_ script. While doing that, stdout > is being redirected to log files, which (for java processes) can override > log4j configuration (if configured with stdout handler). This creates > situation when log4j configuration cannot be relied upon and prevents logging > to stdout. > *Proposed solution* > * Add possibility to log to stdout by using multiple stdout/stderr handlers > * Adjust _runProcess()_ function from _atlas_config.py_ script (and all > functions relying on it) to provide _logconsole_ kwarg which will add > additional handler to Popen stdout/stderr redirection > * Add _ENABLE_LOGGING_TO_CONSOLE_ env variable (defaulting to _False_) for > backwards compatibility. > * Adjust _atlas_start.py_ script to make use of _ENABLE_LOGGING_TO_CONSOLE_ > and provide it in functions invoking atlas/solr/elasticsearch/zookeeper > *Predicted Benefits* > This improvement would bring the benefit of enabling logging to stdout (which > is supressed for now even if configured in spawned components), which could > be very useful when running Atlas in docker container or in Kubernetes/OC, > where some logging scraping is present. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (ATLAS-3610) Enable logging to multiple targets (logdir, stdout)
[ https://issues.apache.org/jira/browse/ATLAS-3610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mariusz Górski updated ATLAS-3610: -- Attachment: (was: ATLAS-3610.patch) > Enable logging to multiple targets (logdir, stdout) > --- > > Key: ATLAS-3610 > URL: https://issues.apache.org/jira/browse/ATLAS-3610 > Project: Atlas > Issue Type: Improvement > Components: atlas-core >Affects Versions: 3.0.0 >Reporter: Mariusz Górski >Priority: Minor > Fix For: 3.0.0 > > Attachments: ATLAS-3610.patch > > Time Spent: 50m > Remaining Estimate: 0h > > *Problem description* > When starting Atlas, Python scripts are invoked to spawn all necessary > processes. This is achieved with the use of _subprocess.Popen()_ in > _runProcess()_ function of _atlas_config.py_ script. While doing that, stdout > is being redirected to log files, which (for java processes) can override > log4j configuration (if configured with stdout handler). This creates > situation when log4j configuration cannot be relied upon and prevents logging > to stdout. > *Proposed solution* > * Add possibility to log to stdout by using multiple stdout/stderr handlers > * Adjust _runProcess()_ function from _atlas_config.py_ script (and all > functions relying on it) to provide _logconsole_ kwarg which will add > additional handler to Popen stdout/stderr redirection > * Add _ENABLE_LOGGING_TO_CONSOLE_ env variable (defaulting to _False_) for > backwards compatibility. > * Adjust _atlas_start.py_ script to make use of _ENABLE_LOGGING_TO_CONSOLE_ > and provide it in functions invoking atlas/solr/elasticsearch/zookeeper > *Predicted Benefits* > This improvement would bring the benefit of having single place where logging > logic is defined (log4j config file) and enable logging to stdout, which > could be very useful when running Atlas in docker container or in > Kubernetes/OC, where some logging scraping is present. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (ATLAS-3610) Enable logging to multiple targets (logdir, stdout)
[ https://issues.apache.org/jira/browse/ATLAS-3610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mariusz Górski updated ATLAS-3610: -- Description: *Problem description* When starting Atlas, Python scripts are invoked to spawn all necessary processes. This is achieved with the use of _subprocess.Popen()_ in _runProcess()_ function of _atlas_config.py_ script. While doing that, stdout is being redirected to log files, which (for java processes) can override log4j configuration (if configured with stdout handler). This creates situation when log4j configuration cannot be relied upon and prevents logging to stdout. *Proposed solution* * Add possibility to log to stdout by using multiple stdout/stderr handlers * Adjust _runProcess()_ function from _atlas_config.py_ script (and all functions relying on it) to provide _logconsole_ kwarg which will add additional handler to Popen stdout/stderr redirection * Add _ENABLE_LOGGING_TO_CONSOLE_ env variable (defaulting to _False_) for backwards compatibility. * Adjust _atlas_start.py_ script to make use of _ENABLE_LOGGING_TO_CONSOLE_ and provide it in functions invoking atlas/solr/elasticsearch/zookeeper *Predicted Benefits* This improvement would bring the benefit of having single place where logging logic is defined (log4j config file) and enable logging to stdout, which could be very useful when running Atlas in docker container or in Kubernetes/OC, where some logging scraping is present. was: *Problem description* When starting Atlas, Python scripts are invoked to spawn all necessary processes. This is achieved with the use of _subprocess.Popen()_ in _runProcess()_ function of _atlas_config.py_ script. While doing that, stdout is being redirected to log files, which (for java processes) can override log4j configuration (if configured with stdout handler). This creates situation when log4j configuration cannot be relied upon and prevents logging to stdout. *Proposed solution* * Add possibility to log to stdout by using multiple stdout/stderr handlers * add _MultiOutput_ class that enables multiple output handlers in _Popen_ * Adjust _runProcess()_ function from _atlas_config.py_ script (and all functions relying on it) to provide _stdout_ kwarg which will add additional handler to Popen stdout/stderr redirection * Add _ENABLE_STDOUT_LOGGING_ env variable (defaulting to False) for backwards compatibility. * Adjust _atlas_start.py_ script to make use of _ENABLE_STDOUT_LOGGING_ and provide it in functions invoking atlas/solr/elasticsearch/zookeeper *Predicted Benefits* This improvement would bring the benefit of having single place where logging logic is defined (log4j config file) and enable logging to stdout, which could be very useful when running Atlas in docker container or in Kubernetes/OC, where some logging scraping is present. > Enable logging to multiple targets (logdir, stdout) > --- > > Key: ATLAS-3610 > URL: https://issues.apache.org/jira/browse/ATLAS-3610 > Project: Atlas > Issue Type: Improvement > Components: atlas-core >Affects Versions: 3.0.0 >Reporter: Mariusz Górski >Priority: Minor > Fix For: 3.0.0 > > Attachments: ATLAS-3610.patch > > Time Spent: 0.5h > Remaining Estimate: 0h > > *Problem description* > When starting Atlas, Python scripts are invoked to spawn all necessary > processes. This is achieved with the use of _subprocess.Popen()_ in > _runProcess()_ function of _atlas_config.py_ script. While doing that, stdout > is being redirected to log files, which (for java processes) can override > log4j configuration (if configured with stdout handler). This creates > situation when log4j configuration cannot be relied upon and prevents logging > to stdout. > *Proposed solution* > * Add possibility to log to stdout by using multiple stdout/stderr handlers > * Adjust _runProcess()_ function from _atlas_config.py_ script (and all > functions relying on it) to provide _logconsole_ kwarg which will add > additional handler to Popen stdout/stderr redirection > * Add _ENABLE_LOGGING_TO_CONSOLE_ env variable (defaulting to _False_) for > backwards compatibility. > * Adjust _atlas_start.py_ script to make use of _ENABLE_LOGGING_TO_CONSOLE_ > and provide it in functions invoking atlas/solr/elasticsearch/zookeeper > *Predicted Benefits* > This improvement would bring the benefit of having single place where logging > logic is defined (log4j config file) and enable logging to stdout, which > could be very useful when running Atlas in docker container or in > Kubernetes/OC, where some logging scraping is present. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (ATLAS-3610) Enable logging to multiple targets (logdir, stdout)
[ https://issues.apache.org/jira/browse/ATLAS-3610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mariusz Górski updated ATLAS-3610: -- Description: *Problem description* When starting Atlas, Python scripts are invoked to spawn all necessary processes. This is achieved with the use of _subprocess.Popen()_ in _runProcess()_ function of _atlas_config.py_ script. While doing that, stdout is being redirected to log files, which (for java processes) can override log4j configuration (if configured with stdout handler). This creates situation when log4j configuration cannot be relied upon and prevents logging to stdout. *Proposed solution* * Add possibility to log to stdout by using multiple stdout/stderr handlers * add _MultiOutput_ class that enables multiple output handlers in _Popen_ * Adjust _runProcess()_ function from _atlas_config.py_ script (and all functions relying on it) to provide _stdout_ kwarg which will add additional handler to Popen stdout/stderr redirection * Add _ENABLE_STDOUT_LOGGING_ env variable (defaulting to False) for backwards compatibility. * Adjust _atlas_start.py_ script to make use of _ENABLE_STDOUT_LOGGING_ and provide it in functions invoking atlas/solr/elasticsearch/zookeeper *Predicted Benefits* This improvement would bring the benefit of having single place where logging logic is defined (log4j config file) and enable logging to stdout, which could be very useful when running Atlas in docker container or in Kubernetes/OC, where some logging scraping is present. was: *Problem description* When starting Atlas, Python scripts are invoked to spawn all necessary processes. This is achieved with the use of _subprocess.Popen()_ in _runProcess()_ function of _atlas_config.py_ script. While doing that, stdout is being redirected to log files, which (for java processes) can override log4j configuration (if configured with stdout handler). This creates situation when log4j configuration cannot be relied upon and prevents logging to stdout. *Proposed solution* * Add possibility to unify logging logic by relying solely on log4j configuration. * Adjust _runProcess()_ function from _atlas_config.py_ script (and all functions relying on it) not to redirect stdout and stderr of spawned processes when logdir is _None_ * Add _RELY_ON_LOG4J_ONLY_ env variable (defaulting to False) for backwards compatibility. * Adjust _atlas_start.py_ script to make use of _RELY_ON_LOG4J_ONLY_ and set logdir as _None_ when _RELY_ON_LOG4J_ONLY_ is true *Predicted Benefits* This improvement would bring the benefit of having single place where logging logic is defined (log4j config file) and enable logging to stdout, which could be very useful when running Atlas in docker container or in Kubernetes/OC, where some logging scraping is present. > Enable logging to multiple targets (logdir, stdout) > --- > > Key: ATLAS-3610 > URL: https://issues.apache.org/jira/browse/ATLAS-3610 > Project: Atlas > Issue Type: Improvement > Components: atlas-core >Affects Versions: 3.0.0 >Reporter: Mariusz Górski >Priority: Minor > Fix For: 3.0.0 > > Attachments: ATLAS-3610.patch > > Time Spent: 0.5h > Remaining Estimate: 0h > > *Problem description* > When starting Atlas, Python scripts are invoked to spawn all necessary > processes. This is achieved with the use of _subprocess.Popen()_ in > _runProcess()_ function of _atlas_config.py_ script. While doing that, stdout > is being redirected to log files, which (for java processes) can override > log4j configuration (if configured with stdout handler). This creates > situation when log4j configuration cannot be relied upon and prevents logging > to stdout. > *Proposed solution* > * Add possibility to log to stdout by using multiple stdout/stderr handlers > * add _MultiOutput_ class that enables multiple output handlers in _Popen_ > * Adjust _runProcess()_ function from _atlas_config.py_ script (and all > functions relying on it) to provide _stdout_ kwarg which will add additional > handler to Popen stdout/stderr redirection > * Add _ENABLE_STDOUT_LOGGING_ env variable (defaulting to False) for > backwards compatibility. > * Adjust _atlas_start.py_ script to make use of _ENABLE_STDOUT_LOGGING_ and > provide it in functions invoking atlas/solr/elasticsearch/zookeeper > *Predicted Benefits* > This improvement would bring the benefit of having single place where logging > logic is defined (log4j config file) and enable logging to stdout, which > could be very useful when running Atlas in docker container or in > Kubernetes/OC, where some logging scraping is present. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (ATLAS-3610) Enable logging to multiple targets (logdir, stdout)
[ https://issues.apache.org/jira/browse/ATLAS-3610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mariusz Górski updated ATLAS-3610: -- Summary: Enable logging to multiple targets (logdir, stdout) (was: Unify logging logic by relying solely on log4j configuration for java processes) > Enable logging to multiple targets (logdir, stdout) > --- > > Key: ATLAS-3610 > URL: https://issues.apache.org/jira/browse/ATLAS-3610 > Project: Atlas > Issue Type: Improvement > Components: atlas-core >Affects Versions: 3.0.0 >Reporter: Mariusz Górski >Priority: Minor > Fix For: 3.0.0 > > Attachments: ATLAS-3610.patch > > Time Spent: 0.5h > Remaining Estimate: 0h > > *Problem description* > When starting Atlas, Python scripts are invoked to spawn all necessary > processes. This is achieved with the use of _subprocess.Popen()_ in > _runProcess()_ function of _atlas_config.py_ script. While doing that, stdout > is being redirected to log files, which (for java processes) can override > log4j configuration (if configured with stdout handler). This creates > situation when log4j configuration cannot be relied upon and prevents logging > to stdout. > *Proposed solution* > * Add possibility to unify logging logic by relying solely on log4j > configuration. > * Adjust _runProcess()_ function from _atlas_config.py_ script (and all > functions relying on it) not to redirect stdout and stderr of spawned > processes when logdir is _None_ > * Add _RELY_ON_LOG4J_ONLY_ env variable (defaulting to False) for backwards > compatibility. > * Adjust _atlas_start.py_ script to make use of _RELY_ON_LOG4J_ONLY_ and set > logdir as _None_ when _RELY_ON_LOG4J_ONLY_ is true > *Predicted Benefits* > This improvement would bring the benefit of having single place where logging > logic is defined (log4j config file) and enable logging to stdout, which > could be very useful when running Atlas in docker container or in > Kubernetes/OC, where some logging scraping is present. -- This message was sent by Atlassian Jira (v8.3.4#803005)
