[jira] [Commented] (ATLAS-3610) Enable logging to multiple targets (logdir, stdout)
[ https://issues.apache.org/jira/browse/ATLAS-3610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17219062#comment-17219062 ] Tobias commented on ATLAS-3610: --- hi [~kevalbhatt] or [~mrp1nk], is there any way this patch can be merged into one of the available atlas branches? we need this feature to log to console because we have no access to our pod and therefore the logs where we will run an Apache Atlas container. > Enable logging to multiple targets (logdir, stdout) > --- > > Key: ATLAS-3610 > URL: https://issues.apache.org/jira/browse/ATLAS-3610 > Project: Atlas > Issue Type: Improvement > Components: atlas-core >Affects Versions: 3.0.0 >Reporter: Mariusz Górski >Priority: Minor > Fix For: 3.0.0, 2.2.0 > > Attachments: ATLAS-3610-1.patch, ATLAS-3610-2.patch, > ATLAS-3610-3.patch, ATLAS-3610.patch > > Time Spent: 50m > Remaining Estimate: 0h > > *Problem description* > When starting Atlas, Python scripts are invoked to spawn all necessary > processes. This is achieved with the use of _subprocess.Popen()_ in > _runProcess()_ function of _atlas_config.py_ script. While doing that, stdout > is being redirected to log files, which (for java processes) can override > log4j configuration (if configured with stdout handler). This creates > situation when log4j configuration cannot be relied upon and prevents logging > to stdout. > *Proposed solution* > * Add possibility to log to stdout by using multiple stdout/stderr handlers > * Adjust _runProcess()_ function from _atlas_config.py_ script (and all > functions relying on it) to provide _logconsole_ kwarg which will add > additional handler to Popen stdout/stderr redirection > * Add _ENABLE_LOGGING_TO_CONSOLE_ env variable (defaulting to _False_) for > backwards compatibility. > * Adjust _atlas_start.py_ script to make use of _ENABLE_LOGGING_TO_CONSOLE_ > and provide it in functions invoking atlas/solr/elasticsearch/zookeeper > *Predicted Benefits* > This improvement would bring the benefit of enabling logging to stdout (which > is supressed for now even if configured in spawned components), which could > be very useful when running Atlas in docker container or in Kubernetes/OC, > where some logging scraping is present. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (ATLAS-3610) Enable logging to multiple targets (logdir, stdout)
[ https://issues.apache.org/jira/browse/ATLAS-3610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17062480#comment-17062480 ] Mariusz Górski commented on ATLAS-3610: --- [~kevalbhatt] does this patch resolve the issue and qualifies for merge ? & for future reference - is there a way to trigger pipeline to spot failure before the code is merged ? > Enable logging to multiple targets (logdir, stdout) > --- > > Key: ATLAS-3610 > URL: https://issues.apache.org/jira/browse/ATLAS-3610 > Project: Atlas > Issue Type: Improvement > Components: atlas-core >Affects Versions: 3.0.0 >Reporter: Mariusz Górski >Priority: Minor > Fix For: 2.1.0, 3.0.0 > > Attachments: ATLAS-3610-1.patch, ATLAS-3610-2.patch, > ATLAS-3610-3.patch, ATLAS-3610.patch > > Time Spent: 50m > Remaining Estimate: 0h > > *Problem description* > When starting Atlas, Python scripts are invoked to spawn all necessary > processes. This is achieved with the use of _subprocess.Popen()_ in > _runProcess()_ function of _atlas_config.py_ script. While doing that, stdout > is being redirected to log files, which (for java processes) can override > log4j configuration (if configured with stdout handler). This creates > situation when log4j configuration cannot be relied upon and prevents logging > to stdout. > *Proposed solution* > * Add possibility to log to stdout by using multiple stdout/stderr handlers > * Adjust _runProcess()_ function from _atlas_config.py_ script (and all > functions relying on it) to provide _logconsole_ kwarg which will add > additional handler to Popen stdout/stderr redirection > * Add _ENABLE_LOGGING_TO_CONSOLE_ env variable (defaulting to _False_) for > backwards compatibility. > * Adjust _atlas_start.py_ script to make use of _ENABLE_LOGGING_TO_CONSOLE_ > and provide it in functions invoking atlas/solr/elasticsearch/zookeeper > *Predicted Benefits* > This improvement would bring the benefit of enabling logging to stdout (which > is supressed for now even if configured in spawned components), which could > be very useful when running Atlas in docker container or in Kubernetes/OC, > where some logging scraping is present. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (ATLAS-3610) Enable logging to multiple targets (logdir, stdout)
[ https://issues.apache.org/jira/browse/ATLAS-3610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17046685#comment-17046685 ] Mariusz Górski commented on ATLAS-3610: --- Thanks for the information [~kevalbhatt] We have successfully run Atlas with this patch in both embedded and not embedded mode in our env. I noticed, however, how tests should be adjusted so that they do not fail - you'll find it in ATLAS-3610-3.patch Re-run tests on localhost and seems the issue is gone: /usr/lib/jvm/java-8-openjdk-amd64/bin/java -Dmaven.multiModuleProjectDirectory=/home/mariusz/IdeaProjects/atlas/distro -Dmaven.home=/snap/intellij-idea-community/208/plugins/maven/lib/maven3 -Dclassworlds.conf=/snap/intellij-idea-community/208/plugins/maven/lib/maven3/bin/m2.conf -Dmaven.ext.class.path=/snap/intellij-idea-community/208/plugins/maven/lib/maven-event-listener.jar -javaagent:/snap/intellij-idea-community/208/lib/idea_rt.jar=33041:/snap/intellij-idea-community/208/bin -Dfile.encoding=UTF-8 -classpath /snap/intellij-idea-community/208/plugins/maven/lib/maven3/boot/plexus-classworlds-2.6.0.jar org.codehaus.classworlds.Launcher -Didea.version2019.3.3 test [INFO] Scanning for projects... [INFO] [INFO] ---< org.apache.atlas:atlas-distro > [INFO] Building Apache Atlas Distribution 3.0.0-SNAPSHOT [INFO] [ jar ]- [INFO] [INFO] --- maven-enforcer-plugin:3.0.0-M1:enforce (enforce-versions) @ atlas-distro --- [INFO] [INFO] --- buildnumber-maven-plugin:1.4:create (default) @ atlas-distro --- [INFO] Executing: /bin/sh -c cd '/home/mariusz/IdeaProjects/atlas/distro' && 'git' 'rev-parse' '--verify' 'HEAD' [INFO] Working directory: /home/mariusz/IdeaProjects/atlas/distro [INFO] Storing buildNumber: 4229cb3c7a9e7109c883431bc5fb502339d077ec at timestamp: 1582813388815 [INFO] Storing buildScmBranch: master [INFO] [INFO] --- apache-rat-plugin:0.13:check (rat-check) @ atlas-distro --- [INFO] Enabled default license matchers. [INFO] Will parse SCM ignores for exclusions... [INFO] Finished adding exclusions from SCM ignore files. [INFO] 62 implicit excludes (use -debug for more details). [INFO] 55 explicit excludes (use -debug for more details). [INFO] 41 resources included (use -debug for more details) [INFO] Rat check: Summary over all files. Unapproved: 0, unknown: 0, generated: 0, approved: 36 licenses. [INFO] [INFO] --- maven-remote-resources-plugin:1.5:process (default) @ atlas-distro --- [INFO] [INFO] --- maven-resources-plugin:2.7:resources (default-resources) @ atlas-distro --- [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] Copying 11 resources to bin [INFO] Copying 16 resources to conf [INFO] Copying 3 resources [INFO] [INFO] --- maven-compiler-plugin:3.7.0:compile (default-compile) @ atlas-distro --- [INFO] No sources to compile [INFO] [INFO] --- maven-resources-plugin:2.7:testResources (default-testResources) @ atlas-distro --- [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] skip non existing resourceDirectory /home/mariusz/IdeaProjects/atlas/distro/src/test/resources [INFO] Copying 3 resources [INFO] [INFO] --- maven-compiler-plugin:3.7.0:testCompile (default-testCompile) @ atlas-distro --- [INFO] No sources to compile [INFO] [INFO] --- maven-surefire-plugin:2.18.1:test (default-test) @ atlas-distro --- [INFO] [INFO] --- exec-maven-plugin:1.2.1:exec (python-test) @ atlas-distro --- test_jar_java_lookups_fail (TestMetadata.TestMetadata) ... ok test_jar_java_lookups_succeed_from_path (TestMetadata.TestMetadata) ... ok test_main_default (TestMetadata.TestMetadata) ... Logging to console is disabled. get_config_mock_side_effect (atlas.graph.storage.backend) Apache Atlas Server started!!! ok test_main_embedded (TestMetadata.TestMetadata) ... Logging to console is disabled. get_config_mock_side_effect (atlas.graph.storage.backend) configured for local hbase. hbase started. configured for local solr. get_config_mock_side_effect (atlas.graph.index.search.solr.zookeeper-url) solr started. setting up solr collections... Apache Atlas Server started!!! ok -- Ran 4 tests in 0.005s OK [INFO] [INFO] BUILD SUCCESS [INFO] [INFO] Total time: 5.710 s [INFO] Finished at: 2020-02-27T15:23:12+01:00 [INFO] Thanks for the support and patience with this ! cc [~mehul] [~vishal.suvagia] [~Rahul_FI] > Enable logging to multiple targets (logdir, stdout) > --- > > Key: ATLAS-3610 > URL: https://issues.apache.org/jira/browse/ATLAS-3610 >
[jira] [Commented] (ATLAS-3610) Enable logging to multiple targets (logdir, stdout)
[ https://issues.apache.org/jira/browse/ATLAS-3610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17046613#comment-17046613 ] ASF subversion and git services commented on ATLAS-3610: Commit 066d67756a5d40cabe190c0f95d344eb74161df5 in atlas's branch refs/heads/branch-2.0 from Keval Bhatt [ https://gitbox.apache.org/repos/asf?p=atlas.git;h=066d677 ] Revert "ATLAS-3610 : Multiple_stdout_stderr_handlers_including_console" distro test cases are failing This reverts commit 6b503eeb8866b67d0cb09d60c389230a28c1efbc. (cherry picked from commit 3c9a437094f800467fa80cdccefefb6538839496) > Enable logging to multiple targets (logdir, stdout) > --- > > Key: ATLAS-3610 > URL: https://issues.apache.org/jira/browse/ATLAS-3610 > Project: Atlas > Issue Type: Improvement > Components: atlas-core >Affects Versions: 3.0.0 >Reporter: Mariusz Górski >Priority: Minor > Fix For: 2.1.0, 3.0.0 > > Attachments: ATLAS-3610-1.patch, ATLAS-3610-2.patch, ATLAS-3610.patch > > Time Spent: 50m > Remaining Estimate: 0h > > *Problem description* > When starting Atlas, Python scripts are invoked to spawn all necessary > processes. This is achieved with the use of _subprocess.Popen()_ in > _runProcess()_ function of _atlas_config.py_ script. While doing that, stdout > is being redirected to log files, which (for java processes) can override > log4j configuration (if configured with stdout handler). This creates > situation when log4j configuration cannot be relied upon and prevents logging > to stdout. > *Proposed solution* > * Add possibility to log to stdout by using multiple stdout/stderr handlers > * Adjust _runProcess()_ function from _atlas_config.py_ script (and all > functions relying on it) to provide _logconsole_ kwarg which will add > additional handler to Popen stdout/stderr redirection > * Add _ENABLE_LOGGING_TO_CONSOLE_ env variable (defaulting to _False_) for > backwards compatibility. > * Adjust _atlas_start.py_ script to make use of _ENABLE_LOGGING_TO_CONSOLE_ > and provide it in functions invoking atlas/solr/elasticsearch/zookeeper > *Predicted Benefits* > This improvement would bring the benefit of enabling logging to stdout (which > is supressed for now even if configured in spawned components), which could > be very useful when running Atlas in docker container or in Kubernetes/OC, > where some logging scraping is present. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (ATLAS-3610) Enable logging to multiple targets (logdir, stdout)
[ https://issues.apache.org/jira/browse/ATLAS-3610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17046604#comment-17046604 ] Keval Bhatt commented on ATLAS-3610: Hi [~mrp1nk], I have committed your patch but looks like related test-cases are failing for the change, you can refer the errors in this pre-commit run -> [https://builds.apache.org/job/PreCommit-ATLAS-Build-Test/1663/console] I have created a revert patch for your change and done a pre-commit run with the revert patch which ran successfully -> [https://builds.apache.org/job/PreCommit-ATLAS-Build-Test/1671/console] For now, I am reverting your commit. > Enable logging to multiple targets (logdir, stdout) > --- > > Key: ATLAS-3610 > URL: https://issues.apache.org/jira/browse/ATLAS-3610 > Project: Atlas > Issue Type: Improvement > Components: atlas-core >Affects Versions: 3.0.0 >Reporter: Mariusz Górski >Priority: Minor > Fix For: 2.1.0, 3.0.0 > > Attachments: ATLAS-3610-1.patch, ATLAS-3610-2.patch, ATLAS-3610.patch > > Time Spent: 50m > Remaining Estimate: 0h > > *Problem description* > When starting Atlas, Python scripts are invoked to spawn all necessary > processes. This is achieved with the use of _subprocess.Popen()_ in > _runProcess()_ function of _atlas_config.py_ script. While doing that, stdout > is being redirected to log files, which (for java processes) can override > log4j configuration (if configured with stdout handler). This creates > situation when log4j configuration cannot be relied upon and prevents logging > to stdout. > *Proposed solution* > * Add possibility to log to stdout by using multiple stdout/stderr handlers > * Adjust _runProcess()_ function from _atlas_config.py_ script (and all > functions relying on it) to provide _logconsole_ kwarg which will add > additional handler to Popen stdout/stderr redirection > * Add _ENABLE_LOGGING_TO_CONSOLE_ env variable (defaulting to _False_) for > backwards compatibility. > * Adjust _atlas_start.py_ script to make use of _ENABLE_LOGGING_TO_CONSOLE_ > and provide it in functions invoking atlas/solr/elasticsearch/zookeeper > *Predicted Benefits* > This improvement would bring the benefit of enabling logging to stdout (which > is supressed for now even if configured in spawned components), which could > be very useful when running Atlas in docker container or in Kubernetes/OC, > where some logging scraping is present. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (ATLAS-3610) Enable logging to multiple targets (logdir, stdout)
[ https://issues.apache.org/jira/browse/ATLAS-3610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17037501#comment-17037501 ] Mariusz Górski commented on ATLAS-3610: --- I have made another go (ATLAS-3610-2.patch) which should scope only *atlas_start* and *cputil* scripts. All remaining invocations are left untouched as logconsole doesn't play well with processes requiring user input. I've added the necessary option to pom.xml and atlas-env.sh (defaulting to false). Please test now [~Rahul_FI] [~vishal.suvagia] [~mehul] > Enable logging to multiple targets (logdir, stdout) > --- > > Key: ATLAS-3610 > URL: https://issues.apache.org/jira/browse/ATLAS-3610 > Project: Atlas > Issue Type: Improvement > Components: atlas-core >Affects Versions: 3.0.0 >Reporter: Mariusz Górski >Priority: Minor > Fix For: 3.0.0 > > Attachments: ATLAS-3610-1.patch, ATLAS-3610-2.patch, ATLAS-3610.patch > > Time Spent: 50m > Remaining Estimate: 0h > > *Problem description* > When starting Atlas, Python scripts are invoked to spawn all necessary > processes. This is achieved with the use of _subprocess.Popen()_ in > _runProcess()_ function of _atlas_config.py_ script. While doing that, stdout > is being redirected to log files, which (for java processes) can override > log4j configuration (if configured with stdout handler). This creates > situation when log4j configuration cannot be relied upon and prevents logging > to stdout. > *Proposed solution* > * Add possibility to log to stdout by using multiple stdout/stderr handlers > * Adjust _runProcess()_ function from _atlas_config.py_ script (and all > functions relying on it) to provide _logconsole_ kwarg which will add > additional handler to Popen stdout/stderr redirection > * Add _ENABLE_LOGGING_TO_CONSOLE_ env variable (defaulting to _False_) for > backwards compatibility. > * Adjust _atlas_start.py_ script to make use of _ENABLE_LOGGING_TO_CONSOLE_ > and provide it in functions invoking atlas/solr/elasticsearch/zookeeper > *Predicted Benefits* > This improvement would bring the benefit of enabling logging to stdout (which > is supressed for now even if configured in spawned components), which could > be very useful when running Atlas in docker container or in Kubernetes/OC, > where some logging scraping is present. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (ATLAS-3610) Enable logging to multiple targets (logdir, stdout)
[ https://issues.apache.org/jira/browse/ATLAS-3610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17037008#comment-17037008 ] Mariusz Górski commented on ATLAS-3610: --- I think in order for logging to console to work, you need the script to be a long running one. So if you need to have access to console you shoud run it like `python2 atlas_start. py &`. Does it fix your problem? > Enable logging to multiple targets (logdir, stdout) > --- > > Key: ATLAS-3610 > URL: https://issues.apache.org/jira/browse/ATLAS-3610 > Project: Atlas > Issue Type: Improvement > Components: atlas-core >Affects Versions: 3.0.0 >Reporter: Mariusz Górski >Priority: Minor > Fix For: 3.0.0 > > Attachments: ATLAS-3610-1.patch, ATLAS-3610.patch > > Time Spent: 50m > Remaining Estimate: 0h > > *Problem description* > When starting Atlas, Python scripts are invoked to spawn all necessary > processes. This is achieved with the use of _subprocess.Popen()_ in > _runProcess()_ function of _atlas_config.py_ script. While doing that, stdout > is being redirected to log files, which (for java processes) can override > log4j configuration (if configured with stdout handler). This creates > situation when log4j configuration cannot be relied upon and prevents logging > to stdout. > *Proposed solution* > * Add possibility to log to stdout by using multiple stdout/stderr handlers > * Adjust _runProcess()_ function from _atlas_config.py_ script (and all > functions relying on it) to provide _logconsole_ kwarg which will add > additional handler to Popen stdout/stderr redirection > * Add _ENABLE_LOGGING_TO_CONSOLE_ env variable (defaulting to _False_) for > backwards compatibility. > * Adjust _atlas_start.py_ script to make use of _ENABLE_LOGGING_TO_CONSOLE_ > and provide it in functions invoking atlas/solr/elasticsearch/zookeeper > *Predicted Benefits* > This improvement would bring the benefit of enabling logging to stdout (which > is supressed for now even if configured in spawned components), which could > be very useful when running Atlas in docker container or in Kubernetes/OC, > where some logging scraping is present. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (ATLAS-3610) Enable logging to multiple targets (logdir, stdout)
[
https://issues.apache.org/jira/browse/ATLAS-3610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17036981#comment-17036981
]
Rahul Kurup commented on ATLAS-3610:
Hi [~mrp1nk], I've applied [^ATLAS-3610-1.patch] on my local, and was able to
start the application and login successfully to Atlas UI.
Observed below differences with and without the changes:
* Without the change, executing the {{atlas_start.py}} script does a clean
exit and the command prompt is available to execute required commands.
* With the change, executing the {{atlas_start.py}} script does not exit
neatly irrespective of whether console-logging is enabled / disabled. Is there
a way to achieve this functionality with a configuration ? This can be required
in cases where atlas_start.py script is being called from other scripts which
will require a clean exit for {{atlas_start.py}}
With the change, When I try to run quick_start.py, no entries are made in the
Atlas application and the script exits with below output.
* With adding export ENABLE_LOGGING_TO_CONSOLE=false
{noformat}
$ ./quick_start.py
No sample data added to Apache Atlas Server.
{noformat}
* With adding export ENABLE_LOGGING_TO_CONSOLE=true
{noformat}
$ ./quick_start.py
Couldn't get a console object for user input
No sample data added to Apache Atlas Server.
{noformat}
> Enable logging to multiple targets (logdir, stdout)
> ---
>
> Key: ATLAS-3610
> URL: https://issues.apache.org/jira/browse/ATLAS-3610
> Project: Atlas
> Issue Type: Improvement
> Components: atlas-core
>Affects Versions: 3.0.0
>Reporter: Mariusz Górski
>Priority: Minor
> Fix For: 3.0.0
>
> Attachments: ATLAS-3610-1.patch, ATLAS-3610.patch
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
> *Problem description*
> When starting Atlas, Python scripts are invoked to spawn all necessary
> processes. This is achieved with the use of _subprocess.Popen()_ in
> _runProcess()_ function of _atlas_config.py_ script. While doing that, stdout
> is being redirected to log files, which (for java processes) can override
> log4j configuration (if configured with stdout handler). This creates
> situation when log4j configuration cannot be relied upon and prevents logging
> to stdout.
> *Proposed solution*
> * Add possibility to log to stdout by using multiple stdout/stderr handlers
> * Adjust _runProcess()_ function from _atlas_config.py_ script (and all
> functions relying on it) to provide _logconsole_ kwarg which will add
> additional handler to Popen stdout/stderr redirection
> * Add _ENABLE_LOGGING_TO_CONSOLE_ env variable (defaulting to _False_) for
> backwards compatibility.
> * Adjust _atlas_start.py_ script to make use of _ENABLE_LOGGING_TO_CONSOLE_
> and provide it in functions invoking atlas/solr/elasticsearch/zookeeper
> *Predicted Benefits*
> This improvement would bring the benefit of enabling logging to stdout (which
> is supressed for now even if configured in spawned components), which could
> be very useful when running Atlas in docker container or in Kubernetes/OC,
> where some logging scraping is present.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (ATLAS-3610) Enable logging to multiple targets (logdir, stdout)
[ https://issues.apache.org/jira/browse/ATLAS-3610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17036400#comment-17036400 ] Mariusz Górski commented on ATLAS-3610: --- [~vishal.suvagia] done for all invocations of mc.java , thanks for catching this ! > Enable logging to multiple targets (logdir, stdout) > --- > > Key: ATLAS-3610 > URL: https://issues.apache.org/jira/browse/ATLAS-3610 > Project: Atlas > Issue Type: Improvement > Components: atlas-core >Affects Versions: 3.0.0 >Reporter: Mariusz Górski >Priority: Minor > Fix For: 3.0.0 > > Attachments: ATLAS-3610.patch > > Time Spent: 50m > Remaining Estimate: 0h > > *Problem description* > When starting Atlas, Python scripts are invoked to spawn all necessary > processes. This is achieved with the use of _subprocess.Popen()_ in > _runProcess()_ function of _atlas_config.py_ script. While doing that, stdout > is being redirected to log files, which (for java processes) can override > log4j configuration (if configured with stdout handler). This creates > situation when log4j configuration cannot be relied upon and prevents logging > to stdout. > *Proposed solution* > * Add possibility to log to stdout by using multiple stdout/stderr handlers > * Adjust _runProcess()_ function from _atlas_config.py_ script (and all > functions relying on it) to provide _logconsole_ kwarg which will add > additional handler to Popen stdout/stderr redirection > * Add _ENABLE_LOGGING_TO_CONSOLE_ env variable (defaulting to _False_) for > backwards compatibility. > * Adjust _atlas_start.py_ script to make use of _ENABLE_LOGGING_TO_CONSOLE_ > and provide it in functions invoking atlas/solr/elasticsearch/zookeeper > *Predicted Benefits* > This improvement would bring the benefit of enabling logging to stdout (which > is supressed for now even if configured in spawned components), which could > be very useful when running Atlas in docker container or in Kubernetes/OC, > where some logging scraping is present. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (ATLAS-3610) Enable logging to multiple targets (logdir, stdout)
[ https://issues.apache.org/jira/browse/ATLAS-3610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17036399#comment-17036399 ] Mariusz Górski commented on ATLAS-3610: --- [~mehul] I've just realised that my change somehow didn't work with atlas-env.sh so rewritten it for complete support, please check now with new patch > Enable logging to multiple targets (logdir, stdout) > --- > > Key: ATLAS-3610 > URL: https://issues.apache.org/jira/browse/ATLAS-3610 > Project: Atlas > Issue Type: Improvement > Components: atlas-core >Affects Versions: 3.0.0 >Reporter: Mariusz Górski >Priority: Minor > Fix For: 3.0.0 > > Attachments: ATLAS-3610.patch > > Time Spent: 50m > Remaining Estimate: 0h > > *Problem description* > When starting Atlas, Python scripts are invoked to spawn all necessary > processes. This is achieved with the use of _subprocess.Popen()_ in > _runProcess()_ function of _atlas_config.py_ script. While doing that, stdout > is being redirected to log files, which (for java processes) can override > log4j configuration (if configured with stdout handler). This creates > situation when log4j configuration cannot be relied upon and prevents logging > to stdout. > *Proposed solution* > * Add possibility to log to stdout by using multiple stdout/stderr handlers > * Adjust _runProcess()_ function from _atlas_config.py_ script (and all > functions relying on it) to provide _logconsole_ kwarg which will add > additional handler to Popen stdout/stderr redirection > * Add _ENABLE_LOGGING_TO_CONSOLE_ env variable (defaulting to _False_) for > backwards compatibility. > * Adjust _atlas_start.py_ script to make use of _ENABLE_LOGGING_TO_CONSOLE_ > and provide it in functions invoking atlas/solr/elasticsearch/zookeeper > *Predicted Benefits* > This improvement would bring the benefit of enabling logging to stdout (which > is supressed for now even if configured in spawned components), which could > be very useful when running Atlas in docker container or in Kubernetes/OC, > where some logging scraping is present. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (ATLAS-3610) Enable logging to multiple targets (logdir, stdout)
[
https://issues.apache.org/jira/browse/ATLAS-3610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17036283#comment-17036283
]
Vishal Suvagia commented on ATLAS-3610:
---
[~mrp1nk], the
[cputil.py|https://github.com/apache/atlas/blob/master/distro/src/bin/cputil.py#L21]
also imports the {{atlas_config.py}}, can you kindly update the references in
{{cputil.py}} as part of the fix.
> Enable logging to multiple targets (logdir, stdout)
> ---
>
> Key: ATLAS-3610
> URL: https://issues.apache.org/jira/browse/ATLAS-3610
> Project: Atlas
> Issue Type: Improvement
> Components: atlas-core
>Affects Versions: 3.0.0
>Reporter: Mariusz Górski
>Priority: Minor
> Fix For: 3.0.0
>
> Attachments: ATLAS-3610.patch
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
> *Problem description*
> When starting Atlas, Python scripts are invoked to spawn all necessary
> processes. This is achieved with the use of _subprocess.Popen()_ in
> _runProcess()_ function of _atlas_config.py_ script. While doing that, stdout
> is being redirected to log files, which (for java processes) can override
> log4j configuration (if configured with stdout handler). This creates
> situation when log4j configuration cannot be relied upon and prevents logging
> to stdout.
> *Proposed solution*
> * Add possibility to log to stdout by using multiple stdout/stderr handlers
> * Adjust _runProcess()_ function from _atlas_config.py_ script (and all
> functions relying on it) to provide _logconsole_ kwarg which will add
> additional handler to Popen stdout/stderr redirection
> * Add _ENABLE_LOGGING_TO_CONSOLE_ env variable (defaulting to _False_) for
> backwards compatibility.
> * Adjust _atlas_start.py_ script to make use of _ENABLE_LOGGING_TO_CONSOLE_
> and provide it in functions invoking atlas/solr/elasticsearch/zookeeper
> *Predicted Benefits*
> This improvement would bring the benefit of enabling logging to stdout (which
> is supressed for now even if configured in spawned components), which could
> be very useful when running Atlas in docker container or in Kubernetes/OC,
> where some logging scraping is present.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (ATLAS-3610) Enable logging to multiple targets (logdir, stdout)
[ https://issues.apache.org/jira/browse/ATLAS-3610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17036254#comment-17036254 ] Mehul Parikh commented on ATLAS-3610: - [~mrp1nk] - tried modifying atlas-env.sh and adding ENABLE_LOGGING_TO_CONSOLE = true, however change was in-effective and atlas_start.py execution showed logging as disabled. Can you please confirm if these changes are effective ? > Enable logging to multiple targets (logdir, stdout) > --- > > Key: ATLAS-3610 > URL: https://issues.apache.org/jira/browse/ATLAS-3610 > Project: Atlas > Issue Type: Improvement > Components: atlas-core >Affects Versions: 3.0.0 >Reporter: Mariusz Górski >Priority: Minor > Fix For: 3.0.0 > > Attachments: ATLAS-3610.patch > > Time Spent: 50m > Remaining Estimate: 0h > > *Problem description* > When starting Atlas, Python scripts are invoked to spawn all necessary > processes. This is achieved with the use of _subprocess.Popen()_ in > _runProcess()_ function of _atlas_config.py_ script. While doing that, stdout > is being redirected to log files, which (for java processes) can override > log4j configuration (if configured with stdout handler). This creates > situation when log4j configuration cannot be relied upon and prevents logging > to stdout. > *Proposed solution* > * Add possibility to log to stdout by using multiple stdout/stderr handlers > * Adjust _runProcess()_ function from _atlas_config.py_ script (and all > functions relying on it) to provide _logconsole_ kwarg which will add > additional handler to Popen stdout/stderr redirection > * Add _ENABLE_LOGGING_TO_CONSOLE_ env variable (defaulting to _False_) for > backwards compatibility. > * Adjust _atlas_start.py_ script to make use of _ENABLE_LOGGING_TO_CONSOLE_ > and provide it in functions invoking atlas/solr/elasticsearch/zookeeper > *Predicted Benefits* > This improvement would bring the benefit of enabling logging to stdout (which > is supressed for now even if configured in spawned components), which could > be very useful when running Atlas in docker container or in Kubernetes/OC, > where some logging scraping is present. -- This message was sent by Atlassian Jira (v8.3.4#803005)
