[jira] [Commented] (ATLAS-3621) update Hive took to not save query-string in 2 attributes - queryText and name
[ https://issues.apache.org/jira/browse/ATLAS-3621?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17037740#comment-17037740 ] ASF subversion and git services commented on ATLAS-3621: Commit f4a9c695dc6d3aeaae116a197909a9af9f60b35d in atlas's branch refs/heads/branch-1.0 from Madhan Neethiraj [ https://gitbox.apache.org/repos/asf?p=atlas.git;h=f4a9c69 ] ATLAS-3621: updated HiveHook to not save query-string in multiple attributes - queryText and name > update Hive took to not save query-string in 2 attributes - queryText and name > -- > > Key: ATLAS-3621 > URL: https://issues.apache.org/jira/browse/ATLAS-3621 > Project: Atlas > Issue Type: Improvement > Components: atlas-intg >Affects Versions: 0.8.4, 1.2.0, 2.0.0 >Reporter: Madhan Neethiraj >Assignee: Madhan Neethiraj >Priority: Major > Fix For: 2.1.0, 3.0.0 > > Attachments: ATLAS-3621-branch-0.8.patch, > ATLAS-3621-branch-1.0.patch, ATLAS-3621-branch-2.0.patch, > ATLAS-3621-master.patch > > > Hive process entities created by HiveHook saves the query-string in 2 > attributes, queryText and name. Since query-string can be large, i.e. in > mega-bytes, it will help to avoid saving such large value in multiple > attributes. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (ATLAS-3621) update Hive took to not save query-string in 2 attributes - queryText and name
[ https://issues.apache.org/jira/browse/ATLAS-3621?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17037734#comment-17037734 ] ASF subversion and git services commented on ATLAS-3621: Commit 40e7e687891a63790ad92f04d914423c22434fd8 in atlas's branch refs/heads/branch-2.0 from Madhan Neethiraj [ https://gitbox.apache.org/repos/asf?p=atlas.git;h=40e7e68 ] ATLAS-3621: updated HiveHook to not save query-string in multiple attributes - queryText and name (cherry picked from commit 10bcaa8091b548efd82e993d6844623f93f1e1f8) > update Hive took to not save query-string in 2 attributes - queryText and name > -- > > Key: ATLAS-3621 > URL: https://issues.apache.org/jira/browse/ATLAS-3621 > Project: Atlas > Issue Type: Improvement > Components: atlas-intg >Affects Versions: 0.8.4, 1.2.0, 2.0.0 >Reporter: Madhan Neethiraj >Assignee: Madhan Neethiraj >Priority: Major > Fix For: 2.1.0, 3.0.0 > > Attachments: ATLAS-3621-branch-0.8.patch, > ATLAS-3621-branch-1.0.patch, ATLAS-3621-branch-2.0.patch, > ATLAS-3621-master.patch > > > Hive process entities created by HiveHook saves the query-string in 2 > attributes, queryText and name. Since query-string can be large, i.e. in > mega-bytes, it will help to avoid saving such large value in multiple > attributes. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (ATLAS-3621) update Hive took to not save query-string in 2 attributes - queryText and name
[ https://issues.apache.org/jira/browse/ATLAS-3621?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17037733#comment-17037733 ] ASF subversion and git services commented on ATLAS-3621: Commit 10bcaa8091b548efd82e993d6844623f93f1e1f8 in atlas's branch refs/heads/master from Madhan Neethiraj [ https://gitbox.apache.org/repos/asf?p=atlas.git;h=10bcaa8 ] ATLAS-3621: updated HiveHook to not save query-string in multiple attributes - queryText and name > update Hive took to not save query-string in 2 attributes - queryText and name > -- > > Key: ATLAS-3621 > URL: https://issues.apache.org/jira/browse/ATLAS-3621 > Project: Atlas > Issue Type: Improvement > Components: atlas-intg >Affects Versions: 0.8.4, 1.2.0, 2.0.0 >Reporter: Madhan Neethiraj >Assignee: Madhan Neethiraj >Priority: Major > Fix For: 2.1.0, 3.0.0 > > Attachments: ATLAS-3621-branch-0.8.patch, > ATLAS-3621-branch-1.0.patch, ATLAS-3621-branch-2.0.patch, > ATLAS-3621-master.patch > > > Hive process entities created by HiveHook saves the query-string in 2 > attributes, queryText and name. Since query-string can be large, i.e. in > mega-bytes, it will help to avoid saving such large value in multiple > attributes. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (ATLAS-3619) Allow to create a namespace typedef without specifying any "applicableEntityTypes"
[ https://issues.apache.org/jira/browse/ATLAS-3619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17037575#comment-17037575 ] ASF subversion and git services commented on ATLAS-3619: Commit b99c84a89d8f15cad5db946eb04869573f5a0bb6 in atlas's branch refs/heads/branch-2.0 from Mandar Ambawane [ https://gitbox.apache.org/repos/asf?p=atlas.git;h=b99c84a ] ATLAS-3619: updated namespace-def to allow attributes with no associated entity-types, mandatory attributes and unique attributes Signed-off-by: Madhan Neethiraj (cherry picked from commit 5e7f8949fd220533510c7ef92e38ac204612a509) > Allow to create a namespace typedef without specifying any > "applicableEntityTypes" > -- > > Key: ATLAS-3619 > URL: https://issues.apache.org/jira/browse/ATLAS-3619 > Project: Atlas > Issue Type: Bug >Reporter: Mandar Ambawane >Assignee: Mandar Ambawane >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (ATLAS-3619) Allow to create a namespace typedef without specifying any "applicableEntityTypes"
[ https://issues.apache.org/jira/browse/ATLAS-3619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17037568#comment-17037568 ] ASF subversion and git services commented on ATLAS-3619: Commit 5e7f8949fd220533510c7ef92e38ac204612a509 in atlas's branch refs/heads/master from Mandar Ambawane [ https://gitbox.apache.org/repos/asf?p=atlas.git;h=5e7f894 ] ATLAS-3619: updated namespace-def to allow attributes with no associated entity-types, mandatory attributes and unique attributes Signed-off-by: Madhan Neethiraj > Allow to create a namespace typedef without specifying any > "applicableEntityTypes" > -- > > Key: ATLAS-3619 > URL: https://issues.apache.org/jira/browse/ATLAS-3619 > Project: Atlas > Issue Type: Bug >Reporter: Mandar Ambawane >Assignee: Mandar Ambawane >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
Re: Review Request 72131: ATLAS-3619 Allow to create a namespace typedef without specifying any applicableEntityTypes
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/72131/#review219595 --- Ship it! Ship It! - Madhan Neethiraj On Feb. 15, 2020, 3:13 p.m., Mandar Ambawane wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/72131/ > --- > > (Updated Feb. 15, 2020, 3:13 p.m.) > > > Review request for atlas, Ashutosh Mestry, Madhan Neethiraj, and Nixon > Rodrigues. > > > Bugs: ATLAS-3619 > https://issues.apache.org/jira/browse/ATLAS-3619 > > > Repository: atlas > > > Description > --- > > Removed throw Exception statement > > > Diffs > - > > intg/src/main/java/org/apache/atlas/AtlasErrorCode.java 1670bda > intg/src/main/java/org/apache/atlas/type/AtlasNamespaceType.java cfbf2b1 > > > Diff: https://reviews.apache.org/r/72131/diff/3/ > > > Testing > --- > > > Thanks, > > Mandar Ambawane > >
Re: Review Request 72131: ATLAS-3619 Allow to create a namespace typedef without specifying any applicableEntityTypes
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/72131/ --- (Updated Feb. 15, 2020, 3:13 p.m.) Review request for atlas, Ashutosh Mestry, Madhan Neethiraj, and Nixon Rodrigues. Changes --- Removed unused Error codes Bugs: ATLAS-3619 https://issues.apache.org/jira/browse/ATLAS-3619 Repository: atlas Description --- Removed throw Exception statement Diffs (updated) - intg/src/main/java/org/apache/atlas/AtlasErrorCode.java 1670bda intg/src/main/java/org/apache/atlas/type/AtlasNamespaceType.java cfbf2b1 Diff: https://reviews.apache.org/r/72131/diff/3/ Changes: https://reviews.apache.org/r/72131/diff/2-3/ Testing --- Thanks, Mandar Ambawane
[jira] [Comment Edited] (ATLAS-3610) Enable logging to multiple targets (logdir, stdout)
[ https://issues.apache.org/jira/browse/ATLAS-3610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17037501#comment-17037501 ] Mariusz Górski edited comment on ATLAS-3610 at 2/15/20 12:55 PM: - I have made another go (*ATLAS-3610-2.patch*) which covers only *atlas_start* and *cputil* scripts. All remaining invocations are left untouched as logconsole doesn't play well with processes requiring user input. I've added the necessary option to *pom.xml* and *atlas-env.sh* (defaulting to *false*). With this approach when *ENABLE_LOGGING_TO_CONSOLE=false* then scripts atlas_start and cputil behave in the old way. When *ENABLE_LOGGING_TO_CONSOLE=true* then they became long running scripts. Please test now [~Rahul_FI] [~vishal.suvagia] [~mehul] was (Author: mrp1nk): I have made another go (*ATLAS-3610-2.patch*) which covers only *atlas_start* and *cputil* scripts. All remaining invocations are left untouched as logconsole doesn't play well with processes requiring user input. I've added the necessary option to *pom.xml* and *atlas-env.sh* (defaulting to *false*). With this approach when *ENABLE_LOGGING_TO_CONSOLE=false* then scripts atlas_start and cputil behave in the old way. When *ENABLE_LOGGING_TO_CONSOLE=true then they became long running scripts. Please test now [~Rahul_FI] [~vishal.suvagia] [~mehul] > Enable logging to multiple targets (logdir, stdout) > --- > > Key: ATLAS-3610 > URL: https://issues.apache.org/jira/browse/ATLAS-3610 > Project: Atlas > Issue Type: Improvement > Components: atlas-core >Affects Versions: 3.0.0 >Reporter: Mariusz Górski >Priority: Minor > Fix For: 3.0.0 > > Attachments: ATLAS-3610-1.patch, ATLAS-3610-2.patch, ATLAS-3610.patch > > Time Spent: 50m > Remaining Estimate: 0h > > *Problem description* > When starting Atlas, Python scripts are invoked to spawn all necessary > processes. This is achieved with the use of _subprocess.Popen()_ in > _runProcess()_ function of _atlas_config.py_ script. While doing that, stdout > is being redirected to log files, which (for java processes) can override > log4j configuration (if configured with stdout handler). This creates > situation when log4j configuration cannot be relied upon and prevents logging > to stdout. > *Proposed solution* > * Add possibility to log to stdout by using multiple stdout/stderr handlers > * Adjust _runProcess()_ function from _atlas_config.py_ script (and all > functions relying on it) to provide _logconsole_ kwarg which will add > additional handler to Popen stdout/stderr redirection > * Add _ENABLE_LOGGING_TO_CONSOLE_ env variable (defaulting to _False_) for > backwards compatibility. > * Adjust _atlas_start.py_ script to make use of _ENABLE_LOGGING_TO_CONSOLE_ > and provide it in functions invoking atlas/solr/elasticsearch/zookeeper > *Predicted Benefits* > This improvement would bring the benefit of enabling logging to stdout (which > is supressed for now even if configured in spawned components), which could > be very useful when running Atlas in docker container or in Kubernetes/OC, > where some logging scraping is present. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (ATLAS-3610) Enable logging to multiple targets (logdir, stdout)
[ https://issues.apache.org/jira/browse/ATLAS-3610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17037501#comment-17037501 ] Mariusz Górski edited comment on ATLAS-3610 at 2/15/20 12:55 PM: - I have made another go (*ATLAS-3610-2.patch*) which covers only *atlas_start* and *cputil* scripts. All remaining invocations are left untouched as logconsole doesn't play well with processes requiring user input. I've added the necessary option to *pom.xml* and *atlas-env.sh* (defaulting to *false*). With this approach when *ENABLE_LOGGING_TO_CONSOLE=false* then scripts atlas_start and cputil behave in the old way. When *ENABLE_LOGGING_TO_CONSOLE=true then they became long running scripts. Please test now [~Rahul_FI] [~vishal.suvagia] [~mehul] was (Author: mrp1nk): I have made another go (*ATLAS-3610-2.patch*) which covers only *atlas_start* and *cputil* scripts. All remaining invocations are left untouched as logconsole doesn't play well with processes requiring user input. I've added the necessary option to *pom.xml* and *atlas-env.sh* (defaulting to *false*). With this approach when *ENABLE_LOGGING_TO_CONSOLE=false* then scripts atlas_start and cputil behave in the old way. When *ENABLE_LOGGING_TO_CONSOLE=true* then *atlas_start.py *is long running script. Please test now [~Rahul_FI] [~vishal.suvagia] [~mehul] > Enable logging to multiple targets (logdir, stdout) > --- > > Key: ATLAS-3610 > URL: https://issues.apache.org/jira/browse/ATLAS-3610 > Project: Atlas > Issue Type: Improvement > Components: atlas-core >Affects Versions: 3.0.0 >Reporter: Mariusz Górski >Priority: Minor > Fix For: 3.0.0 > > Attachments: ATLAS-3610-1.patch, ATLAS-3610-2.patch, ATLAS-3610.patch > > Time Spent: 50m > Remaining Estimate: 0h > > *Problem description* > When starting Atlas, Python scripts are invoked to spawn all necessary > processes. This is achieved with the use of _subprocess.Popen()_ in > _runProcess()_ function of _atlas_config.py_ script. While doing that, stdout > is being redirected to log files, which (for java processes) can override > log4j configuration (if configured with stdout handler). This creates > situation when log4j configuration cannot be relied upon and prevents logging > to stdout. > *Proposed solution* > * Add possibility to log to stdout by using multiple stdout/stderr handlers > * Adjust _runProcess()_ function from _atlas_config.py_ script (and all > functions relying on it) to provide _logconsole_ kwarg which will add > additional handler to Popen stdout/stderr redirection > * Add _ENABLE_LOGGING_TO_CONSOLE_ env variable (defaulting to _False_) for > backwards compatibility. > * Adjust _atlas_start.py_ script to make use of _ENABLE_LOGGING_TO_CONSOLE_ > and provide it in functions invoking atlas/solr/elasticsearch/zookeeper > *Predicted Benefits* > This improvement would bring the benefit of enabling logging to stdout (which > is supressed for now even if configured in spawned components), which could > be very useful when running Atlas in docker container or in Kubernetes/OC, > where some logging scraping is present. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (ATLAS-3610) Enable logging to multiple targets (logdir, stdout)
[ https://issues.apache.org/jira/browse/ATLAS-3610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17037501#comment-17037501 ] Mariusz Górski edited comment on ATLAS-3610 at 2/15/20 12:54 PM: - I have made another go (*ATLAS-3610-2.patch*) which covers only *atlas_start* and *cputil* scripts. All remaining invocations are left untouched as logconsole doesn't play well with processes requiring user input. I've added the necessary option to *pom.xml* and *atlas-env.sh* (defaulting to *false*). With this approach when *ENABLE_LOGGING_TO_CONSOLE=false* then scripts behave in the atlas_start and cputil behave in the old way. When *ENABLE_LOGGING_TO_CONSOLE=true* then *atlas_start.py *is long running script. Please test now [~Rahul_FI] [~vishal.suvagia] [~mehul] was (Author: mrp1nk): I have made another go (*ATLAS-3610-2.patch*) which covers only *atlas_start* and *cputil* scripts. All remaining invocations are left untouched as logconsole doesn't play well with processes requiring user input. I've added the necessary option to *pom.xml* and *atlas-env.sh* (defaulting to *false*). With this approach when ENABLE_LOGGING_TO_CONSOLE=false then scripts behave in the atlas_start and cputil behave in the old way. When ENABLE_LOGGING_TO_CONSOLE=true then atlas_start.py is long running script. Please test now [~Rahul_FI] [~vishal.suvagia] [~mehul] > Enable logging to multiple targets (logdir, stdout) > --- > > Key: ATLAS-3610 > URL: https://issues.apache.org/jira/browse/ATLAS-3610 > Project: Atlas > Issue Type: Improvement > Components: atlas-core >Affects Versions: 3.0.0 >Reporter: Mariusz Górski >Priority: Minor > Fix For: 3.0.0 > > Attachments: ATLAS-3610-1.patch, ATLAS-3610-2.patch, ATLAS-3610.patch > > Time Spent: 50m > Remaining Estimate: 0h > > *Problem description* > When starting Atlas, Python scripts are invoked to spawn all necessary > processes. This is achieved with the use of _subprocess.Popen()_ in > _runProcess()_ function of _atlas_config.py_ script. While doing that, stdout > is being redirected to log files, which (for java processes) can override > log4j configuration (if configured with stdout handler). This creates > situation when log4j configuration cannot be relied upon and prevents logging > to stdout. > *Proposed solution* > * Add possibility to log to stdout by using multiple stdout/stderr handlers > * Adjust _runProcess()_ function from _atlas_config.py_ script (and all > functions relying on it) to provide _logconsole_ kwarg which will add > additional handler to Popen stdout/stderr redirection > * Add _ENABLE_LOGGING_TO_CONSOLE_ env variable (defaulting to _False_) for > backwards compatibility. > * Adjust _atlas_start.py_ script to make use of _ENABLE_LOGGING_TO_CONSOLE_ > and provide it in functions invoking atlas/solr/elasticsearch/zookeeper > *Predicted Benefits* > This improvement would bring the benefit of enabling logging to stdout (which > is supressed for now even if configured in spawned components), which could > be very useful when running Atlas in docker container or in Kubernetes/OC, > where some logging scraping is present. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (ATLAS-3610) Enable logging to multiple targets (logdir, stdout)
[ https://issues.apache.org/jira/browse/ATLAS-3610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17037501#comment-17037501 ] Mariusz Górski edited comment on ATLAS-3610 at 2/15/20 12:54 PM: - I have made another go (*ATLAS-3610-2.patch*) which covers only *atlas_start* and *cputil* scripts. All remaining invocations are left untouched as logconsole doesn't play well with processes requiring user input. I've added the necessary option to *pom.xml* and *atlas-env.sh* (defaulting to *false*). With this approach when *ENABLE_LOGGING_TO_CONSOLE=false* then scripts atlas_start and cputil behave in the old way. When *ENABLE_LOGGING_TO_CONSOLE=true* then *atlas_start.py *is long running script. Please test now [~Rahul_FI] [~vishal.suvagia] [~mehul] was (Author: mrp1nk): I have made another go (*ATLAS-3610-2.patch*) which covers only *atlas_start* and *cputil* scripts. All remaining invocations are left untouched as logconsole doesn't play well with processes requiring user input. I've added the necessary option to *pom.xml* and *atlas-env.sh* (defaulting to *false*). With this approach when *ENABLE_LOGGING_TO_CONSOLE=false* then scripts behave in the atlas_start and cputil behave in the old way. When *ENABLE_LOGGING_TO_CONSOLE=true* then *atlas_start.py *is long running script. Please test now [~Rahul_FI] [~vishal.suvagia] [~mehul] > Enable logging to multiple targets (logdir, stdout) > --- > > Key: ATLAS-3610 > URL: https://issues.apache.org/jira/browse/ATLAS-3610 > Project: Atlas > Issue Type: Improvement > Components: atlas-core >Affects Versions: 3.0.0 >Reporter: Mariusz Górski >Priority: Minor > Fix For: 3.0.0 > > Attachments: ATLAS-3610-1.patch, ATLAS-3610-2.patch, ATLAS-3610.patch > > Time Spent: 50m > Remaining Estimate: 0h > > *Problem description* > When starting Atlas, Python scripts are invoked to spawn all necessary > processes. This is achieved with the use of _subprocess.Popen()_ in > _runProcess()_ function of _atlas_config.py_ script. While doing that, stdout > is being redirected to log files, which (for java processes) can override > log4j configuration (if configured with stdout handler). This creates > situation when log4j configuration cannot be relied upon and prevents logging > to stdout. > *Proposed solution* > * Add possibility to log to stdout by using multiple stdout/stderr handlers > * Adjust _runProcess()_ function from _atlas_config.py_ script (and all > functions relying on it) to provide _logconsole_ kwarg which will add > additional handler to Popen stdout/stderr redirection > * Add _ENABLE_LOGGING_TO_CONSOLE_ env variable (defaulting to _False_) for > backwards compatibility. > * Adjust _atlas_start.py_ script to make use of _ENABLE_LOGGING_TO_CONSOLE_ > and provide it in functions invoking atlas/solr/elasticsearch/zookeeper > *Predicted Benefits* > This improvement would bring the benefit of enabling logging to stdout (which > is supressed for now even if configured in spawned components), which could > be very useful when running Atlas in docker container or in Kubernetes/OC, > where some logging scraping is present. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (ATLAS-3610) Enable logging to multiple targets (logdir, stdout)
[ https://issues.apache.org/jira/browse/ATLAS-3610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17037501#comment-17037501 ] Mariusz Górski edited comment on ATLAS-3610 at 2/15/20 12:53 PM: - I have made another go (*ATLAS-3610-2.patch*) which covers only *atlas_start* and *cputil* scripts. All remaining invocations are left untouched as logconsole doesn't play well with processes requiring user input. I've added the necessary option to *pom.xml* and *atlas-env.sh* (defaulting to *false*). With this approach when ENABLE_LOGGING_TO_CONSOLE=false then scripts behave in the atlas_start and cputil behave in the old way. When ENABLE_LOGGING_TO_CONSOLE=true then atlas_start.py is long running script. Please test now [~Rahul_FI] [~vishal.suvagia] [~mehul] was (Author: mrp1nk): I have made another go (*ATLAS-3610-2.patch*) which covers only *atlas_start* and *cputil* scripts. All remaining invocations are left untouched as logconsole doesn't play well with processes requiring user input. I've added the necessary option to *pom.xml* and *atlas-env.sh* (defaulting to *false*). Please test now [~Rahul_FI] [~vishal.suvagia] [~mehul] > Enable logging to multiple targets (logdir, stdout) > --- > > Key: ATLAS-3610 > URL: https://issues.apache.org/jira/browse/ATLAS-3610 > Project: Atlas > Issue Type: Improvement > Components: atlas-core >Affects Versions: 3.0.0 >Reporter: Mariusz Górski >Priority: Minor > Fix For: 3.0.0 > > Attachments: ATLAS-3610-1.patch, ATLAS-3610-2.patch, ATLAS-3610.patch > > Time Spent: 50m > Remaining Estimate: 0h > > *Problem description* > When starting Atlas, Python scripts are invoked to spawn all necessary > processes. This is achieved with the use of _subprocess.Popen()_ in > _runProcess()_ function of _atlas_config.py_ script. While doing that, stdout > is being redirected to log files, which (for java processes) can override > log4j configuration (if configured with stdout handler). This creates > situation when log4j configuration cannot be relied upon and prevents logging > to stdout. > *Proposed solution* > * Add possibility to log to stdout by using multiple stdout/stderr handlers > * Adjust _runProcess()_ function from _atlas_config.py_ script (and all > functions relying on it) to provide _logconsole_ kwarg which will add > additional handler to Popen stdout/stderr redirection > * Add _ENABLE_LOGGING_TO_CONSOLE_ env variable (defaulting to _False_) for > backwards compatibility. > * Adjust _atlas_start.py_ script to make use of _ENABLE_LOGGING_TO_CONSOLE_ > and provide it in functions invoking atlas/solr/elasticsearch/zookeeper > *Predicted Benefits* > This improvement would bring the benefit of enabling logging to stdout (which > is supressed for now even if configured in spawned components), which could > be very useful when running Atlas in docker container or in Kubernetes/OC, > where some logging scraping is present. -- This message was sent by Atlassian Jira (v8.3.4#803005)
Re: Review Request 72104: Enable logging to STDOUT in Atlas startup scripts
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/72104/ --- (Updated Feb. 15, 2020, 12:51 p.m.) Review request for atlas and Bolke de Bruin. Repository: atlas Description --- This patch enables logging to STDOUT from processes spawned through atlas_start.py or atlas_config.py scripts. For now, even if configured in log4j, logging to STDOUT was supressed (by redirecting to log files only). Diffs (updated) - distro/pom.xml 7159b16cf distro/src/bin/atlas_config.py f09026ff9 distro/src/bin/atlas_start.py 963faf402 distro/src/bin/cputil.py 98a9dc36d distro/src/conf/atlas-env.sh c4241e665 Diff: https://reviews.apache.org/r/72104/diff/3/ Changes: https://reviews.apache.org/r/72104/diff/2-3/ Testing --- The tests were concluded by: 1. Building & running Atlas locally with default log4j configuration and ENABLE_LOGGING_TO_CONSOLE variable set to true. 2. Building & running Atlas on Openshift with log4j configuration pushing everything to STDOUT and ENABLE_LOGGING_TO_CONSOLE variable set to true. In both scenarios logs were available both in the .out/.err files in the Atlas logdir (backwards compatibility), but at the same time I was able to see them in STDOUT of both running process (local installation) and pod (openshift deployment). Thanks, Mariusz Górski
[jira] [Updated] (ATLAS-3610) Enable logging to multiple targets (logdir, stdout)
[ https://issues.apache.org/jira/browse/ATLAS-3610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mariusz Górski updated ATLAS-3610: -- Attachment: (was: ATLAS-3610-2.patch) > Enable logging to multiple targets (logdir, stdout) > --- > > Key: ATLAS-3610 > URL: https://issues.apache.org/jira/browse/ATLAS-3610 > Project: Atlas > Issue Type: Improvement > Components: atlas-core >Affects Versions: 3.0.0 >Reporter: Mariusz Górski >Priority: Minor > Fix For: 3.0.0 > > Attachments: ATLAS-3610-1.patch, ATLAS-3610.patch > > Time Spent: 50m > Remaining Estimate: 0h > > *Problem description* > When starting Atlas, Python scripts are invoked to spawn all necessary > processes. This is achieved with the use of _subprocess.Popen()_ in > _runProcess()_ function of _atlas_config.py_ script. While doing that, stdout > is being redirected to log files, which (for java processes) can override > log4j configuration (if configured with stdout handler). This creates > situation when log4j configuration cannot be relied upon and prevents logging > to stdout. > *Proposed solution* > * Add possibility to log to stdout by using multiple stdout/stderr handlers > * Adjust _runProcess()_ function from _atlas_config.py_ script (and all > functions relying on it) to provide _logconsole_ kwarg which will add > additional handler to Popen stdout/stderr redirection > * Add _ENABLE_LOGGING_TO_CONSOLE_ env variable (defaulting to _False_) for > backwards compatibility. > * Adjust _atlas_start.py_ script to make use of _ENABLE_LOGGING_TO_CONSOLE_ > and provide it in functions invoking atlas/solr/elasticsearch/zookeeper > *Predicted Benefits* > This improvement would bring the benefit of enabling logging to stdout (which > is supressed for now even if configured in spawned components), which could > be very useful when running Atlas in docker container or in Kubernetes/OC, > where some logging scraping is present. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (ATLAS-3610) Enable logging to multiple targets (logdir, stdout)
[ https://issues.apache.org/jira/browse/ATLAS-3610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17037501#comment-17037501 ] Mariusz Górski edited comment on ATLAS-3610 at 2/15/20 11:53 AM: - I have made another go (*ATLAS-3610-2.patch*) which covers only *atlas_start* and *cputil* scripts. All remaining invocations are left untouched as logconsole doesn't play well with processes requiring user input. I've added the necessary option to *pom.xml* and *atlas-env.sh* (defaulting to *false*). Please test now [~Rahul_FI] [~vishal.suvagia] [~mehul] was (Author: mrp1nk): I have made another go (*ATLAS-3610-2.patch*) which covers only *atlas_start* and *cputil* scripts. All remaining invocations are left untouched as logconsole doesn't play well with processes requiring user input. I've added the necessary option to *pom.xml *and *atlas-env.sh* (defaulting to false). Please test now [~Rahul_FI] [~vishal.suvagia] [~mehul] > Enable logging to multiple targets (logdir, stdout) > --- > > Key: ATLAS-3610 > URL: https://issues.apache.org/jira/browse/ATLAS-3610 > Project: Atlas > Issue Type: Improvement > Components: atlas-core >Affects Versions: 3.0.0 >Reporter: Mariusz Górski >Priority: Minor > Fix For: 3.0.0 > > Attachments: ATLAS-3610-1.patch, ATLAS-3610-2.patch, ATLAS-3610.patch > > Time Spent: 50m > Remaining Estimate: 0h > > *Problem description* > When starting Atlas, Python scripts are invoked to spawn all necessary > processes. This is achieved with the use of _subprocess.Popen()_ in > _runProcess()_ function of _atlas_config.py_ script. While doing that, stdout > is being redirected to log files, which (for java processes) can override > log4j configuration (if configured with stdout handler). This creates > situation when log4j configuration cannot be relied upon and prevents logging > to stdout. > *Proposed solution* > * Add possibility to log to stdout by using multiple stdout/stderr handlers > * Adjust _runProcess()_ function from _atlas_config.py_ script (and all > functions relying on it) to provide _logconsole_ kwarg which will add > additional handler to Popen stdout/stderr redirection > * Add _ENABLE_LOGGING_TO_CONSOLE_ env variable (defaulting to _False_) for > backwards compatibility. > * Adjust _atlas_start.py_ script to make use of _ENABLE_LOGGING_TO_CONSOLE_ > and provide it in functions invoking atlas/solr/elasticsearch/zookeeper > *Predicted Benefits* > This improvement would bring the benefit of enabling logging to stdout (which > is supressed for now even if configured in spawned components), which could > be very useful when running Atlas in docker container or in Kubernetes/OC, > where some logging scraping is present. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (ATLAS-3610) Enable logging to multiple targets (logdir, stdout)
[ https://issues.apache.org/jira/browse/ATLAS-3610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mariusz Górski updated ATLAS-3610: -- Attachment: (was: ATLAS-3610-2.patch) > Enable logging to multiple targets (logdir, stdout) > --- > > Key: ATLAS-3610 > URL: https://issues.apache.org/jira/browse/ATLAS-3610 > Project: Atlas > Issue Type: Improvement > Components: atlas-core >Affects Versions: 3.0.0 >Reporter: Mariusz Górski >Priority: Minor > Fix For: 3.0.0 > > Attachments: ATLAS-3610-1.patch, ATLAS-3610-2.patch, ATLAS-3610.patch > > Time Spent: 50m > Remaining Estimate: 0h > > *Problem description* > When starting Atlas, Python scripts are invoked to spawn all necessary > processes. This is achieved with the use of _subprocess.Popen()_ in > _runProcess()_ function of _atlas_config.py_ script. While doing that, stdout > is being redirected to log files, which (for java processes) can override > log4j configuration (if configured with stdout handler). This creates > situation when log4j configuration cannot be relied upon and prevents logging > to stdout. > *Proposed solution* > * Add possibility to log to stdout by using multiple stdout/stderr handlers > * Adjust _runProcess()_ function from _atlas_config.py_ script (and all > functions relying on it) to provide _logconsole_ kwarg which will add > additional handler to Popen stdout/stderr redirection > * Add _ENABLE_LOGGING_TO_CONSOLE_ env variable (defaulting to _False_) for > backwards compatibility. > * Adjust _atlas_start.py_ script to make use of _ENABLE_LOGGING_TO_CONSOLE_ > and provide it in functions invoking atlas/solr/elasticsearch/zookeeper > *Predicted Benefits* > This improvement would bring the benefit of enabling logging to stdout (which > is supressed for now even if configured in spawned components), which could > be very useful when running Atlas in docker container or in Kubernetes/OC, > where some logging scraping is present. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (ATLAS-3610) Enable logging to multiple targets (logdir, stdout)
[ https://issues.apache.org/jira/browse/ATLAS-3610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17037501#comment-17037501 ] Mariusz Górski commented on ATLAS-3610: --- I have made another go (ATLAS-3610-2.patch) which should scope only *atlas_start* and *cputil* scripts. All remaining invocations are left untouched as logconsole doesn't play well with processes requiring user input. I've added the necessary option to pom.xml and atlas-env.sh (defaulting to false). Please test now [~Rahul_FI] [~vishal.suvagia] [~mehul] > Enable logging to multiple targets (logdir, stdout) > --- > > Key: ATLAS-3610 > URL: https://issues.apache.org/jira/browse/ATLAS-3610 > Project: Atlas > Issue Type: Improvement > Components: atlas-core >Affects Versions: 3.0.0 >Reporter: Mariusz Górski >Priority: Minor > Fix For: 3.0.0 > > Attachments: ATLAS-3610-1.patch, ATLAS-3610-2.patch, ATLAS-3610.patch > > Time Spent: 50m > Remaining Estimate: 0h > > *Problem description* > When starting Atlas, Python scripts are invoked to spawn all necessary > processes. This is achieved with the use of _subprocess.Popen()_ in > _runProcess()_ function of _atlas_config.py_ script. While doing that, stdout > is being redirected to log files, which (for java processes) can override > log4j configuration (if configured with stdout handler). This creates > situation when log4j configuration cannot be relied upon and prevents logging > to stdout. > *Proposed solution* > * Add possibility to log to stdout by using multiple stdout/stderr handlers > * Adjust _runProcess()_ function from _atlas_config.py_ script (and all > functions relying on it) to provide _logconsole_ kwarg which will add > additional handler to Popen stdout/stderr redirection > * Add _ENABLE_LOGGING_TO_CONSOLE_ env variable (defaulting to _False_) for > backwards compatibility. > * Adjust _atlas_start.py_ script to make use of _ENABLE_LOGGING_TO_CONSOLE_ > and provide it in functions invoking atlas/solr/elasticsearch/zookeeper > *Predicted Benefits* > This improvement would bring the benefit of enabling logging to stdout (which > is supressed for now even if configured in spawned components), which could > be very useful when running Atlas in docker container or in Kubernetes/OC, > where some logging scraping is present. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (ATLAS-3610) Enable logging to multiple targets (logdir, stdout)
[ https://issues.apache.org/jira/browse/ATLAS-3610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17037501#comment-17037501 ] Mariusz Górski edited comment on ATLAS-3610 at 2/15/20 11:48 AM: - I have made another go (*ATLAS-3610-2.patch*) which covers only *atlas_start* and *cputil* scripts. All remaining invocations are left untouched as logconsole doesn't play well with processes requiring user input. I've added the necessary option to *pom.xml *and *atlas-env.sh* (defaulting to false). Please test now [~Rahul_FI] [~vishal.suvagia] [~mehul] was (Author: mrp1nk): I have made another go (*ATLAS-3610-2.patch*) which should scope only *atlas_start* and *cputil* scripts. All remaining invocations are left untouched as logconsole doesn't play well with processes requiring user input. I've added the necessary option to *pom.xml *and *atlas-env.sh* (defaulting to false). Please test now [~Rahul_FI] [~vishal.suvagia] [~mehul] > Enable logging to multiple targets (logdir, stdout) > --- > > Key: ATLAS-3610 > URL: https://issues.apache.org/jira/browse/ATLAS-3610 > Project: Atlas > Issue Type: Improvement > Components: atlas-core >Affects Versions: 3.0.0 >Reporter: Mariusz Górski >Priority: Minor > Fix For: 3.0.0 > > Attachments: ATLAS-3610-1.patch, ATLAS-3610-2.patch, ATLAS-3610.patch > > Time Spent: 50m > Remaining Estimate: 0h > > *Problem description* > When starting Atlas, Python scripts are invoked to spawn all necessary > processes. This is achieved with the use of _subprocess.Popen()_ in > _runProcess()_ function of _atlas_config.py_ script. While doing that, stdout > is being redirected to log files, which (for java processes) can override > log4j configuration (if configured with stdout handler). This creates > situation when log4j configuration cannot be relied upon and prevents logging > to stdout. > *Proposed solution* > * Add possibility to log to stdout by using multiple stdout/stderr handlers > * Adjust _runProcess()_ function from _atlas_config.py_ script (and all > functions relying on it) to provide _logconsole_ kwarg which will add > additional handler to Popen stdout/stderr redirection > * Add _ENABLE_LOGGING_TO_CONSOLE_ env variable (defaulting to _False_) for > backwards compatibility. > * Adjust _atlas_start.py_ script to make use of _ENABLE_LOGGING_TO_CONSOLE_ > and provide it in functions invoking atlas/solr/elasticsearch/zookeeper > *Predicted Benefits* > This improvement would bring the benefit of enabling logging to stdout (which > is supressed for now even if configured in spawned components), which could > be very useful when running Atlas in docker container or in Kubernetes/OC, > where some logging scraping is present. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (ATLAS-3610) Enable logging to multiple targets (logdir, stdout)
[ https://issues.apache.org/jira/browse/ATLAS-3610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17037501#comment-17037501 ] Mariusz Górski edited comment on ATLAS-3610 at 2/15/20 11:48 AM: - I have made another go (*ATLAS-3610-2.patch*) which should scope only *atlas_start* and *cputil* scripts. All remaining invocations are left untouched as logconsole doesn't play well with processes requiring user input. I've added the necessary option to *pom.xml *and *atlas-env.sh* (defaulting to false). Please test now [~Rahul_FI] [~vishal.suvagia] [~mehul] was (Author: mrp1nk): I have made another go (ATLAS-3610-2.patch) which should scope only *atlas_start* and *cputil* scripts. All remaining invocations are left untouched as logconsole doesn't play well with processes requiring user input. I've added the necessary option to *pom.xml *and *atlas-env.sh* (defaulting to false). Please test now [~Rahul_FI] [~vishal.suvagia] [~mehul] > Enable logging to multiple targets (logdir, stdout) > --- > > Key: ATLAS-3610 > URL: https://issues.apache.org/jira/browse/ATLAS-3610 > Project: Atlas > Issue Type: Improvement > Components: atlas-core >Affects Versions: 3.0.0 >Reporter: Mariusz Górski >Priority: Minor > Fix For: 3.0.0 > > Attachments: ATLAS-3610-1.patch, ATLAS-3610-2.patch, ATLAS-3610.patch > > Time Spent: 50m > Remaining Estimate: 0h > > *Problem description* > When starting Atlas, Python scripts are invoked to spawn all necessary > processes. This is achieved with the use of _subprocess.Popen()_ in > _runProcess()_ function of _atlas_config.py_ script. While doing that, stdout > is being redirected to log files, which (for java processes) can override > log4j configuration (if configured with stdout handler). This creates > situation when log4j configuration cannot be relied upon and prevents logging > to stdout. > *Proposed solution* > * Add possibility to log to stdout by using multiple stdout/stderr handlers > * Adjust _runProcess()_ function from _atlas_config.py_ script (and all > functions relying on it) to provide _logconsole_ kwarg which will add > additional handler to Popen stdout/stderr redirection > * Add _ENABLE_LOGGING_TO_CONSOLE_ env variable (defaulting to _False_) for > backwards compatibility. > * Adjust _atlas_start.py_ script to make use of _ENABLE_LOGGING_TO_CONSOLE_ > and provide it in functions invoking atlas/solr/elasticsearch/zookeeper > *Predicted Benefits* > This improvement would bring the benefit of enabling logging to stdout (which > is supressed for now even if configured in spawned components), which could > be very useful when running Atlas in docker container or in Kubernetes/OC, > where some logging scraping is present. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (ATLAS-3610) Enable logging to multiple targets (logdir, stdout)
[ https://issues.apache.org/jira/browse/ATLAS-3610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17037501#comment-17037501 ] Mariusz Górski edited comment on ATLAS-3610 at 2/15/20 11:48 AM: - I have made another go (ATLAS-3610-2.patch) which should scope only *atlas_start* and *cputil* scripts. All remaining invocations are left untouched as logconsole doesn't play well with processes requiring user input. I've added the necessary option to *pom.xml *and *atlas-env.sh* (defaulting to false). Please test now [~Rahul_FI] [~vishal.suvagia] [~mehul] was (Author: mrp1nk): I have made another go (ATLAS-3610-2.patch) which should scope only *atlas_start* and *cputil* scripts. All remaining invocations are left untouched as logconsole doesn't play well with processes requiring user input. I've added the necessary option to pom.xml and atlas-env.sh (defaulting to false). Please test now [~Rahul_FI] [~vishal.suvagia] [~mehul] > Enable logging to multiple targets (logdir, stdout) > --- > > Key: ATLAS-3610 > URL: https://issues.apache.org/jira/browse/ATLAS-3610 > Project: Atlas > Issue Type: Improvement > Components: atlas-core >Affects Versions: 3.0.0 >Reporter: Mariusz Górski >Priority: Minor > Fix For: 3.0.0 > > Attachments: ATLAS-3610-1.patch, ATLAS-3610-2.patch, ATLAS-3610.patch > > Time Spent: 50m > Remaining Estimate: 0h > > *Problem description* > When starting Atlas, Python scripts are invoked to spawn all necessary > processes. This is achieved with the use of _subprocess.Popen()_ in > _runProcess()_ function of _atlas_config.py_ script. While doing that, stdout > is being redirected to log files, which (for java processes) can override > log4j configuration (if configured with stdout handler). This creates > situation when log4j configuration cannot be relied upon and prevents logging > to stdout. > *Proposed solution* > * Add possibility to log to stdout by using multiple stdout/stderr handlers > * Adjust _runProcess()_ function from _atlas_config.py_ script (and all > functions relying on it) to provide _logconsole_ kwarg which will add > additional handler to Popen stdout/stderr redirection > * Add _ENABLE_LOGGING_TO_CONSOLE_ env variable (defaulting to _False_) for > backwards compatibility. > * Adjust _atlas_start.py_ script to make use of _ENABLE_LOGGING_TO_CONSOLE_ > and provide it in functions invoking atlas/solr/elasticsearch/zookeeper > *Predicted Benefits* > This improvement would bring the benefit of enabling logging to stdout (which > is supressed for now even if configured in spawned components), which could > be very useful when running Atlas in docker container or in Kubernetes/OC, > where some logging scraping is present. -- This message was sent by Atlassian Jira (v8.3.4#803005)
Re: Review Request 72137: ATLAS-3621: updated HiveHook to not save query-string in multiple attributes - queryText and name
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/72137/#review219594 --- Ship it! Ship It! - Nixon Rodrigues On Feb. 15, 2020, 2:11 a.m., Madhan Neethiraj wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/72137/ > --- > > (Updated Feb. 15, 2020, 2:11 a.m.) > > > Review request for atlas, Ashutosh Mestry, Mandar Ambawane, mayank jain, > Mehul Parikh, Nixon Rodrigues, Pinal Shah, Sarath Subramanian, and Sidharth > Mishra. > > > Bugs: ATLAS-3621 > https://issues.apache.org/jira/browse/ATLAS-3621 > > > Repository: atlas > > > Description > --- > > - updated HiveHook to set hive_process.name and hive_column_lineage.name to > the value in qualifiedName (instead of the query-string) > - updated NotificationHookConsumer to update these attributes similarly, to > handle notifications that were generated by earlier hook > > > Diffs > - > > > addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/BaseHiveEvent.java > 0409d8ae0 > > webapp/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java > 14cae4228 > > webapp/src/main/java/org/apache/atlas/notification/preprocessor/HivePreprocessor.java > cc5fe5221 > > webapp/src/main/java/org/apache/atlas/notification/preprocessor/PreprocessorContext.java > 3c58c9fce > > > Diff: https://reviews.apache.org/r/72137/diff/1/ > > > Testing > --- > > - verified that attribute 'name' in hive_process and hive_column_lineage are > set to the value in qualifiedName, instead of query-string, by executing a > hive query > - precommit tests run: > https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/1637 > > > Thanks, > > Madhan Neethiraj > >
