Review Request 48819: As part of atlas startup, "/usr/lib/ambari-logsearch-solr-client/solrCloudCli.sh" script is executed by ambari, which fails with "java.io.FileNotFoundException"
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/48819/ --- Review request for Ambari, Alejandro Fernandez and Nate Cole. Bugs: AMBARI-17274 https://issues.apache.org/jira/browse/AMBARI-17274 Repository: ambari Description --- Setup Atlas to use Solr configuration for Atlas instead of pointing to LogSearch install Solr config. Diffs - ambari-server/src/main/resources/common-services/ATLAS/0.1.0.2.3/configuration/atlas-solrconfig.xml PRE-CREATION ambari-server/src/main/resources/common-services/ATLAS/0.1.0.2.3/metainfo.xml 9f3eda9 ambari-server/src/main/resources/common-services/ATLAS/0.1.0.2.3/package/scripts/metadata.py 1abf2c1 ambari-server/src/main/resources/common-services/ATLAS/0.1.0.2.3/package/scripts/params.py 614fa6b ambari-server/src/test/python/stacks/2.3/ATLAS/test_metadata_server.py de355ea ambari-server/src/test/python/stacks/2.3/configs/default.json 07298f1 ambari-server/src/test/python/stacks/2.3/configs/secure.json 1ca7a2e ambari-server/src/test/python/stacks/2.5/ATLAS/test_atlas_server.py 8729195 ambari-server/src/test/python/stacks/2.5/configs/default.json c839405 Diff: https://reviews.apache.org/r/48819/diff/ Testing --- Manual test install Atlas and LogSearch. Verify Atlas Solr configuration. updated unit tests mvn clean test Thanks, Tom Beerbower
Re: Review Request 48743: Remove {{atlas_conf_dir}} from HADOOP_CLASSPATH in hive-env.
> On June 16, 2016, 6:11 p.m., John Speidel wrote:
> > Curious, how does the Hive hook work now that the Atlas config isn't in
> > it's classpath?
Ambari now writes the atlas configuration file into the Hive conf folder.
- Tom
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/48743/#review138038
---
On June 15, 2016, 6:13 p.m., Tom Beerbower wrote:
>
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/48743/
> ---
>
> (Updated June 15, 2016, 6:13 p.m.)
>
>
> Review request for Ambari and John Speidel.
>
>
> Bugs: AMBARI-17262
> https://issues.apache.org/jira/browse/AMBARI-17262
>
>
> Repository: ambari
>
>
> Description
> ---
>
> In Hive configuration ‘Advanced hive-env’ :
>
>
> export
> HADOOP_CLASSPATH={{atlas_conf_dir}}:{{atlas_home_dir}}/hook/hive:${HADOOP_CLASSPATH}
>
>
> remove
>
>
> {{atlas_conf_dir}}
>
>
> Diffs
> -
>
>
> ambari-server/src/main/resources/stacks/HDP/2.5/services/HIVE/configuration/hive-env.xml
> 0c9aeaf
>
> Diff: https://reviews.apache.org/r/48743/diff/
>
>
> Testing
> ---
>
> Manual test. Verify configuration.
>
> mvn clean test
>
>
> Thanks,
>
> Tom Beerbower
>
>
Re: Review Request 48743: Remove {{atlas_conf_dir}} from HADOOP_CLASSPATH in hive-env.
> On June 16, 2016, 6:11 p.m., John Speidel wrote:
> > Curious, how does the Hive hook work now that the Atlas config isn't in
> > it's classpath?
>
> Tom Beerbower wrote:
> Ambari now writes the atlas configuration file into the Hive conf folder.
Thanks for the review.
- Tom
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/48743/#review138038
---
On June 15, 2016, 6:13 p.m., Tom Beerbower wrote:
>
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/48743/
> ---
>
> (Updated June 15, 2016, 6:13 p.m.)
>
>
> Review request for Ambari and John Speidel.
>
>
> Bugs: AMBARI-17262
> https://issues.apache.org/jira/browse/AMBARI-17262
>
>
> Repository: ambari
>
>
> Description
> ---
>
> In Hive configuration ‘Advanced hive-env’ :
>
>
> export
> HADOOP_CLASSPATH={{atlas_conf_dir}}:{{atlas_home_dir}}/hook/hive:${HADOOP_CLASSPATH}
>
>
> remove
>
>
> {{atlas_conf_dir}}
>
>
> Diffs
> -
>
>
> ambari-server/src/main/resources/stacks/HDP/2.5/services/HIVE/configuration/hive-env.xml
> 0c9aeaf
>
> Diff: https://reviews.apache.org/r/48743/diff/
>
>
> Testing
> ---
>
> Manual test. Verify configuration.
>
> mvn clean test
>
>
> Thanks,
>
> Tom Beerbower
>
>
Re: Review Request 48788: With Atlas HA enabled, atlas instance fails to come up with "Unable to find IDs matching any local host and port binding among id1"
> On June 16, 2016, 5:04 p.m., Alejandro Fernandez wrote:
> > Ship It!
Thanks for the review.
- Tom
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/48788/#review138020
---
On June 16, 2016, 12:23 p.m., Tom Beerbower wrote:
>
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/48788/
> ---
>
> (Updated June 16, 2016, 12:23 p.m.)
>
>
> Review request for Ambari and John Speidel.
>
>
> Bugs: AMBARI-17273
> https://issues.apache.org/jira/browse/AMBARI-17273
>
>
> Repository: ambari
>
>
> Description
> ---
>
> Atlas fails to come up with below error. This is because if ID information
> for all atlas instances is not generated and updated in
> atlas-application.properties.
>
> 2016-06-14 10:48:23,249 WARN - [main:] ~ Failed startup of context
> o.e.j.w.WebAppContext@2ad31d76{/,file:/grid/0/hdp/2.5.0.0-723/atlas/server/webapp/atlas/,STARTING}{/usr/hdp/current/atlas-server/server/webapp/atlas}
> (WebAppContext:514)
> java.lang.RuntimeException: org.apache.atlas.AtlasException: Could not find
> server id for this instance. Unable to find IDs matching any local host and
> port binding among id1
>
> Atlas-application.properties not updated with all other instance information
> leading to atlas startup failure
>
>
> Diffs
> -
>
>
> ambari-server/src/main/resources/common-services/ATLAS/0.1.0.2.3/package/scripts/params.py
> 37236fe
>
> ambari-server/src/main/resources/stacks/HDP/2.5/services/ATLAS/configuration/application-properties.xml
> a5a63ba
> ambari-server/src/main/resources/stacks/HDP/2.5/services/stack_advisor.py
> 614f0b1
> ambari-server/src/test/python/stacks/2.5/common/test_stack_advisor.py
> 4085ad6
>
> Diff: https://reviews.apache.org/r/48788/diff/
>
>
> Testing
> ---
>
> Manual test Atlas install and HA. Verify configuration.
>
> mvn clean test
>
>
> Thanks,
>
> Tom Beerbower
>
>
Review Request 48788: With Atlas HA enabled, atlas instance fails to come up with "Unable to find IDs matching any local host and port binding among id1"
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/48788/
---
Review request for Ambari and John Speidel.
Bugs: AMBARI-17273
https://issues.apache.org/jira/browse/AMBARI-17273
Repository: ambari
Description
---
Atlas fails to come up with below error. This is because if ID information for
all atlas instances is not generated and updated in
atlas-application.properties.
2016-06-14 10:48:23,249 WARN - [main:] ~ Failed startup of context
o.e.j.w.WebAppContext@2ad31d76{/,file:/grid/0/hdp/2.5.0.0-723/atlas/server/webapp/atlas/,STARTING}{/usr/hdp/current/atlas-server/server/webapp/atlas}
(WebAppContext:514)
java.lang.RuntimeException: org.apache.atlas.AtlasException: Could not find
server id for this instance. Unable to find IDs matching any local host and
port binding among id1
Atlas-application.properties not updated with all other instance information
leading to atlas startup failure
Diffs
-
ambari-server/src/main/resources/common-services/ATLAS/0.1.0.2.3/package/scripts/params.py
37236fe
ambari-server/src/main/resources/stacks/HDP/2.5/services/ATLAS/configuration/application-properties.xml
a5a63ba
ambari-server/src/main/resources/stacks/HDP/2.5/services/stack_advisor.py
614f0b1
ambari-server/src/test/python/stacks/2.5/common/test_stack_advisor.py 4085ad6
Diff: https://reviews.apache.org/r/48788/diff/
Testing
---
Manual test Atlas install and HA. Verify configuration.
mvn clean test
Thanks,
Tom Beerbower
Review Request 48743: Remove {{atlas_conf_dir}} from HADOOP_CLASSPATH in hive-env.
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/48743/
---
Review request for Ambari and John Speidel.
Bugs: AMBARI-17262
https://issues.apache.org/jira/browse/AMBARI-17262
Repository: ambari
Description
---
In Hive configuration ‘Advanced hive-env’ :
export
HADOOP_CLASSPATH={{atlas_conf_dir}}:{{atlas_home_dir}}/hook/hive:${HADOOP_CLASSPATH}
remove
{{atlas_conf_dir}}
Diffs
-
ambari-server/src/main/resources/stacks/HDP/2.5/services/HIVE/configuration/hive-env.xml
0c9aeaf
Diff: https://reviews.apache.org/r/48743/diff/
Testing
---
Manual test. Verify configuration.
mvn clean test
Thanks,
Tom Beerbower
Review Request 48710: atlas.authentication.principal should be set to atlas/_h...@example.com
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/48710/ --- Review request for Ambari and Robert Levas. Bugs: AMBARI-17242 https://issues.apache.org/jira/browse/AMBARI-17242 Repository: ambari Description --- In a kerberized env, the value of atlas.authentication.principal should be set to atlas/_h...@example.com. Currently the value is set to atlas which causes Atlas server startup to fail. Diffs - ambari-server/src/main/resources/stacks/HDP/2.5/services/ATLAS/kerberos.json 78ad6ea Diff: https://reviews.apache.org/r/48710/diff/ Testing --- Manual test install Atlas, set Kerberos, verify Atlas configuration. Thanks, Tom Beerbower
Review Request 48707: Atlas Server Web UI alert
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/48707/ --- Review request for Ambari and John Speidel. Bugs: AMBARI-17005 https://issues.apache.org/jira/browse/AMBARI-17005 Repository: ambari Description --- Seeing this HTTP 401 response from http://c6401.ambari.apache.org:21000. Atlas recently added basic authentication. Updates in ATLAS-774 will allow requests to "atlas-server:port/api/atlas/admin/status" without requiring authentication. Diffs - ambari-server/src/main/resources/common-services/ATLAS/0.1.0.2.3/alerts.json e65b5c1 Diff: https://reviews.apache.org/r/48707/diff/ Testing --- Manual test install Atlas. Verify alert not triggered. mvn clean test Thanks, Tom Beerbower
Re: Review Request 48655: ATLAS conf dir needs to be present in all ATLAS hook deployed hosts
> On June 13, 2016, 8:29 p.m., Alejandro Fernandez wrote: > > ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/configuration/hive-env.xml, > > line 238 > > <https://reviews.apache.org/r/48655/diff/1/?file=1417161#file1417161line238> > > > > Does this need to change for existing clusters? > > What about the different stacks? > > I don't believe ATLAS is even available on HDP 2.2 This export was added earlier for HDP 2.5 so it should not be present in any existing clusters. - Tom --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/48655/#review137373 ------- On June 13, 2016, 5:51 p.m., Tom Beerbower wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/48655/ > --- > > (Updated June 13, 2016, 5:51 p.m.) > > > Review request for Ambari, John Speidel and Nate Cole. > > > Bugs: AMBARI-17203 > https://issues.apache.org/jira/browse/AMBARI-17203 > > > Repository: ambari > > > Description > --- > > Atlas atlas-application.properties configuration is required by the Atlas > hooks (Falcon, Storm, Sqoop and Hive). > Currently the atlas configuration is not being pushed to all the hook hosts > which could lead to issues if any config is changed. The Atlas configuration > changes needs to be pushed to all dependent hosts. > > > Diffs > - > > > ambari-server/src/main/resources/common-services/FALCON/0.5.0.2.1/package/scripts/params_linux.py > 441f0da > > ambari-server/src/main/resources/common-services/FALCON/0.5.0.2.1/package/scripts/setup_atlas_falcon.py > 67077c4 > > ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/configuration/hive-env.xml > c20523d > > ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/params_linux.py > fea0635 > > ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/setup_atlas_hive.py > e78190f > > ambari-server/src/main/resources/common-services/SQOOP/1.4.4.2.0/package/scripts/params_linux.py > c154a16 > > ambari-server/src/main/resources/common-services/SQOOP/1.4.4.2.0/package/scripts/setup_atlas_sqoop.py > d18d820 > > ambari-server/src/main/resources/common-services/STORM/0.9.1/package/scripts/params_linux.py > eadbd4a > > ambari-server/src/main/resources/common-services/STORM/0.9.1/package/scripts/setup_atlas_storm.py > 6c3e91f > > ambari-server/src/main/resources/stacks/HDP/2.5/services/FALCON/metainfo.xml > 1998131 > ambari-server/src/main/resources/stacks/HDP/2.5/services/HIVE/metainfo.xml > ac1f9e7 > ambari-server/src/main/resources/stacks/HDP/2.5/services/SQOOP/metainfo.xml > eb67d63 > ambari-server/src/main/resources/stacks/HDP/2.5/services/STORM/metainfo.xml > 3faadc0 > ambari-server/src/main/resources/stacks/HDP/2.5/services/stack_advisor.py > 1bcc09e > > Diff: https://reviews.apache.org/r/48655/diff/ > > > Testing > --- > > Manual test deploy Atlas with all hook integration services on multi-host > cluster. Verify Atlas configuration locations. Update Atlas configuration. > Verify dependant service restart suggestion from Ambari. > > mvn clean test > > > Thanks, > > Tom Beerbower > >
Review Request 48655: ATLAS conf dir needs to be present in all ATLAS hook deployed hosts
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/48655/ --- Review request for Ambari, John Speidel and Nate Cole. Bugs: AMBARI-17203 https://issues.apache.org/jira/browse/AMBARI-17203 Repository: ambari Description --- Atlas atlas-application.properties configuration is required by the Atlas hooks (Falcon, Storm, Sqoop and Hive). Currently the atlas configuration is not being pushed to all the hook hosts which could lead to issues if any config is changed. The Atlas configuration changes needs to be pushed to all dependent hosts. Diffs - ambari-server/src/main/resources/common-services/FALCON/0.5.0.2.1/package/scripts/params_linux.py 441f0da ambari-server/src/main/resources/common-services/FALCON/0.5.0.2.1/package/scripts/setup_atlas_falcon.py 67077c4 ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/configuration/hive-env.xml c20523d ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/params_linux.py fea0635 ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/setup_atlas_hive.py e78190f ambari-server/src/main/resources/common-services/SQOOP/1.4.4.2.0/package/scripts/params_linux.py c154a16 ambari-server/src/main/resources/common-services/SQOOP/1.4.4.2.0/package/scripts/setup_atlas_sqoop.py d18d820 ambari-server/src/main/resources/common-services/STORM/0.9.1/package/scripts/params_linux.py eadbd4a ambari-server/src/main/resources/common-services/STORM/0.9.1/package/scripts/setup_atlas_storm.py 6c3e91f ambari-server/src/main/resources/stacks/HDP/2.5/services/FALCON/metainfo.xml 1998131 ambari-server/src/main/resources/stacks/HDP/2.5/services/HIVE/metainfo.xml ac1f9e7 ambari-server/src/main/resources/stacks/HDP/2.5/services/SQOOP/metainfo.xml eb67d63 ambari-server/src/main/resources/stacks/HDP/2.5/services/STORM/metainfo.xml 3faadc0 ambari-server/src/main/resources/stacks/HDP/2.5/services/stack_advisor.py 1bcc09e Diff: https://reviews.apache.org/r/48655/diff/ Testing --- Manual test deploy Atlas with all hook integration services on multi-host cluster. Verify Atlas configuration locations. Update Atlas configuration. Verify dependant service restart suggestion from Ambari. mvn clean test Thanks, Tom Beerbower
Re: Review Request 48489: Atlas Integration : Required changes for atlas-application.properties
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/48489/
---
(Updated June 9, 2016, 7:30 p.m.)
Review request for Ambari, John Speidel and Robert Levas.
Bugs: AMBARI-17144
https://issues.apache.org/jira/browse/AMBARI-17144
Repository: ambari
Description
---
Changes in ATLAS-820 require following configurations to be present in
atlas-application.properties:
atlas.authentication.method.kerberos=false
atlas.authentication.method.file=true
atlas.authentication.method.ldap=false
atlas.authentication.method.file.filename={{atlas_login_credentials_file}}
atlas.authentication.method.ldap.type=ldap|ad
atlas.authentication.method.ldap.url=
When Kerberos is enabled, following configurations need to be set/added:
atlas.authentication.method.kerberos=true
atlas.authentication.method.kerberos.keytab=/etc/security/keytabs/spnego.service.keytab
atlas.authentication.method.kerberos.principal=HTTP/_h...@example.com
atlas.authentication.method.kerberos.name.rules=
atlas.kafka.sasl.kerberos.service.name=kafka
atlas.kafka.security.protocol=SASL_PLAINTEXT
atlas.jaas.KafkaClient.loginModuleName=com.sun.security.auth.module.Krb5LoginModule
atlas.jaas.KafkaClient.loginModuleControlFlag=required
atlas.jaas.KafkaClient.option.useKeyTab=true
atlas.jaas.KafkaClient.option.storeKey=true
atlas.jaas.KafkaClient.option.serviceName=kafka
atlas.jaas.KafkaClient.option.keyTab={{atlas_keytab_path}}
atlas.jaas.KafkaClient.option.principal={{atlas_jaas_principal}}
Following properties are no more used and need to be removed:
atlas.http.authentication.enabled
atlas.http.authentication.kerberos.keytab
atlas.http.authentication.type
atlas.http.authentication.kerberos.principal
atlas.http.authentication.kerberos.name.rules
atlas.login.method=
atlas.login.credentials.file=
Diffs (updated)
-
ambari-server/src/main/resources/stacks/HDP/2.5/services/ATLAS/configuration/application-properties.xml
97c3e6a
ambari-server/src/main/resources/stacks/HDP/2.5/services/ATLAS/kerberos.json
PRE-CREATION
Diff: https://reviews.apache.org/r/48489/diff/
Testing
---
manual test
install Atlas and verify configuration
enable kerberos and verify configuration
Thanks,
Tom Beerbower
Review Request 48489: Atlas Integration : Required changes for atlas-application.properties
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/48489/
---
Review request for Ambari, John Speidel and Robert Levas.
Bugs: AMBARI-17144
https://issues.apache.org/jira/browse/AMBARI-17144
Repository: ambari
Description
---
Changes in ATLAS-820 require following configurations to be present in
atlas-application.properties:
atlas.authentication.method.kerberos=false
atlas.authentication.method.file=true
atlas.authentication.method.ldap=false
atlas.authentication.method.file.filename={{atlas_login_credentials_file}}
atlas.authentication.method.ldap.type=ldap|ad
atlas.authentication.method.ldap.url=
When Kerberos is enabled, following configurations need to be set/added:
atlas.authentication.method.kerberos=true
atlas.authentication.method.kerberos.keytab=/etc/security/keytabs/spnego.service.keytab
atlas.authentication.method.kerberos.principal=HTTP/_h...@example.com
atlas.authentication.method.kerberos.name.rules=
atlas.kafka.sasl.kerberos.service.name=kafka
atlas.kafka.security.protocol=SASL_PLAINTEXT
atlas.jaas.KafkaClient.loginModuleName=com.sun.security.auth.module.Krb5LoginModule
atlas.jaas.KafkaClient.loginModuleControlFlag=required
atlas.jaas.KafkaClient.option.useKeyTab=true
atlas.jaas.KafkaClient.option.storeKey=true
atlas.jaas.KafkaClient.option.serviceName=kafka
atlas.jaas.KafkaClient.option.keyTab={{atlas_keytab_path}}
atlas.jaas.KafkaClient.option.principal={{atlas_jaas_principal}}
Following properties are no more used and need to be removed:
atlas.http.authentication.enabled
atlas.http.authentication.kerberos.keytab
atlas.http.authentication.type
atlas.http.authentication.kerberos.principal
atlas.http.authentication.kerberos.name.rules
atlas.login.method=
atlas.login.credentials.file=
Diffs
-
ambari-server/src/main/resources/stacks/HDP/2.5/services/ATLAS/configuration/application-properties.xml
2c4426b
ambari-server/src/main/resources/stacks/HDP/2.5/services/ATLAS/kerberos.json
PRE-CREATION
Diff: https://reviews.apache.org/r/48489/diff/
Testing
---
manual test
install Atlas and verify configuration
enable kerberos and verify configuration
Thanks,
Tom Beerbower
Re: Review Request 48134: Remove logsearch-solr-client dependency from Atlas
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/48134/#review135849 --- Ship it! Looks good to me. Thanks for doing that! - Tom Beerbower On June 1, 2016, 4:58 p.m., Oliver Szabo wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/48134/ > --- > > (Updated June 1, 2016, 4:58 p.m.) > > > Review request for Ambari, Robert Nettleton, Sumit Mohanty, and Tom Beerbower. > > > Bugs: AMBARI-16961 > https://issues.apache.org/jira/browse/AMBARI-16961 > > > Repository: ambari > > > Description > --- > > removed LOGSEARCH_SOLR_CLIENT dependency from Atlas stack definition to not > force ambari to install the client (if LOGSEARCH service is not used). > Also I provided a new function in solr_cloud_util to handle permissions on > the solr client files (set the same as logsearch uses, then we wont run into > any permission issues) > > > Diffs > - > > > ambari-common/src/main/python/resource_management/libraries/functions/package_conditions.py > b916f19 > > ambari-common/src/main/python/resource_management/libraries/functions/solr_cloud_util.py > e6e794c > > ambari-server/src/main/resources/common-services/ATLAS/0.1.0.2.3/metainfo.xml > d0d73ad > > ambari-server/src/main/resources/common-services/ATLAS/0.1.0.2.3/package/scripts/metadata.py > a2d78a9 > > ambari-server/src/main/resources/common-services/ATLAS/0.1.0.2.3/package/scripts/params.py > 573857b > ambari-server/src/main/resources/stacks/HDP/2.3/role_command_order.json > ce2528b > ambari-server/src/main/resources/stacks/HDP/2.5/services/ATLAS/metainfo.xml > 4890f00 > ambari-server/src/test/python/stacks/2.3/ATLAS/test_metadata_server.py > faa7350 > ambari-server/src/test/python/stacks/2.3/configs/default.json 157fd7e > ambari-server/src/test/python/stacks/2.3/configs/secure.json 3ad8656 > ambari-server/src/test/python/stacks/2.4/configs/default.json e01bc26 > ambari-server/src/test/python/stacks/2.5/ATLAS/test_atlas_server.py 6839dee > ambari-server/src/test/python/stacks/2.5/configs/default.json 0b67405 > > Diff: https://reviews.apache.org/r/48134/diff/ > > > Testing > --- > > unit tests done. > > FT: I cannot really test it with HDP 2.5 because on the ATLAS side, in the > start script it tries to setup collections anyway, so this patch should be > applied with a fresh ATLAS... so, i opened this review request mostly for > only review. > > > Thanks, > > Oliver Szabo > >
Re: Review Request 47984: Make storm.topology.submission.notifier.plugin.class property optional
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/47984/#review135635 --- Ship it! Ship It! - Tom Beerbower On May 28, 2016, 5:15 a.m., Sivaguru Chendamaraikannan wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/47984/ > --- > > (Updated May 28, 2016, 5:15 a.m.) > > > Review request for Ambari, Srimanth Gunturi and Tom Beerbower. > > > Bugs: AMBARI-16915 > https://issues.apache.org/jira/browse/AMBARI-16915 > > > Repository: ambari > > > Description > --- > > The commit, > https://github.com/apache/ambari/commit/89fd30b40f3108bfcbcc73eb2d74c94a2ba14a7a > which accidentally made the storm.topology.submission.notifier.plugin > property mandatory. If the storm defaults yaml config does not set the > property a " " value is set for the property. This causes storm startup to > fail with a Class Not Found exception when starting with Ambari. This change > fixes the stack advisor to not set the property when no value is specified > for it through command line or storm defaults. > > > Diffs > - > > ambari-server/src/main/resources/stacks/HDP/2.3/services/stack_advisor.py > 79314f5 > > Diff: https://reviews.apache.org/r/47984/diff/ > > > Testing > --- > > mvn test > > > Thanks, > > Sivaguru Chendamaraikannan > >
Re: Review Request 47967: Atlas server start failed after Ambari upgrade due to missing solrCloudCli.sh script
> On May 27, 2016, 8:08 p.m., Robert Levas wrote:
> > Its not clear how the patch helps to prevent:
> >
> > ```
> > Traceback (most recent call last):
> > File
> > "/var/lib/ambari-agent/cache/common-services/ATLAS/0.1.0.2.3/package/scripts/metadata_server.py",
> > line 165, in
> > MetadataServer().execute()
> > raise Fail(err_msg)
> > resource_management.core.exceptions.Fail: Execution of 'export
> > JAVA_HOME=/usr/jdk64/jdk1.7.0_67 ;
> > /usr/lib/ambari-logsearch-solr-client/solrCloudCli.sh -z
> > os-r6-tgvuks-dgm10toeriedwngdha-r6-4.openstacklocal:2181,os-r6-tgvuks-dgm10toeriedwngdha-r6-3.openstacklocal:2181,os-r6-tgvuks-dgm10toeriedwngdha-r6-2.openstacklocal:2181None
> > --upload-config -d
> > /usr/lib/ambari-logsearch-solr/server/solr/configsets/basic_configs/conf
> > -cs basic_configs -rt 5 -i 10' returned 127. -bash:
> > /usr/lib/ambari-logsearch-solr-client/solrCloudCli.sh: No such file or
> > directory
> > ```
Hey Rob. Thanks for reviewing!
Sorry about that.
The problem is that after the upgrade the LogSearch script is being called
unconditionally and Logsearch is not installed.
The stack trace in the review description is truncated. The complete stack is
in the Jira, but here is where the script is being called from in metadata.py...
File
"/var/lib/ambari-agent/cache/common-services/ATLAS/0.1.0.2.3/package/scripts/metadata_server.py",
line 51, in configure
metadata()
File
"/var/lib/ambari-agent/cache/common-services/ATLAS/0.1.0.2.3/package/scripts/metadata.py",
line 113, in metadata
upload_conf_set('basic_configs', random_num)
File
"/var/lib/ambari-agent/cache/common-services/ATLAS/0.1.0.2.3/package/scripts/metadata.py",
line 135, in upload_conf_set
group=params.user_group)
Line 113 in metadata.py is where I added the if check to make sure that the
configuration was set for Solr and that the LogSearch Solr is actually
installed.
if type == 'server' and params.search_backend_solr and
params.has_logsearch_solr:
- Tom
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47967/#review135302
---
On May 27, 2016, 5 p.m., Tom Beerbower wrote:
>
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/47967/
> ---
>
> (Updated May 27, 2016, 5 p.m.)
>
>
> Review request for Ambari, John Speidel and Robert Levas.
>
>
> Bugs: AMBARI-16932
> https://issues.apache.org/jira/browse/AMBARI-16932
>
>
> Repository: ambari
>
>
> Description
> ---
>
> Steps
>
> 1.Deploy HDP-2.4.2.0 cluster with Ambari 2.2.2.0 (including Atlas)
> 2.Upgrade Ambari to 2.4.0.0
> 3.Stop and start all services
>
> Result
>
> Atlas Metadata server start fails with below error:
> Traceback (most recent call last):
> File
> "/var/lib/ambari-agent/cache/common-services/ATLAS/0.1.0.2.3/package/scripts/metadata_server.py",
> line 165, in
> MetadataServer().execute()
> raise Fail(err_msg)
> resource_management.core.exceptions.Fail: Execution of 'export
> JAVA_HOME=/usr/jdk64/jdk1.7.0_67 ;
> /usr/lib/ambari-logsearch-solr-client/solrCloudCli.sh -z
> os-r6-tgvuks-dgm10toeriedwngdha-r6-4.openstacklocal:2181,os-r6-tgvuks-dgm10toeriedwngdha-r6-3.openstacklocal:2181,os-r6-tgvuks-dgm10toeriedwngdha-r6-2.openstacklocal:2181None
> --upload-config -d
> /usr/lib/ambari-logsearch-solr/server/solr/configsets/basic_configs/conf -cs
> basic_configs -rt 5 -i 10' returned 127. -bash:
> /usr/lib/ambari-logsearch-solr-client/solrCloudCli.sh: No such file or
> directory
>
>
> Diffs
> -
>
>
> ambari-server/src/main/resources/common-services/ATLAS/0.1.0.2.3/package/scripts/metadata.py
> 6f287dc
>
> ambari-server/src/main/resources/common-services/ATLAS/0.1.0.2.3/package/scripts/params.py
> 3a9c614
> ambari-server/src/test/python/stacks/2.3/configs/default.json ef04248
> ambari-server/src/test/python/stacks/2.3/configs/secure.json 166fd6f
> ambari-server/src/test/python/stacks/2.5/configs/default.json 0077280
>
> Diff: https://reviews.apache.org/r/47967/diff/
>
>
> Testing
> ---
>
> Manual test
>
> Adjust unit tests
>
> mvn clean test
>
>
> Thanks,
>
> Tom Beerbower
>
>
Review Request 47967: Atlas server start failed after Ambari upgrade due to missing solrCloudCli.sh script
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/47967/ --- Review request for Ambari, John Speidel and Robert Levas. Bugs: AMBARI-16932 https://issues.apache.org/jira/browse/AMBARI-16932 Repository: ambari Description --- Steps 1.Deploy HDP-2.4.2.0 cluster with Ambari 2.2.2.0 (including Atlas) 2.Upgrade Ambari to 2.4.0.0 3.Stop and start all services Result Atlas Metadata server start fails with below error: Traceback (most recent call last): File "/var/lib/ambari-agent/cache/common-services/ATLAS/0.1.0.2.3/package/scripts/metadata_server.py", line 165, in MetadataServer().execute() raise Fail(err_msg) resource_management.core.exceptions.Fail: Execution of 'export JAVA_HOME=/usr/jdk64/jdk1.7.0_67 ; /usr/lib/ambari-logsearch-solr-client/solrCloudCli.sh -z os-r6-tgvuks-dgm10toeriedwngdha-r6-4.openstacklocal:2181,os-r6-tgvuks-dgm10toeriedwngdha-r6-3.openstacklocal:2181,os-r6-tgvuks-dgm10toeriedwngdha-r6-2.openstacklocal:2181None --upload-config -d /usr/lib/ambari-logsearch-solr/server/solr/configsets/basic_configs/conf -cs basic_configs -rt 5 -i 10' returned 127. -bash: /usr/lib/ambari-logsearch-solr-client/solrCloudCli.sh: No such file or directory Diffs - ambari-server/src/main/resources/common-services/ATLAS/0.1.0.2.3/package/scripts/metadata.py 6f287dc ambari-server/src/main/resources/common-services/ATLAS/0.1.0.2.3/package/scripts/params.py 3a9c614 ambari-server/src/test/python/stacks/2.3/configs/default.json ef04248 ambari-server/src/test/python/stacks/2.3/configs/secure.json 166fd6f ambari-server/src/test/python/stacks/2.5/configs/default.json 0077280 Diff: https://reviews.apache.org/r/47967/diff/ Testing --- Manual test Adjust unit tests mvn clean test Thanks, Tom Beerbower
Re: Review Request 47785: Ambari install of Atlas should use external HBase and Logsearch SOLR
> On May 25, 2016, 11:30 p.m., Srimanth Gunturi wrote:
> > ambari-server/src/main/resources/common-services/ATLAS/0.1.0.2.3/package/scripts/params.py,
> > line 150
> > <https://reviews.apache.org/r/47785/diff/2/?file=1394010#file1394010line150>
> >
> > Just wanted to make sure if this always be true?
>
> Nate Cole wrote:
> I think this assumption should be ok - we are already doing the symlink
> magic to point to the right spot.
Thanks for the review Srimanth and Nate.
Looking in the HBase status_params.py, I see this ...
hbase_conf_dir = "/etc/hbase/conf"
if stack_version_formatted and
check_stack_feature(StackFeature.ROLLING_UPGRADE, stack_version_formatted):
hbase_conf_dir = format("{stack_root}/current/{component_directory}/conf")
Do, I need to add the same logic?
- Tom
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47785/#review134876
-------
On May 25, 2016, 10:06 p.m., Tom Beerbower wrote:
>
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/47785/
> ---
>
> (Updated May 25, 2016, 10:06 p.m.)
>
>
> Review request for Ambari, John Speidel, Nate Cole, and Srimanth Gunturi.
>
>
> Bugs: AMBARI-16853 and ATLAS-823
> https://issues.apache.org/jira/browse/AMBARI-16853
> https://issues.apache.org/jira/browse/ATLAS-823
>
>
> Repository: ambari
>
>
> Description
> ---
>
> To support this, add dependency from Atlas to LogSearch SOLR. Also remove
> references to embedded SOLR and embedded HBase.
>
> Use solr_cloud_util to create indexes for Atlas install.
>
> See AMBARI-15865.
> See Atlas-823.
>
>
> Diffs
> -
>
>
> ambari-server/src/main/resources/common-services/ATLAS/0.1.0.2.3/configuration/application-properties.xml
> bf0467e
>
> ambari-server/src/main/resources/common-services/ATLAS/0.1.0.2.3/configuration/atlas-env.xml
> dd4b3e2
>
> ambari-server/src/main/resources/common-services/ATLAS/0.1.0.2.3/configuration/atlas-hbase-site.xml
> 3c4826d
>
> ambari-server/src/main/resources/common-services/ATLAS/0.1.0.2.3/metainfo.xml
> f4115f7
>
> ambari-server/src/main/resources/common-services/ATLAS/0.1.0.2.3/package/scripts/atlas_client.py
> 6b045ca
>
> ambari-server/src/main/resources/common-services/ATLAS/0.1.0.2.3/package/scripts/metadata.py
> e305138
>
> ambari-server/src/main/resources/common-services/ATLAS/0.1.0.2.3/package/scripts/params.py
> f172b79
>
> ambari-server/src/main/resources/stacks/HDP/2.5/services/ATLAS/configuration/application-properties.xml
> 0631b7d
>
> ambari-server/src/main/resources/stacks/HDP/2.5/services/ATLAS/configuration/atlas-env.xml
> 2a5f777
> ambari-server/src/main/resources/stacks/HDP/2.5/services/ATLAS/metainfo.xml
> 15daeea
> ambari-server/src/main/resources/stacks/HDP/2.5/services/stack_advisor.py
> af812fe
> ambari-server/src/test/python/stacks/2.3/ATLAS/test_metadata_server.py
> 98fc678
> ambari-server/src/test/python/stacks/2.3/configs/default.json c8b418b
> ambari-server/src/test/python/stacks/2.3/configs/secure.json 7ebcedf
> ambari-server/src/test/python/stacks/2.5/ATLAS/test_atlas_server.py 9d0f00c
> ambari-server/src/test/python/stacks/2.5/common/test_stack_advisor.py
> 51c0d0c
> ambari-server/src/test/python/stacks/2.5/configs/default.json e4a97f6
>
> Diff: https://reviews.apache.org/r/47785/diff/
>
>
> Testing
> ---
>
> Manual test of Atlas install.
>
> Updated unit tests.
>
> mvn clean test
>
>
> Thanks,
>
> Tom Beerbower
>
>
Review Request 47785: Ambari install of Atlas should use external HBase and Logsearch SOLR
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/47785/ --- Review request for Ambari. Bugs: AMBARI-16853 and ATLAS-823 https://issues.apache.org/jira/browse/AMBARI-16853 https://issues.apache.org/jira/browse/ATLAS-823 Repository: ambari Description --- To support this, add dependency from Atlas to LogSearch SOLR. Also remove references to embedded SOLR and embedded HBase. Use solr_cloud_util to create indexes for Atlas install. See AMBARI-15865. See Atlas-823. Diffs - ambari-server/src/main/resources/common-services/ATLAS/0.1.0.2.3/configuration/application-properties.xml bf0467e ambari-server/src/main/resources/common-services/ATLAS/0.1.0.2.3/configuration/atlas-env.xml 90eabef ambari-server/src/main/resources/common-services/ATLAS/0.1.0.2.3/configuration/atlas-hbase-site.xml 3c4826d ambari-server/src/main/resources/common-services/ATLAS/0.1.0.2.3/metainfo.xml f4115f7 ambari-server/src/main/resources/common-services/ATLAS/0.1.0.2.3/package/scripts/atlas_client.py 6b045ca ambari-server/src/main/resources/common-services/ATLAS/0.1.0.2.3/package/scripts/metadata.py e305138 ambari-server/src/main/resources/common-services/ATLAS/0.1.0.2.3/package/scripts/params.py f172b79 ambari-server/src/main/resources/stacks/HDP/2.5/services/ATLAS/configuration/application-properties.xml 0631b7d ambari-server/src/main/resources/stacks/HDP/2.5/services/ATLAS/configuration/atlas-env.xml bae4de3 ambari-server/src/main/resources/stacks/HDP/2.5/services/ATLAS/metainfo.xml 15daeea ambari-server/src/main/resources/stacks/HDP/2.5/services/stack_advisor.py 11aac72 ambari-server/src/test/python/stacks/2.3/ATLAS/test_metadata_server.py 98fc678 ambari-server/src/test/python/stacks/2.3/configs/default.json c8b418b ambari-server/src/test/python/stacks/2.3/configs/secure.json 7ebcedf ambari-server/src/test/python/stacks/2.5/ATLAS/test_atlas_server.py 9d0f00c ambari-server/src/test/python/stacks/2.5/configs/default.json e4a97f6 Diff: https://reviews.apache.org/r/47785/diff/ Testing --- Manual test of Atlas install. Updated unit tests. mvn clean test Thanks, Tom Beerbower
Re: Review Request 47773: one very careless mistake for ambari agent cancel logic
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/47773/#review134576 --- Ship it! The fix looks okay to me. How did you test it? Could you please complete the "testing done" section of the review? - Tom Beerbower On May 24, 2016, 1:13 p.m., chen jian wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/47773/ > --- > > (Updated May 24, 2016, 1:13 p.m.) > > > Review request for Ambari, Alexandr Antonenko, Robert Nettleton, and Tom > Beerbower. > > > Repository: ambari > > > Description > --- > > one very careless mistake for ambari agent cancel logic > > > Diffs > - > > ambari-agent/src/main/python/ambari_agent/ActionQueue.py 19bd18f > > Diff: https://reviews.apache.org/r/47773/diff/ > > > Testing > --- > > > Thanks, > > chen jian > >
Re: Review Request 47626: Atlas HA configuration Property Changed
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/47626/ --- (Updated May 20, 2016, 10:16 a.m.) Review request for Ambari, Erik Bergenholtz and John Speidel. Changes --- updated diff Bugs: AMBARI-16781 https://issues.apache.org/jira/browse/AMBARI-16781 Repository: ambari Description (updated) --- The Atlas HA properties to designate ha hosts has changed from: atlas.server.host.id to atlas.server.address.id this causes Atlas to not be able to start in HA mode. The following changes need to be made: - resources/common-services/ATLAS/0.1.0.2.3/package/scripts/params.py - resources/stacks/HDP/2.5/services/ATLAS/configuration/application-properties.xml Also, fix new values of renamed lineage configs. Diffs (updated) - ambari-server/src/main/resources/common-services/ATLAS/0.1.0.2.3/package/scripts/params.py 21bc0c5 ambari-server/src/main/resources/stacks/HDP/2.5/services/ATLAS/configuration/application-properties.xml 53c1369 Diff: https://reviews.apache.org/r/47626/diff/ Testing --- manual check Atlas HA - verify configs mvn clean test Thanks, Tom Beerbower
Review Request 47626: Atlas HA configuration Property Changed
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/47626/ --- Review request for Ambari, Erik Bergenholtz and John Speidel. Bugs: AMBARI-16781 https://issues.apache.org/jira/browse/AMBARI-16781 Repository: ambari Description --- The Atlas HA properties to designate ha hosts has changed from: atlas.server.host.id to atlas.server.address.id this causes Atlas to not be able to start in HA mode. The following changes need to be made: - resources/common-services/ATLAS/0.1.0.2.3/package/scripts/params.py - resources/stacks/HDP/2.5/services/ATLAS/configuration/application-properties.xml Diffs - ambari-server/src/main/resources/common-services/ATLAS/0.1.0.2.3/package/scripts/params.py 21bc0c5 ambari-server/src/main/resources/stacks/HDP/2.5/services/ATLAS/configuration/application-properties.xml 53c1369 Diff: https://reviews.apache.org/r/47626/diff/ Testing --- manual check Atlas HA - verify configs mvn clean test Thanks, Tom Beerbower
Review Request 47545: Rename atlas lineage configurations
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/47545/ --- Review request for Ambari, Erik Bergenholtz and John Speidel. Bugs: AMBARI-16730 https://issues.apache.org/jira/browse/AMBARI-16730 Repository: ambari Description --- As part of ATLAS-713, the following changes are required in atlas configuration that is set by Ambari: 1 atlas.lineage.hive.table.schema.query.Table renamed to atlas.lineage.schema.query.Table 2 atlas.lineage.hive.table.schema.query.hive_table renamed to atlas.lineage.schema.query.hive_table 3 delete unused lineage configs Diffs - ambari-server/src/main/resources/stacks/HDP/2.5/services/ATLAS/configuration/application-properties.xml 2dab35a Diff: https://reviews.apache.org/r/47545/diff/ Testing --- manual test Atlas install mvn clean test Thanks, Tom Beerbower
Re: Review Request 47009: Falcon server start fails
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47009/
---
(Updated May 11, 2016, 8:52 p.m.)
Review request for Ambari, Sumit Mohanty and Venkat Ranganathan.
Changes
---
Updated diff
Bugs: AMBARI-16263
https://issues.apache.org/jira/browse/AMBARI-16263
Repository: ambari
Description
---
Deployed HDP-2.5 and Falcon server failed to start.
cat /var/log/falcon/falcon.out.20160504002*
Error: Could not find or load main class org.apache.falcon.FalconServer
Error: Could not find or load main class org.apache.falcon.FalconServer
Error: Could not find or load main class org.apache.falcon.FalconServer
The command to start, in Ambari is:
Execute['/usr/hdp/current/falcon-server/bin/falcon-start -port 15000']
{'environment': {'HADOOP_HOME': '/usr/hdp/current/hadoop-client'}, 'path':
['/usr/hdp/current/hadoop-client/bin'], 'user': 'falcon'}
Link falcon hook jars in falcon classpath - 'ln -s /hook/falcon/*
/server/webapp/falcon/WEB-INF/lib/'
is what is causing it. If falcon webapp is already exploded it will fail for
falcon webapp itself. That step should not be done and we should change it to
add falcon hook into webapp.jar itself post deployment like what we do for
Oozie with prepare-war or added to server classpath explicitly
Diffs (updated)
-
ambari-server/src/main/resources/common-services/FALCON/0.5.0.2.1/configuration/falcon-env.xml
3fdf2e6
ambari-server/src/main/resources/common-services/FALCON/0.5.0.2.1/package/scripts/params_linux.py
bf7f437
ambari-server/src/main/resources/common-services/FALCON/0.5.0.2.1/package/scripts/setup_atlas_falcon.py
4b7c0c3
Diff: https://reviews.apache.org/r/47009/diff/
Testing
---
mvn clean test
Thanks,
Tom Beerbower
Re: Review Request 47256: Atlas Integration : Atlas fails to come up with solr as indexing search when the zookeeper quorum has more than one host
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/47256/ --- (Updated May 11, 2016, 8:41 p.m.) Review request for Ambari, John Speidel and Nate Cole. Bugs: AMBARI-16464 https://issues.apache.org/jira/browse/AMBARI-16464 Repository: ambari Description (updated) --- Atlas fails to communicate with solr when the zookeeper quorum has more than one host, as a result of this atlas fails to come up. The config parameter "atlas.graph.index.search.solr.zookeeper-url" value is set to "zookeeper_quorum/logsearch". Since there are more than one zookeeper servers deployed, this value evaluates to "[os-r6-apathan-atlas-erie-nosec-3.openstacklocal:2181, os-r6-apathan-atlas-erie-nosec-2.openstacklocal:2181/logsearch]". The value was set to match the value of the ZK connection string in the logsearch solr-env.sh ... # Set the ZooKeeper connection string if using an external ZooKeeper ensemble # e.g. host1:2181,host2:2181/chroot # Leave empty if not using SolrCloud ZK_HOST="c6401.ambari.apache.org:2181,c6402.ambari.apache.org:2181,c6403.ambari.apache.org:2181/logsearch" It looks like for atlas.graph.index.search.solr.zookeeper-url it should be ... c6401.ambari.apache.org:2181/logsearch,c6402.ambari.apache.org:2181/logsearch,c6403.ambari.apache.org:2181/logsearch Also needed to update the atlas user-credential.properites and policy-store.txt to get Atlas to deploy (due to recent changes in Atlas). Diffs - ambari-server/src/main/resources/common-services/ATLAS/0.1.0.2.3/package/files/policy-store.txt PRE-CREATION ambari-server/src/main/resources/common-services/ATLAS/0.1.0.2.3/package/files/users-credentials.properties 4d264a0 ambari-server/src/main/resources/common-services/ATLAS/0.1.0.2.3/package/scripts/metadata.py 8e2fcc2 ambari-server/src/main/resources/common-services/ATLAS/0.1.0.2.3/package/scripts/params.py 51fa039 ambari-server/src/main/resources/stacks/HDP/2.5/services/ATLAS/configuration/application-properties.xml 78600ee ambari-server/src/main/resources/stacks/HDP/2.5/services/stack_advisor.py 3d689d7 ambari-server/src/test/python/stacks/2.3/ATLAS/test_metadata_server.py 8ae2c11 ambari-server/src/test/python/stacks/2.5/ATLAS/test_atlas_server.py 422b1c9 ambari-server/src/test/python/stacks/2.5/common/test_stack_advisor.py e9ddc25 Diff: https://reviews.apache.org/r/47256/diff/ Testing --- Updated unit tests. mvn clean test Thanks, Tom Beerbower
Re: Review Request 47009: Falcon server start fails
> On May 9, 2016, 9:49 p.m., Venkat Ranganathan wrote:
> >
>
> Venkat Ranganathan wrote:
> We have committed a change as part of FALCON-1942 to add a environment
> variable called FALCON_EXTRA_CLASS_PATH - this will be appended to the Falcon
> server classpath. Please use it
Great! Thanks Venkat.
- Tom
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47009/#review132337
---
On May 5, 2016, 12:07 p.m., Tom Beerbower wrote:
>
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/47009/
> ---
>
> (Updated May 5, 2016, 12:07 p.m.)
>
>
> Review request for Ambari, Sumit Mohanty and Venkat Ranganathan.
>
>
> Bugs: AMBARI-16263
> https://issues.apache.org/jira/browse/AMBARI-16263
>
>
> Repository: ambari
>
>
> Description
> ---
>
> Deployed HDP-2.5 and Falcon server failed to start.
>
>
> cat /var/log/falcon/falcon.out.20160504002*
> Error: Could not find or load main class org.apache.falcon.FalconServer
> Error: Could not find or load main class org.apache.falcon.FalconServer
> Error: Could not find or load main class org.apache.falcon.FalconServer
>
>
> The command to start, in Ambari is:
>
>
> Execute['/usr/hdp/current/falcon-server/bin/falcon-start -port 15000']
> {'environment': {'HADOOP_HOME': '/usr/hdp/current/hadoop-client'}, 'path':
> ['/usr/hdp/current/hadoop-client/bin'], 'user': 'falcon'}
>
>
> Link falcon hook jars in falcon classpath - 'ln -s /hook/falcon/*
> /server/webapp/falcon/WEB-INF/lib/'
> is what is causing it. If falcon webapp is already exploded it will fail for
> falcon webapp itself. That step should not be done and we should change it to
> add falcon hook into webapp.jar itself post deployment like what we do for
> Oozie with prepare-war or added to server classpath explicitly
>
>
> Diffs
> -
>
>
> ambari-server/src/main/resources/common-services/FALCON/0.5.0.2.1/configuration/falcon-env.xml
> 3fdf2e6
>
> ambari-server/src/main/resources/common-services/FALCON/0.5.0.2.1/package/scripts/params_linux.py
> bf7f437
>
> ambari-server/src/main/resources/common-services/FALCON/0.5.0.2.1/package/scripts/setup_atlas_falcon.py
> 4b7c0c3
>
> Diff: https://reviews.apache.org/r/47009/diff/
>
>
> Testing
> ---
>
> mvn clean test
>
>
> Thanks,
>
> Tom Beerbower
>
>
Re: Review Request 47009: Falcon server start fails
> On May 9, 2016, 9:49 p.m., Venkat Ranganathan wrote:
> > ambari-server/src/main/resources/common-services/FALCON/0.5.0.2.1/configuration/falcon-env.xml,
> > line 162
> > <https://reviews.apache.org/r/47009/diff/1/?file=1372978#file1372978line162>
> >
> > Don't we want the server classpath to be changed in falcon-config?
Thanks for the review. I'm not familiar with falcon-config. Could you give
more details?
- Tom
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47009/#review132337
-------
On May 5, 2016, 12:07 p.m., Tom Beerbower wrote:
>
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/47009/
> ---
>
> (Updated May 5, 2016, 12:07 p.m.)
>
>
> Review request for Ambari, Sumit Mohanty and Venkat Ranganathan.
>
>
> Bugs: AMBARI-16263
> https://issues.apache.org/jira/browse/AMBARI-16263
>
>
> Repository: ambari
>
>
> Description
> ---
>
> Deployed HDP-2.5 and Falcon server failed to start.
>
>
> cat /var/log/falcon/falcon.out.20160504002*
> Error: Could not find or load main class org.apache.falcon.FalconServer
> Error: Could not find or load main class org.apache.falcon.FalconServer
> Error: Could not find or load main class org.apache.falcon.FalconServer
>
>
> The command to start, in Ambari is:
>
>
> Execute['/usr/hdp/current/falcon-server/bin/falcon-start -port 15000']
> {'environment': {'HADOOP_HOME': '/usr/hdp/current/hadoop-client'}, 'path':
> ['/usr/hdp/current/hadoop-client/bin'], 'user': 'falcon'}
>
>
> Link falcon hook jars in falcon classpath - 'ln -s /hook/falcon/*
> /server/webapp/falcon/WEB-INF/lib/'
> is what is causing it. If falcon webapp is already exploded it will fail for
> falcon webapp itself. That step should not be done and we should change it to
> add falcon hook into webapp.jar itself post deployment like what we do for
> Oozie with prepare-war or added to server classpath explicitly
>
>
> Diffs
> -
>
>
> ambari-server/src/main/resources/common-services/FALCON/0.5.0.2.1/configuration/falcon-env.xml
> 3fdf2e6
>
> ambari-server/src/main/resources/common-services/FALCON/0.5.0.2.1/package/scripts/params_linux.py
> bf7f437
>
> ambari-server/src/main/resources/common-services/FALCON/0.5.0.2.1/package/scripts/setup_atlas_falcon.py
> 4b7c0c3
>
> Diff: https://reviews.apache.org/r/47009/diff/
>
>
> Testing
> ---
>
> mvn clean test
>
>
> Thanks,
>
> Tom Beerbower
>
>
Review Request 47009: Falcon server start fails
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47009/
---
Review request for Ambari, Sumit Mohanty and Venkat Ranganathan.
Bugs: AMBARI-16263
https://issues.apache.org/jira/browse/AMBARI-16263
Repository: ambari
Description
---
Deployed HDP-2.5 and Falcon server failed to start.
cat /var/log/falcon/falcon.out.20160504002*
Error: Could not find or load main class org.apache.falcon.FalconServer
Error: Could not find or load main class org.apache.falcon.FalconServer
Error: Could not find or load main class org.apache.falcon.FalconServer
The command to start, in Ambari is:
Execute['/usr/hdp/current/falcon-server/bin/falcon-start -port 15000']
{'environment': {'HADOOP_HOME': '/usr/hdp/current/hadoop-client'}, 'path':
['/usr/hdp/current/hadoop-client/bin'], 'user': 'falcon'}
Link falcon hook jars in falcon classpath - 'ln -s /hook/falcon/*
/server/webapp/falcon/WEB-INF/lib/'
is what is causing it. If falcon webapp is already exploded it will fail for
falcon webapp itself. That step should not be done and we should change it to
add falcon hook into webapp.jar itself post deployment like what we do for
Oozie with prepare-war or added to server classpath explicitly
Diffs
-
ambari-server/src/main/resources/common-services/FALCON/0.5.0.2.1/configuration/falcon-env.xml
3fdf2e6
ambari-server/src/main/resources/common-services/FALCON/0.5.0.2.1/package/scripts/params_linux.py
bf7f437
ambari-server/src/main/resources/common-services/FALCON/0.5.0.2.1/package/scripts/setup_atlas_falcon.py
4b7c0c3
Diff: https://reviews.apache.org/r/47009/diff/
Testing
---
mvn clean test
Thanks,
Tom Beerbower
Review Request 46903: Atlas Integration : Atlas start failed after removing old version of the atlas and installing new one
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/46903/ --- Review request for Ambari, John Speidel and Nate Cole. Bugs: AMBARI-16204 https://issues.apache.org/jira/browse/AMBARI-16204 Repository: ambari Description --- Upgrade from HDP 2.3 stack to 2.5. Stop and delete the Atlas service under 2.3 and then reinstall it after you have upgraded to 2.5. Atlas creates the /var/lib/atlas/server/webapp folder. After removing old service and installing new version of the service the Atlas war file is updated, but the unpacked folder is not. This causes errors like - Can't find main class. Diffs - ambari-server/src/main/resources/common-services/ATLAS/0.1.0.2.3/configuration/application-properties.xml f3ea36f ambari-server/src/main/resources/common-services/ATLAS/0.1.0.2.3/metainfo.xml 5d7766b ambari-server/src/main/resources/common-services/ATLAS/0.1.0.2.3/package/files/users-credentials.properties PRE-CREATION ambari-server/src/main/resources/common-services/ATLAS/0.1.0.2.3/package/scripts/metadata.py 43c1ea6 ambari-server/src/main/resources/common-services/ATLAS/0.1.0.2.3/package/scripts/metadata_server.py 4b08cbc ambari-server/src/main/resources/common-services/ATLAS/0.1.0.2.3/package/scripts/params.py defd6f0 ambari-server/src/main/resources/stacks/HDP/2.5/services/ATLAS/configuration/application-properties.xml 8412fc0 ambari-server/src/main/resources/stacks/HDP/2.5/services/ATLAS/configuration/atlas-env.xml a2ea7e5 ambari-server/src/main/resources/stacks/HDP/2.5/services/ATLAS/metainfo.xml 92f198a ambari-server/src/test/python/stacks/2.3/ATLAS/test_metadata_server.py cf18f92 ambari-server/src/test/python/stacks/2.3/configs/default.json d423ce5 ambari-server/src/test/python/stacks/2.3/configs/secure.json 89ab6f0 ambari-server/src/test/python/stacks/2.5/ATLAS/test_atlas_server.py 83d4c5b ambari-server/src/test/python/stacks/2.5/configs/default.json 5188730 Diff: https://reviews.apache.org/r/46903/diff/ Testing --- Manual test Atlas install. Updated unit tests. mvn clean test. Thanks, Tom Beerbower
Re: Review Request 46740: Atlas Integration : set atlas.cluster.name in hive-site
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/46740/ --- (Updated April 27, 2016, 2:23 p.m.) Review request for Ambari, John Speidel and Nate Cole. Bugs: AMBARI-16089 and AMBARI-16090 https://issues.apache.org/jira/browse/AMBARI-16089 https://issues.apache.org/jira/browse/AMBARI-16090 Repository: ambari Description --- After deploying Hive+Atlas via Ambari, the value of 'atlas.cluster.name‘ in 'Advanced hive-site’ is ‘default’. This results in incorrect value in entity notifications to Ranger. Also, to enable login to Atlas UI, following properties have to be manually added to ‘Custom application-properties’ in Atlas. atlas.login.method=file atlas.login.credentials.file=/etc/atlas/conf/users-credentials.properties Diffs - ambari-server/src/main/resources/common-services/ATLAS/0.1.0.2.3/package/scripts/params.py c86dbf9 ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/configuration/hive-site.xml e56c3e9 ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/params_linux.py 3d6c096 ambari-server/src/main/resources/stacks/HDP/2.3/services/stack_advisor.py f2e9924 ambari-server/src/main/resources/stacks/HDP/2.5/services/ATLAS/configuration/application-properties.xml 99fd20f ambari-server/src/test/python/stacks/2.3/common/test_stack_advisor.py 617f73d Diff: https://reviews.apache.org/r/46740/diff/ Testing --- New stack advisor unit test. mvn clean test Thanks, Tom Beerbower
Review Request 46360: Atlas Integration :Use stack advisor to suggest correct integration related configs
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/46360/ --- Review request for Ambari, John Speidel and Robert Levas. Bugs: AMBARI-15939 https://issues.apache.org/jira/browse/AMBARI-15939 Repository: ambari Description --- Use the stack advisor to suggest the correct configs for the Atlas integrations if the Atlas service is installed. See AMBARI-14920. Also, minor refactoring to move integration specific work into separate scripts. Diffs - ambari-server/src/main/resources/common-services/FALCON/0.5.0.2.1/package/scripts/falcon.py 46457ad ambari-server/src/main/resources/common-services/FALCON/0.5.0.2.1/package/scripts/params_linux.py 84d8b57 ambari-server/src/main/resources/common-services/FALCON/0.5.0.2.1/package/scripts/setup_atlas_falcon.py PRE-CREATION ambari-server/src/main/resources/common-services/SQOOP/1.4.4.2.0/configuration/sqoop-site.xml dd299fe ambari-server/src/main/resources/common-services/SQOOP/1.4.4.2.0/package/scripts/params_linux.py 7db974f ambari-server/src/main/resources/common-services/SQOOP/1.4.4.2.0/package/scripts/setup_atlas_sqoop.py PRE-CREATION ambari-server/src/main/resources/common-services/SQOOP/1.4.4.2.0/package/scripts/sqoop.py 1276a54 ambari-server/src/main/resources/common-services/STORM/0.9.1.2.1/configuration/storm-site.xml cf702c0 ambari-server/src/main/resources/common-services/STORM/0.9.1.2.1/package/scripts/params_linux.py 101ca46 ambari-server/src/main/resources/common-services/STORM/0.9.1.2.1/package/scripts/setup_atlas_storm.py PRE-CREATION ambari-server/src/main/resources/common-services/STORM/0.9.1.2.1/package/scripts/storm.py 74050d8 ambari-server/src/main/resources/stacks/HDP/2.2/services/FALCON/configuration/falcon-startup.properties.xml 5d4fa1e ambari-server/src/main/resources/stacks/HDP/2.3/services/stack_advisor.py 66076f2 ambari-server/src/test/python/stacks/2.3/common/test_stack_advisor.py 6c9fd46 Diff: https://reviews.apache.org/r/46360/diff/ Testing --- Added unit tests for new stack advisor methods. mvn clean test Thanks, Tom Beerbower
Re: Review Request 45937: Support Atlas / Storm integration
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/45937/ --- (Updated April 8, 2016, 8:09 p.m.) Review request for Ambari, John Speidel and Robert Levas. Changes --- updated diff Bugs: AMBARI-14920 https://issues.apache.org/jira/browse/AMBARI-14920 Repository: ambari Description --- Update configurations and copy required libraries. 1 In storm.yaml: - storm.topology.submission.notifier.plugin.class: "org.apache.atlas.storm.hook.StormAtlasHook" - atlas.cluster.name: "cluster_name" 2 In storm_env.ini: - STORM_JAR_JVM_OPTS:"-Datlas.conf=$ATLAS_HOME/conf/" 3 cp $ATLAS_PACKAGE/hook/storm/*.jar to $STORM_HOME/extlib (or symlink'd) Also documented here: https://github.com/apache/incubator-atlas/blob/master/docs/src/site/twiki/StormAtlasHook.twiki Diffs (updated) - ambari-server/src/main/resources/common-services/STORM/0.9.1.2.1/configuration/storm-site.xml aada363 ambari-server/src/main/resources/common-services/STORM/0.9.1.2.1/package/scripts/params_linux.py e9e8ce5 ambari-server/src/main/resources/common-services/STORM/0.9.1.2.1/package/scripts/storm.py d2fafa6 ambari-server/src/main/resources/stacks/HDP/2.3/services/STORM/configuration/storm-env.xml 2bb0862 Diff: https://reviews.apache.org/r/45937/diff/ Testing --- mvn clean test Thanks, Tom Beerbower
Review Request 45896: Atlas Integration : Support Atlas HA
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/45896/ --- Review request for Ambari, John Speidel and Nate Cole. Bugs: AMBARI-15733 https://issues.apache.org/jira/browse/AMBARI-15733 Repository: ambari Description --- 1 set cardinality to 1+ in Atlas stack definition 2 expose Atlas configuration to enable HA (atlas.server.ha.enabled) 3 set Atlas configuration for server id list (atlas.server.ids) 4 set Atlas configuration properties for hosts (atlas.server.host.id1, ...) 5 expose Atlas host component ha_state property in Ambari REST API. 6 Update Ambari quick link definition in Atlas stack definition to show ha_state. (See Yarn RM quick links) * Also refactored HTTPProxyPropertyProvider to remove RM specific code. Diffs - ambari-server/src/main/java/org/apache/ambari/server/controller/internal/AbstractProviderModule.java b77fda2 ambari-server/src/main/java/org/apache/ambari/server/controller/internal/AtlasServerHttpPropertyRequest.java PRE-CREATION ambari-server/src/main/java/org/apache/ambari/server/controller/internal/HostComponentResourceProvider.java 11db913 ambari-server/src/main/java/org/apache/ambari/server/controller/internal/HttpPropertyProvider.java PRE-CREATION ambari-server/src/main/java/org/apache/ambari/server/controller/internal/HttpProxyPropertyProvider.java e92536c ambari-server/src/main/java/org/apache/ambari/server/controller/internal/JsonHttpPropertyRequest.java PRE-CREATION ambari-server/src/main/java/org/apache/ambari/server/controller/internal/ResourceManagerHttpPropertyRequest.java PRE-CREATION ambari-server/src/main/resources/common-services/ATLAS/0.1.0.2.3/package/scripts/params.py b377757 ambari-server/src/main/resources/stacks/HDP/2.5/services/ATLAS/configuration/application-properties.xml PRE-CREATION ambari-server/src/main/resources/stacks/HDP/2.5/services/ATLAS/metainfo.xml 7061d6b ambari-server/src/test/java/org/apache/ambari/server/controller/internal/AtlasServerHttpPropertyRequestTest.java PRE-CREATION ambari-server/src/test/java/org/apache/ambari/server/controller/internal/HttpPropertyProviderTest.java b622728 Diff: https://reviews.apache.org/r/45896/diff/ Testing --- Manual test. mvn clean test new unit tests added Thanks, Tom Beerbower
Review Request 45218: Atlas Integration : Rename Atlas Configurations (2.5 stack definition)
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/45218/ --- Review request for Ambari, John Speidel, Nate Cole, and Sumit Mohanty. Bugs: AMBARI-15431 https://issues.apache.org/jira/browse/AMBARI-15431 Repository: ambari Description --- Move changes for AMBARI-15431 from 2.6 stack definition to 2.5 stack definition. Atlas configuration name application.properties has been changed to atlas-application.properties to avoid name conflicts with other services. See https://issues.apache.org/jira/browse/ATLAS-392. Ambari scripts for Atlas currently use the configuration name application.properties for all stack levels. Stacks which include Atlas > 0.5 should use the configuration name atlas-application.properties. Diffs - ambari-server/src/main/resources/stacks/HDP/2.5/services/ATLAS/configuration/atlas-env.xml PRE-CREATION ambari-server/src/main/resources/stacks/HDP/2.5/services/ATLAS/metainfo.xml 66aea9d ambari-server/src/main/resources/stacks/HDP/2.6/services/ATLAS/configuration/atlas-env.xml 42503b5 ambari-server/src/main/resources/stacks/HDP/2.6/services/ATLAS/metainfo.xml af1a047 ambari-server/src/test/python/stacks/2.5/ATLAS/test_atlas_server.py PRE-CREATION ambari-server/src/test/python/stacks/2.5/configs/default.json PRE-CREATION ambari-server/src/test/python/stacks/2.6/ATLAS/test_atlas_server.py 8e51ea0 ambari-server/src/test/python/stacks/2.6/configs/default.json 2e1bc68 Diff: https://reviews.apache.org/r/45218/diff/ Testing --- Manual test install Atlas (HDP 2.5 stack). Verify configuration. mvn clean test all pass Thanks, Tom Beerbower
Review Request 44878: Atlas Integration : Rename Atlas Configurations
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/44878/ --- Review request for Ambari, John Speidel, Nate Cole, and Sumit Mohanty. Bugs: AMBARI-15431 https://issues.apache.org/jira/browse/AMBARI-15431 Repository: ambari Description --- Atlas configuration name application.properties has been changed to atlas-application.properties to avoid name conflicts with other services. See https://issues.apache.org/jira/browse/ATLAS-392. Ambari scripts for Atlas currently use the configuration name application.properties for all stack levels. Stacks which include Atlas > 0.5 should use the configuration name atlas-application.properties. Diffs - ambari-server/src/main/resources/common-services/ATLAS/0.1.0.2.3/configuration/atlas-env.xml 8500488 ambari-server/src/main/resources/common-services/ATLAS/0.1.0.2.3/package/scripts/metadata.py 6df47b0 ambari-server/src/main/resources/common-services/ATLAS/0.1.0.2.3/package/scripts/metadata_server.py 5a39278 ambari-server/src/main/resources/common-services/ATLAS/0.1.0.2.3/package/scripts/params.py 38c2c9b ambari-server/src/main/resources/common-services/ATLAS/0.1.0.2.3/package/scripts/status_params.py c402721 ambari-server/src/main/resources/common-services/FALCON/0.5.0.2.1/package/scripts/falcon.py b131574 ambari-server/src/main/resources/common-services/FALCON/0.5.0.2.1/package/scripts/params_linux.py 115ed4f ambari-server/src/main/resources/common-services/SQOOP/1.4.4.2.0/package/scripts/params_linux.py ec37243 ambari-server/src/main/resources/common-services/SQOOP/1.4.4.2.0/package/scripts/sqoop.py da67775 ambari-server/src/main/resources/stacks/HDP/2.6/services/ATLAS/configuration/atlas-env.xml PRE-CREATION ambari-server/src/main/resources/stacks/HDP/2.6/services/ATLAS/metainfo.xml 9514576 ambari-server/src/test/python/stacks/2.3/configs/default.json d5102dc ambari-server/src/test/python/stacks/2.3/configs/secure.json 2b60be6 ambari-server/src/test/python/stacks/2.6/ATLAS/test_atlas_server.py PRE-CREATION ambari-server/src/test/python/stacks/2.6/configs/default.json PRE-CREATION Diff: https://reviews.apache.org/r/44878/diff/ Testing --- Manual test install Atlas (HDP 2.6 stack). Verify configuration. mvn clean test all pass Thanks, Tom Beerbower
