from:"Dmytro Grinenko"

Re: Review Request 64983: Setup heartbeat for api endpoint.

2018-01-05 Thread Dmytro Grinenko


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64983/#review194880
---


Ship it!




Ship It!

- Dmytro Grinenko


On Jan. 5, 2018, 5:44 p.m., Myroslav Papirkovskyy wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/64983/
> ---
> 
> (Updated Jan. 5, 2018, 5:44 p.m.)
> 
> 
> Review request for Ambari, Andrew Onischuk, Dmytro Grinenko, Eugene 
> Chekanskiy, and Sid Wagle.
> 
> 
> Bugs: AMBARI-22738
> https://issues.apache.org/jira/browse/AMBARI-22738
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> API endpoint uses sockJS fallback options. The SockJS protocol requires 
> server to set proper heartbeat configuration.
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/java/org/apache/ambari/server/configuration/Configuration.java
>  4dffcaa23ab6a7ca5e2f558ae2bbc3ba89d4cbc4 
>   
> ambari-server/src/main/java/org/apache/ambari/server/configuration/spring/ApiStompConfig.java
>  e968b11f1ae22631cc656003b98ce1da83b25cee 
> 
> 
> Diff: https://reviews.apache.org/r/64983/diff/1/
> 
> 
> Testing
> ---
> 
> manual testing
> 
> 
> Thanks,
> 
> Myroslav Papirkovskyy
> 
>

Re: Review Request 64981: alert_definitions topic doesn't emit any events to client.

2018-01-05 Thread Dmytro Grinenko


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64981/#review194876
---


Ship it!




Ship It!

- Dmytro Grinenko


On Jan. 5, 2018, 4:55 p.m., Myroslav Papirkovskyy wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/64981/
> ---
> 
> (Updated Jan. 5, 2018, 4:55 p.m.)
> 
> 
> Review request for Ambari, Andrew Onischuk, Dmytro Grinenko, Eugene 
> Chekanskiy, and Sid Wagle.
> 
> 
> Bugs: AMBARI-22734
> https://issues.apache.org/jira/browse/AMBARI-22734
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> /alert_definitions topic doesn't emit any events to the client when 
> definition properties are modified.
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/java/org/apache/ambari/server/MessageDestinationIsNotDefinedException.java
>  PRE-CREATION 
>   
> ambari-server/src/main/java/org/apache/ambari/server/configuration/spring/RootStompConfig.java
>  fda0607d52ab6e57d9d925e2b22a3f4186bbbf62 
>   
> ambari-server/src/main/java/org/apache/ambari/server/events/AlertDefinitionsMessageEmitter.java
>  PRE-CREATION 
>   
> ambari-server/src/main/java/org/apache/ambari/server/events/AmbariUpdateEvent.java
>  6991f155bb8942673a4a0be722259b6a7e5bb027 
>   
> ambari-server/src/main/java/org/apache/ambari/server/events/DefaultMessageEmitter.java
>  PRE-CREATION 
>   
> ambari-server/src/main/java/org/apache/ambari/server/events/MessageEmitter.java
>  PRE-CREATION 
>   
> ambari-server/src/main/java/org/apache/ambari/server/events/NamedHostRoleCommandUpdateEvent.java
>  53dc69f7fc0458536698fccdb97e6e97fd0705f6 
>   
> ambari-server/src/main/java/org/apache/ambari/server/events/listeners/requests/StateUpdateListener.java
>  17e8c3d4692cdc0f3fd20ddd5606fef458048d96 
>   
> ambari-server/src/main/java/org/apache/ambari/server/events/listeners/tasks/TaskStatusListener.java
>  58bfcfca289cc2eec8fbef2321683a22eb91e00f 
> 
> 
> Diff: https://reviews.apache.org/r/64981/diff/1/
> 
> 
> Testing
> ---
> 
> manual check
> 
> 
> Thanks,
> 
> Myroslav Papirkovskyy
> 
>

Re: Review Request 64979: Delete host event doesn't have hostName property.

2018-01-05 Thread Dmytro Grinenko


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64979/#review194871
---


Ship it!




Ship It!

- Dmytro Grinenko


On Jan. 5, 2018, 3:57 p.m., Myroslav Papirkovskyy wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/64979/
> ---
> 
> (Updated Jan. 5, 2018, 3:57 p.m.)
> 
> 
> Review request for Ambari, Andrew Onischuk, Dmytro Grinenko, Eugene 
> Chekanskiy, and Sid Wagle.
> 
> 
> Bugs: AMBARI-22735
> https://issues.apache.org/jira/browse/AMBARI-22735
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> When deleting a host, you get an event from /events/ui_topologies, but it 
> doesn't contain hostName property: 
> ```
> {"hash":"922d90764f61dcb2687a33e1e4bfc55b5eb755492b14f48044c940d19324b5e72e669b5af42321383026d46df0775730dd8a0f55a28c625b14f6295bfed02917","eventType":"DELETE","clusters":{"2":{"hosts":[{"hostId":51}]}}}
> ```
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/java/org/apache/ambari/server/agent/stomp/TopologyHolder.java
>  55c0150b9fc54bf608d995155bfbc279711a41bc 
>   
> ambari-server/src/main/java/org/apache/ambari/server/agent/stomp/dto/TopologyHost.java
>  9cdf9442ff96fe730d456037ae05a7becf9a97e9 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/HostResourceProvider.java
>  bd746f57c2dc6a9e948cbb387ce8329d6ff8eb18 
> 
> 
> Diff: https://reviews.apache.org/r/64979/diff/1/
> 
> 
> Testing
> ---
> 
> manual check
> 
> 
> Thanks,
> 
> Myroslav Papirkovskyy
> 
>

Re: Review Request 64977: UI receives a topology event when turning on/off maintenance mode for the host

2018-01-05 Thread Dmytro Grinenko


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64977/#review194867
---


Ship it!




Ship It!

- Dmytro Grinenko


On Jan. 5, 2018, 3:01 p.m., Myroslav Papirkovskyy wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/64977/
> ---
> 
> (Updated Jan. 5, 2018, 3:01 p.m.)
> 
> 
> Review request for Ambari, Andrew Onischuk, Dmytro Grinenko, Eugene 
> Chekanskiy, and Sid Wagle.
> 
> 
> Bugs: AMBARI-22733
> https://issues.apache.org/jira/browse/AMBARI-22733
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> /events/hostcomponents topic should react on maintenance state on/off and 
> stale configs events.
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/java/org/apache/ambari/server/agent/stomp/TopologyHolder.java
>  55c0150b9fc54bf608d995155bfbc279711a41bc 
>   
> ambari-server/src/main/java/org/apache/ambari/server/agent/stomp/dto/TopologyCluster.java
>  c17ec7fd018fcb85672758535fbdc4532f4c895c 
>   
> ambari-server/src/main/java/org/apache/ambari/server/agent/stomp/dto/TopologyComponent.java
>  d96180dc02f362156d135d19e562e874f07545b6 
>   
> ambari-server/src/main/java/org/apache/ambari/server/agent/stomp/dto/TopologyHost.java
>  9cdf9442ff96fe730d456037ae05a7becf9a97e9 
> 
> 
> Diff: https://reviews.apache.org/r/64977/diff/1/
> 
> 
> Testing
> ---
> 
> manual check
> 
> 
> Thanks,
> 
> Myroslav Papirkovskyy
> 
>

Re: Review Request 64951: Idempotent issue on Ambari Upgrade, renameServiceDeletedColumn failed with column already exists exception

2018-01-05 Thread Dmytro Grinenko


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64951/#review194857
---


Ship it!




Ship It!

- Dmytro Grinenko


On Jan. 5, 2018, 1:20 p.m., Vitalyi Brodetskyi wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/64951/
> ---
> 
> (Updated Jan. 5, 2018, 1:20 p.m.)
> 
> 
> Review request for Ambari, Andrew Onischuk, Dmytro Grinenko, and Dmytro Sen.
> 
> 
> Bugs: AMBARI-22724
> https://issues.apache.org/jira/browse/AMBARI-22724
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Issue discovered alongside with another bugs which leads upgrade to fail in 
> the middle with partially applying changes to DB. It leads to exception in 
> code: 
> 
> {code}
>   private void renameServiceDeletedColumn() throws AmbariException, 
> SQLException {
> if (dbAccessor.tableHasColumn(CLUSTER_CONFIG_TABLE, 
> SERVICE_DELETED_COLUMN)) {
>   dbAccessor.renameColumn(CLUSTER_CONFIG_TABLE, SERVICE_DELETED_COLUMN, 
> new DBAccessor.DBColumnInfo(UNMAPPED_COLUMN, Short.class, null, 0, false));
> }
>   }
> {code}
> 
> 
> Exception:  ERROR: column "unmapped" of relation "clusterconfig" already 
> exists   (err.png)
> 
> Table generated DDL from current state: 
> {code}
> -- auto-generated definition
> CREATE TABLE clusterconfig
> (
>   config_id  INT8(19)   NOT NULL
> CONSTRAINT pk_clusterconfig
> PRIMARY KEY,
>   version_tagVARCHAR(255)   NOT NULL,
>   versionINT8(19)   NOT NULL,
>   type_name  VARCHAR(255)   NOT NULL,
>   cluster_id INT8(19)   NOT NULL
> CONSTRAINT fk_clusterconfig_cluster_id
> REFERENCES clusters,
>   stack_id   INT8(19)   NOT NULL
> CONSTRAINT fk_clusterconfig_stack_id
> REFERENCES stack,
>   config_dataTEXT(max)  NOT NULL,
>   config_attributes  TEXT(max),
>   create_timestamp   INT8(19)   NOT NULL,
>   unmapped   INT2(5) DEFAULT 0  NOT NULL,
>   selected   INT2(5) DEFAULT 0  NOT NULL,
>   selected_timestamp INT8(19) DEFAULT 0 NOT NULL,
>   service_deletedINT2(5) DEFAULT 0  NOT NULL
> );
> 
> CREATE UNIQUE INDEX uq_config_type_tag
>   ON clusterconfig (cluster_id, type_name, version_tag);
> 
> CREATE UNIQUE INDEX uq_config_type_version
>   ON clusterconfig (cluster_id, type_name, version);
> 
> 
> {code}
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog252.java
>  e6a7829 
> 
> 
> Diff: https://reviews.apache.org/r/64951/diff/2/
> 
> 
> Testing
> ---
> 
> mvn clean test
> 
> 
> Thanks,
> 
> Vitalyi Brodetskyi
> 
>

Re: Review Request 64951: Idempotent issue on Ambari Upgrade, renameServiceDeletedColumn failed with column already exists exception

2018-01-04 Thread Dmytro Grinenko


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64951/#review194838
---




ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog252.java
Lines 21 (patched)
<https://reviews.apache.org/r/64951/#comment273904>

please check 
https://cwiki.apache.org/confluence/display/AMBARI/Coding+Guidelines+for+Ambarim
 Java Import Order


- Dmytro Grinenko


On Jan. 4, 2018, 5 p.m., Vitalyi Brodetskyi wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/64951/
> ---
> 
> (Updated Jan. 4, 2018, 5 p.m.)
> 
> 
> Review request for Ambari, Andrew Onischuk, Dmytro Grinenko, and Dmytro Sen.
> 
> 
> Bugs: AMBARI-22724
> https://issues.apache.org/jira/browse/AMBARI-22724
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Issue discovered alongside with another bugs which leads upgrade to fail in 
> the middle with partially applying changes to DB. It leads to exception in 
> code: 
> 
> {code}
>   private void renameServiceDeletedColumn() throws AmbariException, 
> SQLException {
> if (dbAccessor.tableHasColumn(CLUSTER_CONFIG_TABLE, 
> SERVICE_DELETED_COLUMN)) {
>   dbAccessor.renameColumn(CLUSTER_CONFIG_TABLE, SERVICE_DELETED_COLUMN, 
> new DBAccessor.DBColumnInfo(UNMAPPED_COLUMN, Short.class, null, 0, false));
> }
>   }
> {code}
> 
> 
> Exception:  ERROR: column "unmapped" of relation "clusterconfig" already 
> exists   (err.png)
> 
> Table generated DDL from current state: 
> {code}
> -- auto-generated definition
> CREATE TABLE clusterconfig
> (
>   config_id  INT8(19)   NOT NULL
> CONSTRAINT pk_clusterconfig
> PRIMARY KEY,
>   version_tagVARCHAR(255)   NOT NULL,
>   versionINT8(19)   NOT NULL,
>   type_name  VARCHAR(255)   NOT NULL,
>   cluster_id INT8(19)   NOT NULL
> CONSTRAINT fk_clusterconfig_cluster_id
> REFERENCES clusters,
>   stack_id   INT8(19)   NOT NULL
> CONSTRAINT fk_clusterconfig_stack_id
> REFERENCES stack,
>   config_dataTEXT(max)  NOT NULL,
>   config_attributes  TEXT(max),
>   create_timestamp   INT8(19)   NOT NULL,
>   unmapped   INT2(5) DEFAULT 0  NOT NULL,
>   selected   INT2(5) DEFAULT 0  NOT NULL,
>   selected_timestamp INT8(19) DEFAULT 0 NOT NULL,
>   service_deletedINT2(5) DEFAULT 0  NOT NULL
> );
> 
> CREATE UNIQUE INDEX uq_config_type_tag
>   ON clusterconfig (cluster_id, type_name, version_tag);
> 
> CREATE UNIQUE INDEX uq_config_type_version
>   ON clusterconfig (cluster_id, type_name, version);
> 
> 
> {code}
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog252.java
>  e6a7829 
> 
> 
> Diff: https://reviews.apache.org/r/64951/diff/1/
> 
> 
> Testing
> ---
> 
> mvn clean test
> 
> 
> Thanks,
> 
> Vitalyi Brodetskyi
> 
>

Re: Review Request 64956: Update Hadoop RPC Encryption Properties During Kerberization and Upgrade

2018-01-04 Thread Dmytro Grinenko


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64956/#review194837
---


Ship it!




Ship It!

- Dmytro Grinenko


On Jan. 4, 2018, 8:16 p.m., Jonathan Hurley wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/64956/
> ---
> 
> (Updated Jan. 4, 2018, 8:16 p.m.)
> 
> 
> Review request for Ambari, Dmytro Grinenko, Dmitro Lisnichenko, and Nate Cole.
> 
> 
> Bugs: AMBARI-22725
> https://issues.apache.org/jira/browse/AMBARI-22725
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Clients should have the ability to choose encrypted communication over RPC 
> when talking to core hadoop components. Today, the properties that control 
> this are:
> 
> - {{core-site.xml : hadoop.rpc.protection = authentication}}
> - {{hdfs-site.xml : dfs.data.transfer.protection = authentication}}
> 
> The new value of {{privacy}} enables clients to choose an encrypted means of 
> communication. By keeping {{authentication}} first, it will be taken as the 
> default mechanism so that wire encryption is not automatically enabled by 
> accident.
> 
> The following properties should be changed to add {{privacy}}:
> 
> - {{core-site.xml : hadoop.rpc.protection = authentication,privacy}}
> - {{hdfs-site.xml : dfs.data.transfer.protection = authentication,privacy}}
> 
> The following are cases when this needs to be performed:
> - During Kerberization, the above two properties should be automatically 
> reconfigured.
> - During a stack upgrade to any version of HDP 2.6, they should be 
> automatically merged
> 
> Blueprint deployment is not a scenario being covered here.
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/java/org/apache/ambari/server/serveraction/upgrades/ConfigureAction.java
>  fbcde51b32 
>   
> ambari-server/src/main/java/org/apache/ambari/server/state/stack/upgrade/ClusterGrouping.java
>  c1a05c03b4 
>   
> ambari-server/src/main/java/org/apache/ambari/server/state/stack/upgrade/StageWrapperBuilder.java
>  7fd8938f7d 
>   
> ambari-server/src/main/java/org/apache/ambari/server/state/stack/upgrade/Task.java
>  2167b7b464 
>   ambari-server/src/main/resources/stacks/HDP/2.2/services/stack_advisor.py 
> 0b0c9c56af 
>   ambari-server/src/main/resources/stacks/HDP/2.6/services/HDFS/kerberos.json 
> f8bdc5cc5c 
>   ambari-server/src/main/resources/stacks/HDP/2.6/upgrades/config-upgrade.xml 
> 94787225f1 
>   
> ambari-server/src/main/resources/stacks/HDP/2.6/upgrades/nonrolling-upgrade-2.6.xml
>  d506f1f16c 
>   ambari-server/src/main/resources/stacks/HDP/2.6/upgrades/upgrade-2.6.xml 
> 17a63943ba 
>   ambari-server/src/main/resources/upgrade-pack.xsd 9e50a087b6 
>   
> ambari-server/src/test/java/org/apache/ambari/server/state/UpgradeHelperTest.java
>  ac9be666c7 
>   
> ambari-server/src/test/resources/stacks/HDP/2.2.0/upgrades/upgrade_test_conditions.xml
>  61c891a4be 
> 
> 
> Diff: https://reviews.apache.org/r/64956/diff/2/
> 
> 
> Testing
> ---
> 
> --
> Total run:1201
> Total errors:0
> Total failures:0
> OK
> [INFO] 
> 
> [INFO] BUILD SUCCESS
> [INFO] 
> 
> [INFO] Total time: 01:01 min
> [INFO] Finished at: 2018-01-04T15:55:20-05:00
> [INFO] Final Memory: 21M/619M
> [INFO] 
> 
> 
> 
> Thanks,
> 
> Jonathan Hurley
> 
>

Re: Review Request 64722: [Patch Hive]webhcat: test_sqoop fails with hdfs:///hdp/apps/2.6.*/sqoop/sqoop.tar.gz does not exist

2017-12-20 Thread Dmytro Grinenko


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64722/#review194244
---


Ship it!




Ship It!

- Dmytro Grinenko


On Dec. 19, 2017, 8:58 p.m., Nate Cole wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/64722/
> ---
> 
> (Updated Dec. 19, 2017, 8:58 p.m.)
> 
> 
> Review request for Ambari, Dmytro Grinenko, Dmitro Lisnichenko, and Jonathan 
> Hurley.
> 
> 
> Bugs: AMBARI-22676
> https://issues.apache.org/jira/browse/AMBARI-22676
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Add SQOOP to the list of services that should go with HIVE
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/resources/stacks/HDP/2.0.6/properties/stack_packages.json
>  62a46b91bd 
> 
> 
> Diff: https://reviews.apache.org/r/64722/diff/1/
> 
> 
> Testing
> ---
> 
> No new tests for json change.
> 
> 
> Thanks,
> 
> Nate Cole
> 
>

Re: Review Request 64652: unable to proceed with cluster install after component install fails.

2017-12-15 Thread Dmytro Grinenko


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64652/#review193938
---


Ship it!




Ship It!

- Dmytro Grinenko


On Dec. 15, 2017, 5 p.m., Myroslav Papirkovskyy wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/64652/
> ---
> 
> (Updated Dec. 15, 2017, 5 p.m.)
> 
> 
> Review request for Ambari, Dmitro Lisnichenko, Jonathan Hurley, and Nate Cole.
> 
> 
> Bugs: AMBARI-22659
> https://issues.apache.org/jira/browse/AMBARI-22659
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Unable go back in install wizard if install was finished at one of hosts.
> 
> ```
> Request 
> URL:http://ambari:8080/api/v1/stacks/HDP/versions/2.6/repository_versions/1
> 
> {
>   "status" : 500,
>   "message" : "org.apache.ambari.server.controller.spi.SystemException: 
> Repository version can't be deleted as it is used by the following hosts: 
> CURRENT on sanjay-divgi-test-2.openstacklocal, 
> sanjay-divgi-test-5.openstacklocal, sanjay-divgi-test-1.openstacklocal, 
> sanjay-divgi-test-4.openstacklocal"
> }
> ```
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/java/org/apache/ambari/server/orm/dao/HostVersionDAO.java
>  158370f3528f796e30a46862e8eb42882001ca02 
>   
> ambari-server/src/main/java/org/apache/ambari/server/orm/entities/HostVersionEntity.java
>  4a030af122dbe3ceb3c3592c000b126c864dc626 
>   
> ambari-server/src/main/java/org/apache/ambari/server/state/cluster/ClusterImpl.java
>  c23b971b85a843433f230b5a94c66ac46c5b1914 
>   
> ambari-server/src/test/java/org/apache/ambari/server/orm/dao/HostVersionDAOTest.java
>  bb077d6667138acdaeaafa8b07f188b2ce3dc2af 
> 
> 
> Diff: https://reviews.apache.org/r/64652/diff/1/
> 
> 
> Testing
> ---
> 
> mvn clean test
> manual check
> 
> 
> Thanks,
> 
> Myroslav Papirkovskyy
> 
>

Re: Review Request 64652: unable to proceed with cluster install after component install fails.

2017-12-15 Thread Dmytro Grinenko


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64652/#review193936
---




ambari-server/src/main/java/org/apache/ambari/server/state/cluster/ClusterImpl.java
Lines 1337 (patched)
<https://reviews.apache.org/r/64652/#comment272602>

i'm not sure on this, for example: if repoVersion were in the installed 
state and then we resetting it,  packages on the hosts would left; however 
initial state would allow DEREGISTER repo version, and register new one with 
same id but different actual version.

Would this cause any issue?


- Dmytro Grinenko


On Dec. 15, 2017, 5 p.m., Myroslav Papirkovskyy wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/64652/
> ---
> 
> (Updated Dec. 15, 2017, 5 p.m.)
> 
> 
> Review request for Ambari, Dmitro Lisnichenko, Jonathan Hurley, and Nate Cole.
> 
> 
> Bugs: AMBARI-22659
> https://issues.apache.org/jira/browse/AMBARI-22659
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Unable go back in install wizard if install was finished at one of hosts.
> 
> ```
> Request 
> URL:http://ambari:8080/api/v1/stacks/HDP/versions/2.6/repository_versions/1
> 
> {
>   "status" : 500,
>   "message" : "org.apache.ambari.server.controller.spi.SystemException: 
> Repository version can't be deleted as it is used by the following hosts: 
> CURRENT on sanjay-divgi-test-2.openstacklocal, 
> sanjay-divgi-test-5.openstacklocal, sanjay-divgi-test-1.openstacklocal, 
> sanjay-divgi-test-4.openstacklocal"
> }
> ```
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/java/org/apache/ambari/server/orm/dao/HostVersionDAO.java
>  158370f3528f796e30a46862e8eb42882001ca02 
>   
> ambari-server/src/main/java/org/apache/ambari/server/orm/entities/HostVersionEntity.java
>  4a030af122dbe3ceb3c3592c000b126c864dc626 
>   
> ambari-server/src/main/java/org/apache/ambari/server/state/cluster/ClusterImpl.java
>  c23b971b85a843433f230b5a94c66ac46c5b1914 
>   
> ambari-server/src/test/java/org/apache/ambari/server/orm/dao/HostVersionDAOTest.java
>  bb077d6667138acdaeaafa8b07f188b2ce3dc2af 
> 
> 
> Diff: https://reviews.apache.org/r/64652/diff/1/
> 
> 
> Testing
> ---
> 
> mvn clean test
> manual check
> 
> 
> Thanks,
> 
> Myroslav Papirkovskyy
> 
>

Re: Review Request 64637: Livy/Livy2 Unable To Start Due to Address Already In Use

2017-12-15 Thread Dmytro Grinenko


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64637/#review193904
---


Ship it!




Ship It!

- Dmytro Grinenko


On Dec. 15, 2017, 3:38 a.m., Jonathan Hurley wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/64637/
> ---
> 
> (Updated Dec. 15, 2017, 3:38 a.m.)
> 
> 
> Review request for Ambari, Dmytro Grinenko, Dmitro Lisnichenko, and Nate Cole.
> 
> 
> Bugs: AMBARI-22655
> https://issues.apache.org/jira/browse/AMBARI-22655
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> While restarting Livy and Livy2 on a non-root cluster, the following is seen:
> 
> ```
> 17/12/14 14:36:23 WARN LivyConf: The configuration key 
> livy.repl.enableHiveContext has been deprecated as of Livy 0.4 and may be 
> removed in the future. Please use the new key livy.repl.enable-hive-context 
> instead.
> 17/12/14 14:36:23 WARN LivyConf: The configuration key 
> livy.server.csrf_protection.enabled has been deprecated as of Livy 0.4 and 
> may be removed in the future. Please use the new key 
> livy.server.csrf-protection.enabled instead.
> 17/12/14 14:36:23 INFO AccessManager: AccessControlManager acls 
> disabled;users with view permission: ;users with modify permission: ;users 
> with super permission: cstm-zeppelin;other allowed users: *
> 17/12/14 14:36:28 INFO LineBufferedStream: stdout: Welcome to
> 17/12/14 14:36:28 INFO LineBufferedStream: stdout:     __
> 17/12/14 14:36:28 INFO LineBufferedStream: stdout:  / __/__  ___ _/ 
> /__
> 17/12/14 14:36:28 INFO LineBufferedStream: stdout: _\ \/ _ \/ _ `/ __/  
> '_/
> 17/12/14 14:36:28 INFO LineBufferedStream: stdout:/___/ .__/_,_/_/ /_/_\  
>  version 2.2.0.2.6.4.0-73
> 17/12/14 14:36:28 INFO LineBufferedStream: stdout:   /_/
> 17/12/14 14:36:28 INFO LineBufferedStream: stdout:
> 17/12/14 14:36:28 INFO LineBufferedStream: stdout: Using Scala version 
> 2.11.8, OpenJDK 64-Bit Server VM, 1.8.0_131
> 17/12/14 14:36:28 INFO LineBufferedStream: stdout: Branch HEAD
> 17/12/14 14:36:28 INFO LineBufferedStream: stdout: Compiled by user jenkins 
> on 2017-12-13T19:08:32Z
> 17/12/14 14:36:28 INFO LineBufferedStream: stdout: Revision 
> a24017869f5450397136ee8b11be818e7cd3facb
> 17/12/14 14:36:28 INFO LineBufferedStream: stdout: Url 
> g...@github.com:hortonworks/spark2.git
> 17/12/14 14:36:28 INFO LineBufferedStream: stdout: Type --help for more 
> information.
> 17/12/14 14:36:29 WARN NativeCodeLoader: Unable to load native-hadoop library 
> for your platform... using builtin-java classes where applicable
> 17/12/14 14:36:31 INFO AHSProxy: Connecting to Application History server at 
> nat-yc-r7-ovvs-ambari-autostart-4-re-2.openstacklocal/172.22.121.144:10200
> 17/12/14 14:36:32 WARN DomainSocketFactory: The short-circuit local reads 
> feature cannot be used because libhadoop cannot be loaded.
> 17/12/14 14:36:33 INFO StateStore$: Using FileSystemStateStore for recovery.
> 17/12/14 14:36:33 INFO BatchSessionManager: Recovered 0 batch sessions. Next 
> session id: 0
> 17/12/14 14:36:33 INFO InteractiveSessionManager: Recovered 0 interactive 
> sessions. Next session id: 0
> 17/12/14 14:36:33 INFO InteractiveSessionManager: Heartbeat watchdog thread 
> started.
> 17/12/14 14:36:33 INFO LivyServer: SPNEGO auth enabled (principal = 
> HTTP/nat-yc-r7-ovvs-ambari-autostart-4-re-3.openstacklo...@example.com)
> 17/12/14 14:36:33 INFO LivyServer: CSRF protection is enabled.
> 17/12/14 14:36:34 INFO KerberosAuthenticationHandler: Login using keytab 
> /etc/security/keytabs/spnego.service.keytab, for principal 
> HTTP/nat-yc-r7-ovvs-ambari-autostart-4-re-3.openstacklo...@example.com
> 17/12/14 14:36:34 INFO KerberosAuthenticationHandler: Map server: 
> nat-yc-r7-ovvs-ambari-autostart-4-re-3.openstacklocal to principal: 
> [HTTP/nat-yc-r7-ovvs-ambari-autostart-4-re-3.openstacklo...@example.com], 
> added = true
> 17/12/14 14:36:34 WARN AbstractLifeCycle: FAILED 
> ServerConnector@df1cff6{SSL-http/1.1}{0.0.0.0:8999}: java.net.BindException: 
> Address already in use
> java.net.BindException: Address already in use
> at sun.nio.ch.Net.bind0(Native Method)
> ```
> 
> This occurs because the PID file cannot be accessed by the non-root agent and 
> returns an exit code of 1:
> {code}
> call returned (1, 'cat: /var/run/livy/livy-cstm-livy-server.pid: Permission 
> denied')
> {code}
> 
> This tricks out PID detection int

Re: Review Request 64637: Livy/Livy2 Unable To Start Due to Address Already In Use

2017-12-14 Thread Dmytro Grinenko


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64637/#review193874
---




ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/livy_service.py
Line 48 (original), 52 (patched)
<https://reviews.apache.org/r/64637/#comment272532>

i don't understand why we removing the pid file, but let's fix this later


- Dmytro Grinenko


On Dec. 15, 2017, 3:03 a.m., Jonathan Hurley wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/64637/
> ---
> 
> (Updated Dec. 15, 2017, 3:03 a.m.)
> 
> 
> Review request for Ambari, Dmytro Grinenko, Dmitro Lisnichenko, and Nate Cole.
> 
> 
> Bugs: AMBARI-22655
> https://issues.apache.org/jira/browse/AMBARI-22655
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> While restarting Livy and Livy2 on a non-root cluster, the following is seen:
> 
> ```
> 17/12/14 14:36:23 WARN LivyConf: The configuration key 
> livy.repl.enableHiveContext has been deprecated as of Livy 0.4 and may be 
> removed in the future. Please use the new key livy.repl.enable-hive-context 
> instead.
> 17/12/14 14:36:23 WARN LivyConf: The configuration key 
> livy.server.csrf_protection.enabled has been deprecated as of Livy 0.4 and 
> may be removed in the future. Please use the new key 
> livy.server.csrf-protection.enabled instead.
> 17/12/14 14:36:23 INFO AccessManager: AccessControlManager acls 
> disabled;users with view permission: ;users with modify permission: ;users 
> with super permission: cstm-zeppelin;other allowed users: *
> 17/12/14 14:36:28 INFO LineBufferedStream: stdout: Welcome to
> 17/12/14 14:36:28 INFO LineBufferedStream: stdout:     __
> 17/12/14 14:36:28 INFO LineBufferedStream: stdout:  / __/__  ___ _/ 
> /__
> 17/12/14 14:36:28 INFO LineBufferedStream: stdout: _\ \/ _ \/ _ `/ __/  
> '_/
> 17/12/14 14:36:28 INFO LineBufferedStream: stdout:/___/ .__/_,_/_/ /_/_\  
>  version 2.2.0.2.6.4.0-73
> 17/12/14 14:36:28 INFO LineBufferedStream: stdout:   /_/
> 17/12/14 14:36:28 INFO LineBufferedStream: stdout:
> 17/12/14 14:36:28 INFO LineBufferedStream: stdout: Using Scala version 
> 2.11.8, OpenJDK 64-Bit Server VM, 1.8.0_131
> 17/12/14 14:36:28 INFO LineBufferedStream: stdout: Branch HEAD
> 17/12/14 14:36:28 INFO LineBufferedStream: stdout: Compiled by user jenkins 
> on 2017-12-13T19:08:32Z
> 17/12/14 14:36:28 INFO LineBufferedStream: stdout: Revision 
> a24017869f5450397136ee8b11be818e7cd3facb
> 17/12/14 14:36:28 INFO LineBufferedStream: stdout: Url 
> g...@github.com:hortonworks/spark2.git
> 17/12/14 14:36:28 INFO LineBufferedStream: stdout: Type --help for more 
> information.
> 17/12/14 14:36:29 WARN NativeCodeLoader: Unable to load native-hadoop library 
> for your platform... using builtin-java classes where applicable
> 17/12/14 14:36:31 INFO AHSProxy: Connecting to Application History server at 
> nat-yc-r7-ovvs-ambari-autostart-4-re-2.openstacklocal/172.22.121.144:10200
> 17/12/14 14:36:32 WARN DomainSocketFactory: The short-circuit local reads 
> feature cannot be used because libhadoop cannot be loaded.
> 17/12/14 14:36:33 INFO StateStore$: Using FileSystemStateStore for recovery.
> 17/12/14 14:36:33 INFO BatchSessionManager: Recovered 0 batch sessions. Next 
> session id: 0
> 17/12/14 14:36:33 INFO InteractiveSessionManager: Recovered 0 interactive 
> sessions. Next session id: 0
> 17/12/14 14:36:33 INFO InteractiveSessionManager: Heartbeat watchdog thread 
> started.
> 17/12/14 14:36:33 INFO LivyServer: SPNEGO auth enabled (principal = 
> HTTP/nat-yc-r7-ovvs-ambari-autostart-4-re-3.openstacklo...@example.com)
> 17/12/14 14:36:33 INFO LivyServer: CSRF protection is enabled.
> 17/12/14 14:36:34 INFO KerberosAuthenticationHandler: Login using keytab 
> /etc/security/keytabs/spnego.service.keytab, for principal 
> HTTP/nat-yc-r7-ovvs-ambari-autostart-4-re-3.openstacklo...@example.com
> 17/12/14 14:36:34 INFO KerberosAuthenticationHandler: Map server: 
> nat-yc-r7-ovvs-ambari-autostart-4-re-3.openstacklocal to principal: 
> [HTTP/nat-yc-r7-ovvs-ambari-autostart-4-re-3.openstacklo...@example.com], 
> added = true
> 17/12/14 14:36:34 WARN AbstractLifeCycle: FAILED 
> ServerConnector@df1cff6{SSL-http/1.1}{0.0.0.0:8999}: java.net.BindException: 
> Address already in use
> java.net.BindException: Address already in use
> at sun.nio.ch.Net.bind0(Native Method)
> ```
> 
> This occurs because the PID f

Re: Review Request 64637: Livy/Livy2 Unable To Start Due to Address Already In Use

2017-12-14 Thread Dmytro Grinenko


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64637/#review193873
---


Ship it!




Ship It!

- Dmytro Grinenko


On Dec. 15, 2017, 3:03 a.m., Jonathan Hurley wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/64637/
> ---
> 
> (Updated Dec. 15, 2017, 3:03 a.m.)
> 
> 
> Review request for Ambari, Dmytro Grinenko, Dmitro Lisnichenko, and Nate Cole.
> 
> 
> Bugs: AMBARI-22655
> https://issues.apache.org/jira/browse/AMBARI-22655
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> While restarting Livy and Livy2 on a non-root cluster, the following is seen:
> 
> ```
> 17/12/14 14:36:23 WARN LivyConf: The configuration key 
> livy.repl.enableHiveContext has been deprecated as of Livy 0.4 and may be 
> removed in the future. Please use the new key livy.repl.enable-hive-context 
> instead.
> 17/12/14 14:36:23 WARN LivyConf: The configuration key 
> livy.server.csrf_protection.enabled has been deprecated as of Livy 0.4 and 
> may be removed in the future. Please use the new key 
> livy.server.csrf-protection.enabled instead.
> 17/12/14 14:36:23 INFO AccessManager: AccessControlManager acls 
> disabled;users with view permission: ;users with modify permission: ;users 
> with super permission: cstm-zeppelin;other allowed users: *
> 17/12/14 14:36:28 INFO LineBufferedStream: stdout: Welcome to
> 17/12/14 14:36:28 INFO LineBufferedStream: stdout:     __
> 17/12/14 14:36:28 INFO LineBufferedStream: stdout:  / __/__  ___ _/ 
> /__
> 17/12/14 14:36:28 INFO LineBufferedStream: stdout: _\ \/ _ \/ _ `/ __/  
> '_/
> 17/12/14 14:36:28 INFO LineBufferedStream: stdout:/___/ .__/_,_/_/ /_/_\  
>  version 2.2.0.2.6.4.0-73
> 17/12/14 14:36:28 INFO LineBufferedStream: stdout:   /_/
> 17/12/14 14:36:28 INFO LineBufferedStream: stdout:
> 17/12/14 14:36:28 INFO LineBufferedStream: stdout: Using Scala version 
> 2.11.8, OpenJDK 64-Bit Server VM, 1.8.0_131
> 17/12/14 14:36:28 INFO LineBufferedStream: stdout: Branch HEAD
> 17/12/14 14:36:28 INFO LineBufferedStream: stdout: Compiled by user jenkins 
> on 2017-12-13T19:08:32Z
> 17/12/14 14:36:28 INFO LineBufferedStream: stdout: Revision 
> a24017869f5450397136ee8b11be818e7cd3facb
> 17/12/14 14:36:28 INFO LineBufferedStream: stdout: Url 
> g...@github.com:hortonworks/spark2.git
> 17/12/14 14:36:28 INFO LineBufferedStream: stdout: Type --help for more 
> information.
> 17/12/14 14:36:29 WARN NativeCodeLoader: Unable to load native-hadoop library 
> for your platform... using builtin-java classes where applicable
> 17/12/14 14:36:31 INFO AHSProxy: Connecting to Application History server at 
> nat-yc-r7-ovvs-ambari-autostart-4-re-2.openstacklocal/172.22.121.144:10200
> 17/12/14 14:36:32 WARN DomainSocketFactory: The short-circuit local reads 
> feature cannot be used because libhadoop cannot be loaded.
> 17/12/14 14:36:33 INFO StateStore$: Using FileSystemStateStore for recovery.
> 17/12/14 14:36:33 INFO BatchSessionManager: Recovered 0 batch sessions. Next 
> session id: 0
> 17/12/14 14:36:33 INFO InteractiveSessionManager: Recovered 0 interactive 
> sessions. Next session id: 0
> 17/12/14 14:36:33 INFO InteractiveSessionManager: Heartbeat watchdog thread 
> started.
> 17/12/14 14:36:33 INFO LivyServer: SPNEGO auth enabled (principal = 
> HTTP/nat-yc-r7-ovvs-ambari-autostart-4-re-3.openstacklo...@example.com)
> 17/12/14 14:36:33 INFO LivyServer: CSRF protection is enabled.
> 17/12/14 14:36:34 INFO KerberosAuthenticationHandler: Login using keytab 
> /etc/security/keytabs/spnego.service.keytab, for principal 
> HTTP/nat-yc-r7-ovvs-ambari-autostart-4-re-3.openstacklo...@example.com
> 17/12/14 14:36:34 INFO KerberosAuthenticationHandler: Map server: 
> nat-yc-r7-ovvs-ambari-autostart-4-re-3.openstacklocal to principal: 
> [HTTP/nat-yc-r7-ovvs-ambari-autostart-4-re-3.openstacklo...@example.com], 
> added = true
> 17/12/14 14:36:34 WARN AbstractLifeCycle: FAILED 
> ServerConnector@df1cff6{SSL-http/1.1}{0.0.0.0:8999}: java.net.BindException: 
> Address already in use
> java.net.BindException: Address already in use
> at sun.nio.ch.Net.bind0(Native Method)
> ```
> 
> This occurs because the PID file cannot be accessed by the non-root agent and 
> returns an exit code of 1:
> {code}
> call returned (1, 'cat: /var/run/livy/livy-cstm-livy-server.pid: Permission 
> denied')
> {code}
> 
> This tricks out PID detection int

Re: Review Request 64579: Node Managers fail to start after Spark2 is patched due to CNF YarnShuffleService

2017-12-13 Thread Dmytro Grinenko


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64579/#review193721
---


Ship it!




Ship It!

- Dmytro Grinenko


On Dec. 13, 2017, 5:12 p.m., Jonathan Hurley wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/64579/
> ---
> 
> (Updated Dec. 13, 2017, 5:12 p.m.)
> 
> 
> Review request for Ambari, Dmytro Grinenko, Dmitro Lisnichenko, and Nate Cole.
> 
> 
> Bugs: AMBARI-22644
> https://issues.apache.org/jira/browse/AMBARI-22644
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> *STR*
> # Deploy HDP-2.6.4.0 cluster with Ambari-2.6.1.0-114
> # Apply HBase patch Upgrade on the cluster (this step is optional)
> # Then apply Spark2 patch Upgrade on the cluster
> # Restart Node Managers
> 
> *Result*
> NM restart fails with below error:
> ```
> 2017-12-10 07:17:02,559 INFO  impl.MetricsSystemImpl 
> (MetricsSystemImpl.java:shutdown(606)) - NodeManager metrics system shutdown 
> complete.
> 2017-12-10 07:17:02,559 FATAL nodemanager.NodeManager 
> (NodeManager.java:initAndStartNodeManager(549)) - Error starting NodeManager
> Caused by: java.lang.ClassNotFoundException: 
> org.apache.spark.network.yarn.YarnShuffleService
> at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> at 
> org.apache.hadoop.util.ApplicationClassLoader.loadClass(ApplicationClassLoader.java:197)
> at 
> org.apache.hadoop.util.ApplicationClassLoader.loadClass(ApplicationClassLoader.java:165)
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:348)
> at 
> org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxiliaryServiceWithCustomClassLoader.getInstance(AuxiliaryServiceWithCustomClassLoader.java:169)
> at 
> org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices.serviceInit(AuxServices.java:131)
> at 
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
> ... 8 more
> 2017-12-10 07:17:02,562 INFO  nodemanager.NodeManager 
> (LogAdapter.java:info(45)) - SHUTDOWN_MSG:
> ```
> 
> The spark properties are correctly being written out as per AMBARI-22525.
> 
> Initially, we had defined Spark properties for ATS like this:
> ```xml
> yarn.nodemanager.aux-services.spark_shuffle.classpath
> {{stack_root}}/${hdp.version}/spark/aux/*
> ```
> 
> When YARN upgrades without Spark, we run into AMBARI-22525. Seems like the 
> shuffle classes are installed as part of RPM dependencies, but not the 
> SparkATSPlugin.
> 
> So:
> - If we use YARN's version for the Spark classes, then ATS can't find 
> SparkATSPlugin since that is not part of YARN.
> - If we use Spark's version for the classes, then Spark can never upgrade 
> without YARN since NodeManager can't find the new Spark classes. 
> 
> However, it seems like shuffle and ATS use different properties. We changed 
> all 3 properties in AMBARI-22525:
> 
> ```
> yarn.nodemanager.aux-services.spark2_shuffle.classpath
> yarn.nodemanager.aux-services.spark_shuffle.classpath
> yarn.timeline-service.entity-group-fs-store.group-id-plugin-classpath
> ```
> 
> It seems like what need to do is change the spark shuffle stuff back to 
> hdp.version, but leave ATS using the new version since we're guaranteed to 
> have Spark installed on the ATS machine.
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/params_linux.py
>  6b5559cf91 
>   
> ambari-server/src/main/resources/stacks/HDP/2.5/services/YARN/configuration/yarn-site.xml
>  29833fbe03 
>   ambari-server/src/main/resources/stacks/HDP/2.6/upgrades/config-upgrade.xml 
> ea0e2cd46b 
> 
> 
> Diff: https://reviews.apache.org/r/64579/diff/2/
> 
> 
> Testing
> ---
> 
> Manual testing on a patched cluster with YARN/Spark
> 
> 
> Thanks,
> 
> Jonathan Hurley
> 
>

Re: Review Request 64554: HBase Cannot Find LZO Classes After Being Patched

2017-12-13 Thread Dmytro Grinenko


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64554/#review193720
---


Ship it!




Ship It!

- Dmytro Grinenko


On Dec. 13, 2017, 3:46 p.m., Jonathan Hurley wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/64554/
> ---
> 
> (Updated Dec. 13, 2017, 3:46 p.m.)
> 
> 
> Review request for Ambari, Dmytro Grinenko, Dmitro Lisnichenko, and Nate Cole.
> 
> 
> Bugs: AMBARI-22640
> https://issues.apache.org/jira/browse/AMBARI-22640
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> After patching HBase where LZO compression is being used, the following is 
> seen:
> 
> ```
> 2017-12-10 22:31:09,244|INFO|MainThread|machine.py:164 - 
> run()||GUID=37b565a7-e164-4641-b335-19884c614ffd|Exception in thread "main" 
> java.lang.RuntimeException: java.lang.ClassNotFoundException: 
> com.hadoop.compression.lzo.LzoCodec
> 2017-12-10 22:31:09,245|INFO|MainThread|machine.py:164 - 
> run()||GUID=37b565a7-e164-4641-b335-19884c614ffd|at 
> org.apache.hadoop.hbase.io.compress.Compression$Algorithm$1.buildCodec(Compression.java:130)
> 2017-12-10 22:31:09,245|INFO|MainThread|machine.py:164 - 
> run()||GUID=37b565a7-e164-4641-b335-19884c614ffd|at 
> org.apache.hadoop.hbase.io.compress.Compression$Algorithm$1.getCodec(Compression.java:116)
> 2017-12-10 22:31:09,245|INFO|MainThread|machine.py:164 - 
> run()||GUID=37b565a7-e164-4641-b335-19884c614ffd|at 
> org.apache.hadoop.hbase.io.compress.Compression$Algorithm.getCompressor(Compression.java:301)
> 2017-12-10 22:31:09,245|INFO|MainThread|machine.py:164 - 
> run()||GUID=37b565a7-e164-4641-b335-19884c614ffd|at 
> org.apache.hadoop.hbase.io.encoding.HFileBlockDefaultEncodingContext.2017-12-10
>  22:31:09,267|INFO|MainThread|machine.py:164 - 
> run()||GUID=37b565a7-e164-4641-b335-19884c614ffd|at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> 2017-12-10 22:31:09,268|INFO|MainThread|machine.py:164 - 
> run()||GUID=37b565a7-e164-4641-b335-19884c614ffd|at 
> java.util.concurrent.FutureTask.run(FutureTask.java:266)
> 2017-12-10 22:31:09,268|INFO|MainThread|machine.py:164 - 
> run()||GUID=37b565a7-e164-4641-b335-19884c614ffd|at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> 2017-12-10 22:31:09,268|INFO|MainThread|machine.py:164 - 
> run()||GUID=37b565a7-e164-4641-b335-19884c614ffd|at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> 2017-12-10 22:31:09,268|INFO|MainThread|machine.py:164 - 
> run()||GUID=37b565a7-e164-4641-b335-19884c614ffd|at 
> java.lang.Thread.run(Thread.java:745)
> 2017-12-10 22:31:09,741|INFO|MainThread|machine.py:189 - 
> run()||GUID=37b565a7-e164-4641-b335-19884c614ffd|Exit Code: 1
> ```
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/resources/common-services/HBASE/0.96.0.2.0/package/scripts/hbase.py
>  ac71ce4b36 
> 
> 
> Diff: https://reviews.apache.org/r/64554/diff/2/
> 
> 
> Testing
> ---
> 
> Manual testing on Hbase cluster.
> 
> 
> Thanks,
> 
> Jonathan Hurley
> 
>

Re: Review Request 64311: AMBARI-22574 Failed to restart services on PPC cluster post Ambari upgrade for IOP/HDP migration

2017-12-05 Thread Dmytro Grinenko



> On Dec. 5, 2017, 1:01 p.m., Jonathan Hurley wrote:
> > ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog261.java
> > Lines 154 (patched)
> > 
> >
> > We really shouldn't be hardcoding stack names and versions like this. 
> > 
> > what change caused this problem? there must be a better workaround
> 
> Di Li wrote:
> Hello Jonathan,
> 
> This is the code change that caused the err >> "AMBARI-22318 
> repositoryFile entity populating for wrong repository for RU (dgrinenko)"
> 
> BigInsights 4.2.5 uses "redhat7" as the os type in the repo_version table 
> instead of "redhat-ppc7" expected by Dmytro's code. Since this blocks the 
> IOP/HDP migration, it has to happen at the ambari-upgrade time and needs to 
> deal with BigInsights 4.2.5 repositories.
> 
> Tim Thorpe wrote:
> Since the next release of 2.6 is supposed to be the last that supports 
> migration, this code should only be there for a short time.
> 
> Di Li wrote:
> Hello Jonathan,
> 
> I provided a more generic fix that handles ALL records in the 
> repo_version table. It also adds a new redhat-ppc7 section instead of 
> modifying the existing redhat7 one. The idea is so that users are able to 
> update repo urls post Ambari upgrade.

@Di Li   it is not my code expects redhat-ppc7, it is change introduced at 
os_family script, which adding ppc fow any platform on PowerPC CPU. It  means 
that on Ambari prior 2.6 we used redhat7 as platform, same as for x86 arch, 
however with 2.6 we start to distinguish them.


- Dmytro


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64311/#review192856
---


On Dec. 5, 2017, 7:11 p.m., Di Li wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/64311/
> ---
> 
> (Updated Dec. 5, 2017, 7:11 p.m.)
> 
> 
> Review request for Ambari, Jonathan Hurley and Tim Thorpe.
> 
> 
> Bugs: AMBARI-22574
> https://issues.apache.org/jira/browse/AMBARI-22574
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> The cluster started as IOP 4.2.5 on PPC with NameNode HA enabled. I upgraded 
> Ambari to Ambari 2.6.1. The NN HA setting does not matter to the issue.
> After the Ambari upgrade, I can't restart services by clicking the "Restart" 
> button on each service with that "restart needed" icon. Nor can I abort the 
> requests.
> Ambari server log contains the following error on unable to find redhat-ppc7 
> os type.
> java.lang.RuntimeException: 
> org.apache.ambari.server.controller.spi.SystemException: Operating System 
> matching redhat-ppc7 could not be found
> at 
> org.apache.ambari.server.actionmanager.ExecutionCommandWrapper.getExecutionCommand(ExecutionCommandWrapper.java:253)
> at 
> org.apache.ambari.server.actionmanager.ActionScheduler.abortOperationsForStage(ActionScheduler.java:880)
> at 
> org.apache.ambari.server.actionmanager.ActionScheduler.processCancelledRequestsList(ActionScheduler.java:1172)
> at 
> org.apache.ambari.server.actionmanager.ActionScheduler.doWork(ActionScheduler.java:333)
> at 
> org.apache.ambari.server.actionmanager.ActionScheduler.run(ActionScheduler.java:310)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: org.apache.ambari.server.controller.spi.SystemException: Operating 
> System matching redhat-ppc7 could not be found
> at 
> org.apache.ambari.server.state.stack.upgrade.RepositoryVersionHelper.getOSEntityForHost(RepositoryVersionHelper.java:422)
> at 
> org.apache.ambari.server.actionmanager.ExecutionCommandWrapper.getExecutionCommand(ExecutionCommandWrapper.java:247)
> repo_version table had "redhat7" as the os type in the repository 
> information. I worked around the issue by updating repo_version table to have 
> redhat-ppc7 as the os type via the following SQL query:
> update repo_version set 
> repositories='[{"OperatingSystems/ambari_managed_repositories":"true","repositories":[
> {"Repositories/repo_id":"IOP-4.2.5","Repositories/base_url":"http://birepo-build.svl.ibm.com/repos/IOP/RHEL7/ppc64le/4.2.5.0/20170602_1749","Repositories/repo_name":"IOP"}
> ,
> {"Repositories/repo_id":"IOP-UTILS-1.3","Repositories/base_url":"http://birepo-build.svl.ibm.com/repos/IOP-UTILS/RHEL7/ppc64le/1.3","Repositories/repo_name":"IOP-UTILS"}
> ],"OperatingSystems/os_type":"redhat-ppc7"}]' where repo_version_id = 1;
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog261.java
>  ba95833 
> 
> 
> Diff: https://reviews.apache.org/r/64311/diff/4/
> 
> 
> Testing
> ---
> 
> upgrade ambari server rpms on an IOP 4.2.5 PPC cluster, replace ambari-server 
> jar with

Re: Review Request 64348: Pig service check failed after PU with LzoCodec CNF

2017-12-05 Thread Dmytro Grinenko


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64348/#review192898
---


Ship it!




Ship It!

- Dmytro Grinenko


On Dec. 5, 2017, 6:33 p.m., Jonathan Hurley wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/64348/
> ---
> 
> (Updated Dec. 5, 2017, 6:33 p.m.)
> 
> 
> Review request for Ambari, Dmytro Grinenko, Dmitro Lisnichenko, and Nate Cole.
> 
> 
> Bugs: AMBARI-22598
> https://issues.apache.org/jira/browse/AMBARI-22598
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> When Pig (and other hadoop clients) are upgraded by themselves, they need to 
> create jobs with versions of the LZO libraries which match _their_ versions.
> 
> *STR*
> # Deployed cluster with Ambari version: 2.5.1.0-159 and HDP version: 
> 2.6.1.0-129 (HDFS has lzo enabled)
> # Upgrade Ambari to Target Version: 2.6.1.0-84
> # Perform full stack upgrade to 2.6.4.0-55
> # Perform 4th digit PU of some services in batches (see attached for list)
> # Run Pig service check
> 
> *Result*
> ```
> Caused by: java.lang.IllegalArgumentException: Compression codec 
> com.hadoop.compression.lzo.LzoCodec not found.
>   at 
> org.apache.hadoop.io.compress.CompressionCodecFactory.getCodecClasses(CompressionCodecFactory.java:139)
>   at 
> org.apache.hadoop.io.compress.CompressionCodecFactory.(CompressionCodecFactory.java:180)
>   at 
> org.apache.hadoop.mapreduce.lib.input.TextInputFormat.isSplitable(TextInputFormat.java:59)
>   at 
> org.apache.hadoop.mapreduce.lib.input.FileInputFormat.getSplits(FileInputFormat.java:399)
>   at 
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInputFormat.getSplits(PigInputFormat.java:265)
>   ... 28 more
> Caused by: java.lang.ClassNotFoundException: Class 
> com.hadoop.compression.lzo.LzoCodec not found
>   at 
> org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2147)
>   at 
> org.apache.hadoop.io.compress.CompressionCodecFactory.getCodecClasses(CompressionCodecFactory.java:132)
>   ... 32 more
> 2017-12-04 12:07:30,322 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 
> 2118: Compression codec com.hadoop.compression.lzo.LzoCodec not found.
> Details at logfile: /home/ambari-qa/pig_1512389247042.log
> 2017-12-04 12:07:30,338 [main] INFO  org.apache.pig.Main - Pig script 
> completed in 3 seconds and 457 milliseconds (3457 ms)
> 2017-12-04 12:07:30,338 [main] INFO  
> org.apache.pig.backend.hadoop.executionengine.tez.TezLauncher - Shutting down 
> thread pool
> ```
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/resources/common-services/MAHOUT/1.0.0.2.3/package/scripts/mahout.py
>  f2c3c18077 
>   
> ambari-server/src/main/resources/common-services/PIG/0.12.0.2.0/package/scripts/pig.py
>  83f70488a3 
>   
> ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/setup_spark.py
>  426509cb0e 
>   
> ambari-server/src/main/resources/common-services/SPARK2/2.0.0/package/scripts/setup_spark.py
>  82b6c63d6d 
> 
> 
> Diff: https://reviews.apache.org/r/64348/diff/1/
> 
> 
> Testing
> ---
> 
> OK
> [INFO] 
> 
> [INFO] BUILD SUCCESS
> [INFO] 
> 
> [INFO] Total time: 01:29 min
> [INFO] Finished at: 2017-12-05T13:27:57-05:00
> [INFO] Final Memory: 21M/619M
> [INFO] 
> 
> 
> 
> Thanks,
> 
> Jonathan Hurley
> 
>

Re: Review Request 64223: Oozie Fails To Restart During Upgrade Because of Missing ExtJS Library

2017-11-30 Thread Dmytro Grinenko


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64223/#review192369
---


Ship it!




Ship It!

- Dmytro Grinenko


On Nov. 30, 2017, 9:18 p.m., Jonathan Hurley wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/64223/
> ---
> 
> (Updated Nov. 30, 2017, 9:18 p.m.)
> 
> 
> Review request for Ambari, Dmytro Grinenko, Dmitro Lisnichenko, and Nate Cole.
> 
> 
> Bugs: AMBARI-22568
> https://issues.apache.org/jira/browse/AMBARI-22568
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> 1) Deployed cluster with Ambari version: 2.6.1.0-65 and HDP version: 
> 2.6.3.0-215
> 2) Express upgrade to target version 2.6.4.0-52
> 
> Oozie Server Restart fails with below error:
> 
> ```
> Traceback (most recent call last):
> ...
>   File 
> "/var/lib/ambari-agent/cache/common-services/OOZIE/4.0.0.2.0/package/scripts/oozie_server.py",
>  line 123, in pre_upgrade_restart
> OozieUpgrade.prepare_libext_directory(upgrade_type=upgrade_type)
>   File 
> "/var/lib/ambari-agent/cache/common-services/OOZIE/4.0.0.2.0/package/scripts/oozie_server_upgrade.py",
>  line 113, in prepare_libext_directory
> raise Fail("Unable to find any Oozie source extension files from the 
> following paths {0}".format(source_ext_zip_paths))
> resource_management.core.exceptions.Fail: Unable to find any Oozie source 
> extension files from the following paths ['/usr/share/HDP-oozie/ext-2.2.zip', 
> '/var/lib/oozie/ext-2.2.zip']
> ```
> 
> This is because we have removed the requirement of having ExtJS in the 
> cluster.
> 
> 
> Diffs
> -
> 
>   
> ambari-common/src/main/python/resource_management/libraries/functions/constants.py
>  b81186144c 
>   
> ambari-server/src/main/resources/common-services/OOZIE/4.0.0.2.0/package/scripts/oozie_server_upgrade.py
>  eb57c22834 
>   
> ambari-server/src/main/resources/stacks/HDP/2.0.6/properties/stack_features.json
>  2109a5d5e9 
>   ambari-server/src/test/python/stacks/2.0.6/OOZIE/test_oozie_server.py 
> 4d94723bbf 
> 
> 
> Diff: https://reviews.apache.org/r/64223/diff/2/
> 
> 
> Testing
> ---
> 
> Upgraded Oozie
> 
> --
> Total run:1196
> Total errors:0
> Total failures:0
> OK
> [INFO] 
> 
> [INFO] BUILD SUCCESS
> [INFO] 
> 
> [INFO] Total time: 58.251 s
> [INFO] Finished at: 2017-11-30T15:16:50-05:00
> [INFO] Final Memory: 21M/619M
> [INFO] 
> 
> 
> 
> Thanks,
> 
> Jonathan Hurley
> 
>

Re: Review Request 64223: Oozie Fails To Restart During Upgrade Because of Missing ExtJS Library

2017-11-30 Thread Dmytro Grinenko


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64223/#review192357
---


Fix it, then Ship it!





ambari-common/src/main/python/resource_management/libraries/functions/constants.py
Line 123 (original), 123 (patched)
<https://reviews.apache.org/r/64223/#comment270458>

is coma needed here? python may think that this is tuple declaration



ambari-server/src/main/resources/common-services/OOZIE/4.0.0.2.0/package/scripts/oozie_server_upgrade.py
Lines 71 (patched)
<https://reviews.apache.org/r/64223/#comment270459>

filles - prolly typo?



ambari-server/src/test/python/stacks/2.0.6/OOZIE/test_oozie_server.py
Lines 1186 (patched)
<https://reviews.apache.org/r/64223/#comment270460>

can we remove this?


- Dmytro Grinenko


On Nov. 30, 2017, 8:28 p.m., Jonathan Hurley wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/64223/
> ---
> 
> (Updated Nov. 30, 2017, 8:28 p.m.)
> 
> 
> Review request for Ambari, Dmytro Grinenko, Dmitro Lisnichenko, and Nate Cole.
> 
> 
> Bugs: AMBARI-22568
> https://issues.apache.org/jira/browse/AMBARI-22568
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> 1) Deployed cluster with Ambari version: 2.6.1.0-65 and HDP version: 
> 2.6.3.0-215
> 2) Express upgrade to target version 2.6.4.0-52
> 
> Oozie Server Restart fails with below error:
> 
> ```
> Traceback (most recent call last):
> ...
>   File 
> "/var/lib/ambari-agent/cache/common-services/OOZIE/4.0.0.2.0/package/scripts/oozie_server.py",
>  line 123, in pre_upgrade_restart
> OozieUpgrade.prepare_libext_directory(upgrade_type=upgrade_type)
>   File 
> "/var/lib/ambari-agent/cache/common-services/OOZIE/4.0.0.2.0/package/scripts/oozie_server_upgrade.py",
>  line 113, in prepare_libext_directory
> raise Fail("Unable to find any Oozie source extension files from the 
> following paths {0}".format(source_ext_zip_paths))
> resource_management.core.exceptions.Fail: Unable to find any Oozie source 
> extension files from the following paths ['/usr/share/HDP-oozie/ext-2.2.zip', 
> '/var/lib/oozie/ext-2.2.zip']
> ```
> 
> This is because we have removed the requirement of having ExtJS in the 
> cluster.
> 
> 
> Diffs
> -
> 
>   
> ambari-common/src/main/python/resource_management/libraries/functions/constants.py
>  b81186144c 
>   
> ambari-server/src/main/resources/common-services/OOZIE/4.0.0.2.0/package/scripts/oozie_server_upgrade.py
>  eb57c22834 
>   
> ambari-server/src/main/resources/stacks/HDP/2.0.6/properties/stack_features.json
>  2109a5d5e9 
>   ambari-server/src/test/python/stacks/2.0.6/OOZIE/test_oozie_server.py 
> 4d94723bbf 
> 
> 
> Diff: https://reviews.apache.org/r/64223/diff/1/
> 
> 
> Testing
> ---
> 
> Upgraded Oozie
> 
> --
> Total run:1196
> Total errors:0
> Total failures:0
> OK
> [INFO] 
> 
> [INFO] BUILD SUCCESS
> [INFO] 
> 
> [INFO] Total time: 58.251 s
> [INFO] Finished at: 2017-11-30T15:16:50-05:00
> [INFO] Final Memory: 21M/619M
> [INFO] 
> 
> 
> 
> Thanks,
> 
> Jonathan Hurley
> 
>

Re: Review Request 64205: Snapshot HBase task failed during IOP migration with TypeError

2017-11-30 Thread Dmytro Grinenko


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64205/#review192269
---


Ship it!





ambari-common/src/main/python/resource_management/libraries/functions/lzo_utils.py
Line 86 (original), 86 (patched)
<https://reviews.apache.org/r/64205/#comment270336>

this is kinda oddly "expect.except"


- Dmytro Grinenko


On Nov. 30, 2017, 11:46 a.m., Dmitro Lisnichenko wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/64205/
> ---
> 
> (Updated Nov. 30, 2017, 11:46 a.m.)
> 
> 
> Review request for Ambari, Dmytro Grinenko, Jonathan Hurley, and Nate Cole.
> 
> 
> Bugs: AMBARI-22558
> https://issues.apache.org/jira/browse/AMBARI-22558
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> *STR*
> # Deployed cluster with Ambari version: 2.2.0 and IOP version: 4.2.0.0
> # Upgrade Ambari to Target Version: 2.5.2.0-298 | Hash: 
> 2453e16418fd964042452b649153dbe45f3c6009
> # Upgrade Ambari to  Target Version: 2.6.1.0-64 | Hash: 
> cd4db8e9ac0ea7ce14fc1253959a121688f34952
> # Register HDP-2.6.4.0-51 and call remove iop-select
> # Install the new HDP version bits and start Express Upgrade
> 
> *Result*
> Snapshot HBase task failed with below error:
> {code}
> Traceback (most recent call last):
> File 
> "/var/lib/ambari-agent/cache/custom_actions/scripts/ru_execute_tasks.py", 
> line 157, in 
> ExecuteUpgradeTasks().execute()
> File 
> "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py",
>  line 367, in execute
> method(env)
> File 
> "/var/lib/ambari-agent/cache/custom_actions/scripts/ru_execute_tasks.py", 
> line 153, in actionexecute
> shell.checked_call(task.command, logoutput=True, quiet=True)
> File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", 
> line 72, in inner
> result = function(command, **kwargs)
> File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", 
> line 102, in checked_call
> tries=tries, try_sleep=try_sleep, timeout_kill_strategy=timeout_kill_strategy)
> File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", 
> line 150, in _call_wrapper
> result = _call(command, **kwargs_copy)
> File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", 
> line 303, in _call
> raise ExecutionFailed(err_msg, code, out, err)
> resource_management.core.exceptions.ExecutionFailed: Execution of 'source 
> /var/lib/ambari-agent/ambari-env.sh ; /usr/bin/ambari-python-wrap 
> /var/lib/ambari-agent/cache/stacks/BigInsights/4.2/services/HBASE/package/scripts/hbase_upgrade.py
>  take_snapshot /var/lib/ambari-agent/data/command-404.json 
> /var/lib/ambari-agent/cache/custom_actions 
> /var/lib/ambari-agent/data/structured-out-404.json INFO 
> /var/lib/ambari-agent/tmp' returned 1. 2017-11-29 07:01:50,049 - Stack 
> Feature Version Info: Cluster Stack=2.6, Command Stack=None, Command 
> Version=4.2.0.0, Upgrade Direction=upgrade -> 4.2.0.0
> 2017-11-29 07:01:50,096 - Using hadoop conf dir: 
> /usr/hdp/current/hadoop-client/conf
> Traceback (most recent call last):
> File 
> "/var/lib/ambari-agent/cache/stacks/BigInsights/4.2/services/HBASE/package/scripts/hbase_upgrade.py",
>  line 37, in 
> HbaseMasterUpgrade().execute()
> File 
> "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py",
>  line 367, in execute
> method(env)
> File 
> "/var/lib/ambari-agent/cache/stacks/BigInsights/4.2/services/HBASE/package/scripts/hbase_upgrade.py",
>  line 28, in take_snapshot
> import params
> File 
> "/var/lib/ambari-agent/cache/stacks/BigInsights/4.2/services/HBASE/package/scripts/params.py",
>  line 107, in 
> regionserver_xmn_percent = 
> expect("/configurations/hbase-env/hbase_regionserver_xmn_ratio", float) 
> #AMBARI-15614
> TypeError: 'module' object is not callable
> {code}
> 
> 
> Upon checking the code 
> [here|https://github.com/hortonworks/ambari/blob/AMBARI-2.6.1.0/ambari-server/src/main/resources/stacks/BigInsights/4.2/services/HBASE/package/scripts/params.py#L30],
>  the issue seems to be with import of 'expect' module
> 
> I tried the following changes in params.py:
> {code}
> L30: from resource_management.libraries.functions import expect
> L107: regionserver_xmn_percent = 
> expect.expe

Re: Review Request 64166: History and Hive server start failed during IOP migration with AttributeError

2017-11-29 Thread Dmytro Grinenko


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64166/#review192153
---


Ship it!





ambari-common/src/main/python/resource_management/libraries/functions/copy_tarball.py
Lines 96 (patched)
<https://reviews.apache.org/r/64166/#comment270177>

seems that tez_version variable is better to rename to something like 
component_version or just version


- Dmytro Grinenko


On Nov. 29, 2017, 4:55 p.m., Jonathan Hurley wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/64166/
> ---
> 
> (Updated Nov. 29, 2017, 4:55 p.m.)
> 
> 
> Review request for Ambari, Dmytro Grinenko, Dmitro Lisnichenko, and Nate Cole.
> 
> 
> Bugs: AMBARI-22547
> https://issues.apache.org/jira/browse/AMBARI-22547
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> *STR*
> # Deployed cluster with Ambari version: 2.4.2.0_IBM_00 and HDP version: 
> 4.2.5.0- (LZO is enabled in HDFS configs)
> # Upgrade Ambari to Target Version: 2.6.1.0-64 | Hash: 
> cd4db8e9ac0ea7ce14fc1253959a121688f34952
> # Register HDP-2.6.4.0-51 version and call remove iop-slect
> # Install the bits and start EU
> 
> *Result*
> History server start failed with below error:
> ```
> Traceback (most recent call last):
>   File 
> "/var/lib/ambari-agent/cache/common-services/YARN/2.1.0.2.0/package/scripts/historyserver.py",
>  line 81, in pre_upgrade_restart
> copy_to_hdfs("tez", params.user_group, params.hdfs_user, 
> skip=params.sysprep_skip_copy_tarballs_hdfs)
>   File 
> "/usr/lib/python2.6/site-packages/resource_management/libraries/functions/copy_tarball.py",
>  line 464, in copy_to_hdfs
> source_file = prepare_function()
>   File 
> "/usr/lib/python2.6/site-packages/resource_management/libraries/functions/copy_tarball.py",
>  line 94, in _prepare_tez_tarball
> hadoop_lib_native_lzo_dir = os.path.join(stack_root, tez_version, 
> "hadoop", "lib", "native")
>   File "/usr/lib64/python2.6/posixpath.py", line 65, in join
> if b.startswith('/'):
> AttributeError: 'NoneType' object has no attribute 'startswith'
> ```
> 
> The error looks to be due to lzo changes, as the following code has the issue:
> ```
> # if enabled, LZO GPL libraries must be copied as well
>  91   if lzo_utils.should_install_lzo():
>  92 stack_root = Script.get_stack_root()
>  93 tez_version = 
> component_version.get_component_repository_version("TEZ")
>  94 hadoop_lib_native_lzo_dir = os.path.join(stack_root, tez_version, 
> "hadoop", "lib", "native")
>  95
> ```
> 
> **In this system, Tez is not installed and Hive is on MapReduce. However, the 
> Tez binaries are installed and should be updated in the event that Tez is 
> installed after upgrade.**
> 
> 
> Diffs
> -
> 
>   
> ambari-common/src/main/python/resource_management/libraries/functions/copy_tarball.py
>  bd1cede834 
>   
> ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/params_linux.py
>  2871dc0ff3 
> 
> 
> Diff: https://reviews.apache.org/r/64166/diff/1/
> 
> 
> Testing
> ---
> 
> Manual install & upgrade
> 
> 
> Thanks,
> 
> Jonathan Hurley
> 
>

Re: Review Request 64161: Tie MapReduce to Hive and Tez For Patch Upgrades

2017-11-29 Thread Dmytro Grinenko


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64161/#review192124
---


Ship it!




Ship It!

- Dmytro Grinenko


On Nov. 29, 2017, 2:49 p.m., Jonathan Hurley wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/64161/
> ---
> 
> (Updated Nov. 29, 2017, 2:49 p.m.)
> 
> 
> Review request for Ambari, Dmytro Grinenko and Nate Cole.
> 
> 
> Bugs: AMBARI-22545
> https://issues.apache.org/jira/browse/AMBARI-22545
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Hive is currently unable to upgrade without MapReduce since the property 
> {{mapreduce.admin.user.env}} cannot properly be detected from 
> {{mapred-site}}. 
> 
> STR:
> - Install a base cluster at 2.6.3.0 with ZK, HDFS, YARN, MapR, Tez, Hive
> - Push YARN/Tez forward to 2.6.3.1-1
> - Push Hive forward to 2.6.3.2-1
> - Push MapR forward to 2.6.3.3-1
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/resources/stacks/HDP/2.0.6/properties/stack_packages.json
>  794d2b0466 
> 
> 
> Diff: https://reviews.apache.org/r/64161/diff/1/
> 
> 
> Testing
> ---
> 
> Manual
> 
> 
> Thanks,
> 
> Jonathan Hurley
> 
>

Re: Review Request 64090: ATS start failed during patch upgrade due to CNF SparkATSPlugin

2017-11-28 Thread Dmytro Grinenko


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64090/#review192021
---


Ship it!




Ship It!

- Dmytro Grinenko


On Nov. 27, 2017, 7:01 p.m., Jonathan Hurley wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/64090/
> ---
> 
> (Updated Nov. 27, 2017, 7:01 p.m.)
> 
> 
> Review request for Ambari, Dmytro Grinenko, Dmitro Lisnichenko, and Nate Cole.
> 
> 
> Bugs: AMBARI-22525
> https://issues.apache.org/jira/browse/AMBARI-22525
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> After patching only YARN, ATS will fail to start with the following because 
> it cannot find the correct Spark JARS:
> ```
> 2017-11-20 11:36:45,457 FATAL 
> applicationhistoryservice.ApplicationHistoryServer 
> (ApplicationHistoryServer.java:launchAppHistoryServer(177)) - Error starting 
> ApplicationHistoryServer
> java.lang.RuntimeException: No class defined for 
> org.apache.spark.deploy.history.yarn.plugin.SparkATSPlugin
> ```
> 
> This is because Spark is configured via properties which currently use 
> ${hdp.version} which will match the version of YARN and not Spark.
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/params_linux.py
>  4079f1866e 
>   
> ambari-server/src/main/resources/stacks/BigInsights/4.0/stack-advisor/stack_advisor_25.py
>  7e77382e2b 
>   
> ambari-server/src/main/resources/stacks/HDP/2.5/services/YARN/configuration/yarn-site.xml
>  b6fadcbec5 
>   ambari-server/src/main/resources/stacks/HDP/2.5/services/stack_advisor.py 
> a3b8263454 
>   ambari-server/src/main/resources/stacks/HDP/2.6/upgrades/config-upgrade.xml 
> 45380728e3 
>   
> ambari-server/src/main/resources/stacks/HDP/2.6/upgrades/nonrolling-upgrade-2.6.xml
>  f3cea4875d 
>   ambari-server/src/main/resources/stacks/HDP/2.6/upgrades/upgrade-2.6.xml 
> 3bb6322fcb 
>   ambari-server/src/test/python/stacks/2.5/common/test_stack_advisor.py 
> bb11969a98 
> 
> 
> Diff: https://reviews.apache.org/r/64090/diff/1/
> 
> 
> Testing
> ---
> 
> Manual install & upgrade
> 
> --
> Total run:1195
> Total errors:0
> Total failures:0
> OK
> [INFO] 
> 
> [INFO] BUILD SUCCESS
> [INFO] 
> 
> [INFO] Total time: 59.118 s
> [INFO] Finished at: 2017-11-27T13:47:56-05:00
> [INFO] Final Memory: 21M/619M
> [INFO] 
> 
> 
> 
> Thanks,
> 
> Jonathan Hurley
> 
>

Re: Review Request 64085: Livy server fails to start during downgrade due to absence of 'conf' directory

2017-11-27 Thread Dmytro Grinenko


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64085/#review191882
---


Ship it!




Ship It!

- Dmytro Grinenko


On Nov. 27, 2017, 2:05 p.m., Jonathan Hurley wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/64085/
> ---
> 
> (Updated Nov. 27, 2017, 2:05 p.m.)
> 
> 
> Review request for Ambari, Dmytro Grinenko and Dmitro Lisnichenko.
> 
> 
> Bugs: AMBARI-22522
> https://issues.apache.org/jira/browse/AMBARI-22522
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> *STR*
> # Deployed cluster with Ambari version: 2.4.3.0-30 and HDP version: 
> 2.5.5.0-157
> # Upgrade Ambari to 2.6.1.0-43
> # Start EU to 2.6.4.0-36 and downgrade at final step
> 
> *Result*
> Observed during downgrade that Livy server start failed with below error:
> {code}
>   File 
> "/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py",
>  line 120, in action_create
> raise Fail("Applying %s failed, parent directory %s doesn't exist" % 
> (self.resource, dirname))
> resource_management.core.exceptions.Fail: Applying 
> File['/usr/hdp/current/livy-server/conf/livy-env.sh'] failed, parent 
> directory /usr/hdp/current/livy-server/conf doesn't exist
> {code}
> 
> On livy node, observed the following:
> {code}
> root@ctr-e134-1499953498516-324773-01-07:/usr/hdp/current/livy-server# ls 
> -la /usr/hdp/current/livy-server/
> total 40
> drwxr-xr-x  8 root root  4096 Nov 20 23:49 .
> drwxr-xr-x 42 root root  4096 Nov 21 00:05 ..
> drwxr-xr-x  2 root root  4096 Nov 20 23:43 bin
> lrwxrwxrwx  1 root root14 Apr 21  2017 conf -> /etc/livy/conf
> drwxr-xr-x  3 root root  4096 Nov 20 23:43 etc
> drwxr-xr-x  2 root root 12288 Nov 20 23:43 jars
> drwxr-xr-x  2 livy livy  4096 Nov 20 23:49 logs
> drwxr-xr-x  2 root root  4096 Nov 20 23:43 repl-jars
> drwxr-xr-x  2 root root  4096 Nov 20 23:43 rsc-jars
> 
> ===
> root@ctr-e134-1499953498516-324773-01-07:~# ls -la /etc/livy/
> total 16
> drwxr-xr-x   4 root root 4096 Nov 21 04:32 .
> drwxr-xr-x 120 root root 4096 Nov 21 03:26 ..
> drwxr-xr-x   3 root root 4096 Nov 21 03:30 2.6.4.0-36
> lrwxrwxrwx   1 root root   21 Nov 21 04:32 conf -> /usr/hdp/current/livy
> drwxr-xr-x   2 root root 4096 Nov 20 23:49 conf.backup
> {code}
> 
> Looks like there is no directory '/etc/livy/2.5.5.0-157/0'
> 
> From downgrade wizard, found that 'Set Version on All Hosts' Upgrade group 
> printed the following for livy:
> {code}
> 2017-11-21 04:32:08,022 - Link['/etc/livy/conf'] {'to': 
> '/usr/hdp/current/livy'}
> 2017-11-21 04:32:08,022 - Link['/etc/livy/conf'] replacing old symlink to 
> /grid/0/hdp/current/livy
> 2017-11-21 04:32:08,022 - Warning: linking to nonexistent location 
> /usr/hdp/current/livy
> 2017-11-21 04:32:08,022 - Creating symbolic Link['/etc/livy/conf'] to 
> /usr/hdp/current/livy
> {code}
> 
> The cause of this appears to be that livy doesn't create a conf directory as 
> expected:
> {code}
> Skipping the conf-select tool on livy since /etc/livy/conf does not exist.
> {code}
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/resources/stacks/BigInsights/4.0/properties/stack_packages.json
>  7a12011674 
>   
> ambari-server/src/main/resources/stacks/HDP/2.0.6/properties/stack_packages.json
>  245449c535 
> 
> 
> Diff: https://reviews.apache.org/r/64085/diff/1/
> 
> 
> Testing
> ---
> 
> Manual
> 
> 
> Thanks,
> 
> Jonathan Hurley
> 
>

Re: Review Request 63962: Conditionally Rebuild MapReduce and Tez Tarballs with LZO if Enabled

2017-11-20 Thread Dmytro Grinenko


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63962/#review191550
---


Fix it, then Ship it!





ambari-common/src/main/python/resource_management/libraries/functions/copy_tarball.py
Lines 96 (patched)
<https://reviews.apache.org/r/63962/#comment269396>

better to use function from the sudo module here



ambari-common/src/main/python/resource_management/libraries/functions/copy_tarball.py
Lines 100 (patched)
<https://reviews.apache.org/r/63962/#comment269397>

we need to use sudo module, as agent may work not from root, thus we can 
get here "permission denied" or not found



ambari-common/src/main/python/resource_management/libraries/functions/copy_tarball.py
Lines 162 (patched)
<https://reviews.apache.org/r/63962/#comment269398>

better to use function from sudo module



ambari-common/src/main/python/resource_management/libraries/functions/copy_tarball.py
Lines 181 (patched)
<https://reviews.apache.org/r/63962/#comment269399>

here native python call is fine, as we operating with our temporary created 
directory and have full ownership on it


- Dmytro Grinenko


On Nov. 20, 2017, 7:27 p.m., Jonathan Hurley wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/63962/
> ---
> 
> (Updated Nov. 20, 2017, 7:27 p.m.)
> 
> 
> Review request for Ambari, Dmytro Grinenko and Nate Cole.
> 
> 
> Bugs: AMBARI-22486
> https://issues.apache.org/jira/browse/AMBARI-22486
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> If LZO is enabled and has been opted-in, then the Tez and MapReduce tarballs 
> should have LZO added to them.
> 
> 
> Diffs
> -
> 
>   
> ambari-common/src/main/python/resource_management/libraries/functions/copy_tarball.py
>  b05c97cdfe 
>   
> ambari-server/src/main/resources/common-services/TEZ/0.4.0.2.1/package/scripts/tez.py
>  dfa650110d 
>   
> ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/configuration-mapred/mapred-site.xml
>  3438c45527 
>   
> ambari-server/src/main/resources/stacks/HDP/2.2/services/TEZ/configuration/tez-site.xml
>  4ffb7a49a4 
>   
> ambari-server/src/main/resources/stacks/HDP/2.2/services/YARN/configuration-mapred/mapred-site.xml
>  084e91254c 
>   ambari-server/src/main/resources/stacks/HDP/2.6/upgrades/config-upgrade.xml 
> 45380728e3 
> 
> 
> Diff: https://reviews.apache.org/r/63962/diff/1/
> 
> 
> Testing
> ---
> 
> Manual testing.
> 
> 
> Thanks,
> 
> Jonathan Hurley
> 
>

Re: Review Request 63807: Add Native Libraries To Tez Tarball

2017-11-14 Thread Dmytro Grinenko


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63807/#review191009
---


Fix it, then Ship it!





ambari-common/src/main/python/resource_management/libraries/functions/copy_tarball.py
Lines 54 (patched)
<https://reviews.apache.org/r/63807/#comment268602>

this is a bit dangerous, as would return the same function and calling it 
would make infinite recursion.

better replace to something like that, by skipping values which r not 
needed:

  _, mapreduce_source_file, _, _ = get_tarball_paths("mapreduce")
  _, tez_source_file, _, _ = get_tarball_paths("tez")
  
Round brackets do not needed here



ambari-common/src/main/python/resource_management/libraries/functions/copy_tarball.py
Lines 73 (patched)
<https://reviews.apache.org/r/63807/#comment268596>

why we trying to copy directory, if it could be even not exists? 

SHoudk we fail at line 70 then?



ambari-common/src/main/python/resource_management/libraries/functions/copy_tarball.py
Lines 219 (patched)
<https://reviews.apache.org/r/63807/#comment268613>

I know, they were here, but that is not pythonic. Please remove round 
brackets, they r reducant



ambari-common/src/main/python/resource_management/libraries/functions/copy_tarball.py
Line 261 (original), 316 (patched)
<https://reviews.apache.org/r/63807/#comment268614>

I know, they were here, but that is not pythonic. Please remove round 
brackets, they r reducant


- Dmytro Grinenko


On Nov. 14, 2017, 9:21 p.m., Jonathan Hurley wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/63807/
> ---
> 
> (Updated Nov. 14, 2017, 9:21 p.m.)
> 
> 
> Review request for Ambari, Dmytro Grinenko, Dmitro Lisnichenko, and Nate Cole.
> 
> 
> Bugs: AMBARI-22444
> https://issues.apache.org/jira/browse/AMBARI-22444
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> As part of the investigation for getting patch upgrades working in Ambari 
> 2.6.1, it was determined that the Tez tarball will need to have the hadoop 
> native libraries added to it so that they can be detected from the tarball.
> 
> STR:
> - Install ZK, MapR, Tez, Yarn, Hive
> - Enable a non-LZO codec, like Snappy
> - Patch Hive to a new version
> - Change the following properties in {{tez-site}}:
> -- tez.am.launch.env = LD_LIBRARY=./tezlib/lib/native
> -- tez.task.launch.env = LD_LIBRARY=./tezlib/lib/native
> 
> When Hive commands run, they will attempt to load the native snappy libraries 
> from the Tez tarball and will fail with:
> {code}
> Caused by: java.io.IOException: Unable to get CompressorType for codec 
> (org.apache.hadoop.io.compress.SnappyCodec). This is most likely due to 
> missing native libraries for the codec.
>   at 
> org.apache.tez.runtime.library.common.sort.impl.ExternalSorter.(ExternalSorter.java:217)
> {code}
> 
> In order to fix this, the Tez tarball should include the native hadoop 
> libraries as well:
> {code}
> ??? tez
> ?   ??? lib
> ?   ?   ??? native
> ?   ?   ?   ??? libhadoop.a
> ?   ?   ?   ??? libhadoop.so -> libhadoop.so.1.0.0
> ?   ?   ?   ??? libhadoop.so.1.0.0
> ?   ?   ?   ??? libhadooppipes.a
> ?   ?   ?   ??? libhadooputils.a
> ?   ?   ?   ??? libhdfs.a
> ?   ?   ?   ??? libhdfs.so -> libhdfs.so.0.0.0
> ?   ?   ?   ??? libhdfs.so.0.0.0
> ?   ?   ?   ??? libsnappy.so -> libsnappy.so.1.1.4
> ?   ?   ?   ??? libsnappy.so.1 -> libsnappy.so.1.1.4
> ?   ?   ?   ??? libsnappy.so.1.1.4
> {code}
> 
> 
> Diffs
> -
> 
>   
> ambari-common/src/main/python/resource_management/libraries/functions/copy_tarball.py
>  03b6213ac4 
>   
> ambari-common/src/main/python/resource_management/libraries/functions/tar_archive.py
>  c682c3e24a 
> 
> 
> Diff: https://reviews.apache.org/r/63807/diff/1/
> 
> 
> Testing
> ---
> 
> Manual testing to ensure the file is created and uploaded.
> 
> UNIT TESTS PENDING...
> 
> 
> Thanks,
> 
> Jonathan Hurley
> 
>

Re: Review Request 63807: Add Native Libraries To Tez Tarball

2017-11-14 Thread Dmytro Grinenko


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63807/#review191012
---



- Dmytro Grinenko


On Nov. 14, 2017, 9:21 p.m., Jonathan Hurley wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/63807/
> ---
> 
> (Updated Nov. 14, 2017, 9:21 p.m.)
> 
> 
> Review request for Ambari, Dmytro Grinenko, Dmitro Lisnichenko, and Nate Cole.
> 
> 
> Bugs: AMBARI-22444
> https://issues.apache.org/jira/browse/AMBARI-22444
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> As part of the investigation for getting patch upgrades working in Ambari 
> 2.6.1, it was determined that the Tez tarball will need to have the hadoop 
> native libraries added to it so that they can be detected from the tarball.
> 
> STR:
> - Install ZK, MapR, Tez, Yarn, Hive
> - Enable a non-LZO codec, like Snappy
> - Patch Hive to a new version
> - Change the following properties in {{tez-site}}:
> -- tez.am.launch.env = LD_LIBRARY=./tezlib/lib/native
> -- tez.task.launch.env = LD_LIBRARY=./tezlib/lib/native
> 
> When Hive commands run, they will attempt to load the native snappy libraries 
> from the Tez tarball and will fail with:
> {code}
> Caused by: java.io.IOException: Unable to get CompressorType for codec 
> (org.apache.hadoop.io.compress.SnappyCodec). This is most likely due to 
> missing native libraries for the codec.
>   at 
> org.apache.tez.runtime.library.common.sort.impl.ExternalSorter.(ExternalSorter.java:217)
> {code}
> 
> In order to fix this, the Tez tarball should include the native hadoop 
> libraries as well:
> {code}
> ??? tez
> ?   ??? lib
> ?   ?   ??? native
> ?   ?   ?   ??? libhadoop.a
> ?   ?   ?   ??? libhadoop.so -> libhadoop.so.1.0.0
> ?   ?   ?   ??? libhadoop.so.1.0.0
> ?   ?   ?   ??? libhadooppipes.a
> ?   ?   ?   ??? libhadooputils.a
> ?   ?   ?   ??? libhdfs.a
> ?   ?   ?   ??? libhdfs.so -> libhdfs.so.0.0.0
> ?   ?   ?   ??? libhdfs.so.0.0.0
> ?   ?   ?   ??? libsnappy.so -> libsnappy.so.1.1.4
> ?   ?   ?   ??? libsnappy.so.1 -> libsnappy.so.1.1.4
> ?   ?   ?   ??? libsnappy.so.1.1.4
> {code}
> 
> 
> Diffs
> -
> 
>   
> ambari-common/src/main/python/resource_management/libraries/functions/copy_tarball.py
>  03b6213ac4 
>   
> ambari-common/src/main/python/resource_management/libraries/functions/tar_archive.py
>  c682c3e24a 
> 
> 
> Diff: https://reviews.apache.org/r/63807/diff/1/
> 
> 
> Testing
> ---
> 
> Manual testing to ensure the file is created and uploaded.
> 
> UNIT TESTS PENDING...
> 
> 
> Thanks,
> 
> Jonathan Hurley
> 
>

Re: Review Request 63807: Add Native Libraries To Tez Tarball

2017-11-14 Thread Dmytro Grinenko


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63807/#review191009
---


Fix it, then Ship it!





ambari-common/src/main/python/resource_management/libraries/functions/copy_tarball.py
Lines 54 (patched)
<https://reviews.apache.org/r/63807/#comment268602>

this is a bit dangerous, as would return the same function and calling it 
would make infinite recursion.

better replace to something like that, by skipping values which r not 
needed:

  _, mapreduce_source_file, _, _ = get_tarball_paths("mapreduce")
  _, tez_source_file, _, _ = get_tarball_paths("tez")
  
Round brackets do not needed here



ambari-common/src/main/python/resource_management/libraries/functions/copy_tarball.py
Lines 73 (patched)
<https://reviews.apache.org/r/63807/#comment268596>

why we trying to copy directory, if it could be even not exists? 

SHoudk we fail at line 70 then?



ambari-common/src/main/python/resource_management/libraries/functions/copy_tarball.py
Lines 219 (patched)
<https://reviews.apache.org/r/63807/#comment268613>

I know, they were here, but that is not pythonic. Please remove round 
brackets, they r reducant



ambari-common/src/main/python/resource_management/libraries/functions/copy_tarball.py
Line 261 (original), 316 (patched)
<https://reviews.apache.org/r/63807/#comment268614>

I know, they were here, but that is not pythonic. Please remove round 
brackets, they r reducant


- Dmytro Grinenko


On Nov. 14, 2017, 9:21 p.m., Jonathan Hurley wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/63807/
> ---
> 
> (Updated Nov. 14, 2017, 9:21 p.m.)
> 
> 
> Review request for Ambari, Dmytro Grinenko, Dmitro Lisnichenko, and Nate Cole.
> 
> 
> Bugs: AMBARI-22444
> https://issues.apache.org/jira/browse/AMBARI-22444
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> As part of the investigation for getting patch upgrades working in Ambari 
> 2.6.1, it was determined that the Tez tarball will need to have the hadoop 
> native libraries added to it so that they can be detected from the tarball.
> 
> STR:
> - Install ZK, MapR, Tez, Yarn, Hive
> - Enable a non-LZO codec, like Snappy
> - Patch Hive to a new version
> - Change the following properties in {{tez-site}}:
> -- tez.am.launch.env = LD_LIBRARY=./tezlib/lib/native
> -- tez.task.launch.env = LD_LIBRARY=./tezlib/lib/native
> 
> When Hive commands run, they will attempt to load the native snappy libraries 
> from the Tez tarball and will fail with:
> {code}
> Caused by: java.io.IOException: Unable to get CompressorType for codec 
> (org.apache.hadoop.io.compress.SnappyCodec). This is most likely due to 
> missing native libraries for the codec.
>   at 
> org.apache.tez.runtime.library.common.sort.impl.ExternalSorter.(ExternalSorter.java:217)
> {code}
> 
> In order to fix this, the Tez tarball should include the native hadoop 
> libraries as well:
> {code}
> ??? tez
> ?   ??? lib
> ?   ?   ??? native
> ?   ?   ?   ??? libhadoop.a
> ?   ?   ?   ??? libhadoop.so -> libhadoop.so.1.0.0
> ?   ?   ?   ??? libhadoop.so.1.0.0
> ?   ?   ?   ??? libhadooppipes.a
> ?   ?   ?   ??? libhadooputils.a
> ?   ?   ?   ??? libhdfs.a
> ?   ?   ?   ??? libhdfs.so -> libhdfs.so.0.0.0
> ?   ?   ?   ??? libhdfs.so.0.0.0
> ?   ?   ?   ??? libsnappy.so -> libsnappy.so.1.1.4
> ?   ?   ?   ??? libsnappy.so.1 -> libsnappy.so.1.1.4
> ?   ?   ?   ??? libsnappy.so.1.1.4
> {code}
> 
> 
> Diffs
> -
> 
>   
> ambari-common/src/main/python/resource_management/libraries/functions/copy_tarball.py
>  03b6213ac4 
>   
> ambari-common/src/main/python/resource_management/libraries/functions/tar_archive.py
>  c682c3e24a 
> 
> 
> Diff: https://reviews.apache.org/r/63807/diff/1/
> 
> 
> Testing
> ---
> 
> Manual testing to ensure the file is created and uploaded.
> 
> UNIT TESTS PENDING...
> 
> 
> Thanks,
> 
> Jonathan Hurley
> 
>

Re: Review Request 63684: Patch Upgrades Broken For Clients Due To Versioned LD Library

2017-11-08 Thread Dmytro Grinenko


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63684/#review190532
---


Ship it!




Ship It!

- Dmytro Grinenko


On Nov. 8, 2017, 9:22 p.m., Jonathan Hurley wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/63684/
> ---
> 
> (Updated Nov. 8, 2017, 9:22 p.m.)
> 
> 
> Review request for Ambari, Dmytro Grinenko, Dmitro Lisnichenko, and Nate Cole.
> 
> 
> Bugs: AMBARI-22386
> https://issues.apache.org/jira/browse/AMBARI-22386
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> When services are patched to different versions, jobs being run in containers 
> are unable to pickup the correct JARs from the {{LD_LIBRARY_PATH}} passed in 
> via the job. 
> 
> Consider the following situation:
> - Base cluster installed at 2.6.3.0-1
> - Enable some sort of codec, like snappy
> - YARN patched to 2.6.3.1-1
> - Hive patched to 2.6.3.2-1
> 
> When Hive goes to run jobs on Tez or when jobs are run on MapR, whatever is 
> sending the job is resolving the {{hdp.version}} to its own version. The box 
> that the job runs on, however, might not have that specific versioned 
> directory. 
> 
> The problem can only be solved externally by providing a way to ensure that 
> the native libraries are on every machine which could potentially run jobs. 
> We could install bits on every host, but that would defeat the space-saving 
> purposes of patch upgrades.
> 
> Another solution is to ensure that the Tez and MapR tarballs have the 
> required libraries and that those libraries are added to the classpath of 
> launched jobs.
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/configuration-mapred/mapred-site.xml
>  a7d8cd67bd 
>   
> ambari-server/src/main/resources/stacks/HDP/2.2/services/TEZ/configuration/tez-site.xml
>  1427a6faae 
>   
> ambari-server/src/main/resources/stacks/HDP/2.2/services/YARN/configuration-mapred/mapred-site.xml
>  4ad08cebf9 
>   ambari-server/src/main/resources/stacks/HDP/2.6/upgrades/config-upgrade.xml 
> 0df0334c53 
>   
> ambari-server/src/main/resources/stacks/HDP/2.6/upgrades/nonrolling-upgrade-2.6.xml
>  b4e3745099 
>   ambari-server/src/main/resources/stacks/HDP/2.6/upgrades/upgrade-2.6.xml 
> 98223904c8 
> 
> 
> Diff: https://reviews.apache.org/r/63684/diff/1/
> 
> 
> Testing
> ---
> 
> Manual upgrade on patched cluster.
> 
> 
> Thanks,
> 
> Jonathan Hurley
> 
>

Re: Review Request 63593: Remove Auto-Installation of Mysql Connector

2017-11-06 Thread Dmytro Grinenko


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63593/#review190222
---


Ship it!




Ship It!

- Dmytro Grinenko


On Nov. 6, 2017, 8:31 p.m., Nate Cole wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/63593/
> ---
> 
> (Updated Nov. 6, 2017, 8:31 p.m.)
> 
> 
> Review request for Ambari, Dmytro Grinenko and Jonathan Hurley.
> 
> 
> Bugs: AMBARI-22369
> https://issues.apache.org/jira/browse/AMBARI-22369
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> The MySQL connector is GPL, and relying on it being in a repository is not 
> sufficient.  Removing auto-installation and will provide documentation 
> surrounding installation.
> 
> 
> Diffs
> -
> 
>   
> ambari-common/src/main/python/resource_management/libraries/functions/package_conditions.py
>  5a16061652 
>   
> ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/metainfo.xml 
> 34a0a0da18 
>   
> ambari-server/src/main/resources/common-services/OOZIE/4.2.0.2.3/metainfo.xml 
> d818074a59 
>   
> ambari-server/src/main/resources/common-services/SQOOP/1.4.4.2.0/metainfo.xml 
> a7ec1ec574 
>   
> ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/HIVE/metainfo.xml 
> d14ce674e1 
>   
> ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/OOZIE/metainfo.xml
>  20c3b32688 
>   
> ambari-server/src/main/resources/stacks/BigInsights/4.0/services/HIVE/metainfo.xml
>  dc1a5b48cd 
>   
> ambari-server/src/main/resources/stacks/BigInsights/4.0/services/OOZIE/metainfo.xml
>  2835c84f7f 
>   
> ambari-server/src/main/resources/stacks/BigInsights/4.0/services/SQOOP/metainfo.xml
>  f8780009bc 
>   
> ambari-server/src/main/resources/stacks/BigInsights/4.1/services/HIVE/metainfo.xml
>  db7d590223 
>   
> ambari-server/src/main/resources/stacks/BigInsights/4.1/services/OOZIE/metainfo.xml
>  c211f3a4be 
>   
> ambari-server/src/main/resources/stacks/BigInsights/4.1/services/SQOOP/metainfo.xml
>  7e247ca98a 
>   
> ambari-server/src/main/resources/stacks/BigInsights/4.2.5/services/HIVE/metainfo.xml
>  ec75b4b47c 
>   
> ambari-server/src/main/resources/stacks/BigInsights/4.2.5/services/OOZIE/metainfo.xml
>  94d37885ab 
>   
> ambari-server/src/main/resources/stacks/BigInsights/4.2.5/services/SQOOP/metainfo.xml
>  87347eb170 
>   
> ambari-server/src/main/resources/stacks/BigInsights/4.2/services/HIVE/metainfo.xml
>  5eb51b7b93 
>   
> ambari-server/src/main/resources/stacks/BigInsights/4.2/services/OOZIE/metainfo.xml
>  293d85069f 
>   
> ambari-server/src/main/resources/stacks/BigInsights/4.2/services/SQOOP/metainfo.xml
>  2be43bece8 
>   
> ambari-server/src/main/resources/stacks/HDP/2.0.6.GlusterFS/services/HIVE/metainfo.xml
>  633f3a6c98 
>   
> ambari-server/src/main/resources/stacks/HDP/2.0.6.GlusterFS/services/OOZIE/metainfo.xml
>  27d4ba4f54 
>   
> ambari-server/src/main/resources/stacks/HDP/2.1.GlusterFS/services/HIVE/metainfo.xml
>  7bad9fc7ff 
>   
> ambari-server/src/main/resources/stacks/HDP/2.1.GlusterFS/services/OOZIE/metainfo.xml
>  a62ef784fb 
>   ambari-server/src/main/resources/stacks/HDP/2.1/services/HIVE/metainfo.xml 
> 28dfb37165 
>   ambari-server/src/main/resources/stacks/HDP/2.2/services/HIVE/metainfo.xml 
> 051cad1595 
>   ambari-server/src/main/resources/stacks/HDP/2.2/services/OOZIE/metainfo.xml 
> cb84a5da2b 
>   ambari-server/src/main/resources/stacks/HDP/2.2/services/SQOOP/metainfo.xml 
> 884019ce94 
>   
> ambari-server/src/main/resources/stacks/HDP/2.3.GlusterFS/services/HIVE/metainfo.xml
>  0e12f091e9 
>   
> ambari-server/src/main/resources/stacks/HDP/2.3.GlusterFS/services/SQOOP/metainfo.xml
>  5238914371 
>   ambari-server/src/main/resources/stacks/HDP/2.3/services/HIVE/metainfo.xml 
> accbedd590 
>   ambari-server/src/main/resources/stacks/HDP/2.5/services/HIVE/metainfo.xml 
> f2a11611ca 
>   
> contrib/management-packs/odpi-ambari-mpack/src/main/resources/stacks/ODPi/2.0/services/HIVE/metainfo.xml
>  5d1c8980cb 
> 
> 
> Diff: https://reviews.apache.org/r/63593/diff/1/
> 
> 
> Testing
> ---
> 
> Manual.  Unit tests pending.
> 
> 
> Thanks,
> 
> Nate Cole
> 
>

Re: Review Request 63305: Downgrade From HDP 2.6 to 2.5 Leaves 2.6 Hosts as CURRENT Instead of INSTALLED

2017-10-25 Thread Dmytro Grinenko


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63305/#review189242
---


Ship it!




Ship It!

- Dmytro Grinenko


On Oct. 25, 2017, 7:57 p.m., Jonathan Hurley wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/63305/
> ---
> 
> (Updated Oct. 25, 2017, 7:57 p.m.)
> 
> 
> Review request for Ambari, Dmytro Grinenko and Nate Cole.
> 
> 
> Bugs: AMBARI-22312
> https://issues.apache.org/jira/browse/AMBARI-22312
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> STR:
> - Install HDP 2.5.4 with ZK
> - Perform an RU to 2.6.0.0
> - Downgrade back to 2.5.4
> 
> All hosts remain current after finalization:
> {code}
> {
>   "href": 
> "http://localhost:8080/api/v1/clusters/c1/stack_versions?fields=ClusterStackVersions/host_states";,
>   "items": [
> {
>   "href": "http://localhost:8080/api/v1/clusters/c1/stack_versions/1";,
>   "ClusterStackVersions": {
> "cluster_name": "c1",
> "id": 1,
> "repository_version": 1,
> "stack": "HDP",
> "version": "2.5",
> "host_states": {
>   "CURRENT": [
> "c6403.ambari.apache.org",
> "c6402.ambari.apache.org",
> "c6401.ambari.apache.org"
>   ],
>   "INSTALLED": [],
>   "INSTALLING": [],
>   "INSTALL_FAILED": [],
>   "NOT_REQUIRED": [],
>   "OUT_OF_SYNC": []
> }
>   }
> },
> {
>   "href": "http://localhost:8080/api/v1/clusters/c1/stack_versions/2";,
>   "ClusterStackVersions": {
> "cluster_name": "c1",
> "id": 2,
> "repository_version": 2,
> "stack": "HDP",
> "version": "2.6",
> "host_states": {
>   "CURRENT": [
> "c6403.ambari.apache.org",
> "c6402.ambari.apache.org",
> "c6401.ambari.apache.org"
>   ],
>   "INSTALLED": [],
>   "INSTALLING": [],
>   "INSTALL_FAILED": [],
>   "NOT_REQUIRED": [],
>   "OUT_OF_SYNC": []
> }
>   }
> }
>   ]
> }
> {code}
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/java/org/apache/ambari/server/serveraction/upgrades/FinalizeUpgradeAction.java
>  4d5f3ba627 
> 
> 
> Diff: https://reviews.apache.org/r/63305/diff/1/
> 
> 
> Testing
> ---
> 
> PENDING
> 
> 
> Thanks,
> 
> Jonathan Hurley
> 
>

Re: Review Request 63150: Ambari Schema Upgrade Failed during Ambari Upgrade

2017-10-19 Thread Dmytro Grinenko


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63150/#review188688
---


Ship it!




Ship It!

- Dmytro Grinenko


On Oct. 19, 2017, 3:38 p.m., Dmitro Lisnichenko wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/63150/
> ---
> 
> (Updated Oct. 19, 2017, 3:38 p.m.)
> 
> 
> Review request for Ambari, Jonathan Hurley and Nate Cole.
> 
> 
> Bugs: AMBARI-22216
> https://issues.apache.org/jira/browse/AMBARI-22216
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Yet another failure when upgrading 2.5.2.0 to 2.6.0.0
> 
> ERROR: Exception in thread "main" org.apache.ambari.server.AmbariException: 
> Cannot drop index 'FK_upgrade_from_repo_version_id': needed in a foreign key 
> constraint
>   at 
> org.apache.ambari.server.upgrade.SchemaUpgradeHelper.executeUpgrade(SchemaUpgradeHelper.java:203)
>   at 
> org.apache.ambari.server.upgrade.SchemaUpgradeHelper.main(SchemaUpgradeHelper.java:418)
> Caused by: java.sql.SQLException: Cannot drop index 
> 'FK_upgrade_from_repo_version_id': needed in a foreign key constraint
>   at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:996)
>   at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3887)
>   at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3823)
>   at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2435)
>   at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2582)
>   at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2526)
>   at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2484)
>   at com.mysql.jdbc.StatementImpl.execute(StatementImpl.java:848)
>   at com.mysql.jdbc.StatementImpl.execute(StatementImpl.java:742)
>   at 
> org.apache.ambari.server.orm.DBAccessorImpl.executeQuery(DBAccessorImpl.java:877)
>   at 
> org.apache.ambari.server.orm.DBAccessorImpl.executeQuery(DBAccessorImpl.java:869)
>   at 
> org.apache.ambari.server.orm.DBAccessorImpl.dropColumn(DBAccessorImpl.java:974)
>   at 
> org.apache.ambari.server.upgrade.UpgradeCatalog260.updateUpgradeTable(UpgradeCatalog260.java:333)
>   at 
> org.apache.ambari.server.upgrade.UpgradeCatalog260.executeDDLUpdates(UpgradeCatalog260.java:200)
>   at 
> org.apache.ambari.server.upgrade.AbstractUpgradeCatalog.upgradeSchema(AbstractUpgradeCatalog.java:923)
>   at 
> org.apache.ambari.server.upgrade.SchemaUpgradeHelper.executeUpgrade(SchemaUpgradeHelper.java:200)
>   ... 1 more
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/java/org/apache/ambari/server/orm/DBAccessorImpl.java 
> 09316be8b3 
>   
> ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog260.java
>  6f379858c0 
> 
> 
> Diff: https://reviews.apache.org/r/63150/diff/1/
> 
> 
> Testing
> ---
> 
> Live cluster check passed
> Tests passed
> 
> 
> Thanks,
> 
> Dmitro Lisnichenko
> 
>

Re: Review Request 63032: YARN Service Checks Fails Because of Old hadoop-client Classpath Entry

2017-10-16 Thread Dmytro Grinenko


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63032/#review188152
---


Ship it!




Ship It!

- Dmytro Grinenko


On Oct. 16, 2017, 4:22 p.m., Jonathan Hurley wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/63032/
> ---
> 
> (Updated Oct. 16, 2017, 4:22 p.m.)
> 
> 
> Review request for Ambari, Dmytro Grinenko, Dmitro Lisnichenko, and Nate Cole.
> 
> 
> Bugs: AMBARI-22245
> https://issues.apache.org/jira/browse/AMBARI-22245
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> STR:
> - Install basic YARN on HDP 2.6.0.0 (no Tez)
> - Upgrade just YARN to HDP 2.6.0.2
> 
> Service checks will fail with the following:
> ```
> 17/10/12 17:55:20 FATAL distributedshell.ApplicationMaster: Error running 
> ApplicationMaster
> java.lang.NoSuchMethodError: 
> org.apache.hadoop.io.retry.RetryPolicies.retryForeverWithFixedSleep(JLjava/util/concurrent/TimeUnit;)Lorg/apache/hadoop/io/retry/RetryPolicy;
>   at 
> org.apache.hadoop.yarn.client.RMProxy.createRetryPolicy(RMProxy.java:280)
> ```
> 
> - All YARN daemons and clients are reporting 2.6.0.2
> - All YARN daemons have loaded JARs for 2.6.0.2
> 
> What is happening here is that the applications being run on YARN are picking 
> up the older hadoop-common JAR file. The method 
> {{retryForeverWithFixedSleep}} did not exist in HDP 2.6.0.0.
> 
> It is picking up the older JARs for running applications because of the 
> {{yarn-site.xml}} property:
> ```
> yarn-site.xml-  yarn.application.classpath
> yarn-site.xml:  
> /etc/hadoop/conf,/usr/hdp/current/hadoop-client/*,/usr/hdp/current/hadoop-client/lib/*,/usr/hdp/current/hadoop-hdfs-client/*,/usr/hdp/current/hadoop-hdfs-client/lib/*,/usr/hdp/current/hadoop-yarn-client/*,/usr/hdp/current/hadoop-yarn-client/lib/*,/usr/hdp/current/ext/hadoop/*
> ```
> 
> Ambari should be parameterizing the {{/usr/hdp/current/hadoop-client}} paths 
> here.
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/params_linux.py
>  f21719261a 
>   
> ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/status_params.py
>  c2e9d92a04 
>   
> ambari-server/src/main/resources/stacks/HDP/2.2/services/YARN/configuration/yarn-site.xml
>  d0b4bb1f0f 
>   ambari-server/src/main/resources/stacks/HDP/2.6/upgrades/config-upgrade.xml 
> 5dbcf2da1b 
>   
> ambari-server/src/main/resources/stacks/HDP/2.6/upgrades/nonrolling-upgrade-2.6.xml
>  b7548eab30 
>   ambari-server/src/main/resources/stacks/HDP/2.6/upgrades/upgrade-2.6.xml 
> d3a096a4fa 
> 
> 
> Diff: https://reviews.apache.org/r/63032/diff/1/
> 
> 
> Testing
> ---
> 
> Manual
> 
> 
> Thanks,
> 
> Jonathan Hurley
> 
>

Re: Review Request 62996: Restart of random service could fail during express downgrade

2017-10-16 Thread Dmytro Grinenko


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62996/#review188112
---


Ship it!




Ship It!

- Dmytro Grinenko


On Oct. 16, 2017, 1:53 a.m., Nate Cole wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/62996/
> ---
> 
> (Updated Oct. 16, 2017, 1:53 a.m.)
> 
> 
> Review request for Ambari, Dmytro Grinenko, Dmitro Lisnichenko, and Jonathan 
> Hurley.
> 
> 
> Bugs: AMBARI-22240
> https://issues.apache.org/jira/browse/AMBARI-22240
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Turns out when we did the import for PU, we neglected to bring over the 
> per-service config changes that are required to support it.  This didn't show 
> during all our development because we were only checking on the same stack.
> 
> This jira brings in those changes such that we are correctly creating and 
> reverting configs when crossing stack versions.
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariManagementController.java
>  91bfe09deb 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariManagementControllerImpl.java
>  18659d44a1 
>   
> ambari-server/src/main/java/org/apache/ambari/server/state/ConfigFactory.java 
> d6cd99786c 
>   
> ambari-server/src/main/java/org/apache/ambari/server/state/ConfigHelper.java 
> 987ff3818d 
>   ambari-server/src/main/java/org/apache/ambari/server/state/ConfigImpl.java 
> 0a861d80c9 
>   
> ambari-server/src/main/java/org/apache/ambari/server/state/UpgradeHelper.java 
> 77e2d47d09 
>   
> ambari-server/src/main/java/org/apache/ambari/server/state/cluster/ClusterImpl.java
>  6bde42c10b 
>   
> ambari-server/src/main/java/org/apache/ambari/server/state/configgroup/ConfigGroupImpl.java
>  1a8b5b61f2 
>   
> ambari-server/src/main/java/org/apache/ambari/server/upgrade/AbstractUpgradeCatalog.java
>  d7cd087001 
>   
> ambari-server/src/test/java/org/apache/ambari/server/actionmanager/ExecutionCommandWrapperTest.java
>  87721e4ccd 
>   
> ambari-server/src/test/java/org/apache/ambari/server/agent/HeartbeatTestHelper.java
>  76f4bb30d9 
>   
> ambari-server/src/test/java/org/apache/ambari/server/agent/TestHeartbeatMonitor.java
>  088ec8ddd2 
>   
> ambari-server/src/test/java/org/apache/ambari/server/upgrade/AbstractUpgradeCatalogTest.java
>  96ccc57f39 
>   
> ambari-server/src/test/java/org/apache/ambari/server/upgrade/UpgradeCatalog250Test.java
>  2590d1c995 
>   
> ambari-server/src/test/java/org/apache/ambari/server/upgrade/UpgradeCatalog252Test.java
>  e6dbb7c0cf 
>   
> ambari-server/src/test/java/org/apache/ambari/server/upgrade/UpgradeCatalog260Test.java
>  23962b7d79 
> 
> 
> Diff: https://reviews.apache.org/r/62996/diff/2/
> 
> 
> Testing
> ---
> 
> Manual IN PROGRESS:
> RU Full through Finalize (DONE)
> RU Full but Downgrade (DONE)
> EU Full through Finalize (DONE)
> EU Full but Downgrade (DONE)
> 
> EU Patch but Downgrade (DONE)
> EU Patch then Finalize (DONE)
> EU Patch then Finalize then Revert (DONE)
> 
> PENDING Unit tests:
> Verify no tests are broken with this change:
> 
> 
> Thanks,
> 
> Nate Cole
> 
>

Re: Review Request 62934: Set current_version for Backward Compatibility

2017-10-12 Thread Dmytro Grinenko


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62934/#review187796
---


Ship it!




Ship It!

- Dmytro Grinenko


On Oct. 12, 2017, 1:57 p.m., Jonathan Hurley wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/62934/
> ---
> 
> (Updated Oct. 12, 2017, 1:57 p.m.)
> 
> 
> Review request for Ambari, Dmytro Grinenko and Nate Cole.
> 
> 
> Bugs: AMBARI-22219
> https://issues.apache.org/jira/browse/AMBARI-22219
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Some stacks, mpacks, and extension services may have been relying on the 
> deprecated and removed {{hostLevelParams/current_version}}. Although this 
> value was removed since there is no longer a single version for the cluster, 
> we can continue to set it using the version of the repository associated with 
> the current command.
> 
> This will allow existing mpacks and services which were relying on this value 
> to continue to using it until they can switch over to 
> {{componentVersionMapping}} structure.
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/java/org/apache/ambari/server/actionmanager/ExecutionCommandWrapper.java
>  f37bdd5e6f 
>   
> ambari-server/src/main/java/org/apache/ambari/server/agent/ExecutionCommand.java
>  6ffc9c11b3 
> 
> 
> Diff: https://reviews.apache.org/r/62934/diff/1/
> 
> 
> Testing
> ---
> 
> Audit done.
> [INFO] 
> 
> [INFO] BUILD SUCCESS
> [INFO] 
> 
> [INFO] Total time: 27:43 min
> [INFO] Finished at: 2017-10-12T10:38:56-04:00
> [INFO] Final Memory: 71M/881M
> [INFO] 
> 
> 
> 
> Thanks,
> 
> Jonathan Hurley
> 
>

Re: Review Request 62922: "ambari-server upgrade" failed on db schema [Upgrade]

2017-10-12 Thread Dmytro Grinenko


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62922/#review187768
---


Ship it!




Ship It!

- Dmytro Grinenko


On Oct. 12, 2017, 5:30 a.m., Dmitro Lisnichenko wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/62922/
> ---
> 
> (Updated Oct. 12, 2017, 5:30 a.m.)
> 
> 
> Review request for Ambari, Jonathan Hurley and Nate Cole.
> 
> 
> Bugs: AMBARI-22213
> https://issues.apache.org/jira/browse/AMBARI-22213
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Manual check:
> {code}
> tr-e134-1499953498516-213280-01-12:~ # ambari-server upgrade
> Using python  /usr/bin/python
> Upgrading ambari-server
> INFO: Upgrade Ambari Server
> INFO: Updating Ambari Server properties in ambari.properties ...
> WARNING: Can not find ambari.properties.rpmsave file from previous version, 
> skipping import of settings
> INFO: Updating Ambari Server properties in ambari-env.sh ...
> INFO: Can not find ambari-env.sh.rpmsave file from previous version, skipping 
> restore of environment settings. ambari-env.sh may not include any user 
> customization.
> INFO: Fixing database objects owner
> Ambari Server configured for Postgres. Confirm you have made a backup of the 
> Ambari Server database [y/n] (y)?
> INFO: Upgrading database schema
> INFO: Return code from schema upgrade command, retcode = 1
> ERROR: Error executing schema upgrade, please check the server logs.
> ERROR: Error output from schema upgrade command:
> ERROR: Exception in thread "main" org.apache.ambari.server.AmbariException: 
> ERROR: relation "servicecomponent_history" does not exist
> Position: 13
> at 
> org.apache.ambari.server.upgrade.SchemaUpgradeHelper.executeUpgrade(SchemaUpgradeHelper.java:203)
> at 
> org.apache.ambari.server.upgrade.SchemaUpgradeHelper.main(SchemaUpgradeHelper.java:418)
> Caused by: org.postgresql.util.PSQLException: ERROR: relation 
> "servicecomponent_history" does not exist
> Position: 13
> at 
> org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2161)
> at 
> org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1890)
> at 
> org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:255)
> at 
> org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:559)
> at 
> org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:403)
> at 
> org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:395)
> at 
> org.apache.ambari.server.orm.DBAccessorImpl.executeQuery(DBAccessorImpl.java:877)
> at 
> org.apache.ambari.server.orm.DBAccessorImpl.executeQuery(DBAccessorImpl.java:869)
> at 
> org.apache.ambari.server.orm.DBAccessorImpl.clearTable(DBAccessorImpl.java:1500)
> at 
> org.apache.ambari.server.upgrade.UpgradeCatalog252.addRepositoryColumnsToUpgradeTable(UpgradeCatalog252.java:169)
> at 
> org.apache.ambari.server.upgrade.UpgradeCatalog252.executeDDLUpdates(UpgradeCatalog252.java:122)
> at 
> org.apache.ambari.server.upgrade.AbstractUpgradeCatalog.upgradeSchema(AbstractUpgradeCatalog.java:923)
> at 
> org.apache.ambari.server.upgrade.SchemaUpgradeHelper.executeUpgrade(SchemaUpgradeHelper.java:200)
> ... 1 more
> 
> 
> ERROR: Ambari server upgrade failed. Please look at 
> /var/log/ambari-server/ambari-server.log, for more details.
> ERROR: Exiting with exit code 11.
> REASON: Schema upgrade failed.
> {code}
> {code}
> 11 Oct 2017 13:49:16,918 ERROR [main] DBAccessorImpl:880 - Error executing 
> query: DELETE FROM servicecomponent_history
> org.postgresql.util.PSQLException: ERROR: relation "servicecomponent_history" 
> does not exist
> Position: 13
> at 
> org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2161)
> at 
> org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1890)
> at 
> org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:255)
> at 
> org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:559)
> at 
> org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:403)
> at 
> org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:395)
> at 
> org.apache.ambari.server.orm.

Re: Review Request 62754: Adding Components On Patched Clusters Can Result In Symlink Issues With conf Directories

2017-10-04 Thread Dmytro Grinenko


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62754/#review187081
---


Ship it!




Ship It!

- Dmytro Grinenko


On Oct. 3, 2017, 9:11 p.m., Jonathan Hurley wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/62754/
> ---
> 
> (Updated Oct. 3, 2017, 9:11 p.m.)
> 
> 
> Review request for Ambari, Dmytro Grinenko, Dmitro Lisnichenko, and Nate Cole.
> 
> 
> Bugs: AMBARI-22123
> https://issues.apache.org/jira/browse/AMBARI-22123
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Essentially, the symlinking was doing a quick-exit if the 
> /etc//conf directory already existed. However, it might have been 
> created by another RPM and not the component we actually care about.
> 
> The refactor here did two things:
> - Invokes conf-select on the necessary packages
> - Combined most of the logic between conf_select.select(...) and 
> conf_select.convert_conf_directories_to_symlinks(...)
> - Removed legacy code from back in the pre-HDP 2.3 days
> 
> 
> Diffs
> -
> 
>   
> ambari-common/src/main/python/resource_management/libraries/functions/conf_select.py
>  86821bf47a 
>   ambari-server/src/main/resources/custom_actions/scripts/install_packages.py 
> 3ace3b1570 
>   
> ambari-server/src/main/resources/stacks/BigInsights/4.0/services/ZOOKEEPER/package/scripts/zookeeper_service.py
>  0727970adb 
>   
> ambari-server/src/main/resources/stacks/BigInsights/4.2/services/ZOOKEEPER/package/scripts/zookeeper_service.py
>  0727970adb 
>   
> ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/after-INSTALL/scripts/shared_initialization.py
>  20fd48de15 
>   ambari-server/src/test/python/TestAmbariServer.py 5db79137ee 
>   ambari-server/src/test/python/TestMpacks.py fd30bf6f88 
>   
> ambari-server/src/test/python/stacks/2.0.6/hooks/after-INSTALL/test_after_install.py
>  92f5011978 
>   ambari-server/src/test/python/stacks/2.2/common/test_conf_select.py 
> 2eeec46b4a 
>   
> contrib/management-packs/odpi-ambari-mpack/src/main/resources/stacks/ODPi/2.0/services/HIVE/package/scripts/hive_client.py
>  3d9bfd7c3e 
>   
> contrib/management-packs/odpi-ambari-mpack/src/main/resources/stacks/ODPi/2.0/services/HIVE/package/scripts/hive_metastore.py
>  b88f3858a7 
>   
> contrib/management-packs/odpi-ambari-mpack/src/main/resources/stacks/ODPi/2.0/services/HIVE/package/scripts/hive_server.py
>  31b083bd0d 
>   
> contrib/management-packs/odpi-ambari-mpack/src/main/resources/stacks/ODPi/2.0/services/HIVE/package/scripts/hive_server_interactive.py
>  2df001cf21 
>   
> contrib/management-packs/odpi-ambari-mpack/src/main/resources/stacks/ODPi/2.0/services/HIVE/package/scripts/webhcat_server.py
>  34687c453b 
>   
> contrib/management-packs/odpi-ambari-mpack/src/main/resources/stacks/ODPi/2.0/services/YARN/package/scripts/application_timeline_server.py
>  4ec6aa788f 
>   
> contrib/management-packs/odpi-ambari-mpack/src/main/resources/stacks/ODPi/2.0/services/YARN/package/scripts/historyserver.py
>  34c683ab2f 
>   
> contrib/management-packs/odpi-ambari-mpack/src/main/resources/stacks/ODPi/2.0/services/YARN/package/scripts/mapreduce2_client.py
>  424157b128 
>   
> contrib/management-packs/odpi-ambari-mpack/src/main/resources/stacks/ODPi/2.0/services/YARN/package/scripts/nodemanager.py
>  b235cad164 
>   
> contrib/management-packs/odpi-ambari-mpack/src/main/resources/stacks/ODPi/2.0/services/YARN/package/scripts/resourcemanager.py
>  71c7bc17a2 
>   
> contrib/management-packs/odpi-ambari-mpack/src/main/resources/stacks/ODPi/2.0/services/YARN/package/scripts/yarn_client.py
>  4d65a404a2 
> 
> 
> Diff: https://reviews.apache.org/r/62754/diff/3/
> 
> 
> Testing
> ---
> 
> Total run:1192
> Total errors:0
> Total failures:0
> OK
> [INFO] 
> 
> [INFO] BUILD SUCCESS
> [INFO] 
> 
> [INFO] Total time: 57.162 s
> [INFO] Finished at: 2017-10-03T23:04:47-04:00
> [INFO] Final Memory: 19M/491M
> [INFO] 
> 
> 
> 
> Thanks,
> 
> Jonathan Hurley
> 
>

Re: Review Request 62364: Set tez.runtime.shuffle.ssl.enable=false in Ambari for HSI

2017-09-18 Thread Dmytro Grinenko


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62364/#review185580
---


Ship it!




Ship It!

- Dmytro Grinenko


On Sept. 18, 2017, 5:46 p.m., Vitalyi Brodetskyi wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/62364/
> ---
> 
> (Updated Sept. 18, 2017, 5:46 p.m.)
> 
> 
> Review request for Ambari, Dmytro Grinenko, Jayush Luniya, Swapan Shridhar, 
> and Sid Wagle.
> 
> 
> Bugs: AMBARI-21973
> https://issues.apache.org/jira/browse/AMBARI-21973
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Add tez.runtime.shuffle.ssl.enable=false to tez-interactive-site.
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/resources/stacks/HDP/2.5/services/HIVE/configuration/tez-interactive-site.xml
>  c1a42b0 
>   ambari-server/src/main/resources/stacks/HDP/2.5/upgrades/config-upgrade.xml 
> 5618e14 
>   
> ambari-server/src/main/resources/stacks/HDP/2.5/upgrades/nonrolling-upgrade-2.5.xml
>  9f77065 
>   
> ambari-server/src/main/resources/stacks/HDP/2.5/upgrades/nonrolling-upgrade-2.6.xml
>  8824de5 
>   ambari-server/src/main/resources/stacks/HDP/2.5/upgrades/upgrade-2.5.xml 
> a6d7a49 
>   ambari-server/src/main/resources/stacks/HDP/2.5/upgrades/upgrade-2.6.xml 
> 2a61034 
>   ambari-server/src/main/resources/stacks/HDP/2.6/upgrades/config-upgrade.xml 
> c67e4cf 
>   
> ambari-server/src/main/resources/stacks/HDP/2.6/upgrades/nonrolling-upgrade-2.6.xml
>  b49fdbf 
>   ambari-server/src/main/resources/stacks/HDP/2.6/upgrades/upgrade-2.6.xml 
> af885e1 
> 
> 
> Diff: https://reviews.apache.org/r/62364/diff/2/
> 
> 
> Testing
> ---
> 
> mvn clean test
> 
> 
> Thanks,
> 
> Vitalyi Brodetskyi
> 
>

Re: Review Request 62259: Upgrade to IOP 4.2.5 from IOP 4.1 failed with combined Solr host names longer than item_text column size in table upgrade_item

2017-09-18 Thread Dmytro Grinenko


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62259/#review185573
---


Ship it!




Ship It!

- Dmytro Grinenko


On Sept. 18, 2017, 5:36 p.m., Sid Wagle wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/62259/
> ---
> 
> (Updated Sept. 18, 2017, 5:36 p.m.)
> 
> 
> Review request for Ambari, Attila Doroszlai, Dmytro Grinenko, Jonathan 
> Hurley, Jayush Luniya, Myroslav Papirkovskyy, and Vitalyi Brodetskyi.
> 
> 
> Bugs: AMBARI-21942
> https://issues.apache.org/jira/browse/AMBARI-21942
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Basically the problem here happens when Ambari is attempting to insert
> into the following upgrade_itme:item_text table:column,
> INSERT INTO upgrade_item (upgrade_item_id, hosts, item_text, stage_id,
> state, tasks, upgrade_group_id) VALUES (?, ?, ?, ?, ?, ?, ?)
> 
> The message string for SOLR includes all the SOLR node hostnames whose 
> combined string length exceeds the column length defined for "item_text" of 
> 1024 characters.
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/java/org/apache/ambari/server/orm/DBAccessorImpl.java 
> 9564934 
>   
> ambari-server/src/main/java/org/apache/ambari/server/orm/entities/UpgradeItemEntity.java
>  35ea769 
>   
> ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog260.java
>  9e145c0 
>   ambari-server/src/main/resources/Ambari-DDL-Derby-CREATE.sql b9c6985 
>   ambari-server/src/main/resources/Ambari-DDL-MySQL-CREATE.sql c53cd7b 
>   ambari-server/src/main/resources/Ambari-DDL-Oracle-CREATE.sql 6f754bb 
>   ambari-server/src/main/resources/Ambari-DDL-Postgres-CREATE.sql 19c0214 
>   ambari-server/src/main/resources/Ambari-DDL-SQLAnywhere-CREATE.sql 8567d09 
>   ambari-server/src/main/resources/Ambari-DDL-SQLServer-CREATE.sql fdbad09 
> 
> 
> Diff: https://reviews.apache.org/r/62259/diff/1/
> 
> 
> Testing
> ---
> 
> Manually verified the upgrade resulted in expansion on PostgreSQL.
> 
> 
> File Attachments
> 
> 
> Removed MySQL special handling
>   
> https://reviews.apache.org/media/uploaded/files/2017/09/18/05ae5778-da1d-4ce6-971e-db5c639f2346__AMBARI-21942-2.patch
> 
> 
> Thanks,
> 
> Sid Wagle
> 
>

Re: Review Request 62331: Cluster provision should allow repo version and repo version id be null to allow default stack version deployment

2017-09-14 Thread Dmytro Grinenko


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62331/#review185412
---


Ship it!




Ship It!

- Dmytro Grinenko


On Sept. 14, 2017, 3:13 p.m., Nate Cole wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/62331/
> ---
> 
> (Updated Sept. 14, 2017, 3:13 p.m.)
> 
> 
> Review request for Ambari, Dmytro Grinenko and Jonathan Hurley.
> 
> 
> Bugs: AMBARI-21690
> https://issues.apache.org/jira/browse/AMBARI-21690
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Remove repository_version and repository_version_id null checks for BP 
> deployments since we want to use the default if-not-specified.
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/java/org/apache/ambari/server/topology/TopologyManager.java
>  092339bb7a 
>   
> ambari-server/src/test/java/org/apache/ambari/server/topology/TopologyManagerTest.java
>  8d891b9534 
> 
> 
> Diff: https://reviews.apache.org/r/62331/diff/1/
> 
> 
> Testing
> ---
> 
> Manual cluster deployment.  Automated pending.
> 
> 
> Thanks,
> 
> Nate Cole
> 
>

Re: Review Request 62258: Stack selection page does not load the HDP stacks [Intermittent]

2017-09-12 Thread Dmytro Grinenko


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62258/#review185235
---


Fix it, then Ship it!





ambari-server/src/main/java/org/apache/ambari/server/stack/StackContext.java
Line 151 (original), 159 (patched)
<https://reviews.apache.org/r/62258/#comment261532>

can we move amount of threads to constant, to make it more easy to tune in 
the future?



ambari-server/src/main/java/org/apache/ambari/server/stack/StackContext.java
Lines 219 (patched)
<https://reviews.apache.org/r/62258/#comment261531>

Looks like not used assign



ambari-server/src/main/java/org/apache/ambari/server/stack/StackModule.java
Lines 1154 (patched)
<https://reviews.apache.org/r/62258/#comment261533>

This could be moved to the top of the function and be used after for the 
check?


- Dmytro Grinenko


On Sept. 12, 2017, 8:10 p.m., Nate Cole wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/62258/
> ---
> 
> (Updated Sept. 12, 2017, 8:10 p.m.)
> 
> 
> Review request for Ambari, Dmytro Grinenko, Dmitro Lisnichenko, and Jonathan 
> Hurley.
> 
> 
> Bugs: AMBARI-21941
> https://issues.apache.org/jira/browse/AMBARI-21941
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Starting Ambari will load all the hdp_urlinfo.json and files and VDF in a 
> sequential manner.  This is all in a single-threaded Executor.  When that 
> happens, the data can take upwards of a full minute to be populated in some 
> cases.  This should be multi-threaded.
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/StackContext.java 
> da7f021059 
>   ambari-server/src/main/java/org/apache/ambari/server/stack/StackModule.java 
> cbbe92e8cc 
>   ambari-server/src/main/java/org/apache/ambari/server/state/StackInfo.java 
> f1412f543e 
>   
> ambari-server/src/main/java/org/apache/ambari/server/state/stack/LatestRepoCallable.java
>  c43ce7c628 
>   
> ambari-server/src/main/java/org/apache/ambari/server/state/stack/RepoUrlInfoCallable.java
>  PRE-CREATION 
>   
> ambari-server/src/main/java/org/apache/ambari/server/state/stack/RepoVdfCallable.java
>  PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/62258/diff/1/
> 
> 
> Testing
> ---
> 
> This is a refactor only.  Existing tests cover the functionality needed while 
> just adding a performance boost to loading VDF.
> 
> Manual.  Automated pending.
> 
> 
> Thanks,
> 
> Nate Cole
> 
>

Re: Review Request 62226: Use Correct Packages For Clients Where Stack Tools Support It

2017-09-12 Thread Dmytro Grinenko


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62226/#review185207
---


Ship it!




Ship It!

- Dmytro Grinenko


On Sept. 12, 2017, 3:50 p.m., Jonathan Hurley wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/62226/
> ---
> 
> (Updated Sept. 12, 2017, 3:50 p.m.)
> 
> 
> Review request for Ambari, Dmytro Grinenko, Dmitro Lisnichenko, and Nate Cole.
> 
> 
> Bugs: AMBARI-21931
> https://issues.apache.org/jira/browse/AMBARI-21931
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> There are several "INVALID" types listed in the stack_packages.json file. 
> These were placeholders that need to be updated with the packages exposed by 
> the stack-select tool. 
> 
> {code}
> ...
> "HDFS_CLIENT": {
>   "STACK-SELECT-PACKAGE": "hadoop-client",
>   "INSTALL": [
> "hadoop-client"
>   ],
>   "PATCH": [
> "INVALID"
>   ],
>   "STANDARD": [
> "hadoop-client"
>   ]
> },
> ...
> {code}
> 
> 
> Diffs
> -
> 
>   
> ambari-common/src/main/python/resource_management/libraries/functions/stack_select.py
>  eac1bef13c 
>   
> ambari-server/src/main/java/org/apache/ambari/server/topology/AmbariContext.java
>  6a2d58d24c 
>   
> ambari-server/src/main/resources/stacks/HDP/2.0.6/properties/stack_packages.json
>  704fb54293 
>   ambari-server/src/test/python/TestStackSelect.py 3d4e5b6a43 
>   ambari-server/src/test/python/stacks/utils/RMFTestCase.py 0341092b69 
> 
> 
> Diff: https://reviews.apache.org/r/62226/diff/4/
> 
> 
> Testing
> ---
> 
> --
> Total run:1191
> Total errors:0
> Total failures:0
> OK
> [INFO] 
> 
> [INFO] BUILD SUCCESS
> [INFO] 
> 
> [INFO] Total time: 58.379 s
> [INFO] Finished at: 2017-09-11T16:25:26-04:00
> [INFO] Final Memory: 21M/619M
> [INFO] 
> 
> 
> 
> Thanks,
> 
> Jonathan Hurley
> 
>

Re: Review Request 62226: Use Correct Packages For Clients Where Stack Tools Support It

2017-09-12 Thread Dmytro Grinenko


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62226/#review185206
---




ambari-server/src/test/python/stacks/utils/RMFTestCase.py
Line 142 (original), 142 (patched)
<https://reviews.apache.org/r/62226/#comment261496>

i like this way more than that big "stairs"


- Dmytro Grinenko


On Sept. 12, 2017, 3:50 p.m., Jonathan Hurley wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/62226/
> ---
> 
> (Updated Sept. 12, 2017, 3:50 p.m.)
> 
> 
> Review request for Ambari, Dmytro Grinenko, Dmitro Lisnichenko, and Nate Cole.
> 
> 
> Bugs: AMBARI-21931
> https://issues.apache.org/jira/browse/AMBARI-21931
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> There are several "INVALID" types listed in the stack_packages.json file. 
> These were placeholders that need to be updated with the packages exposed by 
> the stack-select tool. 
> 
> {code}
> ...
> "HDFS_CLIENT": {
>   "STACK-SELECT-PACKAGE": "hadoop-client",
>   "INSTALL": [
> "hadoop-client"
>   ],
>   "PATCH": [
> "INVALID"
>   ],
>   "STANDARD": [
> "hadoop-client"
>   ]
> },
> ...
> {code}
> 
> 
> Diffs
> -
> 
>   
> ambari-common/src/main/python/resource_management/libraries/functions/stack_select.py
>  eac1bef13c 
>   
> ambari-server/src/main/java/org/apache/ambari/server/topology/AmbariContext.java
>  6a2d58d24c 
>   
> ambari-server/src/main/resources/stacks/HDP/2.0.6/properties/stack_packages.json
>  704fb54293 
>   ambari-server/src/test/python/TestStackSelect.py 3d4e5b6a43 
>   ambari-server/src/test/python/stacks/utils/RMFTestCase.py 0341092b69 
> 
> 
> Diff: https://reviews.apache.org/r/62226/diff/4/
> 
> 
> Testing
> ---
> 
> --
> Total run:1191
> Total errors:0
> Total failures:0
> OK
> [INFO] 
> 
> [INFO] BUILD SUCCESS
> [INFO] 
> 
> [INFO] Total time: 58.379 s
> [INFO] Finished at: 2017-09-11T16:25:26-04:00
> [INFO] Final Memory: 21M/619M
> [INFO] 
> 
> 
> 
> Thanks,
> 
> Jonathan Hurley
> 
>

Re: Review Request 62226: Use Correct Packages For Clients Where Stack Tools Support It

2017-09-12 Thread Dmytro Grinenko



> On Sept. 12, 2017, 3:42 p.m., Dmytro Grinenko wrote:
> > ambari-common/src/main/python/resource_management/libraries/functions/stack_select.py
> > Lines 126 (patched)
> > <https://reviews.apache.org/r/62226/diff/3/?file=1820189#file1820189line126>
> >
> > if not supported_packages:
> >supported_packages = get_supported_packages()
> >
> >
> >  else no sence on this

sense*


- Dmytro


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62226/#review185198
---


On Sept. 12, 2017, 2:21 p.m., Jonathan Hurley wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/62226/
> ---
> 
> (Updated Sept. 12, 2017, 2:21 p.m.)
> 
> 
> Review request for Ambari, Dmytro Grinenko, Dmitro Lisnichenko, and Nate Cole.
> 
> 
> Bugs: AMBARI-21931
> https://issues.apache.org/jira/browse/AMBARI-21931
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> There are several "INVALID" types listed in the stack_packages.json file. 
> These were placeholders that need to be updated with the packages exposed by 
> the stack-select tool. 
> 
> {code}
> ...
> "HDFS_CLIENT": {
>   "STACK-SELECT-PACKAGE": "hadoop-client",
>   "INSTALL": [
> "hadoop-client"
>   ],
>   "PATCH": [
> "INVALID"
>   ],
>   "STANDARD": [
> "hadoop-client"
>   ]
> },
> ...
> {code}
> 
> 
> Diffs
> -
> 
>   
> ambari-common/src/main/python/resource_management/libraries/functions/stack_select.py
>  eac1bef13c 
>   
> ambari-server/src/main/resources/stacks/HDP/2.0.6/properties/stack_packages.json
>  704fb54293 
>   ambari-server/src/test/python/TestStackSelect.py 3d4e5b6a43 
>   ambari-server/src/test/python/stacks/utils/RMFTestCase.py 0341092b69 
> 
> 
> Diff: https://reviews.apache.org/r/62226/diff/3/
> 
> 
> Testing
> ---
> 
> --
> Total run:1191
> Total errors:0
> Total failures:0
> OK
> [INFO] 
> 
> [INFO] BUILD SUCCESS
> [INFO] 
> 
> [INFO] Total time: 58.379 s
> [INFO] Finished at: 2017-09-11T16:25:26-04:00
> [INFO] Final Memory: 21M/619M
> [INFO] 
> 
> 
> 
> Thanks,
> 
> Jonathan Hurley
> 
>

Re: Review Request 62226: Use Correct Packages For Clients Where Stack Tools Support It

2017-09-12 Thread Dmytro Grinenko


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62226/#review185198
---




ambari-common/src/main/python/resource_management/libraries/functions/stack_select.py
Lines 126 (patched)
<https://reviews.apache.org/r/62226/#comment261491>

if not supported_packages:
   supported_packages = get_supported_packages()
   
   
 else no sence on this


- Dmytro Grinenko


On Sept. 12, 2017, 2:21 p.m., Jonathan Hurley wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/62226/
> ---
> 
> (Updated Sept. 12, 2017, 2:21 p.m.)
> 
> 
> Review request for Ambari, Dmytro Grinenko, Dmitro Lisnichenko, and Nate Cole.
> 
> 
> Bugs: AMBARI-21931
> https://issues.apache.org/jira/browse/AMBARI-21931
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> There are several "INVALID" types listed in the stack_packages.json file. 
> These were placeholders that need to be updated with the packages exposed by 
> the stack-select tool. 
> 
> {code}
> ...
> "HDFS_CLIENT": {
>   "STACK-SELECT-PACKAGE": "hadoop-client",
>   "INSTALL": [
> "hadoop-client"
>   ],
>   "PATCH": [
> "INVALID"
>   ],
>   "STANDARD": [
> "hadoop-client"
>   ]
> },
> ...
> {code}
> 
> 
> Diffs
> -
> 
>   
> ambari-common/src/main/python/resource_management/libraries/functions/stack_select.py
>  eac1bef13c 
>   
> ambari-server/src/main/resources/stacks/HDP/2.0.6/properties/stack_packages.json
>  704fb54293 
>   ambari-server/src/test/python/TestStackSelect.py 3d4e5b6a43 
>   ambari-server/src/test/python/stacks/utils/RMFTestCase.py 0341092b69 
> 
> 
> Diff: https://reviews.apache.org/r/62226/diff/3/
> 
> 
> Testing
> ---
> 
> --
> Total run:1191
> Total errors:0
> Total failures:0
> OK
> [INFO] 
> 
> [INFO] BUILD SUCCESS
> [INFO] 
> 
> [INFO] Total time: 58.379 s
> [INFO] Finished at: 2017-09-11T16:25:26-04:00
> [INFO] Final Memory: 21M/619M
> [INFO] 
> 
> 
> 
> Thanks,
> 
> Jonathan Hurley
> 
>

Re: Review Request 62226: Use Correct Packages For Clients Where Stack Tools Support It

2017-09-12 Thread Dmytro Grinenko


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62226/#review185197
---


Ship it!




Ship It!

- Dmytro Grinenko


On Sept. 12, 2017, 2:21 p.m., Jonathan Hurley wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/62226/
> ---
> 
> (Updated Sept. 12, 2017, 2:21 p.m.)
> 
> 
> Review request for Ambari, Dmytro Grinenko, Dmitro Lisnichenko, and Nate Cole.
> 
> 
> Bugs: AMBARI-21931
> https://issues.apache.org/jira/browse/AMBARI-21931
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> There are several "INVALID" types listed in the stack_packages.json file. 
> These were placeholders that need to be updated with the packages exposed by 
> the stack-select tool. 
> 
> {code}
> ...
> "HDFS_CLIENT": {
>   "STACK-SELECT-PACKAGE": "hadoop-client",
>   "INSTALL": [
> "hadoop-client"
>   ],
>   "PATCH": [
> "INVALID"
>   ],
>   "STANDARD": [
> "hadoop-client"
>   ]
> },
> ...
> {code}
> 
> 
> Diffs
> -
> 
>   
> ambari-common/src/main/python/resource_management/libraries/functions/stack_select.py
>  eac1bef13c 
>   
> ambari-server/src/main/resources/stacks/HDP/2.0.6/properties/stack_packages.json
>  704fb54293 
>   ambari-server/src/test/python/TestStackSelect.py 3d4e5b6a43 
>   ambari-server/src/test/python/stacks/utils/RMFTestCase.py 0341092b69 
> 
> 
> Diff: https://reviews.apache.org/r/62226/diff/3/
> 
> 
> Testing
> ---
> 
> --
> Total run:1191
> Total errors:0
> Total failures:0
> OK
> [INFO] 
> 
> [INFO] BUILD SUCCESS
> [INFO] 
> 
> [INFO] Total time: 58.379 s
> [INFO] Finished at: 2017-09-11T16:25:26-04:00
> [INFO] Final Memory: 21M/619M
> [INFO] 
> 
> 
> 
> Thanks,
> 
> Jonathan Hurley
> 
>

Re: Review Request 62226: Use Correct Packages For Clients Where Stack Tools Support It

2017-09-12 Thread Dmytro Grinenko



> On Sept. 12, 2017, 1:07 p.m., Dmytro Grinenko wrote:
> > ambari-common/src/main/python/resource_management/libraries/functions/stack_select.py
> > Lines 145 (patched)
> > <https://reviews.apache.org/r/62226/diff/2/?file=1819598#file1819598line145>
> >
> > while map function is present in python, using generators is more 
> > prefferable
> 
> Jonathan Hurley wrote:
> I've never used generators before - the syntax looks a bit cumbersome for 
> what's needed here. Can you provide an example of what a generator to trim a 
> string would look like?
> 
> Dmytro Grinenko wrote:
> return (s.strip() for s in stdout.splitlines())# this would return 
> gerator, which supports iterative interface
> 
> pros:
>  - actuall method execution would be issues only when you will call 
> iterator, each result would return via "yield" (code would run only once)
> 
> cons:
>  - this would be not list, so no index accessing and only one cycle
> 
> return [s.strip() for s in stdout.splitlines()]# this would return 
> list
>  - this will return list
> 
> Dmytro Grinenko wrote:
> however for python36, it looks that map is faster that list comphersation 
> :)
> 
> Jonathan Hurley wrote:
> Hah! I changed it. If you'd like me to switch it back, let me know.

sience the day, when we will move to python 3.x is even not on horizon, it is 
fine to stay with generators/list comphersation


- Dmytro


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62226/#review185174
---


On Sept. 12, 2017, 2:21 p.m., Jonathan Hurley wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/62226/
> ---
> 
> (Updated Sept. 12, 2017, 2:21 p.m.)
> 
> 
> Review request for Ambari, Dmytro Grinenko, Dmitro Lisnichenko, and Nate Cole.
> 
> 
> Bugs: AMBARI-21931
> https://issues.apache.org/jira/browse/AMBARI-21931
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> There are several "INVALID" types listed in the stack_packages.json file. 
> These were placeholders that need to be updated with the packages exposed by 
> the stack-select tool. 
> 
> {code}
> ...
> "HDFS_CLIENT": {
>   "STACK-SELECT-PACKAGE": "hadoop-client",
>   "INSTALL": [
> "hadoop-client"
>   ],
>   "PATCH": [
> "INVALID"
>   ],
>   "STANDARD": [
> "hadoop-client"
>   ]
> },
> ...
> {code}
> 
> 
> Diffs
> -
> 
>   
> ambari-common/src/main/python/resource_management/libraries/functions/stack_select.py
>  eac1bef13c 
>   
> ambari-server/src/main/resources/stacks/HDP/2.0.6/properties/stack_packages.json
>  704fb54293 
>   ambari-server/src/test/python/TestStackSelect.py 3d4e5b6a43 
>   ambari-server/src/test/python/stacks/utils/RMFTestCase.py 0341092b69 
> 
> 
> Diff: https://reviews.apache.org/r/62226/diff/3/
> 
> 
> Testing
> ---
> 
> --
> Total run:1191
> Total errors:0
> Total failures:0
> OK
> [INFO] 
> 
> [INFO] BUILD SUCCESS
> [INFO] 
> 
> [INFO] Total time: 58.379 s
> [INFO] Finished at: 2017-09-11T16:25:26-04:00
> [INFO] Final Memory: 21M/619M
> [INFO] 
> 
> 
> 
> Thanks,
> 
> Jonathan Hurley
> 
>

Re: Review Request 62226: Use Correct Packages For Clients Where Stack Tools Support It

2017-09-12 Thread Dmytro Grinenko



> On Sept. 12, 2017, 1:07 p.m., Dmytro Grinenko wrote:
> > ambari-common/src/main/python/resource_management/libraries/functions/stack_select.py
> > Lines 145 (patched)
> > <https://reviews.apache.org/r/62226/diff/2/?file=1819598#file1819598line145>
> >
> > while map function is present in python, using generators is more 
> > prefferable
> 
> Jonathan Hurley wrote:
> I've never used generators before - the syntax looks a bit cumbersome for 
> what's needed here. Can you provide an example of what a generator to trim a 
> string would look like?
> 
> Dmytro Grinenko wrote:
> return (s.strip() for s in stdout.splitlines())# this would return 
> gerator, which supports iterative interface
> 
> pros:
>  - actuall method execution would be issues only when you will call 
> iterator, each result would return via "yield" (code would run only once)
> 
> cons:
>  - this would be not list, so no index accessing and only one cycle
> 
> return [s.strip() for s in stdout.splitlines()]# this would return 
> list
>  - this will return list

however for python36, it looks that map is faster that list comphersation :)


- Dmytro


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62226/#review185174
---


On Sept. 11, 2017, 7:43 p.m., Jonathan Hurley wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/62226/
> ---
> 
> (Updated Sept. 11, 2017, 7:43 p.m.)
> 
> 
> Review request for Ambari, Dmytro Grinenko, Dmitro Lisnichenko, and Nate Cole.
> 
> 
> Bugs: AMBARI-21931
> https://issues.apache.org/jira/browse/AMBARI-21931
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> There are several "INVALID" types listed in the stack_packages.json file. 
> These were placeholders that need to be updated with the packages exposed by 
> the stack-select tool. 
> 
> {code}
> ...
> "HDFS_CLIENT": {
>   "STACK-SELECT-PACKAGE": "hadoop-client",
>   "INSTALL": [
> "hadoop-client"
>   ],
>   "PATCH": [
> "INVALID"
>   ],
>   "STANDARD": [
> "hadoop-client"
>   ]
> },
> ...
> {code}
> 
> 
> Diffs
> -
> 
>   
> ambari-common/src/main/python/resource_management/libraries/functions/stack_select.py
>  eac1bef13c 
>   
> ambari-server/src/main/resources/stacks/HDP/2.0.6/properties/stack_packages.json
>  704fb54293 
>   ambari-server/src/test/python/TestStackSelect.py 3d4e5b6a43 
>   ambari-server/src/test/python/stacks/utils/RMFTestCase.py 0341092b69 
> 
> 
> Diff: https://reviews.apache.org/r/62226/diff/2/
> 
> 
> Testing
> ---
> 
> --
> Total run:1191
> Total errors:0
> Total failures:0
> OK
> [INFO] 
> 
> [INFO] BUILD SUCCESS
> [INFO] 
> 
> [INFO] Total time: 58.379 s
> [INFO] Finished at: 2017-09-11T16:25:26-04:00
> [INFO] Final Memory: 21M/619M
> [INFO] 
> 
> 
> 
> Thanks,
> 
> Jonathan Hurley
> 
>

Re: Review Request 62226: Use Correct Packages For Clients Where Stack Tools Support It

2017-09-12 Thread Dmytro Grinenko



> On Sept. 12, 2017, 1:07 p.m., Dmytro Grinenko wrote:
> > ambari-common/src/main/python/resource_management/libraries/functions/stack_select.py
> > Lines 125 (patched)
> > <https://reviews.apache.org/r/62226/diff/2/?file=1819598#file1819598line125>
> >
> > According to code below, this function would be called 
> > "get_supported_packages" each time in for cycle and would do/return the 
> > same. Can we optimize this?
> 
> Jonathan Hurley wrote:
> Yes, I thought about the fact it's called a lot. The problem is that 
> caching the result wouldn't work since we don't know when to clear the cache. 
> How would you recommend that we optimize this?

No need to cache it in the script-life cycle, it is enough only for 
function-call lifetime, as i dont see any change, which may cause output change 
per function call.

so it could be like:

def is_package_supported(package, pkgs=None):
  if not pkgs:
pkgs = get_supported_packages()

  ..


def get_packages():

 ...

 pkgs = get_supported_packages()

 for .:
  if not is_package_supported(package, pkgs)
 .


> On Sept. 12, 2017, 1:07 p.m., Dmytro Grinenko wrote:
> > ambari-common/src/main/python/resource_management/libraries/functions/stack_select.py
> > Lines 145 (patched)
> > <https://reviews.apache.org/r/62226/diff/2/?file=1819598#file1819598line145>
> >
> > while map function is present in python, using generators is more 
> > prefferable
> 
> Jonathan Hurley wrote:
> I've never used generators before - the syntax looks a bit cumbersome for 
> what's needed here. Can you provide an example of what a generator to trim a 
> string would look like?

return (s.strip() for s in stdout.splitlines())# this would return gerator, 
which supports iterative interface

pros:
 - actuall method execution would be issues only when you will call iterator, 
each result would return via "yield" (code would run only once)

cons:
 - this would be not list, so no index accessing and only one cycle

return [s.strip() for s in stdout.splitlines()]# this would return list
 - this will return list


- Dmytro


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62226/#review185174
---


On Sept. 11, 2017, 7:43 p.m., Jonathan Hurley wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/62226/
> ---
> 
> (Updated Sept. 11, 2017, 7:43 p.m.)
> 
> 
> Review request for Ambari, Dmytro Grinenko, Dmitro Lisnichenko, and Nate Cole.
> 
> 
> Bugs: AMBARI-21931
> https://issues.apache.org/jira/browse/AMBARI-21931
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> There are several "INVALID" types listed in the stack_packages.json file. 
> These were placeholders that need to be updated with the packages exposed by 
> the stack-select tool. 
> 
> {code}
> ...
> "HDFS_CLIENT": {
>   "STACK-SELECT-PACKAGE": "hadoop-client",
>   "INSTALL": [
> "hadoop-client"
>   ],
>   "PATCH": [
> "INVALID"
>   ],
>   "STANDARD": [
> "hadoop-client"
>   ]
> },
> ...
> {code}
> 
> 
> Diffs
> -
> 
>   
> ambari-common/src/main/python/resource_management/libraries/functions/stack_select.py
>  eac1bef13c 
>   
> ambari-server/src/main/resources/stacks/HDP/2.0.6/properties/stack_packages.json
>  704fb54293 
>   ambari-server/src/test/python/TestStackSelect.py 3d4e5b6a43 
>   ambari-server/src/test/python/stacks/utils/RMFTestCase.py 0341092b69 
> 
> 
> Diff: https://reviews.apache.org/r/62226/diff/2/
> 
> 
> Testing
> ---
> 
> --
> Total run:1191
> Total errors:0
> Total failures:0
> OK
> [INFO] 
> 
> [INFO] BUILD SUCCESS
> [INFO] 
> 
> [INFO] Total time: 58.379 s
> [INFO] Finished at: 2017-09-11T16:25:26-04:00
> [INFO] Final Memory: 21M/619M
> [INFO] 
> 
> 
> 
> Thanks,
> 
> Jonathan Hurley
> 
>

Re: Review Request 62226: Use Correct Packages For Clients Where Stack Tools Support It

2017-09-12 Thread Dmytro Grinenko


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62226/#review185174
---


Fix it, then Ship it!




Ship It!


ambari-common/src/main/python/resource_management/libraries/functions/stack_select.py
Lines 125 (patched)
<https://reviews.apache.org/r/62226/#comment261467>

According to code below, this function would be called 
"get_supported_packages" each time in for cycle and would do/return the same. 
Can we optimize this?



ambari-common/src/main/python/resource_management/libraries/functions/stack_select.py
Lines 138 (patched)
<https://reviews.apache.org/r/62226/#comment261465>

No parentheses needed to define tuple in such way, just list variables 
using coma (it is pythonic way)



ambari-common/src/main/python/resource_management/libraries/functions/stack_select.py
Lines 145 (patched)
<https://reviews.apache.org/r/62226/#comment261466>

while map function is present in python, using generators is more 
prefferable


- Dmytro Grinenko


On Sept. 11, 2017, 7:43 p.m., Jonathan Hurley wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/62226/
> ---
> 
> (Updated Sept. 11, 2017, 7:43 p.m.)
> 
> 
> Review request for Ambari, Dmytro Grinenko, Dmitro Lisnichenko, and Nate Cole.
> 
> 
> Bugs: AMBARI-21931
> https://issues.apache.org/jira/browse/AMBARI-21931
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> There are several "INVALID" types listed in the stack_packages.json file. 
> These were placeholders that need to be updated with the packages exposed by 
> the stack-select tool. 
> 
> {code}
> ...
> "HDFS_CLIENT": {
>   "STACK-SELECT-PACKAGE": "hadoop-client",
>   "INSTALL": [
> "hadoop-client"
>   ],
>   "PATCH": [
> "INVALID"
>   ],
>   "STANDARD": [
> "hadoop-client"
>   ]
> },
> ...
> {code}
> 
> 
> Diffs
> -
> 
>   
> ambari-common/src/main/python/resource_management/libraries/functions/stack_select.py
>  eac1bef13c 
>   
> ambari-server/src/main/resources/stacks/HDP/2.0.6/properties/stack_packages.json
>  704fb54293 
>   ambari-server/src/test/python/TestStackSelect.py 3d4e5b6a43 
>   ambari-server/src/test/python/stacks/utils/RMFTestCase.py 0341092b69 
> 
> 
> Diff: https://reviews.apache.org/r/62226/diff/2/
> 
> 
> Testing
> ---
> 
> --
> Total run:1191
> Total errors:0
> Total failures:0
> OK
> [INFO] 
> 
> [INFO] BUILD SUCCESS
> [INFO] 
> 
> [INFO] Total time: 58.379 s
> [INFO] Finished at: 2017-09-11T16:25:26-04:00
> [INFO] Final Memory: 21M/619M
> [INFO] 
> 
> 
> 
> Thanks,
> 
> Jonathan Hurley
> 
>

Re: Review Request 62095: Installation should ignore OS that are not managed by Ambari

2017-09-06 Thread Dmytro Grinenko


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62095/#review184684
---


Ship it!




Ship It!

- Dmytro Grinenko


On Sept. 6, 2017, 1:55 p.m., Nate Cole wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/62095/
> ---
> 
> (Updated Sept. 6, 2017, 1:55 p.m.)
> 
> 
> Review request for Ambari, Dmytro Grinenko and Jonathan Hurley.
> 
> 
> Bugs: AMBARI-21884
> https://issues.apache.org/jira/browse/AMBARI-21884
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Added fixes that broke RH Satellite option.
> 
> @dgrinenko, this will likely conflict with your legacy patch.
> 
> 
> Diffs
> -
> 
>   
> ambari-common/src/main/python/resource_management/libraries/functions/repository_util.py
>  91009b0d2d 
>   
> ambari-server/src/main/java/org/apache/ambari/server/agent/CommandRepository.java
>  3d961222b4 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariCustomCommandExecutionHelper.java
>  26309fed99 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/ClusterStackVersionResourceProvider.java
>  f96f257f9e 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/HostStackVersionResourceProvider.java
>  fb4831edac 
>   
> ambari-server/src/main/java/org/apache/ambari/server/state/stack/upgrade/RepositoryVersionHelper.java
>  227d42d419 
>   ambari-server/src/main/resources/version_definition.xsd 851e0d5c3c 
>   
> ambari-server/src/test/java/org/apache/ambari/server/actionmanager/ExecutionCommandWrapperTest.java
>  94ad0d57fe 
>   
> ambari-server/src/test/java/org/apache/ambari/server/controller/AmbariCustomCommandExecutionHelperTest.java
>  a60b696808 
>   
> ambari-server/src/test/java/org/apache/ambari/server/controller/internal/ClusterStackVersionResourceProviderTest.java
>  42b2a4caef 
> 
> 
> Diff: https://reviews.apache.org/r/62095/diff/2/
> 
> 
> Testing
> ---
> 
> Manual.  Automated:
> 
> [WARNING] Tests run: 4843, Failures: 0, Errors: 0, Skipped: 35
> [INFO]
> [INFO] 
> 
> [INFO] BUILD SUCCESS
> [INFO] 
> 
> [INFO] Total time: 24:13.075s
> [INFO] Finished at: Tue Sep 05 17:26:36 EDT 2017
> [INFO] Final Memory: 68M/1908M
> [INFO] 
> 
> 
> 
> Thanks,
> 
> Nate Cole
> 
>

Re: Review Request 62095: Installation should ignore OS that are not managed by Ambari

2017-09-06 Thread Dmytro Grinenko



> On Sept. 6, 2017, 12:43 a.m., Dmytro Grinenko wrote:
> > ambari-server/src/main/java/org/apache/ambari/server/agent/CommandRepository.java
> > Lines 113 (patched)
> > <https://reviews.apache.org/r/62095/diff/1/?file=1815904#file1815904line113>
> >
> > what is the sense even to pass repositories if Ambari doesn't manage 
> > them ?   CommandRepository should be enough
> > 
> > optional: pass not-managed flag to tell that explicitly to the Agent as 
> > i dont really like the think to guessing on baseurl

i even could say, that we need to pass "not-managed" flag, as in yumrpm.py, 
zypper.py, apt.py we trying to scope some commands to repoID, which is not 
available for ambari in case of non-managed repositories. And the one supplied 
in repositoryFile, generated by Ambari is most likely wrong


- Dmytro


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62095/#review184611
---


On Sept. 5, 2017, 9:04 p.m., Nate Cole wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/62095/
> -------
> 
> (Updated Sept. 5, 2017, 9:04 p.m.)
> 
> 
> Review request for Ambari, Dmytro Grinenko and Jonathan Hurley.
> 
> 
> Bugs: AMBARI-21884
> https://issues.apache.org/jira/browse/AMBARI-21884
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Added fixes that broke RH Satellite option.
> 
> @dgrinenko, this will likely conflict with your legacy patch.
> 
> 
> Diffs
> -
> 
>   
> ambari-common/src/main/python/resource_management/libraries/functions/repository_util.py
>  91009b0d2d 
>   
> ambari-server/src/main/java/org/apache/ambari/server/agent/CommandRepository.java
>  3d961222b4 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariCustomCommandExecutionHelper.java
>  26309fed99 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/ClusterStackVersionResourceProvider.java
>  f96f257f9e 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/HostStackVersionResourceProvider.java
>  fb4831edac 
>   
> ambari-server/src/main/java/org/apache/ambari/server/state/stack/upgrade/RepositoryVersionHelper.java
>  227d42d419 
>   ambari-server/src/main/resources/version_definition.xsd 851e0d5c3c 
>   
> ambari-server/src/test/java/org/apache/ambari/server/actionmanager/ExecutionCommandWrapperTest.java
>  94ad0d57fe 
>   
> ambari-server/src/test/java/org/apache/ambari/server/controller/AmbariCustomCommandExecutionHelperTest.java
>  a60b696808 
>   
> ambari-server/src/test/java/org/apache/ambari/server/controller/internal/ClusterStackVersionResourceProviderTest.java
>  42b2a4caef 
> 
> 
> Diff: https://reviews.apache.org/r/62095/diff/1/
> 
> 
> Testing
> ---
> 
> Manual.  Automated:
> 
> [WARNING] Tests run: 4843, Failures: 0, Errors: 0, Skipped: 35
> [INFO]
> [INFO] 
> 
> [INFO] BUILD SUCCESS
> [INFO] 
> 
> [INFO] Total time: 24:13.075s
> [INFO] Finished at: Tue Sep 05 17:26:36 EDT 2017
> [INFO] Final Memory: 68M/1908M
> [INFO] 
> 
> 
> 
> Thanks,
> 
> Nate Cole
> 
>

Re: Review Request 62095: Installation should ignore OS that are not managed by Ambari

2017-09-05 Thread Dmytro Grinenko


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62095/#review184611
---


Fix it, then Ship it!




Ship It!


ambari-server/src/main/java/org/apache/ambari/server/agent/CommandRepository.java
Lines 113 (patched)
<https://reviews.apache.org/r/62095/#comment260774>

what is the sense even to pass repositories if Ambari doesn't manage them ? 
  CommandRepository should be enough

optional: pass not-managed flag to tell that explicitly to the Agent as i 
dont really like the think to guessing on baseurl



ambari-server/src/main/java/org/apache/ambari/server/controller/internal/ClusterStackVersionResourceProvider.java
Lines 89 (patched)
<https://reviews.apache.org/r/62095/#comment260776>

why to not use apache commons instead of spring internals?


- Dmytro Grinenko


On Sept. 5, 2017, 9:04 p.m., Nate Cole wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/62095/
> ---
> 
> (Updated Sept. 5, 2017, 9:04 p.m.)
> 
> 
> Review request for Ambari, Dmytro Grinenko and Jonathan Hurley.
> 
> 
> Bugs: AMBARI-21884
> https://issues.apache.org/jira/browse/AMBARI-21884
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Added fixes that broke RH Satellite option.
> 
> @dgrinenko, this will likely conflict with your legacy patch.
> 
> 
> Diffs
> -
> 
>   
> ambari-common/src/main/python/resource_management/libraries/functions/repository_util.py
>  91009b0d2d 
>   
> ambari-server/src/main/java/org/apache/ambari/server/agent/CommandRepository.java
>  3d961222b4 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariCustomCommandExecutionHelper.java
>  26309fed99 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/ClusterStackVersionResourceProvider.java
>  f96f257f9e 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/HostStackVersionResourceProvider.java
>  fb4831edac 
>   
> ambari-server/src/main/java/org/apache/ambari/server/state/stack/upgrade/RepositoryVersionHelper.java
>  227d42d419 
>   ambari-server/src/main/resources/version_definition.xsd 851e0d5c3c 
>   
> ambari-server/src/test/java/org/apache/ambari/server/actionmanager/ExecutionCommandWrapperTest.java
>  94ad0d57fe 
>   
> ambari-server/src/test/java/org/apache/ambari/server/controller/AmbariCustomCommandExecutionHelperTest.java
>  a60b696808 
>   
> ambari-server/src/test/java/org/apache/ambari/server/controller/internal/ClusterStackVersionResourceProviderTest.java
>  42b2a4caef 
> 
> 
> Diff: https://reviews.apache.org/r/62095/diff/1/
> 
> 
> Testing
> ---
> 
> Manual.  Automated:
> 
> [WARNING] Tests run: 4843, Failures: 0, Errors: 0, Skipped: 35
> [INFO]
> [INFO] 
> 
> [INFO] BUILD SUCCESS
> [INFO] 
> 
> [INFO] Total time: 24:13.075s
> [INFO] Finished at: Tue Sep 05 17:26:36 EDT 2017
> [INFO] Final Memory: 68M/1908M
> [INFO] 
> 
> 
> 
> Thanks,
> 
> Nate Cole
> 
>

Re: Review Request 61949: Reject PATCH VDFs with Services that are not Included in the Cluster

2017-09-01 Thread Dmytro Grinenko


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61949/#review184353
---


Ship it!




Ship It!

- Dmytro Grinenko


On Sept. 1, 2017, 1:20 p.m., Dmitro Lisnichenko wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61949/
> ---
> 
> (Updated Sept. 1, 2017, 1:20 p.m.)
> 
> 
> Review request for Ambari, Dmytro Grinenko, Jonathan Hurley, and Nate Cole.
> 
> 
> Bugs: AMBARI-21832
> https://issues.apache.org/jira/browse/AMBARI-21832
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Currently there is an odd scenario which can occur when patch repositories 
> are registered which have services not yet installed. Consider the following 
> scenario:
> 
> - Install ZooKeeper, Storm on HDP 2.6.0.0-1234
> - Register/patch a {{PATCH}} VDF for Storm and Accumulo for 2.6.0.1-
> - Install Accumulo
> 
> Which version does Accumulo use - the {{STANDARD}} repository or the 
> {{PATCH}}? If the {{PATCH}} repository is chosen, this will now prevent 
> reversion of the patch since there's no prior version for Accumulo to revert 
> back to.
> 
> If Accumulo uses the {{STANDARD}} repo, then there needs to be a lot of 
> design and UX flow work provided to indicate that a {{PATCH}} which was 
> previously applied can be re-applied for the new service. This also causes 
> problems for patch reversion since now there would be two upgrades which need 
> to be reverted to "get rid" of the patch.
> 
> For the timeframe for Ambari 2.6, we should reject VDFs that include services 
> which are not installed. This will prevent the problem.
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/ClusterStackVersionResourceProvider.java
>  7a53e91bb0 
>   
> ambari-server/src/test/java/org/apache/ambari/server/controller/internal/ClusterStackVersionResourceProviderTest.java
>  282f159a66 
> 
> 
> Diff: https://reviews.apache.org/r/61949/diff/3/
> 
> 
> Testing
> ---
> 
> mvn clean test
> 
> UI misbehaves when VDF validation triggers error. Will open a UI jira for that
> 
> 
> Thanks,
> 
> Dmitro Lisnichenko
> 
>

Re: Review Request 61715: package_regex in get_package_from_available() can match wrong pkg

2017-08-30 Thread Dmytro Grinenko


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61715/#review184142
---


Ship it!




Ship It!

- Dmytro Grinenko


On Aug. 29, 2017, 3:07 p.m., Dmitro Lisnichenko wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61715/
> ---
> 
> (Updated Aug. 29, 2017, 3:07 p.m.)
> 
> 
> Review request for Ambari, Dmytro Grinenko and Vitalyi Brodetskyi.
> 
> 
> Bugs: AMBARI-21744
> https://issues.apache.org/jira/browse/AMBARI-21744
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Due to the issue with regex (missing ^ and $ boundaries), 
> resource_management.libraries.script.script.Script#get_package_from_available 
> may return wrong package.
> {code}
> >>> list=['hbase_3_0_0_0_229-master', 'hbase_3_0_0_0_229']
> >>> if re.match('hbase_(\d|_)+', 'hbase_3_0_0_0_229-master'):
> ...print 'YES'
> ...
> YES
> >>> if re.match('hbase_(\d|_)+', 'hbase_3_0_0_0_229'):
> ...print 'YES'
> ...
> YES
> {code}
> 
> In this case, the first package name from a list of available packages will 
> be returned.
> The impact of bug is that we may install a wrong package if it's simillary 
> named and goes first at list. Patch is a single-line fix.
> 
> 
> Diffs
> -
> 
>   
> ambari-common/src/main/python/resource_management/libraries/script/script.py 
> 313b666878 
>   ambari-server/src/test/python/custom_actions/TestInstallPackages.py 
> de2cced554 
> 
> 
> Diff: https://reviews.apache.org/r/61715/diff/3/
> 
> 
> Testing
> ---
> 
> mvn clean test
> 
> 
> Thanks,
> 
> Dmitro Lisnichenko
> 
>

Re: Review Request 61715: PREVIEW: package_regex in get_package_from_available() can match wrong pkg

2017-08-28 Thread Dmytro Grinenko


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61715/#review183939
---




ambari-common/src/main/python/resource_management/libraries/script/script.py
Line 455 (original), 455 (patched)
<https://reviews.apache.org/r/61715/#comment259985>

it looks that this function is cross-linux version, could we move it to 
https://github.com/apache/ambari/tree/trunk/ambari-common/src/main/python/resource_management/core/providers/package
   to respective os provider?


- Dmytro Grinenko


On Aug. 28, 2017, 3:49 p.m., Dmitro Lisnichenko wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61715/
> ---
> 
> (Updated Aug. 28, 2017, 3:49 p.m.)
> 
> 
> Review request for Ambari, Dmytro Grinenko and Vitalyi Brodetskyi.
> 
> 
> Bugs: AMBARI-21744
> https://issues.apache.org/jira/browse/AMBARI-21744
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Due to the issue with regex (missing ^ and $ boundaries), 
> resource_management.libraries.script.script.Script#get_package_from_available 
> may return wrong package.
> {code}
> >>> list=['hbase_3_0_0_0_229-master', 'hbase_3_0_0_0_229']
> >>> if re.match('hbase_(\d|_)+', 'hbase_3_0_0_0_229-master'):
> ...print 'YES'
> ...
> YES
> >>> if re.match('hbase_(\d|_)+', 'hbase_3_0_0_0_229'):
> ...print 'YES'
> ...
> YES
> {code}
> 
> In this case, the first package name from a list of available packages will 
> be returned.
> The impact of bug is that we may install a wrong package if it's simillary 
> named and goes first at list. Patch is a single-line fix.
> 
> 
> Diffs
> -
> 
>   
> ambari-common/src/main/python/resource_management/libraries/script/script.py 
> 313b666878 
> 
> 
> Diff: https://reviews.apache.org/r/61715/diff/2/
> 
> 
> Testing
> ---
> 
> Will fix unit tests a bit later
> 
> 
> Thanks,
> 
> Dmitro Lisnichenko
> 
>

Re: Review Request 61746: Prevent New Clusters from Being Provisioned With PATCH/MAINT Repos

2017-08-21 Thread Dmytro Grinenko


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61746/#review183324
---


Ship it!




Ship It!

- Dmytro Grinenko


On Aug. 19, 2017, 1:23 a.m., Jonathan Hurley wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61746/
> ---
> 
> (Updated Aug. 19, 2017, 1:23 a.m.)
> 
> 
> Review request for Ambari, Dmytro Grinenko, Dmitro Lisnichenko, Nate Cole, 
> and Robert Levas.
> 
> 
> Bugs: AMBARI-21758
> https://issues.apache.org/jira/browse/AMBARI-21758
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> New clusters should only be able to be provisioned using a {{STANDARD}} 
> repository. 
> 
> STR:
> - Register a 2.5.0.0-1234 {{STANDARD}} repo
> - Register a 2.5.4.0- {{MAINT} repo
> - Use a blueprint go create a repo for 2.5.4.0-
> 
> The cluster is created and is in a weird state. We should reject these types 
> of repos...
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariManagementControllerImpl.java
>  60f5ccea67 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/ClusterRequest.java
>  aea2072c65 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/ServiceResourceProvider.java
>  2b4bf3bba9 
>   
> ambari-server/src/main/java/org/apache/ambari/server/orm/entities/RepositoryVersionEntity.java
>  27b1654fd3 
>   
> ambari-server/src/main/java/org/apache/ambari/server/topology/AmbariContext.java
>  d77d13c00b 
>   
> ambari-server/src/test/java/org/apache/ambari/server/topology/AmbariContextTest.java
>  fc4fa868b5 
> 
> 
> Diff: https://reviews.apache.org/r/61746/diff/2/
> 
> 
> Testing
> ---
> 
> mvn clean test
> 
> 
> Thanks,
> 
> Jonathan Hurley
> 
>

Re: Review Request 61622: Begin Using Service Versions In Python stack_feature Code

2017-08-14 Thread Dmytro Grinenko


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61622/#review182913
---


Ship it!




Ship It!

- Dmytro Grinenko


On Aug. 14, 2017, 9:01 p.m., Jonathan Hurley wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61622/
> ---
> 
> (Updated Aug. 14, 2017, 9:01 p.m.)
> 
> 
> Review request for Ambari, Dmytro Grinenko, Dmitro Lisnichenko, and Nate Cole.
> 
> 
> Bugs: AMBARI-21722
> https://issues.apache.org/jira/browse/AMBARI-21722
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> The command generators in Ambari are sending down the 
> {{KeyName.CURRENT_VERSION}} ({{current_version}}) property in the JSON to the 
> agents. This is being used to determine the "current cluster version" which 
> doesn't really exist anymore:
> 
> {code:title=stack_features.py}
>   # something like 2.4.0.0-1234
>   # (or None if this is a cluster install and it hasn't been calculated yet)
>   current_cluster_version = default("/hostLevelParams/current_version", None)
> {code}
> 
> This really needs to be calculated from the service's repository version 
> instead.
> 
> 
> Diffs
> -
> 
>   
> ambari-common/src/main/python/resource_management/libraries/functions/component_version.py
>  PRE-CREATION 
>   
> ambari-common/src/main/python/resource_management/libraries/functions/copy_tarball.py
>  31a9be414c 
>   
> ambari-common/src/main/python/resource_management/libraries/functions/stack_features.py
>  3fcce820ca 
>   
> ambari-common/src/main/python/resource_management/libraries/functions/upgrade_summary.py
>  PRE-CREATION 
>   
> ambari-server/src/main/java/org/apache/ambari/server/actionmanager/ExecutionCommandWrapper.java
>  e4b2540696 
>   
> ambari-server/src/main/java/org/apache/ambari/server/agent/ExecutionCommand.java
>  9b6b2f58dd 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariManagementControllerImpl.java
>  7e26fd790d 
>   
> ambari-server/src/main/java/org/apache/ambari/server/state/UpgradeContext.java
>  e6a420e090 
>   
> ambari-server/src/main/resources/common-services/FALCON/0.5.0.2.1/package/scripts/falcon.py
>  c9c66ac1dd 
>   
> ambari-server/src/main/resources/common-services/FALCON/0.5.0.2.1/package/scripts/params_linux.py
>  3773918ef7 
>   
> ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/hive_metastore.py
>  f94248bd8b 
>   
> ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/hive_server_upgrade.py
>  17db489a03 
>   
> ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/hive_service.py
>  22b4061d5d 
>   
> ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/params_linux.py
>  d46b6ce721 
>   
> ambari-server/src/main/resources/common-services/KAFKA/0.8.1/package/scripts/kafka.py
>  e6d73393f5 
>   
> ambari-server/src/main/resources/common-services/KAFKA/0.8.1/package/scripts/kafka_broker.py
>  266bb429ec 
>   
> ambari-server/src/main/resources/common-services/KAFKA/0.8.1/package/scripts/params.py
>  8aa4fc26e4 
>   
> ambari-server/src/main/resources/common-services/KNOX/0.5.0.2.2/package/scripts/params_linux.py
>  9b0bbfc836 
>   
> ambari-server/src/main/resources/common-services/OOZIE/4.0.0.2.0/package/scripts/oozie.py
>  aa5bc308fe 
>   
> ambari-server/src/main/resources/common-services/ZOOKEEPER/3.4.5/package/scripts/params_linux.py
>  b8e8f78df1 
>   
> ambari-server/src/main/resources/stacks/BigInsights/4.0/services/HIVE/package/scripts/hive_metastore.py
>  88bb2e09e8 
>   
> ambari-server/src/main/resources/stacks/BigInsights/4.0/services/HIVE/package/scripts/hive_server_upgrade.py
>  318fcca23d 
>   
> ambari-server/src/main/resources/stacks/BigInsights/4.0/services/HIVE/package/scripts/params.py
>  e5fe128873 
>   
> ambari-server/src/main/resources/stacks/BigInsights/4.0/services/KAFKA/package/scripts/kafka_broker.py
>  cb5954e306 
>   
> ambari-server/src/main/resources/stacks/BigInsights/4.0/services/KAFKA/package/scripts/params.py
>  bc197042b1 
>   
> ambari-server/src/main/resources/stacks/BigInsights/4.0/services/SOLR/package/scripts/params.py
>  d5d90f615b 
>   
> ambari-server/src/main/resources/stacks/BigInsights/4.0/services/SOLR/package/scripts/solr_service.py
>  105aac63fe 
>   
> ambari-s

Re: Review Request 61622: Begin Using Service Versions In Python stack_feature Code

2017-08-14 Thread Dmytro Grinenko


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61622/#review182912
---




ambari-common/src/main/python/resource_management/libraries/functions/component_version.py
Lines 21 (patched)
<https://reviews.apache.org/r/61622/#comment258850>

it looks that script getting more and more arguments required to be parsed 
and a lot of modules need to be imported like linux_params. Why we cant create 
class named ScriptParams for example and put all params there with initializing 
that class on script instance creation. In such way we can decrease amount of 
work on each script we have and would need in the future with likely good side 
effect - keeping same functionality at the same place.

But it looks like a separate issue


- Dmytro Grinenko


On Aug. 14, 2017, 9:01 p.m., Jonathan Hurley wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61622/
> ---
> 
> (Updated Aug. 14, 2017, 9:01 p.m.)
> 
> 
> Review request for Ambari, Dmytro Grinenko, Dmitro Lisnichenko, and Nate Cole.
> 
> 
> Bugs: AMBARI-21722
> https://issues.apache.org/jira/browse/AMBARI-21722
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> The command generators in Ambari are sending down the 
> {{KeyName.CURRENT_VERSION}} ({{current_version}}) property in the JSON to the 
> agents. This is being used to determine the "current cluster version" which 
> doesn't really exist anymore:
> 
> {code:title=stack_features.py}
>   # something like 2.4.0.0-1234
>   # (or None if this is a cluster install and it hasn't been calculated yet)
>   current_cluster_version = default("/hostLevelParams/current_version", None)
> {code}
> 
> This really needs to be calculated from the service's repository version 
> instead.
> 
> 
> Diffs
> -
> 
>   
> ambari-common/src/main/python/resource_management/libraries/functions/component_version.py
>  PRE-CREATION 
>   
> ambari-common/src/main/python/resource_management/libraries/functions/copy_tarball.py
>  31a9be414c 
>   
> ambari-common/src/main/python/resource_management/libraries/functions/stack_features.py
>  3fcce820ca 
>   
> ambari-common/src/main/python/resource_management/libraries/functions/upgrade_summary.py
>  PRE-CREATION 
>   
> ambari-server/src/main/java/org/apache/ambari/server/actionmanager/ExecutionCommandWrapper.java
>  e4b2540696 
>   
> ambari-server/src/main/java/org/apache/ambari/server/agent/ExecutionCommand.java
>  9b6b2f58dd 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariManagementControllerImpl.java
>  7e26fd790d 
>   
> ambari-server/src/main/java/org/apache/ambari/server/state/UpgradeContext.java
>  e6a420e090 
>   
> ambari-server/src/main/resources/common-services/FALCON/0.5.0.2.1/package/scripts/falcon.py
>  c9c66ac1dd 
>   
> ambari-server/src/main/resources/common-services/FALCON/0.5.0.2.1/package/scripts/params_linux.py
>  3773918ef7 
>   
> ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/hive_metastore.py
>  f94248bd8b 
>   
> ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/hive_server_upgrade.py
>  17db489a03 
>   
> ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/hive_service.py
>  22b4061d5d 
>   
> ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/params_linux.py
>  d46b6ce721 
>   
> ambari-server/src/main/resources/common-services/KAFKA/0.8.1/package/scripts/kafka.py
>  e6d73393f5 
>   
> ambari-server/src/main/resources/common-services/KAFKA/0.8.1/package/scripts/kafka_broker.py
>  266bb429ec 
>   
> ambari-server/src/main/resources/common-services/KAFKA/0.8.1/package/scripts/params.py
>  8aa4fc26e4 
>   
> ambari-server/src/main/resources/common-services/KNOX/0.5.0.2.2/package/scripts/params_linux.py
>  9b0bbfc836 
>   
> ambari-server/src/main/resources/common-services/OOZIE/4.0.0.2.0/package/scripts/oozie.py
>  aa5bc308fe 
>   
> ambari-server/src/main/resources/common-services/ZOOKEEPER/3.4.5/package/scripts/params_linux.py
>  b8e8f78df1 
>   
> ambari-server/src/main/resources/stacks/BigInsights/4.0/services/HIVE/package/scripts/hive_metastore.py
>  88bb2e09e8 
>   
> ambari-server/src/main/resources/stacks/BigInsights/4.0/services/HIVE/package/scripts/hive_server_upgrade.py
>  318fcca23d 
>   
> amb

Re: Review Request 61490: Upgrade Pre-Checks Should Take PATCH/SERVICE Types Into Account

2017-08-07 Thread Dmytro Grinenko


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61490/#review182361
---


Fix it, then Ship it!




Ship It!


ambari-server/src/main/java/org/apache/ambari/server/checks/AbstractCheckDescriptor.java
Line 249 (original), 255 (patched)
<https://reviews.apache.org/r/61490/#comment258234>

could we put here some usefull information about request and kind of error 
occurs?


- Dmytro Grinenko


On Aug. 8, 2017, 3:48 a.m., Jonathan Hurley wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61490/
> ---
> 
> (Updated Aug. 8, 2017, 3:48 a.m.)
> 
> 
> Review request for Ambari, Dmytro Grinenko, Dmitro Lisnichenko, and Nate Cole.
> 
> 
> Bugs: AMBARI-21677
> https://issues.apache.org/jira/browse/AMBARI-21677
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> With {{PATCH}} and {{SERVICE}} style upgrades, many of the upgrade pre-checks 
> would flag issues related to services or hosts which may not be a part of the 
> upgrade. If the upgrade is not a full-stack upgrade, then warnings and errors 
> related to services which are not part of the upgrade should be ignored.
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/java/org/apache/ambari/server/checks/AbstractCheckDescriptor.java
>  4f8a39d8a5 
>   
> ambari-server/src/main/java/org/apache/ambari/server/checks/AtlasPresenceCheck.java
>  04b73fa160 
>   
> ambari-server/src/main/java/org/apache/ambari/server/checks/AutoStartDisabledCheck.java
>  c41ad207a5 
>   
> ambari-server/src/main/java/org/apache/ambari/server/checks/ClientRetryPropertyCheck.java
>  226d82c049 
>   
> ambari-server/src/main/java/org/apache/ambari/server/checks/ComponentsExistInRepoCheck.java
>  d60433d141 
>   
> ambari-server/src/main/java/org/apache/ambari/server/checks/ComponentsInstallationCheck.java
>  a77d72b520 
>   
> ambari-server/src/main/java/org/apache/ambari/server/checks/ConfigurationMergeCheck.java
>  28d7d7884a 
>   
> ambari-server/src/main/java/org/apache/ambari/server/checks/HealthCheck.java 
> 8feb77a8dd 
>   
> ambari-server/src/main/java/org/apache/ambari/server/checks/HiveDynamicServiceDiscoveryCheck.java
>  c2ef4ad337 
>   
> ambari-server/src/main/java/org/apache/ambari/server/checks/HiveMultipleMetastoreCheck.java
>  ea20a5535b 
>   
> ambari-server/src/main/java/org/apache/ambari/server/checks/HiveNotRollingWarning.java
>  2b1c62ede2 
>   
> ambari-server/src/main/java/org/apache/ambari/server/checks/HostsMasterMaintenanceCheck.java
>  8cd935bf1f 
>   
> ambari-server/src/main/java/org/apache/ambari/server/checks/HostsRepositoryVersionCheck.java
>  613c5fc431 
>   
> ambari-server/src/main/java/org/apache/ambari/server/checks/InstallPackagesCheck.java
>  7c84e5c630 
>   
> ambari-server/src/main/java/org/apache/ambari/server/checks/KafkaKerberosCheck.java
>  69721d9e3c 
>   
> ambari-server/src/main/java/org/apache/ambari/server/checks/MapReduce2JobHistoryStatePreservingCheck.java
>  06ca1629ce 
>   
> ambari-server/src/main/java/org/apache/ambari/server/checks/RangerAuditDbCheck.java
>  ec4ed09da4 
>   
> ambari-server/src/main/java/org/apache/ambari/server/checks/RangerPasswordCheck.java
>  a55a1481d6 
>   
> ambari-server/src/main/java/org/apache/ambari/server/checks/RangerSSLConfigCheck.java
>  02f6559fe4 
>   
> ambari-server/src/main/java/org/apache/ambari/server/checks/SecondaryNamenodeDeletedCheck.java
>  0dbb1b5da1 
>   
> ambari-server/src/main/java/org/apache/ambari/server/checks/ServicesMaintenanceModeCheck.java
>  08f4fd8eed 
>   
> ambari-server/src/main/java/org/apache/ambari/server/checks/ServicesMapReduceDistributedCacheCheck.java
>  3970e9e698 
>   
> ambari-server/src/main/java/org/apache/ambari/server/checks/ServicesNamenodeHighAvailabilityCheck.java
>  38a6702b2e 
>   
> ambari-server/src/main/java/org/apache/ambari/server/checks/ServicesNamenodeTruncateCheck.java
>  35be754bdf 
>   
> ambari-server/src/main/java/org/apache/ambari/server/checks/ServicesTezDistributedCacheCheck.java
>  5dadcddc20 
>   
> ambari-server/src/main/java/org/apache/ambari/server/checks/ServicesUpCheck.java
>  d838f6a20e 
>   
> ambari-server/src/main/java/org/apache/ambari/server/checks/ServicesYarnWorkPreservingCheck.java
>  77605c1835 
>   
> ambari-server/src/main/java/org/apache/ambari/server/checks/StormShutdownWarning.java
>  b5435f1830 
>   
> ambari-server/src/mai

Re: Review Request 61186: Service repos are not updated with "latest" url in repoinfo.xml

2017-07-27 Thread Dmytro Grinenko


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61186/#review181581
---


Ship it!




Ship It!

- Dmytro Grinenko


On July 27, 2017, 5:05 p.m., Eugene Chekanskiy wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61186/
> ---
> 
> (Updated July 27, 2017, 5:05 p.m.)
> 
> 
> Review request for Ambari, Dmitro Lisnichenko, Robert Levas, and Vitalyi 
> Brodetskyi.
> 
> 
> Bugs: AMBARI-21589
> https://issues.apache.org/jira/browse/AMBARI-21589
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Service level repoinfo.xml not updated automatically.
> 
> 
> Diffs
> -
> 
>   ambari-server/src/main/java/org/apache/ambari/server/stack/StackModule.java 
> dd23e76 
> 
> 
> Diff: https://reviews.apache.org/r/61186/diff/1/
> 
> 
> Testing
> ---
> 
> mvn clean test
> 
> 
> Thanks,
> 
> Eugene Chekanskiy
> 
>

Re: Review Request 61146: Replace Hard Coded stack-select Structures

2017-07-27 Thread Dmytro Grinenko


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61146/#review181559
---


Ship it!




Ship It!

- Dmytro Grinenko


On July 26, 2017, 7:04 p.m., Jonathan Hurley wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61146/
> ---
> 
> (Updated July 26, 2017, 7:04 p.m.)
> 
> 
> Review request for Ambari, Alejandro Fernandez, Dmytro Grinenko, Dmitro 
> Lisnichenko, and Nate Cole.
> 
> 
> Bugs: AMBARI-21580
> https://issues.apache.org/jira/browse/AMBARI-21580
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Currently, both the stack-select and conf-select python files have hard-coded 
> structures inside of them which contain mappings for things like 
> configuration directories and stack-select component names. 
> 
> This is mainly used for pairing the Ambari role name (defined from 
> metainfo.xml) with the {{stack-select}} package name. There are places, like 
> {{params.py}} where we have not yet entered the target python file (like 
> {{AccumuloScript}}) where we could get this information. Some components need 
> this to be able to build things like the Hadoop conf dir:
> {code:title=stack_select.py}
> SERVER_ROLE_DIRECTORY_MAP = {
>   'ACCUMULO_MASTER' : 'accumulo-master',
>   'ACCUMULO_MONITOR' : 'accumulo-monitor',
>   'ACCUMULO_GC' : 'accumulo-gc',
>   'ACCUMULO_TRACER' : 'accumulo-tracer',
>   'ACCUMULO_TSERVER' : 'accumulo-tablet',
>   'ATLAS_SERVER' : 'atlas-server',
>   'FLUME_HANDLER' : 'flume-server',
>   'FALCON_SERVER' : 'falcon-server',
> ...
> {code}
> 
> With the coming of management packs replacing stacks, we can no longer hard 
> code this in Python. My suggestion is to begin moving this data into a 
> property of some sort. Today, {{cluster-env}} exists and would be the only 
> place for it (where the stack-feature tools are now). However, I believe that 
> {{cluster-env}} is also going away and being replaced with something similar 
> to "Cluster Settings". 
> 
> In any event, this Jira is to track the work needed to replace this logic.
> 
> 
> Diffs
> -
> 
>   
> ambari-common/src/main/python/resource_management/libraries/functions/stack_select.py
>  265e7df9a0 
>   
> ambari-common/src/main/python/resource_management/libraries/script/script.py 
> cce3ac461f 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/BlueprintConfigurationProcessor.java
>  50cea9e00b 
>   
> ambari-server/src/main/java/org/apache/ambari/server/state/ConfigHelper.java 
> 974ad4f29e 
>   
> ambari-server/src/main/java/org/apache/ambari/server/upgrade/FinalUpgradeCatalog.java
>  dad0ecf8b0 
>   
> ambari-server/src/main/resources/common-services/ACCUMULO/1.6.1.2.2.0/package/scripts/accumulo_client.py
>  67ca525f15 
>   
> ambari-server/src/main/resources/common-services/ACCUMULO/1.6.1.2.2.0/package/scripts/accumulo_script.py
>  ebd418d855 
>   
> ambari-server/src/main/resources/common-services/ATLAS/0.1.0.2.3/package/scripts/atlas_client.py
>  d01ff84842 
>   
> ambari-server/src/main/resources/common-services/ATLAS/0.1.0.2.3/package/scripts/metadata_server.py
>  38f9a4118d 
>   
> ambari-server/src/main/resources/common-services/DRUID/0.9.2/package/scripts/druid_node.py
>  7c6bf39d5f 
>   
> ambari-server/src/main/resources/common-services/DRUID/0.9.2/package/scripts/superset.py
>  b837b2492c 
>   
> ambari-server/src/main/resources/common-services/FALCON/0.5.0.2.1/package/scripts/falcon_client.py
>  365f661a1d 
>   
> ambari-server/src/main/resources/common-services/FALCON/0.5.0.2.1/package/scripts/falcon_server.py
>  5b2db4477b 
>   
> ambari-server/src/main/resources/common-services/FLUME/1.4.0.2.0/package/scripts/flume_handler.py
>  107ce6d8ae 
>   
> ambari-server/src/main/resources/common-services/HBASE/0.96.0.2.0/package/scripts/hbase_client.py
>  f18a96af8d 
>   
> ambari-server/src/main/resources/common-services/HBASE/0.96.0.2.0/package/scripts/hbase_master.py
>  815157220b 
>   
> ambari-server/src/main/resources/common-services/HBASE/0.96.0.2.0/package/scripts/hbase_regionserver.py
>  9194991aa9 
>   
> ambari-server/src/main/resources/common-services/HBASE/0.96.0.2.0/package/scripts/phoenix_queryserver.py
>  b1bdb785a9 
>   
>

Re: Review Request 61146: Replace Hard Coded stack-select Structures

2017-07-26 Thread Dmytro Grinenko



> On July 26, 2017, 7:39 p.m., Dmytro Grinenko wrote:
> > ambari-common/src/main/python/resource_management/libraries/functions/stack_select.py
> > Lines 131 (patched)
> > <https://reviews.apache.org/r/61146/diff/1/?file=1782632#file1782632line175>
> >
> > it is not overkill to post big json each time with request or with 
> > cluster-env posted? i believe that this structure is not something subject 
> > for often updates and proly better to put it to resources and sync on 
> > changes or post to agent initial version and after just post on changes? 
> > 
> > 
> > Current request r not small entites, and proly it is not good idea to 
> > make them even more larger.
> 
> Jonathan Hurley wrote:
> The problem with storing it as a resource is that we'd need to update it 
> for new stacks. That means we'd need to read from resources and then write 
> back to resources dynamically instead of just updating a property. 
> 
> Also, I timed it:
> !!! Decoding the JSON took 1ms
> 
> So, not very long :)

it is not really about decoding, i'm more concerning about amount of transfered 
data with each task and amount of spam on 2-3k node cluster. Roughtly, this 
json eating 23 kb of the space, what means 44 mb of additional data on 2k 
cluster per stage from one task.

 About updating data with new stacks, putting to resource is just what comes 
first to my head.  We can sent data on agent registration for initial session 
and later with harbeats or even real commands for non-restart updates, this 
will allow us to instantly update cache.

 It is ok to keep those data on cluster-env, but prolly we should limit huge 
properties to be posted often and update them only when they really updated.


- Dmytro


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61146/#review181481
---


On July 26, 2017, 7:04 p.m., Jonathan Hurley wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61146/
> -------
> 
> (Updated July 26, 2017, 7:04 p.m.)
> 
> 
> Review request for Ambari, Alejandro Fernandez, Dmytro Grinenko, Dmitro 
> Lisnichenko, and Nate Cole.
> 
> 
> Bugs: AMBARI-21580
> https://issues.apache.org/jira/browse/AMBARI-21580
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Currently, both the stack-select and conf-select python files have hard-coded 
> structures inside of them which contain mappings for things like 
> configuration directories and stack-select component names. 
> 
> This is mainly used for pairing the Ambari role name (defined from 
> metainfo.xml) with the {{stack-select}} package name. There are places, like 
> {{params.py}} where we have not yet entered the target python file (like 
> {{AccumuloScript}}) where we could get this information. Some components need 
> this to be able to build things like the Hadoop conf dir:
> {code:title=stack_select.py}
> SERVER_ROLE_DIRECTORY_MAP = {
>   'ACCUMULO_MASTER' : 'accumulo-master',
>   'ACCUMULO_MONITOR' : 'accumulo-monitor',
>   'ACCUMULO_GC' : 'accumulo-gc',
>   'ACCUMULO_TRACER' : 'accumulo-tracer',
>   'ACCUMULO_TSERVER' : 'accumulo-tablet',
>   'ATLAS_SERVER' : 'atlas-server',
>   'FLUME_HANDLER' : 'flume-server',
>   'FALCON_SERVER' : 'falcon-server',
> ...
> {code}
> 
> With the coming of management packs replacing stacks, we can no longer hard 
> code this in Python. My suggestion is to begin moving this data into a 
> property of some sort. Today, {{cluster-env}} exists and would be the only 
> place for it (where the stack-feature tools are now). However, I believe that 
> {{cluster-env}} is also going away and being replaced with something similar 
> to "Cluster Settings". 
> 
> In any event, this Jira is to track the work needed to replace this logic.
> 
> 
> Diffs
> -
> 
>   
> ambari-common/src/main/python/resource_management/libraries/functions/stack_select.py
>  265e7df9a0 
>   
> ambari-common/src/main/python/resource_management/libraries/script/script.py 
> cce3ac461f 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/BlueprintConfigurationProcessor.java
>  50cea9e00b 
>   
> ambari-server/src/main/java/org/apache/ambari/server/state/ConfigHel

Re: Review Request 61146: Replace Hard Coded stack-select Structures

2017-07-26 Thread Dmytro Grinenko


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61146/#review181481
---




ambari-common/src/main/python/resource_management/libraries/functions/stack_select.py
Lines 131 (patched)
<https://reviews.apache.org/r/61146/#comment257092>

it is not overkill to post big json each time with request or with 
cluster-env posted? i believe that this structure is not something subject for 
often updates and proly better to put it to resources and sync on changes or 
post to agent initial version and after just post on changes? 

Current request r not small entites, and proly it is not good idea to make 
them even more larger.


- Dmytro Grinenko


On July 26, 2017, 7:04 p.m., Jonathan Hurley wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61146/
> ---
> 
> (Updated July 26, 2017, 7:04 p.m.)
> 
> 
> Review request for Ambari, Alejandro Fernandez, Dmytro Grinenko, Dmitro 
> Lisnichenko, and Nate Cole.
> 
> 
> Bugs: AMBARI-21580
> https://issues.apache.org/jira/browse/AMBARI-21580
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Currently, both the stack-select and conf-select python files have hard-coded 
> structures inside of them which contain mappings for things like 
> configuration directories and stack-select component names. 
> 
> This is mainly used for pairing the Ambari role name (defined from 
> metainfo.xml) with the {{stack-select}} package name. There are places, like 
> {{params.py}} where we have not yet entered the target python file (like 
> {{AccumuloScript}}) where we could get this information. Some components need 
> this to be able to build things like the Hadoop conf dir:
> {code:title=stack_select.py}
> SERVER_ROLE_DIRECTORY_MAP = {
>   'ACCUMULO_MASTER' : 'accumulo-master',
>   'ACCUMULO_MONITOR' : 'accumulo-monitor',
>   'ACCUMULO_GC' : 'accumulo-gc',
>   'ACCUMULO_TRACER' : 'accumulo-tracer',
>   'ACCUMULO_TSERVER' : 'accumulo-tablet',
>   'ATLAS_SERVER' : 'atlas-server',
>   'FLUME_HANDLER' : 'flume-server',
>   'FALCON_SERVER' : 'falcon-server',
> ...
> {code}
> 
> With the coming of management packs replacing stacks, we can no longer hard 
> code this in Python. My suggestion is to begin moving this data into a 
> property of some sort. Today, {{cluster-env}} exists and would be the only 
> place for it (where the stack-feature tools are now). However, I believe that 
> {{cluster-env}} is also going away and being replaced with something similar 
> to "Cluster Settings". 
> 
> In any event, this Jira is to track the work needed to replace this logic.
> 
> 
> Diffs
> -
> 
>   
> ambari-common/src/main/python/resource_management/libraries/functions/stack_select.py
>  265e7df9a0 
>   
> ambari-common/src/main/python/resource_management/libraries/script/script.py 
> cce3ac461f 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/BlueprintConfigurationProcessor.java
>  50cea9e00b 
>   
> ambari-server/src/main/java/org/apache/ambari/server/state/ConfigHelper.java 
> 974ad4f29e 
>   
> ambari-server/src/main/java/org/apache/ambari/server/upgrade/FinalUpgradeCatalog.java
>  dad0ecf8b0 
>   
> ambari-server/src/main/resources/common-services/ACCUMULO/1.6.1.2.2.0/package/scripts/accumulo_client.py
>  67ca525f15 
>   
> ambari-server/src/main/resources/common-services/ACCUMULO/1.6.1.2.2.0/package/scripts/accumulo_script.py
>  ebd418d855 
>   
> ambari-server/src/main/resources/common-services/ATLAS/0.1.0.2.3/package/scripts/atlas_client.py
>  d01ff84842 
>   
> ambari-server/src/main/resources/common-services/ATLAS/0.1.0.2.3/package/scripts/metadata_server.py
>  38f9a4118d 
>   
> ambari-server/src/main/resources/common-services/DRUID/0.9.2/package/scripts/druid_node.py
>  7c6bf39d5f 
>   
> ambari-server/src/main/resources/common-services/DRUID/0.9.2/package/scripts/superset.py
>  b837b2492c 
>   
> ambari-server/src/main/resources/common-services/FALCON/0.5.0.2.1/package/scripts/falcon_client.py
>  365f661a1d 
>   
> ambari-server/src/main/resources/common-services/FALCON/0.5.0.2.1/package/scripts/falcon_server.py
>  5b2db4477b 
>   
> ambari-server/src/main/resources/common-services/FLUME/1.4.0.2.0/package/scripts/flume_handler

Re: Review Request 60919: AMBARI-21502. Cross-stack migration from BigInsights to HDP, EU needs to set hive-site custom.hive.warehouse.mode to 0770

2017-07-17 Thread Dmytro Grinenko


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/60919/#review180748
---


Ship it!




Ship It!

- Dmytro Grinenko


On July 17, 2017, 9:31 p.m., Alejandro Fernandez wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/60919/
> ---
> 
> (Updated July 17, 2017, 9:31 p.m.)
> 
> 
> Review request for Ambari, Andrii Tkach, Dmytro Grinenko, Jonathan Hurley, 
> Sumit Mohanty, and Sid Wagle.
> 
> 
> Bugs: AMBARI-21502
> https://issues.apache.org/jira/browse/AMBARI-21502
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> During cross stack upgrade from BigInsights, if user has not set 
> custom.hive.warehouse.mode in hive-site, as the default value is changed in 
> HDFS from BigInsights default (0770) to HDP default (0777).
> EU should set the value of custom.hive.warehouse.mode to 0770
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/resources/stacks/BigInsights/4.2.5/upgrades/config-upgrade.xml
>  f1a954e 
>   
> ambari-server/src/main/resources/stacks/BigInsights/4.2.5/upgrades/nonrolling-upgrade-to-hdp-2.6.xml
>  5f1e06c 
>   
> ambari-server/src/main/resources/stacks/BigInsights/4.2/upgrades/config-upgrade.xml
>  310e504 
>   
> ambari-server/src/main/resources/stacks/BigInsights/4.2/upgrades/nonrolling-upgrade-to-hdp-2.6.xml
>  5b8f8d9 
> 
> 
> Diff: https://reviews.apache.org/r/60919/diff/2/
> 
> 
> Testing
> ---
> 
> Python unit tests passed.
> 
> --
> Total run:1161
> Total errors:0
> Total failures:0
> OK
> 
> 
> Thanks,
> 
> Alejandro Fernandez
> 
>

Re: Review Request 60897: Add UID/GID related enhancements

2017-07-16 Thread Dmytro Grinenko


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/60897/#review180641
---


Ship it!




Ship It!

- Dmytro Grinenko


On July 16, 2017, 5:07 p.m., Eugene Chekanskiy wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/60897/
> ---
> 
> (Updated July 16, 2017, 5:07 p.m.)
> 
> 
> Review request for Ambari, Aleksandr Kovalenko, Andrii Tkach, Dmitro 
> Lisnichenko, Robert Levas, and Vitalyi Brodetskyi.
> 
> 
> Bugs: AMBARI-21483
> https://issues.apache.org/jira/browse/AMBARI-21483
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Ability to change UID for ambari users.
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/java/org/apache/ambari/server/state/PropertyInfo.java 
> 62396e3 
>   
> ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/before-ANY/files/changeToSecureUid.sh
>  08542c4 
>   
> ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/before-ANY/scripts/shared_initialization.py
>  39f5a47 
>   
> ambari-server/src/test/python/stacks/2.0.6/hooks/before-ANY/test_before_any.py
>  75c6543 
>   ambari-web/app/controllers/wizard/step7_controller.js 9a897d0 
>   ambari-web/app/mappers/configs/stack_config_properties_mapper.js 9b4b920 
>   ambari-web/app/styles/application.less 29788bc 
>   
> ambari-web/app/templates/wizard/controls_service_config_usergroup_with_id.hbs 
> PRE-CREATION 
>   ambari-web/app/utils/config.js 00cc2a3 
>   ambari-web/app/views/common/configs/service_configs_by_category_view.js 
> 4058020 
>   ambari-web/app/views/common/controls_view.js edeaf0a 
> 
> 
> Diff: https://reviews.apache.org/r/60897/diff/1/
> 
> 
> Testing
> ---
> 
> mvn clean test, manual deploy
> 
> 
> Thanks,
> 
> Eugene Chekanskiy
> 
>

Re: Review Request 60866: AMBARI-21477: Remove Falcon proxy entries from Knox kerberos.json

2017-07-14 Thread Dmytro Grinenko


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/60866/#review180558
---


Ship it!




Ship It!

- Dmytro Grinenko


On July 14, 2017, 1:34 p.m., Di Li wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/60866/
> ---
> 
> (Updated July 14, 2017, 1:34 p.m.)
> 
> 
> Review request for Ambari, Dmytro Grinenko, Di Li, Dmitro Lisnichenko, 
> Jonathan Hurley, Sumit Mohanty, Sid Wagle, and Tim Thorpe.
> 
> 
> Bugs: AMBARI-21477
> https://issues.apache.org/jira/browse/AMBARI-21477
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Bi 4.2 stack Knox kerberos.json has falcon proxy entries that are not needed 
> anymore. remove them.
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/resources/stacks/BigInsights/4.2/services/KNOX/kerberos.json
>  8ee2acc 
> 
> 
> Diff: https://reviews.apache.org/r/60866/diff/1/
> 
> 
> Testing
> ---
> 
> to be tested in integration tests.
> 
> 
> Thanks,
> 
> Di Li
> 
>

Re: Review Request 60543: DB consistency checker throws errors for missing 'parquet-logging' and 'product-info' configs after Ambari upgrade

2017-07-05 Thread Dmytro Grinenko


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/60543/#review179631
---


Ship it!




Ship It!

- Dmytro Grinenko


On June 29, 2017, 4:28 p.m., Dmitro Lisnichenko wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/60543/
> ---
> 
> (Updated June 29, 2017, 4:28 p.m.)
> 
> 
> Review request for Ambari, Dmytro Grinenko and Jonathan Hurley.
> 
> 
> Bugs: AMBARI-21364
> https://issues.apache.org/jira/browse/AMBARI-21364
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> *STR*
> # Deployed cluster with Ambari version: 2.5.1.0-159 and HDP version: 
> 2.6.1.0-129
> # Upgrade Ambari to 2.5.2.0-74 (hash: 
> fd30644590991deb41241454d6e9091ed7a38e92)
> # Run "ambari-server start"
> 
> {code}
> root@ctr-e133-1493418528701-156570-01-05:/hwqe/hadoopqe# ambari-server 
> restart
> Using python  /usr/bin/python
> Restarting ambari-server
> Waiting for server stop...
> Ambari Server stopped
> Ambari Server running with administrator privileges.
> Organizing resource files at /var/lib/ambari-server/resources...
> Ambari database consistency check started...
> Server PID at: /var/run/ambari-server/ambari-server.pid
> Server out at: /var/log/ambari-server/ambari-server.out
> Server log at: /var/log/ambari-server/ambari-server.log
> Waiting for server start..
> DB configs consistency check failed. Run "ambari-server start 
> --skip-database-check" to skip. You may try --auto-fix-database flag to 
> attempt to fix issues automatically. If you use this "--skip-database-check" 
> option, do not make any changes to your cluster topology or perform a cluster 
> upgrade until you correct the database consistency issues. See 
> /var/log/ambari-server/ambari-server-check-database.log for more details on 
> the consistency issues.
> ERROR: Exiting with exit code -1.
> REASON: Ambari Server java process has stopped. Please check the logs for 
> more information.
> {code}
> 
> DB log: ambari-server-check-database.log
> {code}
> 2017-06-27 13:51:38,743  INFO - Executing query 'GET_SERVICES_WITH_CONFIGS'
> 2017-06-27 13:51:38,748  INFO - Comparing service configs from stack with 
> configs that we got from db
> 2017-06-27 13:51:38,748  INFO - Getting services from metainfo
> 2017-06-27 13:51:38,748  INFO - Processing HDP-2.6 / KAFKA
> 2017-06-27 13:51:38,748  INFO - Processing HDP-2.6 / PIG
> 2017-06-27 13:51:38,748  INFO - Processing HDP-2.6 / ZEPPELIN
> 2017-06-27 13:51:38,748  INFO - Processing HDP-2.6 / LOGSEARCH
> 2017-06-27 13:51:38,748  INFO - Processing HDP-2.6 / MAPREDUCE2
> 2017-06-27 13:51:38,748  INFO - Processing HDP-2.6 / SLIDER
> 2017-06-27 13:51:38,748  INFO - Processing HDP-2.6 / HIVE
> 2017-06-27 13:51:38,748  INFO - Processing HDP-2.6 / TEZ
> 2017-06-27 13:51:38,748  INFO - Processing HDP-2.6 / HBASE
> 2017-06-27 13:51:38,748  INFO - Processing HDP-2.6 / OOZIE
> 2017-06-27 13:51:38,748  INFO - Processing HDP-2.6 / FLUME
> 2017-06-27 13:51:38,748  INFO - Processing HDP-2.6 / MAHOUT
> 2017-06-27 13:51:38,748  INFO - Processing HDP-2.6 / HDFS
> 2017-06-27 13:51:38,748  INFO - Processing HDP-2.6 / DRUID
> 2017-06-27 13:51:38,748  INFO - Processing HDP-2.6 / AMBARI_METRICS
> 2017-06-27 13:51:38,749  INFO - Processing HDP-2.6 / SPARK
> 2017-06-27 13:51:38,749  INFO - Processing HDP-2.6 / SMARTSENSE
> 2017-06-27 13:51:38,749  INFO - Processing HDP-2.6 / AMBARI_INFRA
> 2017-06-27 13:51:38,749  INFO - Processing HDP-2.6 / YARN
> 2017-06-27 13:51:38,749  INFO - Processing HDP-2.6 / FALCON
> 2017-06-27 13:51:38,749  INFO - Processing HDP-2.6 / SPARK2
> 2017-06-27 13:51:38,749  INFO - Processing HDP-2.6 / ZOOKEEPER
> 2017-06-27 13:51:38,749  INFO - Processing HDP-2.6 / ATLAS
> 2017-06-27 13:51:38,749  INFO - Processing HDP-2.6 / SQOOP
> 2017-06-27 13:51:38,749  INFO - Processing HDP-2.6 / STORM
> 2017-06-27 13:51:38,749  INFO - Processing HDP-2.6 / KNOX
> 2017-06-27 13:51:38,749  INFO - Processing HDP-2.6 / KERBEROS
> 2017-06-27 13:51:38,749  INFO - Comparing required service configs from stack 
> with mapped service configs from db
> 2017-06-27 13:51:38,751 ERROR - Required config(s): product-info is(are) not 
> available for service SMARTSENSE with service config version 2 in cluster cl1
> 2017-06-27 13:51:38,751 ERROR - Required config(s): parquet-logging is(are) 
> not available for service HIVE with service config version 7 in cluster cl1
> 2017-06-27

Re: Review Request 60543: DB consistency checker throws errors for missing 'parquet-logging' and 'product-info' configs after Ambari upgrade

2017-07-05 Thread Dmytro Grinenko



> On June 29, 2017, 5:13 p.m., Alejandro Fernandez wrote:
> > ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/metainfo.xml
> > Lines 232 (patched)
> > <https://reviews.apache.org/r/60543/diff/1/?file=1766953#file1766953line232>
> >
> > Same change is needed in trunk for Hive in 3.0
> 
> Jonathan Hurley wrote:
> Ah - right - good catch... I keep forgetting we have essentially 2 
> versions of every service in trunk now.

nach, HIVE 3.0 linking to 2.1.0.3.0 which even not defining this configuration 
type


- Dmytro


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/60543/#review179277
---


On June 29, 2017, 4:28 p.m., Dmitro Lisnichenko wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/60543/
> ---
> 
> (Updated June 29, 2017, 4:28 p.m.)
> 
> 
> Review request for Ambari, Dmytro Grinenko and Jonathan Hurley.
> 
> 
> Bugs: AMBARI-21364
> https://issues.apache.org/jira/browse/AMBARI-21364
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> *STR*
> # Deployed cluster with Ambari version: 2.5.1.0-159 and HDP version: 
> 2.6.1.0-129
> # Upgrade Ambari to 2.5.2.0-74 (hash: 
> fd30644590991deb41241454d6e9091ed7a38e92)
> # Run "ambari-server start"
> 
> {code}
> root@ctr-e133-1493418528701-156570-01-05:/hwqe/hadoopqe# ambari-server 
> restart
> Using python  /usr/bin/python
> Restarting ambari-server
> Waiting for server stop...
> Ambari Server stopped
> Ambari Server running with administrator privileges.
> Organizing resource files at /var/lib/ambari-server/resources...
> Ambari database consistency check started...
> Server PID at: /var/run/ambari-server/ambari-server.pid
> Server out at: /var/log/ambari-server/ambari-server.out
> Server log at: /var/log/ambari-server/ambari-server.log
> Waiting for server start..
> DB configs consistency check failed. Run "ambari-server start 
> --skip-database-check" to skip. You may try --auto-fix-database flag to 
> attempt to fix issues automatically. If you use this "--skip-database-check" 
> option, do not make any changes to your cluster topology or perform a cluster 
> upgrade until you correct the database consistency issues. See 
> /var/log/ambari-server/ambari-server-check-database.log for more details on 
> the consistency issues.
> ERROR: Exiting with exit code -1.
> REASON: Ambari Server java process has stopped. Please check the logs for 
> more information.
> {code}
> 
> DB log: ambari-server-check-database.log
> {code}
> 2017-06-27 13:51:38,743  INFO - Executing query 'GET_SERVICES_WITH_CONFIGS'
> 2017-06-27 13:51:38,748  INFO - Comparing service configs from stack with 
> configs that we got from db
> 2017-06-27 13:51:38,748  INFO - Getting services from metainfo
> 2017-06-27 13:51:38,748  INFO - Processing HDP-2.6 / KAFKA
> 2017-06-27 13:51:38,748  INFO - Processing HDP-2.6 / PIG
> 2017-06-27 13:51:38,748  INFO - Processing HDP-2.6 / ZEPPELIN
> 2017-06-27 13:51:38,748  INFO - Processing HDP-2.6 / LOGSEARCH
> 2017-06-27 13:51:38,748  INFO - Processing HDP-2.6 / MAPREDUCE2
> 2017-06-27 13:51:38,748  INFO - Processing HDP-2.6 / SLIDER
> 2017-06-27 13:51:38,748  INFO - Processing HDP-2.6 / HIVE
> 2017-06-27 13:51:38,748  INFO - Processing HDP-2.6 / TEZ
> 2017-06-27 13:51:38,748  INFO - Processing HDP-2.6 / HBASE
> 2017-06-27 13:51:38,748  INFO - Processing HDP-2.6 / OOZIE
> 2017-06-27 13:51:38,748  INFO - Processing HDP-2.6 / FLUME
> 2017-06-27 13:51:38,748  INFO - Processing HDP-2.6 / MAHOUT
> 2017-06-27 13:51:38,748  INFO - Processing HDP-2.6 / HDFS
> 2017-06-27 13:51:38,748  INFO - Processing HDP-2.6 / DRUID
> 2017-06-27 13:51:38,748  INFO - Processing HDP-2.6 / AMBARI_METRICS
> 2017-06-27 13:51:38,749  INFO - Processing HDP-2.6 / SPARK
> 2017-06-27 13:51:38,749  INFO - Processing HDP-2.6 / SMARTSENSE
> 2017-06-27 13:51:38,749  INFO - Processing HDP-2.6 / AMBARI_INFRA
> 2017-06-27 13:51:38,749  INFO - Processing HDP-2.6 / YARN
> 2017-06-27 13:51:38,749  INFO - Processing HDP-2.6 / FALCON
> 2017-06-27 13:51:38,749  INFO - Processing HDP-2.6 / SPARK2
> 2017-06-27 13:51:38,749  INFO - Processing HDP-2.6 / ZOOKEEPER
> 2017-06-27 13:51:38,749  INFO - Processing HDP-2.6 / ATLAS
> 2017-06-27 13:51:38,749  INFO - Processing HDP-2.6 / SQOOP
> 2017-06-27 13:51:38,749  INFO - Processing HDP-2.6 / STORM
> 2017-06-27 13:51:38,749  INFO - Proc

Re: Review Request 60543: DB consistency checker throws errors for missing 'parquet-logging' and 'product-info' configs after Ambari upgrade

2017-06-29 Thread Dmytro Grinenko


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/60543/#review179275
---


Ship it!




Ship It!

- Dmytro Grinenko


On June 29, 2017, 4:28 p.m., Dmitro Lisnichenko wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/60543/
> ---
> 
> (Updated June 29, 2017, 4:28 p.m.)
> 
> 
> Review request for Ambari, Dmytro Grinenko and Jonathan Hurley.
> 
> 
> Bugs: AMBARI-21364
> https://issues.apache.org/jira/browse/AMBARI-21364
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> *STR*
> # Deployed cluster with Ambari version: 2.5.1.0-159 and HDP version: 
> 2.6.1.0-129
> # Upgrade Ambari to 2.5.2.0-74 (hash: 
> fd30644590991deb41241454d6e9091ed7a38e92)
> # Run "ambari-server start"
> 
> {code}
> root@ctr-e133-1493418528701-156570-01-05:/hwqe/hadoopqe# ambari-server 
> restart
> Using python  /usr/bin/python
> Restarting ambari-server
> Waiting for server stop...
> Ambari Server stopped
> Ambari Server running with administrator privileges.
> Organizing resource files at /var/lib/ambari-server/resources...
> Ambari database consistency check started...
> Server PID at: /var/run/ambari-server/ambari-server.pid
> Server out at: /var/log/ambari-server/ambari-server.out
> Server log at: /var/log/ambari-server/ambari-server.log
> Waiting for server start..
> DB configs consistency check failed. Run "ambari-server start 
> --skip-database-check" to skip. You may try --auto-fix-database flag to 
> attempt to fix issues automatically. If you use this "--skip-database-check" 
> option, do not make any changes to your cluster topology or perform a cluster 
> upgrade until you correct the database consistency issues. See 
> /var/log/ambari-server/ambari-server-check-database.log for more details on 
> the consistency issues.
> ERROR: Exiting with exit code -1.
> REASON: Ambari Server java process has stopped. Please check the logs for 
> more information.
> {code}
> 
> DB log: ambari-server-check-database.log
> {code}
> 2017-06-27 13:51:38,743  INFO - Executing query 'GET_SERVICES_WITH_CONFIGS'
> 2017-06-27 13:51:38,748  INFO - Comparing service configs from stack with 
> configs that we got from db
> 2017-06-27 13:51:38,748  INFO - Getting services from metainfo
> 2017-06-27 13:51:38,748  INFO - Processing HDP-2.6 / KAFKA
> 2017-06-27 13:51:38,748  INFO - Processing HDP-2.6 / PIG
> 2017-06-27 13:51:38,748  INFO - Processing HDP-2.6 / ZEPPELIN
> 2017-06-27 13:51:38,748  INFO - Processing HDP-2.6 / LOGSEARCH
> 2017-06-27 13:51:38,748  INFO - Processing HDP-2.6 / MAPREDUCE2
> 2017-06-27 13:51:38,748  INFO - Processing HDP-2.6 / SLIDER
> 2017-06-27 13:51:38,748  INFO - Processing HDP-2.6 / HIVE
> 2017-06-27 13:51:38,748  INFO - Processing HDP-2.6 / TEZ
> 2017-06-27 13:51:38,748  INFO - Processing HDP-2.6 / HBASE
> 2017-06-27 13:51:38,748  INFO - Processing HDP-2.6 / OOZIE
> 2017-06-27 13:51:38,748  INFO - Processing HDP-2.6 / FLUME
> 2017-06-27 13:51:38,748  INFO - Processing HDP-2.6 / MAHOUT
> 2017-06-27 13:51:38,748  INFO - Processing HDP-2.6 / HDFS
> 2017-06-27 13:51:38,748  INFO - Processing HDP-2.6 / DRUID
> 2017-06-27 13:51:38,748  INFO - Processing HDP-2.6 / AMBARI_METRICS
> 2017-06-27 13:51:38,749  INFO - Processing HDP-2.6 / SPARK
> 2017-06-27 13:51:38,749  INFO - Processing HDP-2.6 / SMARTSENSE
> 2017-06-27 13:51:38,749  INFO - Processing HDP-2.6 / AMBARI_INFRA
> 2017-06-27 13:51:38,749  INFO - Processing HDP-2.6 / YARN
> 2017-06-27 13:51:38,749  INFO - Processing HDP-2.6 / FALCON
> 2017-06-27 13:51:38,749  INFO - Processing HDP-2.6 / SPARK2
> 2017-06-27 13:51:38,749  INFO - Processing HDP-2.6 / ZOOKEEPER
> 2017-06-27 13:51:38,749  INFO - Processing HDP-2.6 / ATLAS
> 2017-06-27 13:51:38,749  INFO - Processing HDP-2.6 / SQOOP
> 2017-06-27 13:51:38,749  INFO - Processing HDP-2.6 / STORM
> 2017-06-27 13:51:38,749  INFO - Processing HDP-2.6 / KNOX
> 2017-06-27 13:51:38,749  INFO - Processing HDP-2.6 / KERBEROS
> 2017-06-27 13:51:38,749  INFO - Comparing required service configs from stack 
> with mapped service configs from db
> 2017-06-27 13:51:38,751 ERROR - Required config(s): product-info is(are) not 
> available for service SMARTSENSE with service config version 2 in cluster cl1
> 2017-06-27 13:51:38,751 ERROR - Required config(s): parquet-logging is(are) 
> not available for service HIVE with service config version 7 in cluster cl1
> 2017-06-27

Re: Review Request 60498: AMBARI-21362. Ambari upgrade not idempotent due to column move

2017-06-28 Thread Dmytro Grinenko



> On June 28, 2017, 8:09 p.m., Alejandro Fernandez wrote:
> > ambari-server/src/main/java/org/apache/ambari/server/orm/DBAccessorImpl.java
> > Lines 1416 (patched)
> > <https://reviews.apache.org/r/60498/diff/2/?file=1766142#file1766142line1416>
> >
> > This should be !tableHasColumn(targetTableName, targetIDFieldName)

why it should be so?   Both targetID and sourceID need to be exists so realize 
one-to-one connection


- Dmytro


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/60498/#review179160
---


On June 28, 2017, 12:27 p.m., Attila Doroszlai wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/60498/
> ---
> 
> (Updated June 28, 2017, 12:27 p.m.)
> 
> 
> Review request for Ambari, Alejandro Fernandez, Dmytro Grinenko, Dmitro 
> Lisnichenko, and Jonathan Hurley.
> 
> 
> Bugs: AMBARI-21362
> https://issues.apache.org/jira/browse/AMBARI-21362
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Check for source column existence, too.
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/java/org/apache/ambari/server/orm/DBAccessorImpl.java 
> 13e7d7d148795325cd0d1c698eae084f2f00a411 
>   
> ambari-server/src/test/java/org/apache/ambari/server/orm/DBAccessorImplTest.java
>  ca2674c7109d8f2e4b75edbb85dbe5e05c8b1927 
> 
> 
> Diff: https://reviews.apache.org/r/60498/diff/2/
> 
> 
> Testing
> ---
> 
> Manually tested:
>  * real upgrade from 2.4.2 for any regression
>  * simulated retry: upgrade from 2.5.2 where metainfo says it's 2.5.0
> 
> New unit test.
> 
> 
> Thanks,
> 
> Attila Doroszlai
> 
>

Re: Review Request 60498: AMBARI-21362. Ambari upgrade not idempotent due to column move

2017-06-28 Thread Dmytro Grinenko


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/60498/#review179096
---


Ship it!




Ship It!

- Dmytro Grinenko


On June 28, 2017, 12:27 p.m., Attila Doroszlai wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/60498/
> ---
> 
> (Updated June 28, 2017, 12:27 p.m.)
> 
> 
> Review request for Ambari, Alejandro Fernandez, Dmytro Grinenko, Dmitro 
> Lisnichenko, and Jonathan Hurley.
> 
> 
> Bugs: AMBARI-21362
> https://issues.apache.org/jira/browse/AMBARI-21362
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Check for source column existence, too.
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/java/org/apache/ambari/server/orm/DBAccessorImpl.java 
> 13e7d7d148795325cd0d1c698eae084f2f00a411 
>   
> ambari-server/src/test/java/org/apache/ambari/server/orm/DBAccessorImplTest.java
>  ca2674c7109d8f2e4b75edbb85dbe5e05c8b1927 
> 
> 
> Diff: https://reviews.apache.org/r/60498/diff/2/
> 
> 
> Testing
> ---
> 
> Manually tested:
>  * real upgrade from 2.4.2 for any regression
>  * simulated retry: upgrade from 2.5.2 where metainfo says it's 2.5.0
> 
> New unit test.
> 
> 
> Thanks,
> 
> Attila Doroszlai
> 
>

Re: Review Request 60498: AMBARI-21362. Ambari upgrade not idempotent due to column move

2017-06-28 Thread Dmytro Grinenko



> On June 28, 2017, 12:10 p.m., Dmytro Grinenko wrote:
> > ambari-server/src/test/java/org/apache/ambari/server/orm/DBAccessorImplTest.java
> > Lines 683 (patched)
> > <https://reviews.apache.org/r/60498/diff/1/?file=1766136#file1766136line683>
> >
> > If we checking here for idempotent issue, does we need to check the 
> > resulset?
> > 
> > In case if function would be not idempotent, second update call will 
> > cause exception, hence we need to check for this
> 
> Attila Doroszlai wrote:
> Yes, it's not strictly necessary.  Should I remove everything after the 
> `moveColumnToAnotherTable` call then?

yes, it will help to better understand for what this test is needed. thanks


- Dmytro


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/60498/#review179091
---


On June 28, 2017, 8:59 a.m., Attila Doroszlai wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/60498/
> ---
> 
> (Updated June 28, 2017, 8:59 a.m.)
> 
> 
> Review request for Ambari, Alejandro Fernandez, Dmytro Grinenko, Dmitro 
> Lisnichenko, and Jonathan Hurley.
> 
> 
> Bugs: AMBARI-21362
> https://issues.apache.org/jira/browse/AMBARI-21362
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Check for source column existence, too.
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/java/org/apache/ambari/server/orm/DBAccessorImpl.java 
> 13e7d7d148795325cd0d1c698eae084f2f00a411 
>   
> ambari-server/src/test/java/org/apache/ambari/server/orm/DBAccessorImplTest.java
>  ca2674c7109d8f2e4b75edbb85dbe5e05c8b1927 
> 
> 
> Diff: https://reviews.apache.org/r/60498/diff/1/
> 
> 
> Testing
> ---
> 
> Manually tested:
>  * real upgrade from 2.4.2 for any regression
>  * simulated retry: upgrade from 2.5.2 where metainfo says it's 2.5.0
> 
> New unit test.
> 
> 
> Thanks,
> 
> Attila Doroszlai
> 
>

Re: Review Request 60498: AMBARI-21362. Ambari upgrade not idempotent due to column move

2017-06-28 Thread Dmytro Grinenko


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/60498/#review179091
---




ambari-server/src/test/java/org/apache/ambari/server/orm/DBAccessorImplTest.java
Lines 683 (patched)
<https://reviews.apache.org/r/60498/#comment253544>

If we checking here for idempotent issue, does we need to check the 
resulset?

In case if function would be not idempotent, second update call will cause 
exception, hence we need to check for this


- Dmytro Grinenko


On June 28, 2017, 8:59 a.m., Attila Doroszlai wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/60498/
> ---
> 
> (Updated June 28, 2017, 8:59 a.m.)
> 
> 
> Review request for Ambari, Alejandro Fernandez, Dmytro Grinenko, Dmitro 
> Lisnichenko, and Jonathan Hurley.
> 
> 
> Bugs: AMBARI-21362
> https://issues.apache.org/jira/browse/AMBARI-21362
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Check for source column existence, too.
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/java/org/apache/ambari/server/orm/DBAccessorImpl.java 
> 13e7d7d148795325cd0d1c698eae084f2f00a411 
>   
> ambari-server/src/test/java/org/apache/ambari/server/orm/DBAccessorImplTest.java
>  ca2674c7109d8f2e4b75edbb85dbe5e05c8b1927 
> 
> 
> Diff: https://reviews.apache.org/r/60498/diff/1/
> 
> 
> Testing
> ---
> 
> Manually tested:
>  * real upgrade from 2.4.2 for any regression
>  * simulated retry: upgrade from 2.5.2 where metainfo says it's 2.5.0
> 
> New unit test.
> 
> 
> Thanks,
> 
> Attila Doroszlai
> 
>

Re: Review Request 59912: Run status commands with real configurations and parameters information

2017-06-08 Thread Dmytro Grinenko


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59912/#review177293
---


Ship it!




Ship It!

- Dmytro Grinenko


On June 8, 2017, 10:38 a.m., Andrew Onischuk wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/59912/
> ---
> 
> (Updated June 8, 2017, 10:38 a.m.)
> 
> 
> Review request for Ambari and Dmitro Lisnichenko.
> 
> 
> Bugs: AMBARI-21199
> https://issues.apache.org/jira/browse/AMBARI-21199
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> 
> Diffs
> -
> 
>   ambari-agent/src/main/python/ambari_agent/ActionQueue.py dbd9f4c 
>   ambari-agent/src/main/python/ambari_agent/ClusterCache.py 8e91afe 
>   ambari-agent/src/main/python/ambari_agent/ClusterTopologyCache.py 5810e67 
>   ambari-agent/src/main/python/ambari_agent/ComponentStatusExecutor.py 
> 3a2e105 
>   ambari-agent/src/main/python/ambari_agent/CustomServiceOrchestrator.py 
> 656e9a1 
>   ambari-agent/src/main/python/ambari_agent/FileCache.py 139dcba 
>   ambari-agent/src/main/python/ambari_agent/InitializerModule.py 4d0ac9b 
>   ambari-agent/src/main/python/ambari_agent/Utils.py 845eb30 
>   ambari-agent/src/test/python/ambari_agent/TestAgentStompResponses.py 
> c969c75 
>   
> ambari-agent/src/test/python/ambari_agent/dummy_files/stomp/configurations_update.json
>  c415c7d 
>   
> ambari-agent/src/test/python/ambari_agent/dummy_files/stomp/execution_commands.json
>  536233d 
>   
> ambari-agent/src/test/python/ambari_agent/dummy_files/stomp/metadata_after_registration.json
>  0dc5aff 
>   
> ambari-agent/src/test/python/ambari_agent/dummy_files/stomp/topology_add_component.json
>  2c37111 
>   
> ambari-agent/src/test/python/ambari_agent/dummy_files/stomp/topology_add_host.json
>  a9407c3 
>   
> ambari-agent/src/test/python/ambari_agent/dummy_files/stomp/topology_cache_expected.json
>  9894420 
>   
> ambari-agent/src/test/python/ambari_agent/dummy_files/stomp/topology_create.json
>  cf1afa7 
>   
> ambari-server/src/main/java/org/apache/ambari/server/agent/stomp/AgentReportsController.java
>  68b7f3b 
> 
> 
> Diff: https://reviews.apache.org/r/59912/diff/1/
> 
> 
> Testing
> ---
> 
> mvn clean test
> 
> 
> Thanks,
> 
> Andrew Onischuk
> 
>

Re: Review Request 59835: Agent Host Disk Usage Alert Hardcodes the Stack Directory

2017-06-06 Thread Dmytro Grinenko


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59835/#review177020
---


Ship it!




Ship It!

- Dmytro Grinenko


On June 6, 2017, 10:45 a.m., Andrew Onischuk wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/59835/
> ---
> 
> (Updated June 6, 2017, 10:45 a.m.)
> 
> 
> Review request for Ambari and Vitalyi Brodetskyi.
> 
> 
> Bugs: AMBARI-21182
> https://issues.apache.org/jira/browse/AMBARI-21182
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> The Host Disk Usage alert currently hard codes the stack location directly
> into the script:
> 
> 
> 
> 
> # the location where HDP installs components when using HDP 2.2+
> STACK_HOME_DIR = "/usr/hdp"
> # the location where HDP installs components when using HDP 2.0 to 2.1
> STACK_HOME_LEGACY_DIR = "/usr/lib"
> # determine the location of HDP home
>   stack_home = None
>   if os.path.isdir(STACK_HOME_DIR):
> stack_home = STACK_HOME_DIR
>   elif os.path.isdir(STACK_HOME_LEGACY_DIR):
> stack_home = STACK_HOME_LEGACY_DIR
> 
> 
> On clusters where a different stack is installed (such as `/usr/hdf`, the
> above logic incorrectly checks the `STACK_HOME_LEGACY_DIR`.
> 
>   * The 2.0 and 2.1 code paths should be removed since they are not supported 
> anymore.
>   * We should parameterize STACK_HOME_DIR (or even better, use the stack 
> features JSON structure) to determine the home location to check.
> 
> 
> Diffs
> -
> 
>   ambari-server/src/main/resources/host_scripts/alert_disk_space.py d2b4f36 
> 
> 
> Diff: https://reviews.apache.org/r/59835/diff/1/
> 
> 
> Testing
> ---
> 
> mvn clean test
> 
> 
> Thanks,
> 
> Andrew Onischuk
> 
>

Re: Review Request 59394: Race condition: webhdfs call mkdir /tmp/druid-indexing before /tmp making tmp not writable.

2017-06-06 Thread Dmytro Grinenko


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59394/#review177018
---


Ship it!




Ship It!

- Dmytro Grinenko


On May 19, 2017, 9:54 a.m., Andrew Onischuk wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/59394/
> ---
> 
> (Updated May 19, 2017, 9:54 a.m.)
> 
> 
> Review request for Ambari and Vitalyi Brodetskyi.
> 
> 
> Bugs: AMBARI-21070
> https://issues.apache.org/jira/browse/AMBARI-21070
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Race condition: webhdfs call mkdir /tmp/druid-indexing before /tmp making tmp
> not writable.
> 
> @HDP install through ambari , just at the step start components on host< > we
> have some webhdfs operations in background which is creating HDFS directory
> structures required for specific components like (/tmp, /tmp/hive /user/druid
> /tmp/druid-indexing ...)
> 
> generally the expected order is getfileInfo : /tmp --> mkdir: /tmp
> changePermission: /tmp to 777 (hdfs:hdfs) so that /tmp is accessible to all ,
> hence hivemetastore able to create /tmp/hive(hive scratch directory)
> 
> But here in this case specific to druid install , most of the times mkdir of
> /tmp/druid-indexing called before(actual /tmp creation) and thus /tmp is
> having just default directory permission(755).
> 
> ->So next call of getfileInfo : /tmp says already exist it will not further 
> create and change permission
> 
> This made /tmp not accessible to write, So HiveServer process gets shutdown as
> it unable to create/access /tmp/hive.
> 
> hdfs-audit log:
> 
> 
> 
> 
> 2017-05-12 06:39:51,067 INFO FSNamesystem.audit: allowed=true   ugi=hdfs 
> (auth:SIMPLE)  ip=/172.27.26.3 cmd=getfileinfo src=/tmp/druid-indexing 
> dst=nullperm=null   proto=webhdfs
> 2017-05-12 06:39:51,120 INFO FSNamesystem.audit: allowed=true   ugi=hdfs 
> (auth:SIMPLE)  ip=/172.27.22.81cmd=contentSummary  
> src=/user/druid dst=nullperm=null   proto=webhdfs
> 2017-05-12 06:39:51,133 INFO FSNamesystem.audit: allowed=true   ugi=hdfs 
> (auth:SIMPLE)  ip=/172.27.37.200   cmd=setPermission   
> src=/ats/active dst=nullperm=hdfs:hadoop:rwxr-xr-x  proto=webhdfs
> 2017-05-12 06:39:51,155 INFO FSNamesystem.audit: allowed=true   ugi=hdfs 
> (auth:SIMPLE)  ip=/172.27.26.3 cmd=mkdirs  src=/tmp/druid-indexing 
> dst=nullperm=hdfs:hdfs:rwxr-xr-xproto=webhdfs
> 2017-05-12 06:39:51,206 INFO FSNamesystem.audit: allowed=true   ugi=hdfs 
> (auth:SIMPLE)  ip=/172.27.22.81cmd=listStatus  src=/user/druid 
> dst=nullperm=null   proto=webhdfs
> 2017-05-12 06:39:51,235 INFO FSNamesystem.audit: allowed=true   ugi=hdfs 
> (auth:SIMPLE)  ip=/172.27.37.200   cmd=setPermission   src=/ats/  
>  dst=nullperm=yarn:hadoop:rwxr-xr-x  proto=webhdfs
> 2017-05-12 06:39:51,249 INFO FSNamesystem.audit: allowed=true   ugi=hdfs 
> (auth:SIMPLE)  ip=/172.27.26.3 cmd=setPermission   
> src=/tmp/druid-indexing dst=nullperm=hdfs:hdfs:rwxr-xr-x
> proto=webhdfs
> 2017-05-12 06:39:51,290 INFO FSNamesystem.audit: allowed=true   ugi=hdfs 
> (auth:SIMPLE)  ip=/172.27.22.81cmd=listStatus  src=/user/druid/data   
>  dst=nullperm=null   proto=webhdfs
> 2017-05-12 06:39:51,339 INFO FSNamesystem.audit: allowed=true   ugi=hdfs 
> (auth:SIMPLE)  ip=/172.27.37.200   cmd=setPermission   
> src=/ats/active/dst=nullperm=hdfs:hadoop:rwxr-xr-x  
> proto=webhdfs
> 2017-05-12 06:39:51,341 INFO FSNamesystem.audit: allowed=true   ugi=hdfs 
> (auth:SIMPLE)  ip=/172.27.26.3 cmd=setOwnersrc=/tmp/druid-indexing 
> dst=nullperm=druid:hdfs:rwxr-xr-x   proto=webhdfs
> 2017-05-12 06:39:51,380 INFO FSNamesystem.audit: allowed=true   ugi=hdfs 
> (auth:SIMPLE)  ip=/172.27.22.81cmd=setOwnersrc=/user/druid/data   
>  dst=nullperm=druid:hdfs:rwxr-xr-x   proto=webhdfs
> 2017-05-12 06:39:51,431 INFO FSNamesystem.audit: allowed=true   ugi=hdfs 
> (auth:SIMPLE)  ip=/172.27.37.200   cmd=setOwnersrc=/ats/active 
> dst=nullperm=yarn:hadoop:rwxr-xr-x  proto=webhdfs
> 2017-05-12 06:39:51,526 INFO FSNamesystem.audit: allowed=true   ugi=hdfs 
> (auth:SIMPLE)  ip=/172.27.37.200   cmd=setOwnersrc=/ats/   
> dst=nullperm=yarn:hadoop:rwxr-xr-x  proto=webhdfs
>

Re: Review Request 59365: Add ppc as a new OS for User.

2017-06-06 Thread Dmytro Grinenko


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59365/#review177019
---


Ship it!




Ship It!

- Dmytro Grinenko


On May 18, 2017, 11:49 a.m., Andrew Onischuk wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/59365/
> ---
> 
> (Updated May 18, 2017, 11:49 a.m.)
> 
> 
> Review request for Ambari and Vitalyi Brodetskyi.
> 
> 
> Bugs: AMBARI-21054
> https://issues.apache.org/jira/browse/AMBARI-21054
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Add ppc as a new OS for User.
> 
> As centos 6 - there should be a centos6-ppc for ppc users.
> 
> 
> Diffs
> -
> 
>   ambari-common/src/main/python/ambari_commons/os_check.py b430c86 
>   ambari-common/src/main/python/ambari_commons/resources/os_family.json 
> 859ce56 
>   
> ambari-common/src/main/python/resource_management/core/providers/__init__.py 
> 21ae0d5 
>   
> ambari-common/src/main/python/resource_management/libraries/providers/__init__.py
>  bd7c98a 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariManagementControllerImpl.java
>  01f93b4 
>   
> ambari-server/src/main/java/org/apache/ambari/server/state/stack/OsFamily.java
>  1300172 
>   ambari-server/src/main/resources/stacks/HDP/2.6/repos/repoinfo.xml 81a70a5 
> 
> 
> Diff: https://reviews.apache.org/r/59365/diff/1/
> 
> 
> Testing
> ---
> 
> mvn clean test
> 
> 
> Thanks,
> 
> Andrew Onischuk
> 
>

Re: Review Request 59277: Upgrades Should Be Associated With Repositories Instead of String Versions

2017-05-16 Thread Dmytro Grinenko



> On May 16, 2017, 9 a.m., Dmitro Lisnichenko wrote:
> > ambari-server/src/main/java/org/apache/ambari/server/controller/internal/UpgradeResourceProvider.java
> > Lines 576 (patched)
> > <https://reviews.apache.org/r/59277/diff/1/?file=1718429#file1718429line619>
> >
> > :)

lol, but...well,  better to make it via // ToDo: .as this name have good 
chances to leave in the code


- Dmytro


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59277/#review175070
---


On May 15, 2017, 5:35 p.m., Jonathan Hurley wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/59277/
> ---
> 
> (Updated May 15, 2017, 5:35 p.m.)
> 
> 
> Review request for Ambari, Dmytro Grinenko, Dmitro Lisnichenko, and Nate Cole.
> 
> 
> Bugs: AMBARI-21022
> https://issues.apache.org/jira/browse/AMBARI-21022
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> The upgrade entity is modeled currently to store only references to versions 
> For example, to_version is "2.5.0.0-1234" and from_version is "2.3.0.0-1". 
> This presents a problem with support for multiple repositories since two 
> stacks could potentially have the same version, thus preventing a repository 
> from being retrieved simply by its no-longer-unique version.
> 
> Instead, the upgrade should be associated with a {{RepositoryVersionEntity}} 
> and enforced in the database.
> 
> Using repository associations will also help to track the to/from history for 
> components included in the upgrade.
> 
> Some new terminology here:
> - "associated_repository" is the repository you're upgrade to (UPGRADE) or 
> downgrading from (DOWNGRADE). It's always singular
> - "target_repositories" are the repositories you're trying to get to. In an 
> UPGRADE, these are all the same for all components. In a downgrade, they are 
> the original values of those components before the failed upgrade
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/java/org/apache/ambari/server/alerts/ComponentVersionAlertRunnable.java
>  ec5c85ea35 
>   
> ambari-server/src/main/java/org/apache/ambari/server/audit/request/eventcreator/UpgradeEventCreator.java
>  456aa00df4 
>   
> ambari-server/src/main/java/org/apache/ambari/server/checks/PreviousUpgradeCompleted.java
>  ef165a5ba5 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariCustomCommandExecutionHelper.java
>  617d7c0771 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariManagementControllerImpl.java
>  882f583a2d 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/AbstractControllerResourceProvider.java
>  a762e2b156 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/UpgradeResourceProvider.java
>  7ca6164e75 
>   
> ambari-server/src/main/java/org/apache/ambari/server/orm/dao/ServiceComponentDesiredStateDAO.java
>  92f1d09ff0 
>   
> ambari-server/src/main/java/org/apache/ambari/server/orm/entities/ServiceComponentDesiredStateEntity.java
>  7576e00e54 
>   
> ambari-server/src/main/java/org/apache/ambari/server/orm/entities/ServiceComponentHistoryEntity.java
>  15214684e5 
>   
> ambari-server/src/main/java/org/apache/ambari/server/orm/entities/UpgradeEntity.java
>  e5e2de385f 
>   
> ambari-server/src/main/java/org/apache/ambari/server/orm/entities/UpgradeHistoryEntity.java
>  PRE-CREATION 
>   
> ambari-server/src/main/java/org/apache/ambari/server/serveraction/upgrades/AbstractUpgradeServerAction.java
>  de0f282fbb 
>   
> ambari-server/src/main/java/org/apache/ambari/server/serveraction/upgrades/ComponentVersionCheckAction.java
>  4a3bd9b4c7 
>   
> ambari-server/src/main/java/org/apache/ambari/server/serveraction/upgrades/FinalizeUpgradeAction.java
>  1b9fb23100 
>   
> ambari-server/src/main/java/org/apache/ambari/server/serveraction/upgrades/UpdateDesiredStackAction.java
>  4500b5d179 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/MasterHostResolver.java
>  ce105686f5 
>   
> ambari-server/src/main/java/org/apache/ambari/server/state/UpgradeContext.java
>  a68a2e1235 
>   
> ambari-server/src/main/java/org/apache/ambari/server/state/UpgradeContextFactory.java
>  4f15ee2af8 
>   
> ambari-server/src/main/java/org/apache/ambari/server/state/UpgradeHelper.java 
> 3ec9

Re: Review Request 59176: Add missing 'cluster_host_info' column to 'request' table to MSSQL DDL

2017-05-11 Thread Dmytro Grinenko



> On May 11, 2017, 4:38 p.m., Alejandro Fernandez wrote:
> > ambari-server/src/main/resources/Ambari-DDL-SQLServer-CREATE.sql
> > Line 365 (original)
> > <https://reviews.apache.org/r/59176/diff/1/?file=1715296#file1715296line366>
> >
> > Does the Upgrade Catalog for 2.5.1 also have to change?
> > 
> > If any user is on Ambari 2.5 and SQLServer, were they able to even 
> > install Ambari?

1) Nope, as we not support upgrades on SQLServer
2) what you mean ? With this fix they do


- Dmytro


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59176/#review174677
---


On May 11, 2017, 1:17 p.m., Sebastian Toader wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/59176/
> ---
> 
> (Updated May 11, 2017, 1:17 p.m.)
> 
> 
> Review request for Ambari, Dmytro Grinenko and Nate Cole.
> 
> 
> Bugs: AMBARI-20988
> https://issues.apache.org/jira/browse/AMBARI-20988
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> https://issues.apache.org/jira/browse/AMBARI-20919 moved the 
> cluster_host_info column from stage table into request table. The change 
> missed to cover the Ambari database schema DDL for MS SQL server.
> 
> 
> Diffs
> -
> 
>   ambari-server/src/main/resources/Ambari-DDL-SQLServer-CREATE.sql b78e2a4 
> 
> 
> Diff: https://reviews.apache.org/r/59176/diff/1/
> 
> 
> Testing
> ---
> 
> Unit tests:
> Results :
> 
> Tests run: 4065, Failures: 0, Errors: 0, Skipped: 32
> 
> 
> ==
> docker exec -it mssql /opt/mssql-tools/bin/sqlcmd -S localhost -U sa -P 
> "$MSSQL_PASSWORD" -i /tmp/ambari/Ambari-DDL-SQLServer-CREATE.sql
> 
> (54 rows affected)
> 
> (3 rows affected)
> 
> (1 rows affected)
> 
> (3 rows affected)
> 
> (8 rows affected)
> 
> (1 rows affected)
> 
> (7 rows affected)
> 
> (53 rows affected)
> 
> (1 rows affected)
> 
> (14 rows affected)
> 
> (19 rows affected)
> 
> (24 rows affected)
> 
> (31 rows affected)
> 
> (42 rows affected)
> 
> (53 rows affected)
> 
> (1 rows affected)
> 
> (1 rows affected)
> 
> 
> docker exec -it mssql /opt/mssql-tools/bin/sqlcmd -S localhost -U sa -P 
> "$MSSQL_PASSWORD"
> 1> select count(*) from ambari_sequences;
> 2> go
>
> ---
>  54
> 
> (1 rows affected)
> 
> 
> Thanks,
> 
> Sebastian Toader
> 
>

Re: Review Request 59176: Add missing 'cluster_host_info' column to 'request' table to MSSQL DDL

2017-05-11 Thread Dmytro Grinenko


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59176/#review174642
---


Ship it!




Ship It!

- Dmytro Grinenko


On May 11, 2017, 1:17 p.m., Sebastian Toader wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/59176/
> ---
> 
> (Updated May 11, 2017, 1:17 p.m.)
> 
> 
> Review request for Ambari, Dmytro Grinenko and Nate Cole.
> 
> 
> Bugs: AMBARI-20988
> https://issues.apache.org/jira/browse/AMBARI-20988
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> https://issues.apache.org/jira/browse/AMBARI-20919 moved the 
> cluster_host_info column from stage table into request table. The change 
> missed to cover the Ambari database schema DDL for MS SQL server.
> 
> 
> Diffs
> -
> 
>   ambari-server/src/main/resources/Ambari-DDL-SQLServer-CREATE.sql b78e2a4 
> 
> 
> Diff: https://reviews.apache.org/r/59176/diff/1/
> 
> 
> Testing
> ---
> 
> Unit tests:
> Results :
> 
> Tests run: 4065, Failures: 0, Errors: 0, Skipped: 32
> 
> 
> ==
> docker exec -it mssql /opt/mssql-tools/bin/sqlcmd -S localhost -U sa -P 
> "$MSSQL_PASSWORD" -i /tmp/ambari/Ambari-DDL-SQLServer-CREATE.sql
> 
> (54 rows affected)
> 
> (3 rows affected)
> 
> (1 rows affected)
> 
> (3 rows affected)
> 
> (8 rows affected)
> 
> (1 rows affected)
> 
> (7 rows affected)
> 
> (53 rows affected)
> 
> (1 rows affected)
> 
> (14 rows affected)
> 
> (19 rows affected)
> 
> (24 rows affected)
> 
> (31 rows affected)
> 
> (42 rows affected)
> 
> (53 rows affected)
> 
> (1 rows affected)
> 
> (1 rows affected)
> 
> 
> docker exec -it mssql /opt/mssql-tools/bin/sqlcmd -S localhost -U sa -P 
> "$MSSQL_PASSWORD"
> 1> select count(*) from ambari_sequences;
> 2> go
>
> ---
>  54
> 
> (1 rows affected)
> 
> 
> Thanks,
> 
> Sebastian Toader
> 
>

Re: Review Request 57940: JAVA_LIBRARY_PATH in hadoop-env.sh is different on host and downloaded configs

2017-03-26 Thread Dmytro Grinenko


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/57940/#review170123
---


Ship it!




Ship It!

- Dmytro Grinenko


On March 26, 2017, 9:34 p.m., Andrew Onischuk wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/57940/
> ---
> 
> (Updated March 26, 2017, 9:34 p.m.)
> 
> 
> Review request for Ambari and Eugene Chekanskiy.
> 
> 
> Bugs: AMBARI-20575
> https://issues.apache.org/jira/browse/AMBARI-20575
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> * Download client configs from UI (Service page or host details page)
>   * Compare the contents of downloaded configs and the conf file at host
>   * Value of export JAVA_LIBRARY_PATH in hadoop-env.sh is different on host 
> and downloaded config.
> 
> Downloaded config has
> 
> 
> 
> 
>   export 
> JAVA_LIBRARY_PATH=${JAVA_LIBRARY_PATH}:/usr/lib/hadoop/lib/native/Linux-amd64-64
> 
> 
> where as /etc/hadoop/conf/hadoop-env.sh has
> 
> 
> 
> 
>   export 
> JAVA_LIBRARY_PATH=${JAVA_LIBRARY_PATH}:/usr/lib/hadoop/lib/native/Linux--64
> 
> 
> This test has been passing and has failed in todays run. Please see the test
> case history at  
> <http://dashboard.qe.hortonworks.com:5000/#/testHistory?tc_id=3019558>  
> Could you please help check and let us know of this difference  
> Artifacts can be found [here](http://qelog.hortonworks.com/log/nat-yc-d7-mzps-
> ambari-config-5-re-re/test-logs/ambari-config/artifacts/screenshots/com.hw.amb
> ari.ui.tests.monitoring.TestDownloadConfigurations/testA_DownloadClientConfigu
> rationsOnServicesTab/_25_15_27_19_file_diff_is_not_empty_for_file_hadoop_env_s
> h__/)
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/before-ANY/scripts/params.py
>  d1eeaa5 
> 
> 
> Diff: https://reviews.apache.org/r/57940/diff/1/
> 
> 
> Testing
> ---
> 
> mvn clean test
> 
> 
> Thanks,
> 
> Andrew Onischuk
> 
>

Re: Review Request 57295: Atlas service check fails during EU on wire encrypted cluster

2017-03-06 Thread Dmytro Grinenko


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/57295/#review168006
---


Ship it!




Ship It!

- Dmytro Grinenko


On March 6, 2017, 2:57 p.m., Jonathan Hurley wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/57295/
> ---
> 
> (Updated March 6, 2017, 2:57 p.m.)
> 
> 
> Review request for Ambari, Alejandro Fernandez, Dmytro Grinenko, Dmitro 
> Lisnichenko, and Nate Cole.
> 
> 
> Bugs: AMBARI-20308
> https://issues.apache.org/jira/browse/AMBARI-20308
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> During an upgrade to HDP 2.6, Atlas service checks are failing. This is 
> because in ATLAS-1427, `TLSv1.2` was added as the default protocol and 
> `TLSv1, TLSv1.1` were excluded. Some versions of cURL do not support TLSv1.2.
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/resources/stacks/HDP/2.5/services/ATLAS/configuration/application-properties.xml
>  PRE-CREATION 
>   ambari-server/src/main/resources/stacks/HDP/2.5/upgrades/config-upgrade.xml 
> e5f07ba 
>   
> ambari-server/src/main/resources/stacks/HDP/2.5/upgrades/nonrolling-upgrade-2.5.xml
>  2dbc468 
>   
> ambari-server/src/main/resources/stacks/HDP/2.5/upgrades/nonrolling-upgrade-2.6.xml
>  0d2f1bc 
>   ambari-server/src/main/resources/stacks/HDP/2.5/upgrades/upgrade-2.5.xml 
> a102d13 
>   ambari-server/src/main/resources/stacks/HDP/2.5/upgrades/upgrade-2.6.xml 
> f0a4f05 
>   
> ambari-server/src/main/resources/stacks/HDP/2.6/services/ATLAS/configuration/application-properties.xml
>  47e1fb5 
> 
> 
> Diff: https://reviews.apache.org/r/57295/diff/2/
> 
> 
> Testing
> ---
> 
> Performed an EU to HDP 2.6 for Atlas and verified the property.
> 
> 
> Thanks,
> 
> Jonathan Hurley
> 
>

Re: Review Request 56950: Error during EU while updating Ranger Log4J service configs

2017-02-23 Thread Dmytro Grinenko


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/56950/#review166516
---


Ship it!




Ship It!

- Dmytro Grinenko


On Feb. 22, 2017, 9:11 p.m., Jonathan Hurley wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/56950/
> ---
> 
> (Updated Feb. 22, 2017, 9:11 p.m.)
> 
> 
> Review request for Ambari, Dmytro Grinenko, Dmitro Lisnichenko, and Nate Cole.
> 
> 
> Bugs: AMBARI-20120
> https://issues.apache.org/jira/browse/AMBARI-20120
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Perform an HDP upgrade from 2.3 to 2.4 where Ranger is an installed service.
> 
> *Result*
> Observed errors while updating below Ranger service configs:
> Updating configuration usersync-log4j
> Updating configuration admin-log4j
> 
> {code}
> Server action failed
> Could not find desired config type with name admin-log4j
> {code}
> 
> - Both {{usersync-log4j}} and {{admin-log4j}} appear in Ranger 0.6.0 which is 
> first used by HDP 2.5. Therefore, the configuration changes should simply be 
> removed from upgrades on stacks before HDP 2.5
> 
> - Additionally, I found {{RANGER_TAGSYNC}} references in HDP 2.3 and HDP 2.4. 
> This component also first appeared in Ranger 0.6.0, so it should also be 
> removed from earlier upgrade packs.
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/UpgradeResourceProvider.java
>  13a6c36 
>   
> ambari-server/src/main/java/org/apache/ambari/server/state/ConfigHelper.java 
> 6572bbb 
>   ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/config-upgrade.xml 
> 8589e2d 
>   
> ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.4.xml
>  52421d9 
>   
> ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.5.xml
>  28d 
>   
> ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.6.xml
>  d675986 
>   ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/upgrade-2.4.xml 
> b662b28 
>   ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/upgrade-2.5.xml 
> b53ff23 
>   ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/upgrade-2.6.xml 
> 9917ee1 
>   ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/config-upgrade.xml 
> 14feab6 
>   
> ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/nonrolling-upgrade-2.5.xml
>  f093cb1 
>   
> ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/nonrolling-upgrade-2.6.xml
>  e856288 
>   ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/upgrade-2.5.xml 
> 2a3e6b2 
>   ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/upgrade-2.6.xml 
> f736796 
> 
> Diff: https://reviews.apache.org/r/56950/diff/
> 
> 
> Testing
> ---
> 
> Performed upgrades from HDP 2.3 to 2.4 and then to 2.5. Observed the new 
> configurations were created in Ambari 2.5 and no errors were seen during 
> upgrade.
> 
> 
> Thanks,
> 
> Jonathan Hurley
> 
>

Re: Review Request 56790: Getting Internal Server Error (500) on services API while trying to start all services with atleast one component in INSTALL_FAILED state

2017-02-22 Thread Dmytro Grinenko


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/56790/#review166351
---


Ship it!




Ship It!

- Dmytro Grinenko


On Feb. 22, 2017, 10:51 a.m., Dmitro Lisnichenko wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/56790/
> ---
> 
> (Updated Feb. 22, 2017, 10:51 a.m.)
> 
> 
> Review request for Ambari, Dmytro Grinenko and Nate Cole.
> 
> 
> Bugs: AMBARI-20068
> https://issues.apache.org/jira/browse/AMBARI-20068
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Steps to reproduce:
> 1) Have at-least one component in INSTALL_FAILED state
> 2) Click on "Start All" button in the Actions menu in the left pane.
> 
> Expected: UI throws an error
> 
> Actual: No errors thrown. Getting 500 error from server with the following 
> exception found in ambari-server.log
> 
> {code:java}
> 07 Feb 2017 23:01:59,273  INFO [ambari-client-thread-25] 
> AbstractResourceProvider:517 - Received a updateService request, 
> clusterName=c1, serviceName=HBASE, request=clusterName=c1, serviceName=HBASE, 
> desiredState=STARTED, credentialStoreEnabled=null
> 07 Feb 2017 23:01:59,279 ERROR [ambari-client-thread-25] 
> AbstractResourceProvider:353 - Caught AmbariException when modifying a 
> resource
> org.apache.ambari.server.AmbariException: Invalid transition for 
> servicecomponenthost, clusterName=c1, clusterId=2, serviceName=HBASE, 
> componentName=HBASE_MASTER, hostname=c6401.ambari.apache.org, 
> currentState=INSTALL_FAILED, newDesiredState=STARTED
> at 
> org.apache.ambari.server.controller.internal.ServiceResourceProvider.updateServiceComponents(ServiceResourceProvider.java:785)
> at 
> org.apache.ambari.server.controller.internal.ServiceResourceProvider.updateServices(ServiceResourceProvider.java:634)
> at 
> org.apache.ambari.server.controller.internal.ServiceResourceProvider$4.invoke(ServiceResourceProvider.java:315)
> at 
> org.apache.ambari.server.controller.internal.ServiceResourceProvider$4.invoke(ServiceResourceProvider.java:312)
> at 
> org.apache.ambari.server.controller.internal.AbstractResourceProvider.invokeWithRetry(AbstractResourceProvider.java:465)
> at 
> org.apache.ambari.server.controller.internal.AbstractResourceProvider.modifyResources(AbstractResourceProvider.java:346)
> at 
> org.apache.ambari.server.controller.internal.ServiceResourceProvider.doUpdateResources(ServiceResourceProvider.java:312)
> at 
> org.apache.ambari.server.controller.internal.ServiceResourceProvider.updateResourcesAuthorized(ServiceResourceProvider.java:226)
> at 
> org.apache.ambari.server.controller.internal.AbstractAuthorizedResourceProvider.updateResources(AbstractAuthorizedResourceProvider.java:301)
> at 
> org.apache.ambari.server.controller.internal.ClusterControllerImpl.updateResources(ClusterControllerImpl.java:319)
> at 
> org.apache.ambari.server.api.services.persistence.PersistenceManagerImpl.update(PersistenceManagerImpl.java:125)
> at 
> org.apache.ambari.server.api.handlers.UpdateHandler.persist(UpdateHandler.java:45)
> at 
> org.apache.ambari.server.api.handlers.BaseManagementHandler.handleRequest(BaseManagementHandler.java:76)
> at 
> org.apache.ambari.server.api.services.BaseRequest.process(BaseRequest.java:144)
> at 
> org.apache.ambari.server.api.services.BaseService.handleRequest(BaseService.java:125)
> at 
> org.apache.ambari.server.api.services.BaseService.handleRequest(BaseService.java:89)
> at 
> org.apache.ambari.server.api.services.ServiceService.updateServices(ServiceService.java:160)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at 
> com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
> at 
> com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:205)
> at 
> com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
> at 
> com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:302)
> at 
> com.sun.jersey.server.impl.uri.rules.SubLocatorRule.accept(SubLocatorRule.java:137)
> at

Re: Review Request 56020: Ambari HDFS Metric alerts turns to UNKNOWN status with error "argument of type 'NoneType' is not iterable"

2017-01-27 Thread Dmytro Grinenko


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/56020/#review163273
---




ambari-common/src/main/python/ambari_commons/ambari_metrics_helper.py (line 49)
<https://reviews.apache.org/r/56020/#comment234708>

load_properties_from_file doesn't provide possibility to check if target 
file is exists.

We should add logging here, to be able to track further that conf_select 
failed and we used default hardcoded location.

This should help use to debug issues here later


- Dmytro Grinenko


On Jan. 27, 2017, 2:22 p.m., Dmytro Sen wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/56020/
> ---
> 
> (Updated Jan. 27, 2017, 2:22 p.m.)
> 
> 
> Review request for Ambari, Aravindan Vijayan, Sid Wagle, and Vitalyi 
> Brodetskyi.
> 
> 
> Bugs: AMBARI-19746
> https://issues.apache.org/jira/browse/AMBARI-19746
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Build used :  ambari-2.5.0.0-724
> 
> Test is updating the alert definition minimumValue to "0.0".
> Payload used to update the minimumValue
> 
> /{
>   "AlertDefinition": {
>   "cluster_name": "cl1",
>   "id": 81,
>   "name": "increase_nn_heap_usage_daily",
>   "label": "NameNode Heap Usage (Daily)",
>   "component_name": "NAMENODE",
>   "description": "This service-level alert is triggered if the 
> NameNode heap usage deviation has grown beyond the specified threshold within 
> a day period.",
>   "enabled": true,
>   "ignore_host": false,
>   "interval": 1,
>   "scope": "ANY",
>   "service_name": "HDFS",
>   "source": {
>   "parameters": [{
>   "name": "mergeHaMetrics",
>   "display_name": "Whether active and stanby 
> NameNodes metrics should be merged",
>   "value": "false",
>   "description": "Whether active and stanby 
> NameNodes metrics should be merged.",
>   "type": "STRING",
>   "visibility": "HIDDEN"
>   }, {
>   "name": "interval",
>   "display_name": "Time interval in minutes",
>   "value": "1440.0",
>   "description": "Time interval in minutes.",
>   "type": "NUMERIC",
>   "visibility": "HIDDEN"
>   }, {
>   "name": "appId",
>   "display_name": "AMS application id",
>   "value": "NAMENODE",
>   "description": "The application id used to 
> retrieve the metric.",
>   "type": "STRING",
>   "visibility": "HIDDEN"
>   }, {
>   "name": "metricName",
>   "display_name": "Metric Name",
>   "value": "jvm.JvmMetrics.MemHeapUsedM",
>   "description": "The metric to monitor.",
>   "type": "STRING",
>   "visibility": "HIDDEN"
>   }, {
>   "name": "metric.deviation.warning.threshold",
>   "display_name": "Growth Rate",
>   "value": "20.0",
>   "description": "

Re: Review Request 56023: Upgrade default JDK installed by Ambari to be >8u100

2017-01-27 Thread Dmytro Grinenko


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/56023/#review163271
---


Ship it!




Ship It!

- Dmytro Grinenko


On Jan. 27, 2017, 3:14 p.m., Vitalyi Brodetskyi wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/56023/
> ---
> 
> (Updated Jan. 27, 2017, 3:14 p.m.)
> 
> 
> Review request for Ambari, Andrew Onischuk, Dmitro Lisnichenko, and Dmytro 
> Sen.
> 
> 
> Bugs: AMBARI-19748
> https://issues.apache.org/jira/browse/AMBARI-19748
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Te current version installed is 8u77
> Would be good to get done early, to maximize testing time since the entire 
> stack is affected
> 
> 
> Diffs
> -
> 
>   ambari-server/conf/unix/ambari.properties 99fea741 
>   ambari-server/src/main/python/ambari_server/serverSetup.py cf91a6d 
> 
> Diff: https://reviews.apache.org/r/56023/diff/
> 
> 
> Testing
> ---
> 
> mvn clean test
> 
> 
> Thanks,
> 
> Vitalyi Brodetskyi
> 
>

Re: Review Request 55424: Stack Downgrading Potentially Corrupts Kerberos Descriptor

2017-01-16 Thread Dmytro Grinenko


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/55424/#review161689
---


Ship it!




Ship It!

- Dmytro Grinenko


On Jan. 12, 2017, 2:09 p.m., Eugene Chekanskiy wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/55424/
> ---
> 
> (Updated Jan. 12, 2017, 2:09 p.m.)
> 
> 
> Review request for Ambari, Dmytro Grinenko, Jonathan Hurley, and Robert Levas.
> 
> 
> Bugs: AMBARI-19464
> https://issues.apache.org/jira/browse/AMBARI-19464
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> AMBARI-18335 addresses an issue where configuration properties in the 
> Kerberos descriptor don't change during a stack upgrade. It accounts for both 
> upgrade/downgrade scenarios and situations where properties are added/removed.
> However, it does not take into account the potential for a property to be 
> customized by the user (not a new property, but a property which the user 
> changes the value of). In this case, the downgrade would be destructive, 
> overriding the user's values with those from the downgrade stack.
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/java/org/apache/ambari/server/orm/entities/ArtifactEntity.java
>  8972e6d 
>   
> ambari-server/src/main/java/org/apache/ambari/server/serveraction/upgrades/UpgradeUserKerberosDescriptor.java
>  f1eab38 
>   
> ambari-server/src/test/java/org/apache/ambari/server/serveraction/upgrades/UpgradeUserKerberosDescriptorTest.java
>  PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/55424/diff/
> 
> 
> Testing
> ---
> 
> mvn clean test
> 
> 
> Thanks,
> 
> Eugene Chekanskiy
> 
>

Re: Review Request 55424: Stack Downgrading Potentially Corrupts Kerberos Descriptor

2017-01-15 Thread Dmytro Grinenko


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/55424/#review161688
---




ambari-server/src/main/java/org/apache/ambari/server/serveraction/upgrades/UpgradeUserKerberosDescriptor.java
 (line 252)
<https://reviews.apache.org/r/55424/#comment233001>

should be this an atomic operation?


- Dmytro Grinenko


On Jan. 12, 2017, 2:09 p.m., Eugene Chekanskiy wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/55424/
> ---
> 
> (Updated Jan. 12, 2017, 2:09 p.m.)
> 
> 
> Review request for Ambari, Dmytro Grinenko, Jonathan Hurley, and Robert Levas.
> 
> 
> Bugs: AMBARI-19464
> https://issues.apache.org/jira/browse/AMBARI-19464
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> AMBARI-18335 addresses an issue where configuration properties in the 
> Kerberos descriptor don't change during a stack upgrade. It accounts for both 
> upgrade/downgrade scenarios and situations where properties are added/removed.
> However, it does not take into account the potential for a property to be 
> customized by the user (not a new property, but a property which the user 
> changes the value of). In this case, the downgrade would be destructive, 
> overriding the user's values with those from the downgrade stack.
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/java/org/apache/ambari/server/orm/entities/ArtifactEntity.java
>  8972e6d 
>   
> ambari-server/src/main/java/org/apache/ambari/server/serveraction/upgrades/UpgradeUserKerberosDescriptor.java
>  f1eab38 
>   
> ambari-server/src/test/java/org/apache/ambari/server/serveraction/upgrades/UpgradeUserKerberosDescriptorTest.java
>  PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/55424/diff/
> 
> 
> Testing
> ---
> 
> mvn clean test
> 
> 
> Thanks,
> 
> Eugene Chekanskiy
> 
>

Re: Review Request 55424: Stack Downgrading Potentially Corrupts Kerberos Descriptor

2017-01-12 Thread Dmytro Grinenko



> On Jan. 11, 2017, 2:30 p.m., Robert Levas wrote:
> > ambari-server/src/main/java/org/apache/ambari/server/serveraction/upgrades/UpgradeUserKerberosDescriptor.java,
> >  lines 241-242
> > <https://reviews.apache.org/r/55424/diff/1/?file=1602491#file1602491line241>
> >
> > If backupEntity is `null`, this will generate an NPE

and even more, "artifactDAO.remove(entity);" shouldn't be removed, if there no 
backup entity available.


- Dmytro


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/55424/#review161238
---


On Jan. 11, 2017, 1:32 p.m., Eugene Chekanskiy wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/55424/
> ---
> 
> (Updated Jan. 11, 2017, 1:32 p.m.)
> 
> 
> Review request for Ambari, Dmytro Grinenko, Jonathan Hurley, and Robert Levas.
> 
> 
> Bugs: AMBARI-19464
> https://issues.apache.org/jira/browse/AMBARI-19464
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> AMBARI-18335 addresses an issue where configuration properties in the 
> Kerberos descriptor don't change during a stack upgrade. It accounts for both 
> upgrade/downgrade scenarios and situations where properties are added/removed.
> However, it does not take into account the potential for a property to be 
> customized by the user (not a new property, but a property which the user 
> changes the value of). In this case, the downgrade would be destructive, 
> overriding the user's values with those from the downgrade stack.
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/java/org/apache/ambari/server/orm/entities/ArtifactEntity.java
>  8972e6d 
>   
> ambari-server/src/main/java/org/apache/ambari/server/serveraction/upgrades/UpgradeUserKerberosDescriptor.java
>  f1eab38 
>   
> ambari-server/src/test/java/org/apache/ambari/server/serveraction/upgrades/UpgradeUserKerberosDescriptorTest.java
>  PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/55424/diff/
> 
> 
> Testing
> ---
> 
> mvn clean test
> 
> 
> Thanks,
> 
> Eugene Chekanskiy
> 
>

Re: Review Request 55424: Stack Downgrading Potentially Corrupts Kerberos Descriptor

2017-01-12 Thread Dmytro Grinenko



> On Jan. 11, 2017, 1:48 p.m., Jonathan Hurley wrote:
> > ambari-server/src/main/java/org/apache/ambari/server/serveraction/upgrades/UpgradeUserKerberosDescriptor.java,
> >  lines 246-248
> > <https://reviews.apache.org/r/55424/diff/1/?file=1602491#file1602491line246>
> >
> > I'm worried about this part. If something goes wrong on the downgrade 
> > in the next method and the user re-tries the command, it won't be able to 
> > find the backup, right? 
> > 
> > I know you've already restored it - but it's good to keep the backup 
> > just in case you need it.
> 
> Robert Levas wrote:
> I think that I am indifferent about this, but it can't hurt to leave the 
> data around.  If we leave it around, would it be usefil to append the 
> relevant stack version to it?  For example: `kerberos_descriptor_backup_2.5`

well, i'm not fan of making a backup storage from Ambari Database. In general, 
ttl for descriptor is from one stack version to another or forced regeneration. 
That means, that after sucessfull downgrade, that backup would not be used 
anymore and will just live with Ambari next 3 years.

I seen the code above, that we will remove previous backup if found one. That 
could be enough and here we could just keep current backup, so user can re-try 
as much as he wants. This should solve problem


> On Jan. 11, 2017, 1:48 p.m., Jonathan Hurley wrote:
> > ambari-server/src/main/java/org/apache/ambari/server/serveraction/upgrades/UpgradeUserKerberosDescriptor.java,
> >  line 124
> > <https://reviews.apache.org/r/55424/diff/1/?file=1602491#file1602491line124>
> >
> > Kind of a double-negative - can you make this if(isUpgrade()) instead?

or just swap content of if...else ... endif block and remove one negative "!"


- Dmytro


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/55424/#review161228
---


On Jan. 11, 2017, 1:32 p.m., Eugene Chekanskiy wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/55424/
> ---
> 
> (Updated Jan. 11, 2017, 1:32 p.m.)
> 
> 
> Review request for Ambari, Dmytro Grinenko, Jonathan Hurley, and Robert Levas.
> 
> 
> Bugs: AMBARI-19464
> https://issues.apache.org/jira/browse/AMBARI-19464
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> AMBARI-18335 addresses an issue where configuration properties in the 
> Kerberos descriptor don't change during a stack upgrade. It accounts for both 
> upgrade/downgrade scenarios and situations where properties are added/removed.
> However, it does not take into account the potential for a property to be 
> customized by the user (not a new property, but a property which the user 
> changes the value of). In this case, the downgrade would be destructive, 
> overriding the user's values with those from the downgrade stack.
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/java/org/apache/ambari/server/orm/entities/ArtifactEntity.java
>  8972e6d 
>   
> ambari-server/src/main/java/org/apache/ambari/server/serveraction/upgrades/UpgradeUserKerberosDescriptor.java
>  f1eab38 
>   
> ambari-server/src/test/java/org/apache/ambari/server/serveraction/upgrades/UpgradeUserKerberosDescriptorTest.java
>  PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/55424/diff/
> 
> 
> Testing
> ---
> 
> mvn clean test
> 
> 
> Thanks,
> 
> Eugene Chekanskiy
> 
>

Re: Review Request 55424: Stack Downgrading Potentially Corrupts Kerberos Descriptor

2017-01-12 Thread Dmytro Grinenko


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/55424/#review161375
---




ambari-server/src/main/java/org/apache/ambari/server/serveraction/upgrades/UpgradeUserKerberosDescriptor.java
 (line 123)
<https://reviews.apache.org/r/55424/#comment232620>

or just swap content of if...else ... endif block and remove one negative 
"!"



ambari-server/src/main/java/org/apache/ambari/server/serveraction/upgrades/UpgradeUserKerberosDescriptor.java
 (lines 245 - 247)
<https://reviews.apache.org/r/55424/#comment232621>

well, i'm not fan of making a backup storage from Ambari Database. In 
general, ttl for descriptor is from one stack version to another or forced 
regeneration. That means, that after sucessfull downgrade, that backup would 
not be used anymore and will just live with Ambari next 3 years.

I seen the code above, that we will remove previous backup if found one. 
That could be enough and here we could just keep current backup, so user can 
re-try as much as he wants. This should solve problem.


- Dmytro Grinenko


On Jan. 11, 2017, 1:32 p.m., Eugene Chekanskiy wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/55424/
> ---
> 
> (Updated Jan. 11, 2017, 1:32 p.m.)
> 
> 
> Review request for Ambari, Dmytro Grinenko, Jonathan Hurley, and Robert Levas.
> 
> 
> Bugs: AMBARI-19464
> https://issues.apache.org/jira/browse/AMBARI-19464
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> AMBARI-18335 addresses an issue where configuration properties in the 
> Kerberos descriptor don't change during a stack upgrade. It accounts for both 
> upgrade/downgrade scenarios and situations where properties are added/removed.
> However, it does not take into account the potential for a property to be 
> customized by the user (not a new property, but a property which the user 
> changes the value of). In this case, the downgrade would be destructive, 
> overriding the user's values with those from the downgrade stack.
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/java/org/apache/ambari/server/orm/entities/ArtifactEntity.java
>  8972e6d 
>   
> ambari-server/src/main/java/org/apache/ambari/server/serveraction/upgrades/UpgradeUserKerberosDescriptor.java
>  f1eab38 
>   
> ambari-server/src/test/java/org/apache/ambari/server/serveraction/upgrades/UpgradeUserKerberosDescriptorTest.java
>  PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/55424/diff/
> 
> 
> Testing
> ---
> 
> mvn clean test
> 
> 
> Thanks,
> 
> Eugene Chekanskiy
> 
>

Re: Review Request 55230: HOU Fails To Restart NameNode in non-HA Cluster

2017-01-06 Thread Dmytro Grinenko



> On Jan. 6, 2017, 7:29 a.m., Dmytro Grinenko wrote:
> > ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/hdfs_namenode.py,
> >  line 22
> > <https://reviews.apache.org/r/55230/diff/1/?file=1597766#file1597766line22>
> >
> > why not import via coma in one line(more pythonic) or for example:
> > 
> >  from ambari_commons import constants
> >   .
> >  
> >   constants.UPGRADE_TYPE_NON_ROLLING
> 
> Jonathan Hurley wrote:
> Sure, I can change that.
> 
> Jonathan Hurley wrote:
> I just thought that it was more readable as this:
> 
> if upgrade_type == UPGRADE_TYPE_ROLLING
> 
> vs
> 
> if upgrade_type == constants.UPGRADE_TYPE_ROLLING
> 
> Do you think the 2nd option is clearer?

second options allow to not extend import each time when new constant were 
added.


- Dmytro


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/55230/#review160687
---


On Jan. 6, 2017, 2:21 p.m., Jonathan Hurley wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/55230/
> ---
> 
> (Updated Jan. 6, 2017, 2:21 p.m.)
> 
> 
> Review request for Ambari, Dmytro Grinenko and Nate Cole.
> 
> 
> Bugs: AMBARI-19344
> https://issues.apache.org/jira/browse/AMBARI-19344
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> When restarting NameNode during an HOU, NN throws an error because it was 
> improperly put into rollingUpgrade mode.
> 
> *Steps*
> # Deploy HDP-2.5.0.0 with Ambari-2.5.0.0-547 build (non-HA cluster)
> # Start Host Ordered Upgrade to HDP-2.5.3
>  
> *Result:* Error at pre Upgrade HDFS step
> 
> ```
> PREPARE rolling upgrade ...
> 17/01/03 17:07:51 WARN retry.RetryInvocationHandler: Exception while invoking 
> ClientNamenodeProtocolTranslatorPB.rollingUpgrade over null. Not retrying 
> because try once and fail.
> org.apache.hadoop.ipc.RemoteException(java.io.IOException): Safe mode should 
> be turned ON in order to create namespace image.
>   at 
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startRollingUpgradeInternalForNonHA(FSNamesystem.java:7839)
> 
> rollingUpgrade: Safe mode should be turned ON in order to create namespace 
> image.
> ```
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/hdfs_namenode.py
>  23119f0 
>   
> ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/namenode.py
>  86f68e5 
>   
> ambari-server/src/main/resources/common-services/HDFS/3.0.0.3.0/package/scripts/hdfs_namenode.py
>  23119f0 
>   
> ambari-server/src/main/resources/common-services/HDFS/3.0.0.3.0/package/scripts/namenode.py
>  86f68e5 
>   
> ambari-server/src/main/resources/stacks/HDP/2.5/upgrades/host-ordered-upgrade.xml
>  f6480bf 
>   
> ambari-server/src/main/resources/stacks/HDP/2.6/upgrades/host-ordered-upgrade.xml
>  1b29af3 
>   ambari-server/src/test/python/stacks/2.0.6/HDFS/test_namenode.py 2627ddf 
> 
> Diff: https://reviews.apache.org/r/55230/diff/
> 
> 
> Testing
> ---
> 
> --
> Total run:1157
> Total errors:0
> Total failures:0
> OK
> [INFO] 
> 
> [INFO] BUILD SUCCESS
> [INFO] 
> 
> [INFO] Total time: 50.822 s
> [INFO] Finished at: 2017-01-05T16:29:21-05:00
> [INFO] Final Memory: 17M/310M
> [INFO] 
> 
> 
> 
> Thanks,
> 
> Jonathan Hurley
> 
>

Re: Review Request 55230: HOU Fails To Restart NameNode in non-HA Cluster

2017-01-06 Thread Dmytro Grinenko


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/55230/#review160732
---


Ship it!




Ship It!

- Dmytro Grinenko


On Jan. 6, 2017, 2:21 p.m., Jonathan Hurley wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/55230/
> ---
> 
> (Updated Jan. 6, 2017, 2:21 p.m.)
> 
> 
> Review request for Ambari, Dmytro Grinenko and Nate Cole.
> 
> 
> Bugs: AMBARI-19344
> https://issues.apache.org/jira/browse/AMBARI-19344
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> When restarting NameNode during an HOU, NN throws an error because it was 
> improperly put into rollingUpgrade mode.
> 
> *Steps*
> # Deploy HDP-2.5.0.0 with Ambari-2.5.0.0-547 build (non-HA cluster)
> # Start Host Ordered Upgrade to HDP-2.5.3
>  
> *Result:* Error at pre Upgrade HDFS step
> 
> ```
> PREPARE rolling upgrade ...
> 17/01/03 17:07:51 WARN retry.RetryInvocationHandler: Exception while invoking 
> ClientNamenodeProtocolTranslatorPB.rollingUpgrade over null. Not retrying 
> because try once and fail.
> org.apache.hadoop.ipc.RemoteException(java.io.IOException): Safe mode should 
> be turned ON in order to create namespace image.
>   at 
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startRollingUpgradeInternalForNonHA(FSNamesystem.java:7839)
> 
> rollingUpgrade: Safe mode should be turned ON in order to create namespace 
> image.
> ```
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/hdfs_namenode.py
>  23119f0 
>   
> ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/namenode.py
>  86f68e5 
>   
> ambari-server/src/main/resources/common-services/HDFS/3.0.0.3.0/package/scripts/hdfs_namenode.py
>  23119f0 
>   
> ambari-server/src/main/resources/common-services/HDFS/3.0.0.3.0/package/scripts/namenode.py
>  86f68e5 
>   
> ambari-server/src/main/resources/stacks/HDP/2.5/upgrades/host-ordered-upgrade.xml
>  f6480bf 
>   
> ambari-server/src/main/resources/stacks/HDP/2.6/upgrades/host-ordered-upgrade.xml
>  1b29af3 
>   ambari-server/src/test/python/stacks/2.0.6/HDFS/test_namenode.py 2627ddf 
> 
> Diff: https://reviews.apache.org/r/55230/diff/
> 
> 
> Testing
> ---
> 
> --
> Total run:1157
> Total errors:0
> Total failures:0
> OK
> [INFO] 
> 
> [INFO] BUILD SUCCESS
> [INFO] 
> 
> [INFO] Total time: 50.822 s
> [INFO] Finished at: 2017-01-05T16:29:21-05:00
> [INFO] Final Memory: 17M/310M
> [INFO] 
> 
> 
> 
> Thanks,
> 
> Jonathan Hurley
> 
>

Re: Review Request 55230: HOU Fails To Restart NameNode in non-HA Cluster

2017-01-05 Thread Dmytro Grinenko


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/55230/#review160687
---




ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/hdfs_namenode.py
 (line 22)
<https://reviews.apache.org/r/55230/#comment231898>

why not import via coma in one line(more pythonic) or for example:

 from ambari_commons import constants
  .
 
  constants.UPGRADE_TYPE_NON_ROLLING


- Dmytro Grinenko


On Jan. 5, 2017, 9:47 p.m., Jonathan Hurley wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/55230/
> ---
> 
> (Updated Jan. 5, 2017, 9:47 p.m.)
> 
> 
> Review request for Ambari, Dmytro Grinenko and Nate Cole.
> 
> 
> Bugs: AMBARI-19344
> https://issues.apache.org/jira/browse/AMBARI-19344
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> When restarting NameNode during an HOU, NN throws an error because it was 
> improperly put into rollingUpgrade mode.
> 
> *Steps*
> # Deploy HDP-2.5.0.0 with Ambari-2.5.0.0-547 build (non-HA cluster)
> # Start Host Ordered Upgrade to HDP-2.5.3
>  
> *Result:* Error at pre Upgrade HDFS step
> 
> ```
> PREPARE rolling upgrade ...
> 17/01/03 17:07:51 WARN retry.RetryInvocationHandler: Exception while invoking 
> ClientNamenodeProtocolTranslatorPB.rollingUpgrade over null. Not retrying 
> because try once and fail.
> org.apache.hadoop.ipc.RemoteException(java.io.IOException): Safe mode should 
> be turned ON in order to create namespace image.
>   at 
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startRollingUpgradeInternalForNonHA(FSNamesystem.java:7839)
> 
> rollingUpgrade: Safe mode should be turned ON in order to create namespace 
> image.
> ```
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/hdfs_namenode.py
>  23119f0 
>   
> ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/namenode.py
>  86f68e5 
>   
> ambari-server/src/main/resources/common-services/HDFS/3.0.0.3.0/package/scripts/hdfs_namenode.py
>  23119f0 
>   
> ambari-server/src/main/resources/common-services/HDFS/3.0.0.3.0/package/scripts/namenode.py
>  86f68e5 
>   
> ambari-server/src/main/resources/stacks/HDP/2.5/upgrades/host-ordered-upgrade.xml
>  f6480bf 
>   
> ambari-server/src/main/resources/stacks/HDP/2.6/upgrades/host-ordered-upgrade.xml
>  1b29af3 
>   ambari-server/src/test/python/stacks/2.0.6/HDFS/test_namenode.py 2627ddf 
> 
> Diff: https://reviews.apache.org/r/55230/diff/
> 
> 
> Testing
> ---
> 
> --
> Total run:1157
> Total errors:0
> Total failures:0
> OK
> [INFO] 
> 
> [INFO] BUILD SUCCESS
> [INFO] 
> 
> [INFO] Total time: 50.822 s
> [INFO] Finished at: 2017-01-05T16:29:21-05:00
> [INFO] Final Memory: 17M/310M
> [INFO] 
> 
> 
> 
> Thanks,
> 
> Jonathan Hurley
> 
>

Re: Review Request 55018: Fix hive-site.xml and hive-env.sh permissions for /etc/hive/conf (client) folder from 600 to 644.

2016-12-23 Thread Dmytro Grinenko


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/55018/#review160095
---


Ship it!




Ship It!

- Dmytro Grinenko


On Dec. 23, 2016, 6:30 p.m., Vitalyi Brodetskyi wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/55018/
> ---
> 
> (Updated Dec. 23, 2016, 6:30 p.m.)
> 
> 
> Review request for Ambari, Dmytro Sen, Myroslav Papirkovskyy, Sumit Mohanty, 
> and Sid Wagle.
> 
> 
> Bugs: AMBARI-19119
> https://issues.apache.org/jira/browse/AMBARI-19119
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> As part of change for AMBARI-19005, permissions for hive-site.xml and 
> hive-env.sh files in conf folder were also getting set as 600, whereas they 
> are expected to stay 644 (earlier it was 644).
> Adding fix for that.
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/hive.py
>  6344ed7 
>   ambari-server/src/test/python/stacks/2.0.6/HIVE/test_hive_client.py 1a7256f 
> 
> Diff: https://reviews.apache.org/r/55018/diff/
> 
> 
> Testing
> ---
> 
> mvn clean test
> 
> 
> Thanks,
> 
> Vitalyi Brodetskyi
> 
>

Re: Review Request 55000: Perf: Deploy 3000 Agent cluster and find perf bugs. Part 4

2016-12-23 Thread Dmytro Grinenko


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/55000/#review160076
---


Ship it!




Ship It!

- Dmytro Grinenko


On Dec. 23, 2016, 12:22 p.m., Vitalyi Brodetskyi wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/55000/
> ---
> 
> (Updated Dec. 23, 2016, 12:22 p.m.)
> 
> 
> Review request for Ambari, Alejandro Fernandez, Dmytro Sen, Sumit Mohanty, 
> and Sid Wagle.
> 
> 
> Bugs: AMBARI-19058
> https://issues.apache.org/jira/browse/AMBARI-19058
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> 1) Fixed problem with too low packet size in mysql.
> 2) Fixed problem with too short time to leave for kerberos ticket.
> 
> 
> Diffs
> -
> 
>   contrib/utils/perf/deploy-gce-perf-cluster.py 73c353e 
> 
> Diff: https://reviews.apache.org/r/55000/diff/
> 
> 
> Testing
> ---
> 
> mvn clean test
> 
> 
> Thanks,
> 
> Vitalyi Brodetskyi
> 
>

Re: Review Request 55002: 'conf.server' dir for HIVE1 and HIVE2 should have 700 permission and files in it should have 600 permission.

2016-12-23 Thread Dmytro Grinenko


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/55002/#review160077
---


Ship it!




Ship It!

- Dmytro Grinenko


On Dec. 23, 2016, 12:35 p.m., Vitalyi Brodetskyi wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/55002/
> ---
> 
> (Updated Dec. 23, 2016, 12:35 p.m.)
> 
> 
> Review request for Ambari, Alejandro Fernandez, Dmytro Sen, Sumit Mohanty, 
> and Sid Wagle.
> 
> 
> Bugs: AMBARI-19005
> https://issues.apache.org/jira/browse/AMBARI-19005
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Observed an issue that affects earlier releases of HIVE as well. The 
> conf.server directory has hive-site.xml with metastore server password in it.
> That directory is only populated on machines with hs2/metastore (the good 
> part).
> But the permissions of that directory and the files in it are not restricted, 
> it should be set to 700 (directory) and 600 (files). Currently, it is set to 
> 755 and 644 for dir and files.
> I believe intention of creating separate directory was to restrict 
> permissions to only the user hive.
> I see this problem in ambari installed HDP 2.5 cluster and HDP 2.2 and 2.3 
> sandboxes, so it seems to affect the corresponding ambari versions as well.
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/hive.py
>  792aac3 
>   
> ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/hive_interactive.py
>  298db2a 
> 
> Diff: https://reviews.apache.org/r/55002/diff/
> 
> 
> Testing
> ---
> 
> mvn clean test
> 
> 
> Thanks,
> 
> Vitalyi Brodetskyi
> 
>

Re: Review Request 54942: AMBARI-19272. Ignored mount points logged for each mount

2016-12-22 Thread Dmytro Grinenko


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/54942/#review160018
---


Ship it!




Ship It!

- Dmytro Grinenko


On Dec. 21, 2016, 4:52 p.m., Attila Doroszlai wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/54942/
> ---
> 
> (Updated Dec. 21, 2016, 4:52 p.m.)
> 
> 
> Review request for Ambari, Balázs Bence Sári, Dmytro Grinenko, Laszlo Puskas, 
> Sumit Mohanty, and Sebastian Toader.
> 
> 
> Bugs: AMBARI-19272
> https://issues.apache.org/jira/browse/AMBARI-19272
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> * Moved log out of the loop.
>  * Fixed typo ("was" -> "were")
> 
> 
> Diffs
> -
> 
>   ambari-agent/src/main/python/ambari_agent/Hardware.py 
> 0d431a389ee492fc8557ae04b56a6c178d5a8fdd 
> 
> Diff: https://reviews.apache.org/r/54942/diff/
> 
> 
> Testing
> ---
> 
> Verified that only the message with all ignored mounts is logged.  (Note: 
> /vagrant is really mounted twice.)
> 
> ```
> INFO 2016-12-21 16:41:03,424 Hardware.py:168 - Some mount points were 
> ignored: /, /dev/shm, /boot, /vagrant, /vagrant
> ```
> 
> Ran unit tests:
> 
> ```
> $ mvn -am -pl ambari-agent clean test
> ...
> Ran 451 tests in 15.070s
> 
> OK
> ...
> [INFO] Rat check: Summary of files. Unapproved: 0 unknown: 0 generated: 0 
> approved: 151 licence.
> ...
> [INFO] Ambari Agent .. SUCCESS [17.537s]
> ```
> 
> 
> Thanks,
> 
> Attila Doroszlai
> 
>

Re: Review Request 54674: AMBARI-19170. NPE during Ambari server schema upgrade

2016-12-12 Thread Dmytro Grinenko



> On Dec. 12, 2016, 8:37 p.m., Jonathan Hurley wrote:
> > ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog240.java,
> >  line 1555
> > <https://reviews.apache.org/r/54674/diff/1/?file=1582171#file1582171line1555>
> >
> > Because this is changing data, should the data manipulation be wrapped 
> > in an `executeInTransaction(Runnable)` ?
> 
> Dmytro Grinenko wrote:
> moreover, i'm not sure that @Transactional annotation will have any 
> effect on DDL changes via dbAccessor or any other way to make schema changes 
> in scope of database transaction. (not all database engines support this)

P.S. i mean revert of schema alteration, as MySQL for exmple done explicit 
commit on alter commands


- Dmytro


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/54674/#review158900
---


On Dec. 12, 2016, 8:01 p.m., Attila Doroszlai wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/54674/
> ---
> 
> (Updated Dec. 12, 2016, 8:01 p.m.)
> 
> 
> Review request for Ambari, Jonathan Hurley, Sebastian Toader, and Sid Wagle.
> 
> 
> Bugs: AMBARI-19170
> https://issues.apache.org/jira/browse/AMBARI-19170
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Removed @Transactional
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog240.java
>  063c29539e135433b7b9ce37fa5c32d0de2234f8 
> 
> Diff: https://reviews.apache.org/r/54674/diff/
> 
> 
> Testing
> ---
> 
> Installed 2.2.2.0-460, upgraded to 2.5.0.0-453.
> 
> ```
> [root@c6401 ~]# ambari-server upgrade
> Using python  /usr/bin/python
> Upgrading ambari-server
> INFO: Upgrade Ambari Server
> INFO: Updating Ambari Server properties in ambari.properties ...
> INFO: Updating Ambari Server properties in ambari-env.sh ...
> WARNING: Original file ambari-env.sh kept
> INFO: Fixing database objects owner
> Ambari Server configured for Embedded Postgres. Confirm you have made a 
> backup of the Ambari Server database [y/n] (y)? y
> INFO: Upgrading database schema
> INFO: Return code from schema upgrade command, retcode = 0
> INFO: Schema upgrade completed
> Adjusting ambari-server permissions and ownership...
> Ambari Server 'upgrade' completed successfully.
> ```
> 
> 
> Thanks,
> 
> Attila Doroszlai
> 
>

1 2 >

1 - 100 of 146 matches

Mail list logo