Re: Review Request 48817: Hive Metastore Upgrade Fails Because Of Missing Hive Interactive Directory
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/48817/
---
(Updated June 16, 2016, 11:11 p.m.)
Review request for Ambari, Alejandro Fernandez, Nate Cole, and Vitalyi
Brodetskyi.
Changes
---
Updated tests
Bugs: AMBARI-17284
https://issues.apache.org/jira/browse/AMBARI-17284
Repository: ambari
Description
---
During an upgrade from HDP 2.3/2.4 to HDP 2.5, the following error is seen:
```
resource_management.core.exceptions.Fail: Execution of 'cp --remove-destination
/var/lib/ambari-agent/tmp/mysql-connector-java.jar
/usr/hdp/current/hive-server2-hive2/lib/mysql-connector-java.jar' returned 1.
cp: cannot create regular file
`/usr/hdp/current/hive-server2-hive2/lib/mysql-connector-java.jar': No such
file or directory
```
This is because with HDP 2.5, Hive introduced a new component called
`hive-server2-hive2`:
```
/usr/hdp/current/hive-metastore -> /usr/hdp/2.5.0.0-1234/hive
/usr/hdp/current/hive-server2 -> /usr/hdp/2.5.0.0-1234/hive
/usr/hdp/current/hive-server2-hive2 -> /usr/hdp/2.5.0.0-1234/hive2
```
Normally, this would be fine. However, we must use the `schematool` from
`hive2` and not {{hive}}:
```
/usr/hdp/2.5.0.0-1234/hive2/bin/schematool
```
Since Metastore is uggraded first, it's trying to write the custom JDBC
connector to
`/usr/hdp/current/hive-server2-hive2/lib`
But since this component doesn't exist or hasn't been `hdp-select`'d yet, it
fails.
Diffs (updated)
-
ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/hive.py
ef0fd6b
ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/hive_metastore.py
55cddd6
ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/hive_service.py
0a3d921
ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/params_linux.py
81207d6
ambari-server/src/test/python/stacks/2.0.6/HIVE/test_hive_metastore.py
9039ea4
ambari-server/src/test/python/stacks/2.0.6/HIVE/test_hive_server.py 3c224a8
ambari-server/src/test/python/stacks/2.0.6/HIVE/test_hive_service_check.py
3405a65
ambari-server/src/test/python/stacks/2.0.6/HIVE/test_webhcat_server.py
d5ab0b6
ambari-server/src/test/python/stacks/2.1/HIVE/test_hive_metastore.py c36b428
ambari-server/src/test/python/stacks/utils/RMFTestCase.py b85fe90
Diff: https://reviews.apache.org/r/48817/diff/
Testing (updated)
---
--
Total run:1062
Total errors:0
Total failures:0
OK
Thanks,
Jonathan Hurley
Re: Review Request 48817: Hive Metastore Upgrade Fails Because Of Missing Hive Interactive Directory
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/48817/#review138094
---
Ship it!
Any new tests required for these changes?
- Nate Cole
On June 16, 2016, 5:02 p.m., Jonathan Hurley wrote:
>
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/48817/
> ---
>
> (Updated June 16, 2016, 5:02 p.m.)
>
>
> Review request for Ambari, Alejandro Fernandez, Nate Cole, and Vitalyi
> Brodetskyi.
>
>
> Bugs: AMBARI-17284
> https://issues.apache.org/jira/browse/AMBARI-17284
>
>
> Repository: ambari
>
>
> Description
> ---
>
> During an upgrade from HDP 2.3/2.4 to HDP 2.5, the following error is seen:
>
> ```
> resource_management.core.exceptions.Fail: Execution of 'cp
> --remove-destination /var/lib/ambari-agent/tmp/mysql-connector-java.jar
> /usr/hdp/current/hive-server2-hive2/lib/mysql-connector-java.jar' returned 1.
> cp: cannot create regular file
> `/usr/hdp/current/hive-server2-hive2/lib/mysql-connector-java.jar': No such
> file or directory
> ```
>
> This is because with HDP 2.5, Hive introduced a new component called
> `hive-server2-hive2`:
>
> ```
> /usr/hdp/current/hive-metastore -> /usr/hdp/2.5.0.0-1234/hive
> /usr/hdp/current/hive-server2 -> /usr/hdp/2.5.0.0-1234/hive
> /usr/hdp/current/hive-server2-hive2 -> /usr/hdp/2.5.0.0-1234/hive2
> ```
>
> Normally, this would be fine. However, we must use the `schematool` from
> `hive2` and not {{hive}}:
>
> ```
> /usr/hdp/2.5.0.0-1234/hive2/bin/schematool
> ```
>
> Since Metastore is uggraded first, it's trying to write the custom JDBC
> connector to
> `/usr/hdp/current/hive-server2-hive2/lib`
>
> But since this component doesn't exist or hasn't been `hdp-select`'d yet, it
> fails.
>
>
> Diffs
> -
>
>
> ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/hive.py
> ef0fd6b
>
> ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/hive_metastore.py
> 55cddd6
>
> ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/hive_service.py
> 0a3d921
>
> ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/params_linux.py
> 81207d6
>
> Diff: https://reviews.apache.org/r/48817/diff/
>
>
> Testing
> ---
>
> PENDING
>
>
> Thanks,
>
> Jonathan Hurley
>
>
Re: Review Request 48817: Hive Metastore Upgrade Fails Because Of Missing Hive Interactive Directory
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/48817/#review138082
---
Ship it!
Ship It!
- Alejandro Fernandez
On June 16, 2016, 9:02 p.m., Jonathan Hurley wrote:
>
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/48817/
> ---
>
> (Updated June 16, 2016, 9:02 p.m.)
>
>
> Review request for Ambari, Alejandro Fernandez, Nate Cole, and Vitalyi
> Brodetskyi.
>
>
> Bugs: AMBARI-17284
> https://issues.apache.org/jira/browse/AMBARI-17284
>
>
> Repository: ambari
>
>
> Description
> ---
>
> During an upgrade from HDP 2.3/2.4 to HDP 2.5, the following error is seen:
>
> ```
> resource_management.core.exceptions.Fail: Execution of 'cp
> --remove-destination /var/lib/ambari-agent/tmp/mysql-connector-java.jar
> /usr/hdp/current/hive-server2-hive2/lib/mysql-connector-java.jar' returned 1.
> cp: cannot create regular file
> `/usr/hdp/current/hive-server2-hive2/lib/mysql-connector-java.jar': No such
> file or directory
> ```
>
> This is because with HDP 2.5, Hive introduced a new component called
> `hive-server2-hive2`:
>
> ```
> /usr/hdp/current/hive-metastore -> /usr/hdp/2.5.0.0-1234/hive
> /usr/hdp/current/hive-server2 -> /usr/hdp/2.5.0.0-1234/hive
> /usr/hdp/current/hive-server2-hive2 -> /usr/hdp/2.5.0.0-1234/hive2
> ```
>
> Normally, this would be fine. However, we must use the `schematool` from
> `hive2` and not {{hive}}:
>
> ```
> /usr/hdp/2.5.0.0-1234/hive2/bin/schematool
> ```
>
> Since Metastore is uggraded first, it's trying to write the custom JDBC
> connector to
> `/usr/hdp/current/hive-server2-hive2/lib`
>
> But since this component doesn't exist or hasn't been `hdp-select`'d yet, it
> fails.
>
>
> Diffs
> -
>
>
> ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/hive.py
> ef0fd6b
>
> ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/hive_metastore.py
> 55cddd6
>
> ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/hive_service.py
> 0a3d921
>
> ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/params_linux.py
> 81207d6
>
> Diff: https://reviews.apache.org/r/48817/diff/
>
>
> Testing
> ---
>
> PENDING
>
>
> Thanks,
>
> Jonathan Hurley
>
>
Re: Review Request 48817: Hive Metastore Upgrade Fails Because Of Missing Hive Interactive Directory
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/48817/#review138083
---
I'm really sorry about this review; Our python code is a mess. 90% of the
changes here are to cleanup HDP 2.1 things and try to make the variable names
clearer. The real change is this:
```
if upgrade_direction:
hive_jdbc_target = format("{hive_version_lib}/{jdbc_jar_name}")
hive2_jdbc_target =
format("{hive_server2_hive2_version_lib}/{jdbc_jar_name}") if hive2_jdbc_target
is not None else None
```
Basically, if you're in an upgrade, write out the JDBC connector to the
"versioned" location (and don't use current)
- Jonathan Hurley
On June 16, 2016, 5:02 p.m., Jonathan Hurley wrote:
>
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/48817/
> ---
>
> (Updated June 16, 2016, 5:02 p.m.)
>
>
> Review request for Ambari, Alejandro Fernandez, Nate Cole, and Vitalyi
> Brodetskyi.
>
>
> Bugs: AMBARI-17284
> https://issues.apache.org/jira/browse/AMBARI-17284
>
>
> Repository: ambari
>
>
> Description
> ---
>
> During an upgrade from HDP 2.3/2.4 to HDP 2.5, the following error is seen:
>
> ```
> resource_management.core.exceptions.Fail: Execution of 'cp
> --remove-destination /var/lib/ambari-agent/tmp/mysql-connector-java.jar
> /usr/hdp/current/hive-server2-hive2/lib/mysql-connector-java.jar' returned 1.
> cp: cannot create regular file
> `/usr/hdp/current/hive-server2-hive2/lib/mysql-connector-java.jar': No such
> file or directory
> ```
>
> This is because with HDP 2.5, Hive introduced a new component called
> `hive-server2-hive2`:
>
> ```
> /usr/hdp/current/hive-metastore -> /usr/hdp/2.5.0.0-1234/hive
> /usr/hdp/current/hive-server2 -> /usr/hdp/2.5.0.0-1234/hive
> /usr/hdp/current/hive-server2-hive2 -> /usr/hdp/2.5.0.0-1234/hive2
> ```
>
> Normally, this would be fine. However, we must use the `schematool` from
> `hive2` and not {{hive}}:
>
> ```
> /usr/hdp/2.5.0.0-1234/hive2/bin/schematool
> ```
>
> Since Metastore is uggraded first, it's trying to write the custom JDBC
> connector to
> `/usr/hdp/current/hive-server2-hive2/lib`
>
> But since this component doesn't exist or hasn't been `hdp-select`'d yet, it
> fails.
>
>
> Diffs
> -
>
>
> ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/hive.py
> ef0fd6b
>
> ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/hive_metastore.py
> 55cddd6
>
> ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/hive_service.py
> 0a3d921
>
> ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/params_linux.py
> 81207d6
>
> Diff: https://reviews.apache.org/r/48817/diff/
>
>
> Testing
> ---
>
> PENDING
>
>
> Thanks,
>
> Jonathan Hurley
>
>
Re: Review Request 48817: Hive Metastore Upgrade Fails Because Of Missing Hive Interactive Directory
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/48817/
---
(Updated June 16, 2016, 5:02 p.m.)
Review request for Ambari, Alejandro Fernandez, Nate Cole, and Vitalyi
Brodetskyi.
Bugs: AMBARI-17284
https://issues.apache.org/jira/browse/AMBARI-17284
Repository: ambari
Description (updated)
---
During an upgrade from HDP 2.3/2.4 to HDP 2.5, the following error is seen:
```
resource_management.core.exceptions.Fail: Execution of 'cp --remove-destination
/var/lib/ambari-agent/tmp/mysql-connector-java.jar
/usr/hdp/current/hive-server2-hive2/lib/mysql-connector-java.jar' returned 1.
cp: cannot create regular file
`/usr/hdp/current/hive-server2-hive2/lib/mysql-connector-java.jar': No such
file or directory
```
This is because with HDP 2.5, Hive introduced a new component called
`hive-server2-hive2`:
```
/usr/hdp/current/hive-metastore -> /usr/hdp/2.5.0.0-1234/hive
/usr/hdp/current/hive-server2 -> /usr/hdp/2.5.0.0-1234/hive
/usr/hdp/current/hive-server2-hive2 -> /usr/hdp/2.5.0.0-1234/hive2
```
Normally, this would be fine. However, we must use the `schematool` from
`hive2` and not {{hive}}:
```
/usr/hdp/2.5.0.0-1234/hive2/bin/schematool
```
Since Metastore is uggraded first, it's trying to write the custom JDBC
connector to
`/usr/hdp/current/hive-server2-hive2/lib`
But since this component doesn't exist or hasn't been `hdp-select`'d yet, it
fails.
Diffs
-
ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/hive.py
ef0fd6b
ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/hive_metastore.py
55cddd6
ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/hive_service.py
0a3d921
ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/params_linux.py
81207d6
Diff: https://reviews.apache.org/r/48817/diff/
Testing
---
PENDING
Thanks,
Jonathan Hurley
Review Request 48817: Hive Metastore Upgrade Fails Because Of Missing Hive Interactive Directory
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/48817/
---
Review request for Ambari, Alejandro Fernandez, Nate Cole, and Vitalyi
Brodetskyi.
Bugs: AMBARI-17284
https://issues.apache.org/jira/browse/AMBARI-17284
Repository: ambari
Description
---
During an upgrade from HDP 2.3/2.4 to HDP 2.5, the following error is seen:
```
resource_management.core.exceptions.Fail: Execution of 'cp --remove-destination
/var/lib/ambari-agent/tmp/mysql-connector-java.jar
/usr/hdp/current/hive-server2-hive2/lib/mysql-connector-java.jar' returned 1.
cp: cannot create regular file
`/usr/hdp/current/hive-server2-hive2/lib/mysql-connector-java.jar': No such
file or directory
```
This is because with HDP 2.5, Hive introduced a new component called
{{hive-server2-hive2}}:
```
/usr/hdp/current/hive-metastore -> /usr/hdp/2.5.0.0-1234/hive
/usr/hdp/current/hive-server2 -> /usr/hdp/2.5.0.0-1234/hive
/usr/hdp/current/hive-server2-hive2 -> /usr/hdp/2.5.0.0-1234/hive2
```
Normally, this would be fine. However, we must use the {{schematool}} from
{{hive2}} and not {{hive}}:
```
/usr/hdp/2.5.0.0-1234/hive2/bin/schematool
```
Since Metastore is uggraded first, it's trying to write the custom JDBC
connector to
`/usr/hdp/current/hive-server2-hive2/lib}}`
But since this component doesn't exist or hasn't been `hdp-select`'d yet, it
fails.
Diffs
-
ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/hive.py
ef0fd6b
ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/hive_metastore.py
55cddd6
ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/hive_service.py
0a3d921
ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/params_linux.py
81207d6
Diff: https://reviews.apache.org/r/48817/diff/
Testing
---
PENDING
Thanks,
Jonathan Hurley
