[jira] [Updated] (HDFS-12473) Change hosts JSON file format
[
https://issues.apache.org/jira/browse/HDFS-12473?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Junping Du updated HDFS-12473:
--
Fix Version/s: (was: 2.8.3)
> Change hosts JSON file format
> -
>
> Key: HDFS-12473
> URL: https://issues.apache.org/jira/browse/HDFS-12473
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Ming Ma
>Assignee: Ming Ma
> Fix For: 2.9.0, 3.0.0-beta1, 2.8.2, 3.1.0
>
> Attachments: HDFS-12473-2.patch, HDFS-12473-3.patch,
> HDFS-12473-4.patch, HDFS-12473-5.patch, HDFS-12473-6.patch,
> HDFS-12473-branch-2.patch, HDFS-12473.patch
>
>
> The existing host JSON file format doesn't have a top-level token.
> {noformat}
> {"hostName": "host1"}
> {"hostName": "host2", "upgradeDomain": "ud0"}
> {"hostName": "host3", "adminState": "DECOMMISSIONED"}
> {"hostName": "host4", "upgradeDomain": "ud2", "adminState":
> "DECOMMISSIONED"}
> {"hostName": "host5", "port": 8090}
> {"hostName": "host6", "adminState": "IN_MAINTENANCE"}
> {"hostName": "host7", "adminState": "IN_MAINTENANCE",
> "maintenanceExpireTimeInMS": "112233"}
> {noformat}
> Instead, to conform with the JSON standard it should be like
> {noformat}
> [
> {"hostName": "host1"},
> {"hostName": "host2", "upgradeDomain": "ud0"},
> {"hostName": "host3", "adminState": "DECOMMISSIONED"},
> {"hostName": "host4", "upgradeDomain": "ud2", "adminState":
> "DECOMMISSIONED"},
> {"hostName": "host5", "port": 8090},
> {"hostName": "host6", "adminState": "IN_MAINTENANCE"},
> {"hostName": "host7", "adminState": "IN_MAINTENANCE",
> "maintenanceExpireTimeInMS": "112233"}
> ]
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12473) Change hosts JSON file format
[
https://issues.apache.org/jira/browse/HDFS-12473?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ming Ma updated HDFS-12473:
---
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 3.1.0
2.8.3
2.8.2
3.0.0-beta1
2.9.0
Status: Resolved (was: Patch Available)
Thanks [~manojg] and [~zhz]. I have committed patch to trunk, branch-3.0,
branch-2, branch-2.8 and branch 2.8.2. Besides the branch-2 diff mentioned
above, the patch for branch-2.8/branch.2.8.2 is slightly different in the unit
tests as maintenance state only exists in branch-2 and above.
> Change hosts JSON file format
> -
>
> Key: HDFS-12473
> URL: https://issues.apache.org/jira/browse/HDFS-12473
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Ming Ma
>Assignee: Ming Ma
> Fix For: 2.9.0, 3.0.0-beta1, 2.8.2, 2.8.3, 3.1.0
>
> Attachments: HDFS-12473-2.patch, HDFS-12473-3.patch,
> HDFS-12473-4.patch, HDFS-12473-5.patch, HDFS-12473-6.patch,
> HDFS-12473-branch-2.patch, HDFS-12473.patch
>
>
> The existing host JSON file format doesn't have a top-level token.
> {noformat}
> {"hostName": "host1"}
> {"hostName": "host2", "upgradeDomain": "ud0"}
> {"hostName": "host3", "adminState": "DECOMMISSIONED"}
> {"hostName": "host4", "upgradeDomain": "ud2", "adminState":
> "DECOMMISSIONED"}
> {"hostName": "host5", "port": 8090}
> {"hostName": "host6", "adminState": "IN_MAINTENANCE"}
> {"hostName": "host7", "adminState": "IN_MAINTENANCE",
> "maintenanceExpireTimeInMS": "112233"}
> {noformat}
> Instead, to conform with the JSON standard it should be like
> {noformat}
> [
> {"hostName": "host1"},
> {"hostName": "host2", "upgradeDomain": "ud0"},
> {"hostName": "host3", "adminState": "DECOMMISSIONED"},
> {"hostName": "host4", "upgradeDomain": "ud2", "adminState":
> "DECOMMISSIONED"},
> {"hostName": "host5", "port": 8090},
> {"hostName": "host6", "adminState": "IN_MAINTENANCE"},
> {"hostName": "host7", "adminState": "IN_MAINTENANCE",
> "maintenanceExpireTimeInMS": "112233"}
> ]
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12473) Change hosts JSON file format
[
https://issues.apache.org/jira/browse/HDFS-12473?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ming Ma updated HDFS-12473:
---
Attachment: HDFS-12473-branch-2.patch
Compared to the patch for trunk, branch-2's version is slightly different
because it depends on different version of jackson. Specifically trunk can use
{{com.fasterxml.jackson.databind.ObjectMapper}} while branch-2 should use
{{org.codehaus.jackson.map.ObjectMapper}}. {{CombinedHostsFileReader.java}}
also needs to be modified slightly due to different exception in case of empty
file.
> Change hosts JSON file format
> -
>
> Key: HDFS-12473
> URL: https://issues.apache.org/jira/browse/HDFS-12473
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Ming Ma
>Assignee: Ming Ma
> Attachments: HDFS-12473-2.patch, HDFS-12473-3.patch,
> HDFS-12473-4.patch, HDFS-12473-5.patch, HDFS-12473-6.patch,
> HDFS-12473-branch-2.patch, HDFS-12473.patch
>
>
> The existing host JSON file format doesn't have a top-level token.
> {noformat}
> {"hostName": "host1"}
> {"hostName": "host2", "upgradeDomain": "ud0"}
> {"hostName": "host3", "adminState": "DECOMMISSIONED"}
> {"hostName": "host4", "upgradeDomain": "ud2", "adminState":
> "DECOMMISSIONED"}
> {"hostName": "host5", "port": 8090}
> {"hostName": "host6", "adminState": "IN_MAINTENANCE"}
> {"hostName": "host7", "adminState": "IN_MAINTENANCE",
> "maintenanceExpireTimeInMS": "112233"}
> {noformat}
> Instead, to conform with the JSON standard it should be like
> {noformat}
> [
> {"hostName": "host1"},
> {"hostName": "host2", "upgradeDomain": "ud0"},
> {"hostName": "host3", "adminState": "DECOMMISSIONED"},
> {"hostName": "host4", "upgradeDomain": "ud2", "adminState":
> "DECOMMISSIONED"},
> {"hostName": "host5", "port": 8090},
> {"hostName": "host6", "adminState": "IN_MAINTENANCE"},
> {"hostName": "host7", "adminState": "IN_MAINTENANCE",
> "maintenanceExpireTimeInMS": "112233"}
> ]
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12473) Change hosts JSON file format
[
https://issues.apache.org/jira/browse/HDFS-12473?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ming Ma updated HDFS-12473:
---
Attachment: HDFS-12473-6.patch
Thanks [~manojg]. Here is the new patch to address your comments, except for
the exception handling where I prefer the code swallow as fewer exception as
possible.
> Change hosts JSON file format
> -
>
> Key: HDFS-12473
> URL: https://issues.apache.org/jira/browse/HDFS-12473
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Ming Ma
>Assignee: Ming Ma
> Attachments: HDFS-12473-2.patch, HDFS-12473-3.patch,
> HDFS-12473-4.patch, HDFS-12473-5.patch, HDFS-12473-6.patch, HDFS-12473.patch
>
>
> The existing host JSON file format doesn't have a top-level token.
> {noformat}
> {"hostName": "host1"}
> {"hostName": "host2", "upgradeDomain": "ud0"}
> {"hostName": "host3", "adminState": "DECOMMISSIONED"}
> {"hostName": "host4", "upgradeDomain": "ud2", "adminState":
> "DECOMMISSIONED"}
> {"hostName": "host5", "port": 8090}
> {"hostName": "host6", "adminState": "IN_MAINTENANCE"}
> {"hostName": "host7", "adminState": "IN_MAINTENANCE",
> "maintenanceExpireTimeInMS": "112233"}
> {noformat}
> Instead, to conform with the JSON standard it should be like
> {noformat}
> [
> {"hostName": "host1"},
> {"hostName": "host2", "upgradeDomain": "ud0"},
> {"hostName": "host3", "adminState": "DECOMMISSIONED"},
> {"hostName": "host4", "upgradeDomain": "ud2", "adminState":
> "DECOMMISSIONED"},
> {"hostName": "host5", "port": 8090},
> {"hostName": "host6", "adminState": "IN_MAINTENANCE"},
> {"hostName": "host7", "adminState": "IN_MAINTENANCE",
> "maintenanceExpireTimeInMS": "112233"}
> ]
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12473) Change hosts JSON file format
[
https://issues.apache.org/jira/browse/HDFS-12473?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ming Ma updated HDFS-12473:
---
Attachment: HDFS-12473-5.patch
>From discussion with [~zhz], 2.8.2 hasn't been released yet. Thus we don't
>need to deal with the backward compatibility issue of old JSON format being
>used in HDFS-7541, assuming we can get the patch in 2.8.2 and branch-3.0.
>[~manojg] here is the latest patch.
> Change hosts JSON file format
> -
>
> Key: HDFS-12473
> URL: https://issues.apache.org/jira/browse/HDFS-12473
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Ming Ma
>Assignee: Ming Ma
> Attachments: HDFS-12473-2.patch, HDFS-12473-3.patch,
> HDFS-12473-4.patch, HDFS-12473-5.patch, HDFS-12473.patch
>
>
> The existing host JSON file format doesn't have a top-level token.
> {noformat}
> {"hostName": "host1"}
> {"hostName": "host2", "upgradeDomain": "ud0"}
> {"hostName": "host3", "adminState": "DECOMMISSIONED"}
> {"hostName": "host4", "upgradeDomain": "ud2", "adminState":
> "DECOMMISSIONED"}
> {"hostName": "host5", "port": 8090}
> {"hostName": "host6", "adminState": "IN_MAINTENANCE"}
> {"hostName": "host7", "adminState": "IN_MAINTENANCE",
> "maintenanceExpireTimeInMS": "112233"}
> {noformat}
> Instead, to conform with the JSON standard it should be like
> {noformat}
> [
> {"hostName": "host1"},
> {"hostName": "host2", "upgradeDomain": "ud0"},
> {"hostName": "host3", "adminState": "DECOMMISSIONED"},
> {"hostName": "host4", "upgradeDomain": "ud2", "adminState":
> "DECOMMISSIONED"},
> {"hostName": "host5", "port": 8090},
> {"hostName": "host6", "adminState": "IN_MAINTENANCE"},
> {"hostName": "host7", "adminState": "IN_MAINTENANCE",
> "maintenanceExpireTimeInMS": "112233"}
> ]
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12473) Change hosts JSON file format
[
https://issues.apache.org/jira/browse/HDFS-12473?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ming Ma updated HDFS-12473:
---
Attachment: HDFS-12473-4.patch
> Change hosts JSON file format
> -
>
> Key: HDFS-12473
> URL: https://issues.apache.org/jira/browse/HDFS-12473
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Ming Ma
>Assignee: Ming Ma
> Attachments: HDFS-12473-2.patch, HDFS-12473-3.patch,
> HDFS-12473-4.patch, HDFS-12473.patch
>
>
> The existing host JSON file format doesn't have a top-level token.
> {noformat}
> {"hostName": "host1"}
> {"hostName": "host2", "upgradeDomain": "ud0"}
> {"hostName": "host3", "adminState": "DECOMMISSIONED"}
> {"hostName": "host4", "upgradeDomain": "ud2", "adminState":
> "DECOMMISSIONED"}
> {"hostName": "host5", "port": 8090}
> {"hostName": "host6", "adminState": "IN_MAINTENANCE"}
> {"hostName": "host7", "adminState": "IN_MAINTENANCE",
> "maintenanceExpireTimeInMS": "112233"}
> {noformat}
> Instead, to conform with the JSON standard it should be like
> {noformat}
> [
> {"hostName": "host1"},
> {"hostName": "host2", "upgradeDomain": "ud0"},
> {"hostName": "host3", "adminState": "DECOMMISSIONED"},
> {"hostName": "host4", "upgradeDomain": "ud2", "adminState":
> "DECOMMISSIONED"},
> {"hostName": "host5", "port": 8090},
> {"hostName": "host6", "adminState": "IN_MAINTENANCE"},
> {"hostName": "host7", "adminState": "IN_MAINTENANCE",
> "maintenanceExpireTimeInMS": "112233"}
> ]
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12473) Change hosts JSON file format
[
https://issues.apache.org/jira/browse/HDFS-12473?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ming Ma updated HDFS-12473:
---
Attachment: HDFS-12473-2.patch
Thanks Manoj. Here is the updated patch to address your comments.
bq. What happens when the hosts file has improper json format?
I was hoping we can get it before 3.0 beta release thus without worrying about
compatibility issue. But it looks like upgrade domain feature has been
backported to 2.8.2. Unfortunately that means we have to support the old format.
bq. #readFile can now return null object
The updated patch will return empty array instead.
bq. If MAPPER is no more used, can be removed.
It was removed. Maybe you referred to the existing file.
bq. CombinedHostsFileReader.readFile() can return null if the input hosts file
has no entries.
test case testEmptyCombinedHostsFileReader
> Change hosts JSON file format
> -
>
> Key: HDFS-12473
> URL: https://issues.apache.org/jira/browse/HDFS-12473
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Ming Ma
>Assignee: Ming Ma
> Attachments: HDFS-12473-2.patch, HDFS-12473.patch
>
>
> The existing host JSON file format doesn't have a top-level token.
> {noformat}
> {"hostName": "host1"}
> {"hostName": "host2", "upgradeDomain": "ud0"}
> {"hostName": "host3", "adminState": "DECOMMISSIONED"}
> {"hostName": "host4", "upgradeDomain": "ud2", "adminState":
> "DECOMMISSIONED"}
> {"hostName": "host5", "port": 8090}
> {"hostName": "host6", "adminState": "IN_MAINTENANCE"}
> {"hostName": "host7", "adminState": "IN_MAINTENANCE",
> "maintenanceExpireTimeInMS": "112233"}
> {noformat}
> Instead, to conform with the JSON standard it should be like
> {noformat}
> [
> {"hostName": "host1"},
> {"hostName": "host2", "upgradeDomain": "ud0"},
> {"hostName": "host3", "adminState": "DECOMMISSIONED"},
> {"hostName": "host4", "upgradeDomain": "ud2", "adminState":
> "DECOMMISSIONED"},
> {"hostName": "host5", "port": 8090},
> {"hostName": "host6", "adminState": "IN_MAINTENANCE"},
> {"hostName": "host7", "adminState": "IN_MAINTENANCE",
> "maintenanceExpireTimeInMS": "112233"}
> ]
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12473) Change hosts JSON file format
[
https://issues.apache.org/jira/browse/HDFS-12473?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Manoj Govindassamy updated HDFS-12473:
--
Summary: Change hosts JSON file format (was: Change host JSON file format)
> Change hosts JSON file format
> -
>
> Key: HDFS-12473
> URL: https://issues.apache.org/jira/browse/HDFS-12473
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Ming Ma
>Assignee: Ming Ma
> Attachments: HDFS-12473.patch
>
>
> The existing host JSON file format doesn't have a top-level token.
> {noformat}
> {"hostName": "host1"}
> {"hostName": "host2", "upgradeDomain": "ud0"}
> {"hostName": "host3", "adminState": "DECOMMISSIONED"}
> {"hostName": "host4", "upgradeDomain": "ud2", "adminState":
> "DECOMMISSIONED"}
> {"hostName": "host5", "port": 8090}
> {"hostName": "host6", "adminState": "IN_MAINTENANCE"}
> {"hostName": "host7", "adminState": "IN_MAINTENANCE",
> "maintenanceExpireTimeInMS": "112233"}
> {noformat}
> Instead, to conform with the JSON standard it should be like
> {noformat}
> [
> {"hostName": "host1"},
> {"hostName": "host2", "upgradeDomain": "ud0"},
> {"hostName": "host3", "adminState": "DECOMMISSIONED"},
> {"hostName": "host4", "upgradeDomain": "ud2", "adminState":
> "DECOMMISSIONED"},
> {"hostName": "host5", "port": 8090},
> {"hostName": "host6", "adminState": "IN_MAINTENANCE"},
> {"hostName": "host7", "adminState": "IN_MAINTENANCE",
> "maintenanceExpireTimeInMS": "112233"}
> ]
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
