[jira] [Updated] (HDFS-12591) [READ] Implement LevelDBFileRegionFormat
[
https://issues.apache.org/jira/browse/HDFS-12591?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Virajith Jalaparti updated HDFS-12591:
--
Hadoop Flags: Reviewed
> [READ] Implement LevelDBFileRegionFormat
>
>
> Key: HDFS-12591
> URL: https://issues.apache.org/jira/browse/HDFS-12591
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: hdfs
>Reporter: Ewan Higgs
>Assignee: Ewan Higgs
>Priority: Minor
> Fix For: 3.1.0
>
> Attachments: HDFS-12591-HDFS-9806.001.patch,
> HDFS-12591-HDFS-9806.002.patch, HDFS-12591-HDFS-9806.003.patch,
> HDFS-12591-HDFS-9806.004.patch, HDFS-12591-HDFS-9806.005.patch,
> HDFS-12591-HDFS-9806.006.patch, HDFS-12591-HDFS-9806.007.patch
>
>
> The existing work for HDFS-9806 uses an implementation of the {{FileRegion}}
> read from a csv file. This is good for testability and diagnostic purposes,
> but it is not very efficient for larger systems.
> There should be a version that is similar to the {{TextFileRegionFormat}}
> that instead uses LevelDB.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12591) [READ] Implement LevelDBFileRegionFormat
[
https://issues.apache.org/jira/browse/HDFS-12591?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Virajith Jalaparti updated HDFS-12591:
--
Status: Patch Available (was: Open)
> [READ] Implement LevelDBFileRegionFormat
>
>
> Key: HDFS-12591
> URL: https://issues.apache.org/jira/browse/HDFS-12591
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: hdfs
>Reporter: Ewan Higgs
>Assignee: Ewan Higgs
>Priority: Minor
> Attachments: HDFS-12591-HDFS-9806.001.patch,
> HDFS-12591-HDFS-9806.002.patch, HDFS-12591-HDFS-9806.003.patch,
> HDFS-12591-HDFS-9806.004.patch, HDFS-12591-HDFS-9806.005.patch
>
>
> The existing work for HDFS-9806 uses an implementation of the {{FileRegion}}
> read from a csv file. This is good for testability and diagnostic purposes,
> but it is not very efficient for larger systems.
> There should be a version that is similar to the {{TextFileRegionFormat}}
> that instead uses LevelDB.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12591) [READ] Implement LevelDBFileRegionFormat
[
https://issues.apache.org/jira/browse/HDFS-12591?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Virajith Jalaparti updated HDFS-12591:
--
Status: Open (was: Patch Available)
> [READ] Implement LevelDBFileRegionFormat
>
>
> Key: HDFS-12591
> URL: https://issues.apache.org/jira/browse/HDFS-12591
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: hdfs
>Reporter: Ewan Higgs
>Assignee: Ewan Higgs
>Priority: Minor
> Attachments: HDFS-12591-HDFS-9806.001.patch,
> HDFS-12591-HDFS-9806.002.patch, HDFS-12591-HDFS-9806.003.patch,
> HDFS-12591-HDFS-9806.004.patch, HDFS-12591-HDFS-9806.005.patch
>
>
> The existing work for HDFS-9806 uses an implementation of the {{FileRegion}}
> read from a csv file. This is good for testability and diagnostic purposes,
> but it is not very efficient for larger systems.
> There should be a version that is similar to the {{TextFileRegionFormat}}
> that instead uses LevelDB.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12591) [READ] Implement LevelDBFileRegionFormat
[
https://issues.apache.org/jira/browse/HDFS-12591?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Virajith Jalaparti updated HDFS-12591:
--
Attachment: HDFS-12591-HDFS-9806.005.patch
Fixing the checkstyle and findbugs issues in patch v4 in patch v5.
> [READ] Implement LevelDBFileRegionFormat
>
>
> Key: HDFS-12591
> URL: https://issues.apache.org/jira/browse/HDFS-12591
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: hdfs
>Reporter: Ewan Higgs
>Assignee: Ewan Higgs
>Priority: Minor
> Attachments: HDFS-12591-HDFS-9806.001.patch,
> HDFS-12591-HDFS-9806.002.patch, HDFS-12591-HDFS-9806.003.patch,
> HDFS-12591-HDFS-9806.004.patch, HDFS-12591-HDFS-9806.005.patch
>
>
> The existing work for HDFS-9806 uses an implementation of the {{FileRegion}}
> read from a csv file. This is good for testability and diagnostic purposes,
> but it is not very efficient for larger systems.
> There should be a version that is similar to the {{TextFileRegionFormat}}
> that instead uses LevelDB.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12591) [READ] Implement LevelDBFileRegionFormat
[
https://issues.apache.org/jira/browse/HDFS-12591?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Virajith Jalaparti updated HDFS-12591:
--
Status: Patch Available (was: Open)
> [READ] Implement LevelDBFileRegionFormat
>
>
> Key: HDFS-12591
> URL: https://issues.apache.org/jira/browse/HDFS-12591
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: hdfs
>Reporter: Ewan Higgs
>Assignee: Ewan Higgs
>Priority: Minor
> Attachments: HDFS-12591-HDFS-9806.001.patch,
> HDFS-12591-HDFS-9806.002.patch, HDFS-12591-HDFS-9806.003.patch,
> HDFS-12591-HDFS-9806.004.patch, HDFS-12591-HDFS-9806.005.patch
>
>
> The existing work for HDFS-9806 uses an implementation of the {{FileRegion}}
> read from a csv file. This is good for testability and diagnostic purposes,
> but it is not very efficient for larger systems.
> There should be a version that is similar to the {{TextFileRegionFormat}}
> that instead uses LevelDB.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12591) [READ] Implement LevelDBFileRegionFormat
[
https://issues.apache.org/jira/browse/HDFS-12591?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Virajith Jalaparti updated HDFS-12591:
--
Status: Open (was: Patch Available)
> [READ] Implement LevelDBFileRegionFormat
>
>
> Key: HDFS-12591
> URL: https://issues.apache.org/jira/browse/HDFS-12591
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: hdfs
>Reporter: Ewan Higgs
>Assignee: Ewan Higgs
>Priority: Minor
> Attachments: HDFS-12591-HDFS-9806.001.patch,
> HDFS-12591-HDFS-9806.002.patch, HDFS-12591-HDFS-9806.003.patch,
> HDFS-12591-HDFS-9806.004.patch, HDFS-12591-HDFS-9806.005.patch
>
>
> The existing work for HDFS-9806 uses an implementation of the {{FileRegion}}
> read from a csv file. This is good for testability and diagnostic purposes,
> but it is not very efficient for larger systems.
> There should be a version that is similar to the {{TextFileRegionFormat}}
> that instead uses LevelDB.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12591) [READ] Implement LevelDBFileRegionFormat
[
https://issues.apache.org/jira/browse/HDFS-12591?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Virajith Jalaparti updated HDFS-12591:
--
Attachment: HDFS-12591-HDFS-9806.004.patch
Posting a new patch (v4) that rebases v3 on the latest feature branch, and
removes the shared reference to the leveldb between {{LevelDBReader}} and
{{LevelDBWriter}}. The reader and writer are not intended to be used
simultaneously as provided storage just supports read-only as of now (hence,
removed {{testSimultaneousWriteAndRead}}). [~ehiggs], can you take a look at
this?
> [READ] Implement LevelDBFileRegionFormat
>
>
> Key: HDFS-12591
> URL: https://issues.apache.org/jira/browse/HDFS-12591
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: hdfs
>Reporter: Ewan Higgs
>Assignee: Ewan Higgs
>Priority: Minor
> Attachments: HDFS-12591-HDFS-9806.001.patch,
> HDFS-12591-HDFS-9806.002.patch, HDFS-12591-HDFS-9806.003.patch,
> HDFS-12591-HDFS-9806.004.patch
>
>
> The existing work for HDFS-9806 uses an implementation of the {{FileRegion}}
> read from a csv file. This is good for testability and diagnostic purposes,
> but it is not very efficient for larger systems.
> There should be a version that is similar to the {{TextFileRegionFormat}}
> that instead uses LevelDB.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12591) [READ] Implement LevelDBFileRegionFormat
[
https://issues.apache.org/jira/browse/HDFS-12591?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ewan Higgs updated HDFS-12591:
--
Attachment: HDFS-12591-HDFS-9806.003.patch
Attaching a patch rebased on HDFS-12665 so the two LevelDB alias maps (file
based and NN in-memory) use the same code for reading and writing data.
> [READ] Implement LevelDBFileRegionFormat
>
>
> Key: HDFS-12591
> URL: https://issues.apache.org/jira/browse/HDFS-12591
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: hdfs
>Reporter: Ewan Higgs
>Assignee: Ewan Higgs
>Priority: Minor
> Attachments: HDFS-12591-HDFS-9806.001.patch,
> HDFS-12591-HDFS-9806.002.patch, HDFS-12591-HDFS-9806.003.patch
>
>
> The existing work for HDFS-9806 uses an implementation of the {{FileRegion}}
> read from a csv file. This is good for testability and diagnostic purposes,
> but it is not very efficient for larger systems.
> There should be a version that is similar to the {{TextFileRegionFormat}}
> that instead uses LevelDB.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12591) [READ] Implement LevelDBFileRegionFormat
[
https://issues.apache.org/jira/browse/HDFS-12591?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Virajith Jalaparti updated HDFS-12591:
--
Attachment: HDFS-12591-HDFS-9806.002.patch
Posting a modified version of [~ehiggs]'s patch, rebased on the latest
HDFS-9806 feature branch. This applies on top of the HDFS-11902 v009 patch.
I also fixed the memory leak issues pointed out by [~jeagles]. Thanks
[~jeagles]!
> [READ] Implement LevelDBFileRegionFormat
>
>
> Key: HDFS-12591
> URL: https://issues.apache.org/jira/browse/HDFS-12591
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: hdfs
>Reporter: Ewan Higgs
>Priority: Minor
> Attachments: HDFS-12591-HDFS-9806.001.patch,
> HDFS-12591-HDFS-9806.002.patch
>
>
> The existing work for HDFS-9806 uses an implementation of the {{FileRegion}}
> read from a csv file. This is good for testability and diagnostic purposes,
> but it is not very efficient for larger systems.
> There should be a version that is similar to the {{TextFileRegionFormat}}
> that instead uses LevelDB.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12591) [READ] Implement LevelDBFileRegionFormat
[
https://issues.apache.org/jira/browse/HDFS-12591?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ewan Higgs updated HDFS-12591:
--
Attachment: HDFS-12591-HDFS-9806.001.patch
Attaching work I've previously done for this. It needs to be rebased onto the
HDFS-9806 branch.
> [READ] Implement LevelDBFileRegionFormat
>
>
> Key: HDFS-12591
> URL: https://issues.apache.org/jira/browse/HDFS-12591
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: hdfs
>Reporter: Ewan Higgs
>Priority: Minor
> Attachments: HDFS-12591-HDFS-9806.001.patch
>
>
> The existing work for HDFS-9806 uses an implementation of the {{FileRegion}}
> read from a csv file. This is good for testability and diagnostic purposes,
> but it is not very efficient for larger systems.
> There should be a version that is similar to the {{TextFileRegionFormat}}
> that instead uses LevelDB.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12591) [READ] Implement LevelDBFileRegionFormat
[
https://issues.apache.org/jira/browse/HDFS-12591?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ewan Higgs updated HDFS-12591:
--
Status: Patch Available (was: Open)
> [READ] Implement LevelDBFileRegionFormat
>
>
> Key: HDFS-12591
> URL: https://issues.apache.org/jira/browse/HDFS-12591
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: hdfs
>Reporter: Ewan Higgs
>Priority: Minor
> Attachments: HDFS-12591-HDFS-9806.001.patch
>
>
> The existing work for HDFS-9806 uses an implementation of the {{FileRegion}}
> read from a csv file. This is good for testability and diagnostic purposes,
> but it is not very efficient for larger systems.
> There should be a version that is similar to the {{TextFileRegionFormat}}
> that instead uses LevelDB.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
