[
https://issues.apache.org/jira/browse/HDFS-7244?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Amir Langer updated HDFS-7244:
--
Attachment: SampleCode.tar.gz
HDFS-7244.docx
attached doc that describes an outline of
[
https://issues.apache.org/jira/browse/HDFS-7246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Amir Langer updated HDFS-7246:
--
Summary: Use ids for DatanodeStorageInfo in the BlockInfo triplets - HDFS
6660 (was: Use ids for
[
https://issues.apache.org/jira/browse/HDFS-7244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14178063#comment-14178063
]
Amir Langer commented on HDFS-7244:
---
Thanks for the comments [~cmccabe] all valid points.
[
https://issues.apache.org/jira/browse/HDFS-6658?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14178078#comment-14178078
]
Amir Langer commented on HDFS-6658:
---
Thanks [~shv]
I agree that the scenario of a remove
Amir Langer created HDFS-7244:
-
Summary: Reduce Namenode memory using Flyweight pattern
Key: HDFS-7244
URL: https://issues.apache.org/jira/browse/HDFS-7244
Project: Hadoop HDFS
Issue Type:
[
https://issues.apache.org/jira/browse/HDFS-7244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14170801#comment-14170801
]
Amir Langer commented on HDFS-7244:
---
Although not focusing on off-heap specifically this
Amir Langer created HDFS-7245:
-
Summary: Introduce Slab code in HDFS
Key: HDFS-7245
URL: https://issues.apache.org/jira/browse/HDFS-7245
Project: Hadoop HDFS
Issue Type: Sub-task
Amir Langer created HDFS-7246:
-
Summary: Use ids for DatanodeStorageInfo in the BlockInfo triplets
Key: HDFS-7246
URL: https://issues.apache.org/jira/browse/HDFS-7246
Project: Hadoop HDFS
Issue
Amir Langer created HDFS-7247:
-
Summary: Use ids for Block collection in Block
Key: HDFS-7247
URL: https://issues.apache.org/jira/browse/HDFS-7247
Project: Hadoop HDFS
Issue Type: Sub-task
Amir Langer created HDFS-7248:
-
Summary: Use ids for blocks in InodeFile
Key: HDFS-7248
URL: https://issues.apache.org/jira/browse/HDFS-7248
Project: Hadoop HDFS
Issue Type: Sub-task
Amir Langer created HDFS-7249:
-
Summary: Define block slabs per replication factor and initialise
them in advance (inc. size config)
Key: HDFS-7249
URL: https://issues.apache.org/jira/browse/HDFS-7249
Amir Langer created HDFS-7250:
-
Summary: Store blocks in slabs rather than a Map inside BlocksMap
Key: HDFS-7250
URL: https://issues.apache.org/jira/browse/HDFS-7250
Project: Hadoop HDFS
Issue
[
https://issues.apache.org/jira/browse/HDFS-7244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14170806#comment-14170806
]
Amir Langer commented on HDFS-7244:
---
Although orthogonal, HDFS-6658 is a first step in
[
https://issues.apache.org/jira/browse/HDFS-6658?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Amir Langer updated HDFS-6658:
--
Attachment: HDFS-6658.patch
Namenode memory optimization - Block replicas list
[
https://issues.apache.org/jira/browse/HDFS-6658?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Amir Langer updated HDFS-6658:
--
Status: Patch Available (was: Open)
Path includes all sub-tasks
Namenode memory optimization - Block
[
https://issues.apache.org/jira/browse/HDFS-6660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14128110#comment-14128110
]
Amir Langer commented on HDFS-6660:
---
Re: [~daryn]
Yes. It is an intermediate step but I
[
https://issues.apache.org/jira/browse/HDFS-6659?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Amir Langer updated HDFS-6659:
--
Labels: perfomance (was: )
Status: Patch Available (was: Open)
Create a Block List
[
https://issues.apache.org/jira/browse/HDFS-6659?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Amir Langer updated HDFS-6659:
--
Attachment: HDFS-6659.patch
Create a Block List
---
Key: HDFS-6659
[
https://issues.apache.org/jira/browse/HDFS-6659?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Amir Langer updated HDFS-6659:
--
Attachment: (was:
0001-BlockList-a-list-of-Blocks-that-saves-memory-by-mana.patch)
Create a Block
[
https://issues.apache.org/jira/browse/HDFS-6660?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Amir Langer updated HDFS-6660:
--
Attachment: (was:
0002-add-an-integer-id-to-all-storages-in-DatanodeManager.patch)
Use int instead
[
https://issues.apache.org/jira/browse/HDFS-6660?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Amir Langer updated HDFS-6660:
--
Labels: performance (was: )
Status: Patch Available (was: Open)
Use int instead of object
[
https://issues.apache.org/jira/browse/HDFS-6660?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Amir Langer updated HDFS-6660:
--
Attachment: HDFS-6660.patch
Use int instead of object reference to DatanodeStorageInfo in BlockInfo
[
https://issues.apache.org/jira/browse/HDFS-6659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14121339#comment-14121339
]
Amir Langer commented on HDFS-6659:
---
Hi [~vinayrpet]
1. It will be used by the patch I'm
[
https://issues.apache.org/jira/browse/HDFS-6658?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Amir Langer updated HDFS-6658:
--
Attachment: BlockListOptimizationComparison.xlsx
Added comparison of memory with and without
[
https://issues.apache.org/jira/browse/HDFS-6659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14096987#comment-14096987
]
Amir Langer commented on HDFS-6659:
---
Re: [~daryn]
The problems I had with using
[
https://issues.apache.org/jira/browse/HDFS-6658?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14075574#comment-14075574
]
Amir Langer commented on HDFS-6658:
---
Re [~drankye]: Using on or off heap memory is an
[
https://issues.apache.org/jira/browse/HDFS-6658?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14062498#comment-14062498
]
Amir Langer commented on HDFS-6658:
---
We explored the idea of off-heap memory for the
[
https://issues.apache.org/jira/browse/HDFS-6658?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14063072#comment-14063072
]
Amir Langer commented on HDFS-6658:
---
~szetszwo - yes I thought about using just one and
[
https://issues.apache.org/jira/browse/HDFS-6658?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14063077#comment-14063077
]
Amir Langer commented on HDFS-6658:
---
[~ cmccabe] When I looked at off heap I was using
[
https://issues.apache.org/jira/browse/HDFS-6658?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14060023#comment-14060023
]
Amir Langer commented on HDFS-6658:
---
Hi [~kihwal] - In response to the scenario of
Amir Langer created HDFS-6658:
-
Summary: Namenode memory optimization - Block replicas list
Key: HDFS-6658
URL: https://issues.apache.org/jira/browse/HDFS-6658
Project: Hadoop HDFS
Issue Type:
[
https://issues.apache.org/jira/browse/HDFS-6658?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Amir Langer updated HDFS-6658:
--
Attachment: Namenode Memory Optimizations - Block replicas list.docx
Design doc. + Evaluation results
Amir Langer created HDFS-6659:
-
Summary: Create a Block List
Key: HDFS-6659
URL: https://issues.apache.org/jira/browse/HDFS-6659
Project: Hadoop HDFS
Issue Type: Sub-task
Components:
Amir Langer created HDFS-6660:
-
Summary: Use int instead of object reference to
DatanodeStorageInfo in BlockInfo triplets,
Key: HDFS-6660
URL: https://issues.apache.org/jira/browse/HDFS-6660
Project:
Amir Langer created HDFS-6661:
-
Summary: Use BlockList instead of double linked list i.e
BlockInfo triplets
Key: HDFS-6661
URL: https://issues.apache.org/jira/browse/HDFS-6661
Project: Hadoop HDFS
[
https://issues.apache.org/jira/browse/HDFS-6659?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Amir Langer updated HDFS-6659:
--
Attachment: 0001-BlockList-a-list-of-Blocks-that-saves-memory-by-mana.patch
Create a Block List
[
https://issues.apache.org/jira/browse/HDFS-6660?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Amir Langer updated HDFS-6660:
--
Attachment: 0002-add-an-integer-id-to-all-storages-in-DatanodeManager.patch
Use int instead of object
[
https://issues.apache.org/jira/browse/HDFS-6658?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14057385#comment-14057385
]
Amir Langer commented on HDFS-6658:
---
Attached patches for every sub-task.
Namenode
[
https://issues.apache.org/jira/browse/HDFS-6661?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Amir Langer updated HDFS-6661:
--
Attachment: 0003-use-BlockList-instead-of-a-double-linked-list-of-blo.patch
Use BlockList instead of
[
https://issues.apache.org/jira/browse/HDFS-5477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Amir Langer updated HDFS-5477:
--
Attachment: (was: Block Manager as a Service - Implementation
decisions.pdf)
Block manager as a
[
https://issues.apache.org/jira/browse/HDFS-5477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Amir Langer updated HDFS-5477:
--
Attachment: (was: Remote BM.pdf)
Block manager as a service
--
[
https://issues.apache.org/jira/browse/HDFS-5477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Amir Langer updated HDFS-5477:
--
Attachment: (was: patches.tar.gz)
Block manager as a service
--
[
https://issues.apache.org/jira/browse/HDFS-6552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Amir Langer updated HDFS-6552:
--
Target Version/s: 2.4.0
Status: Patch Available (was: Open)
added tests that cover this
[
https://issues.apache.org/jira/browse/HDFS-6552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Amir Langer updated HDFS-6552:
--
Status: Open (was: Patch Available)
add DN storage to a BlockInfo will not replace the different
[
https://issues.apache.org/jira/browse/HDFS-6552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Amir Langer updated HDFS-6552:
--
Attachment: HDFS-6552.patch
patch - added tests that cover this scenario and fixed the issue.
add DN
[
https://issues.apache.org/jira/browse/HDFS-6552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Amir Langer updated HDFS-6552:
--
Status: Patch Available (was: Open)
Not sure what's the correct procedure - but I attached the patch
Amir Langer created HDFS-6552:
-
Summary: add DN storage to a BlockInfo will not replace the
different storage from same DN
Key: HDFS-6552
URL: https://issues.apache.org/jira/browse/HDFS-6552
Project:
[
https://issues.apache.org/jira/browse/HDFS-6552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Amir Langer updated HDFS-6552:
--
Description:
In BlockInfo - addStorage code looks wrong.
At line 10 (below) - we remove the storage
[
https://issues.apache.org/jira/browse/HDFS-6552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Amir Langer updated HDFS-6552:
--
Description:
In BlockInfo - addStorage code looks wrong.
At line 10 (below) - we remove the storage
[
https://issues.apache.org/jira/browse/HDFS-6552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Amir Langer updated HDFS-6552:
--
Description:
In BlockInfo - addStorage code looks wrong.
At line 10 (below) - we remove the storage
[
https://issues.apache.org/jira/browse/HDFS-6552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Amir Langer updated HDFS-6552:
--
Description:
In BlockInfo - addStorage code looks wrong.
At line 10 (below) - we remove the storage
[
https://issues.apache.org/jira/browse/HDFS-5477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13977968#comment-13977968
]
Amir Langer commented on HDFS-5477:
---
Hi Milind,
I believe the work - though it is
[
https://issues.apache.org/jira/browse/HDFS-5477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13961208#comment-13961208
]
Amir Langer commented on HDFS-5477:
---
I am on vacation until April 20th with no access to
[
https://issues.apache.org/jira/browse/HDFS-5477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Amir Langer updated HDFS-5477:
--
Attachment: Remote BM.pdf
Attached design doc. RemoteBM.pdf to its JIRA
Block manager as a service
[
https://issues.apache.org/jira/browse/HDFS-5477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Amir Langer updated HDFS-5477:
--
Attachment: Block Manager as a Service - Implementation decisions.pdf
Attached some more documentation
[
https://issues.apache.org/jira/browse/HDFS-5477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Amir Langer updated HDFS-5477:
--
Attachment: patches.tar.gz
All 38 patches which were used to create the 2 patches attached to the first
[
https://issues.apache.org/jira/browse/HDFS-5477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13919397#comment-13919397
]
Amir Langer commented on HDFS-5477:
---
Sorry - Previous comment meant to begin - All 38
[
https://issues.apache.org/jira/browse/HDFS-5732?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Amir Langer updated HDFS-5732:
--
Attachment: 0002-Separation-of-BM-from-NN-Step-2-Separate-memory-spac.patch
incremental patch after
[
https://issues.apache.org/jira/browse/HDFS-5732?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Amir Langer updated HDFS-5732:
--
Attachment: (was:
0002-Separation-of-BM-from-NN-Step-2-Separate-memory-spac.patch)
Separate
[
https://issues.apache.org/jira/browse/HDFS-5732?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Amir Langer updated HDFS-5732:
--
Attachment: 0002-Separation-of-BM-from-NN-Step-2-Separate-memory-spac.patch
Separation of memory space
[
https://issues.apache.org/jira/browse/HDFS-5731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Amir Langer updated HDFS-5731:
--
Description:
Start the separation of BlockManager (BM) from NameNode (NN) by simplifying the
flow
[
https://issues.apache.org/jira/browse/HDFS-5731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Amir Langer updated HDFS-5731:
--
Description:
Start the separation of BlockManager (BM) from NameNode (NN) by simplifying the
flow
Amir Langer created HDFS-5731:
-
Summary: Refactoring to define interfaces between BM and NN and
simplify the flow between them
Key: HDFS-5731
URL: https://issues.apache.org/jira/browse/HDFS-5731
Project:
[
https://issues.apache.org/jira/browse/HDFS-5731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Amir Langer updated HDFS-5731:
--
Description:
Start the separation of BlockManager (BM) from NameNode (NN) by simplifying the
flow
Amir Langer created HDFS-5732:
-
Summary: Separate memory space between BM and NN
Key: HDFS-5732
URL: https://issues.apache.org/jira/browse/HDFS-5732
Project: Hadoop HDFS
Issue Type: Sub-task
Amir Langer created HDFS-5733:
-
Summary: Separate concurrency control between BM and NN
Key: HDFS-5733
URL: https://issues.apache.org/jira/browse/HDFS-5733
Project: Hadoop HDFS
Issue Type:
Amir Langer created HDFS-5734:
-
Summary: A NN-internal RPC BM service
Key: HDFS-5734
URL: https://issues.apache.org/jira/browse/HDFS-5734
Project: Hadoop HDFS
Issue Type: Sub-task
Amir Langer created HDFS-5735:
-
Summary: Testing support for BM as a service
Key: HDFS-5735
URL: https://issues.apache.org/jira/browse/HDFS-5735
Project: Hadoop HDFS
Issue Type: Sub-task
Amir Langer created HDFS-5736:
-
Summary: BM service as a separate process
Key: HDFS-5736
URL: https://issues.apache.org/jira/browse/HDFS-5736
Project: Hadoop HDFS
Issue Type: Sub-task
[
https://issues.apache.org/jira/browse/HDFS-5731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Amir Langer updated HDFS-5731:
--
Attachment: 0001-Separation-of-BM-from-NN-Step1-introduce-APIs-as-int.patch
patch are changes done on
[
https://issues.apache.org/jira/browse/HDFS-5734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13866342#comment-13866342
]
Amir Langer commented on HDFS-5734:
---
BM = Block Manager
A NN-internal RPC BM service
[
https://issues.apache.org/jira/browse/HDFS-5731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Amir Langer updated HDFS-5731:
--
Description:
Start the separation of BlockManager (BM) from NameNode (NN) by simplifying the
flow
[
https://issues.apache.org/jira/browse/HDFS-5453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13858664#comment-13858664
]
Amir Langer commented on HDFS-5453:
---
The ASYNC performance of get block locations with
[
https://issues.apache.org/jira/browse/HDFS-5453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13856257#comment-13856257
]
Amir Langer commented on HDFS-5453:
---
Hi [~sureshms]
I worked on the async simulation
74 matches
Mail list logo
