Hi Andrew,
Implementation:
The cache layer is written in Go and wraps around the DataNode so that
all traffic between the DataNode and NameNode as well as DataNode and
client flow through the cache layer. The layer currently employs a simple
LRU cache, where files that are requested are placed
Hi Dhaivat,
I did a good chunk of the design and implementation of HDFS-4949, so if you
could post a longer writeup of your envisioned use cases and
implementation, I'd definitely be interested in taking a look.
It's also good to note that HDFS-4949 is only the foundation for a whole
slew of
Any other ideas?
On Sun, Dec 22, 2013 at 10:38 PM, Dhaivat Pandya dhaivatpan...@gmail.comwrote:
I understand that that is how the port is *later* retrieved, but how does
the namenode know the port in the first place? i.e. if the datanode sends a
packet to the namenode, how does the namenode
Hi,
On Mon, Dec 23, 2013 at 9:41 AM, Dhaivat Pandya dhaivatpan...@gmail.com wrote:
Hi,
I'm currently trying to build a cache layer that should sit on top of the
datanode. Essentially, the namenode should know the port number of the
cache layer instead of that of the datanode (since the
Hi Harsh,
Thanks a lot for the response. As it turns out, I figured out the
registration mechanism this evening and how the sourceId is relayed to the
NN.
As for your question about the cache layer it is a similar basic concept as
the plan mentioned, but the technical details differ
Hi,
I'm currently trying to build a cache layer that should sit on top of the
datanode. Essentially, the namenode should know the port number of the
cache layer instead of that of the datanode (since the namenode then relays
this information to the default HDFS client). All of the communication
Check the version file
During the startup Datanode sends out a heart beat signal and NN
assigns a storageID for it which is unique for every datanode.
Using the storage id it identify the ip and port of a datanode
Regards,
Som Shekhar Sharma
+91-8197243810
On Mon, Dec 23, 2013 at 9:41 AM,
I understand that that is how the port is *later* retrieved, but how does
the namenode know the port in the first place? i.e. if the datanode sends a
packet to the namenode, how does the namenode know what port # the datanode
server is running on?
On Sun, Dec 22, 2013 at 10:28 PM, Shekhar Sharma
