[jira] [Commented] (HDFS-1094) Intelligent block placement policy to decrease probability of block loss

[ https://issues.apache.org/jira/browse/HDFS-1094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14079618#comment-14079618 ] Allen Wittenauer commented on HDFS-1094: With pluggable block placement, I'm not

[jira] Commented: (HDFS-1094) Intelligent block placement policy to decrease probability of block loss

[ https://issues.apache.org/jira/browse/HDFS-1094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12891837#action_12891837 ] Konstantin Shvachko commented on HDFS-1094: --- We can ignore rack failure, which is

[jira] Commented: (HDFS-1094) Intelligent block placement policy to decrease probability of block loss

[ https://issues.apache.org/jira/browse/HDFS-1094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12889910#action_12889910 ] Scott Carey commented on HDFS-1094: --- This needs to change p from a constant, to a function

[jira] Commented: (HDFS-1094) Intelligent block placement policy to decrease probability of block loss

[ https://issues.apache.org/jira/browse/HDFS-1094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12889976#action_12889976 ] Rodrigo Schmidt commented on HDFS-1094: --- Hi Scott, I totally understand your

[jira] Commented: (HDFS-1094) Intelligent block placement policy to decrease probability of block loss

[ https://issues.apache.org/jira/browse/HDFS-1094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=1289#action_1289 ] Scott Carey commented on HDFS-1094: --- bq. I think we are more interested in permanent

[jira] Commented: (HDFS-1094) Intelligent block placement policy to decrease probability of block loss

[ https://issues.apache.org/jira/browse/HDFS-1094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12889732#action_12889732 ] Rodrigo Schmidt commented on HDFS-1094: --- I got permission from Alex Smith to post his

[jira] Commented: (HDFS-1094) Intelligent block placement policy to decrease probability of block loss

[ https://issues.apache.org/jira/browse/HDFS-1094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12887866#action_12887866 ] Tsz Wo (Nicholas), SZE commented on HDFS-1094: -- Thanks Dhruba for the clear

[jira] Commented: (HDFS-1094) Intelligent block placement policy to decrease probability of block loss

[ https://issues.apache.org/jira/browse/HDFS-1094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12887151#action_12887151 ] dhruba borthakur commented on HDFS-1094: This is a really an excellent discussion

[jira] Commented: (HDFS-1094) Intelligent block placement policy to decrease probability of block loss

[ https://issues.apache.org/jira/browse/HDFS-1094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12886998#action_12886998 ] Rodrigo Schmidt commented on HDFS-1094: --- I might be wrong, but it seems to me that you

[jira] Commented: (HDFS-1094) Intelligent block placement policy to decrease probability of block loss

[ https://issues.apache.org/jira/browse/HDFS-1094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12887012#action_12887012 ] Joydeep Sen Sarma commented on HDFS-1094: - (this is the closest i will get to

[jira] Commented: (HDFS-1094) Intelligent block placement policy to decrease probability of block loss

[ https://issues.apache.org/jira/browse/HDFS-1094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12887061#action_12887061 ] Rodrigo Schmidt commented on HDFS-1094: --- According to your definition, what prevents

[jira] Commented: (HDFS-1094) Intelligent block placement policy to decrease probability of block loss

[ https://issues.apache.org/jira/browse/HDFS-1094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12887075#action_12887075 ] Joydeep Sen Sarma commented on HDFS-1094: - @Rodrigo - it's evident u have not read

[jira] Commented: (HDFS-1094) Intelligent block placement policy to decrease probability of block loss

[ https://issues.apache.org/jira/browse/HDFS-1094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12886632#action_12886632 ] Rodrigo Schmidt commented on HDFS-1094: --- Hairong, would you mind explaining how this

[jira] Commented: (HDFS-1094) Intelligent block placement policy to decrease probability of block loss

[ https://issues.apache.org/jira/browse/HDFS-1094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12886768#action_12886768 ] Tsz Wo (Nicholas), SZE commented on HDFS-1094: -- Here the assumption is,

[jira] Commented: (HDFS-1094) Intelligent block placement policy to decrease probability of block loss

[ https://issues.apache.org/jira/browse/HDFS-1094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12886771#action_12886771 ] Joydeep Sen Sarma commented on HDFS-1094: - @Rodrigo - how about using a term to

[jira] Commented: (HDFS-1094) Intelligent block placement policy to decrease probability of block loss

[ https://issues.apache.org/jira/browse/HDFS-1094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12886783#action_12886783 ] Joydeep Sen Sarma commented on HDFS-1094: - regarding T and simulataneous failures:

[jira] Commented: (HDFS-1094) Intelligent block placement policy to decrease probability of block loss

[ https://issues.apache.org/jira/browse/HDFS-1094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12886797#action_12886797 ] Scott Carey commented on HDFS-1094: --- bq. I've had some general discussions with Dhruba

[jira] Commented: (HDFS-1094) Intelligent block placement policy to decrease probability of block loss

[ https://issues.apache.org/jira/browse/HDFS-1094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12886823#action_12886823 ] Rodrigo Schmidt commented on HDFS-1094: --- @Joydeep: Using a term to describe the groups

[jira] Commented: (HDFS-1094) Intelligent block placement policy to decrease probability of block loss

[ https://issues.apache.org/jira/browse/HDFS-1094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12886861#action_12886861 ] Joydeep Sen Sarma commented on HDFS-1094: - one can't have a different node-group for

[jira] Commented: (HDFS-1094) Intelligent block placement policy to decrease probability of block loss

[ https://issues.apache.org/jira/browse/HDFS-1094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12886867#action_12886867 ] Rodrigo Schmidt commented on HDFS-1094: --- I'm not sure I understand what you mean by

[jira] Commented: (HDFS-1094) Intelligent block placement policy to decrease probability of block loss

[ https://issues.apache.org/jira/browse/HDFS-1094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12886900#action_12886900 ] Konstantin Shvachko commented on HDFS-1094: --- Rodrigo, I understand now that you

[jira] Commented: (HDFS-1094) Intelligent block placement policy to decrease probability of block loss

[ https://issues.apache.org/jira/browse/HDFS-1094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12886903#action_12886903 ] Joydeep Sen Sarma commented on HDFS-1094: - my mental model is that a node group is a

[jira] Commented: (HDFS-1094) Intelligent block placement policy to decrease probability of block loss

[ https://issues.apache.org/jira/browse/HDFS-1094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12886976#action_12886976 ] Joydeep Sen Sarma commented on HDFS-1094: - - Pick random rack r2 that is within R

[jira] Commented: (HDFS-1094) Intelligent block placement policy to decrease probability of block loss

[ https://issues.apache.org/jira/browse/HDFS-1094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12886979#action_12886979 ] Rodrigo Schmidt commented on HDFS-1094: --- @Konstantin: We have a script that calculates

[jira] Commented: (HDFS-1094) Intelligent block placement policy to decrease probability of block loss

[ https://issues.apache.org/jira/browse/HDFS-1094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12886993#action_12886993 ] Joydeep Sen Sarma commented on HDFS-1094: - :-) - of course i meant a node group =

[jira] Commented: (HDFS-1094) Intelligent block placement policy to decrease probability of block loss

[ https://issues.apache.org/jira/browse/HDFS-1094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12886254#action_12886254 ] Hairong Kuang commented on HDFS-1094: - I did not mean #racks per block. I mean racks for

[jira] Commented: (HDFS-1094) Intelligent block placement policy to decrease probability of block loss

[ https://issues.apache.org/jira/browse/HDFS-1094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12886257#action_12886257 ] Rodrigo Schmidt commented on HDFS-1094: --- As Joydeep wrote, we didn't think this was a

[jira] Commented: (HDFS-1094) Intelligent block placement policy to decrease probability of block loss

[ https://issues.apache.org/jira/browse/HDFS-1094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12886363#action_12886363 ] Hairong Kuang commented on HDFS-1094: - For a large file, it does matter especially in

[jira] Commented: (HDFS-1094) Intelligent block placement policy to decrease probability of block loss

[ https://issues.apache.org/jira/browse/HDFS-1094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12886418#action_12886418 ] Rodrigo Schmidt commented on HDFS-1094: --- I don't think I understand your use case. It

[jira] Commented: (HDFS-1094) Intelligent block placement policy to decrease probability of block loss

[ https://issues.apache.org/jira/browse/HDFS-1094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12886430#action_12886430 ] Konstantin Shvachko commented on HDFS-1094: --- The math looks good to me (in pdf

[jira] Commented: (HDFS-1094) Intelligent block placement policy to decrease probability of block loss

[ https://issues.apache.org/jira/browse/HDFS-1094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12886521#action_12886521 ] Rodrigo Schmidt commented on HDFS-1094: --- @Konstantin: The new policy will have, for

[jira] Commented: (HDFS-1094) Intelligent block placement policy to decrease probability of block loss

[ https://issues.apache.org/jira/browse/HDFS-1094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12886595#action_12886595 ] Hairong Kuang commented on HDFS-1094: - I don't think I understand your use case. It

[jira] Commented: (HDFS-1094) Intelligent block placement policy to decrease probability of block loss

[ https://issues.apache.org/jira/browse/HDFS-1094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12886599#action_12886599 ] Rodrigo Schmidt commented on HDFS-1094: --- I think I get your point. I don't think you

[jira] Commented: (HDFS-1094) Intelligent block placement policy to decrease probability of block loss

[ https://issues.apache.org/jira/browse/HDFS-1094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12886600#action_12886600 ] Rodrigo Schmidt commented on HDFS-1094: --- Just complementing my previous comment, I

[jira] Commented: (HDFS-1094) Intelligent block placement policy to decrease probability of block loss

[ https://issues.apache.org/jira/browse/HDFS-1094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12886613#action_12886613 ] Hairong Kuang commented on HDFS-1094: - Hi Rodrigo, I do not know your algorithm so I

[jira] Commented: (HDFS-1094) Intelligent block placement policy to decrease probability of block loss

[ https://issues.apache.org/jira/browse/HDFS-1094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12885844#action_12885844 ] Rodrigo Schmidt commented on HDFS-1094: --- @Hairong: Just a few observations 1) The

[jira] Commented: (HDFS-1094) Intelligent block placement policy to decrease probability of block loss

[ https://issues.apache.org/jira/browse/HDFS-1094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12885845#action_12885845 ] Hairong Kuang commented on HDFS-1094: - Right, in case of a map/reduce application,

[jira] Commented: (HDFS-1094) Intelligent block placement policy to decrease probability of block loss

[ https://issues.apache.org/jira/browse/HDFS-1094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12885848#action_12885848 ] Hairong Kuang commented on HDFS-1094: - Rodrigo, thanks for your explanation. Now I

[jira] Commented: (HDFS-1094) Intelligent block placement policy to decrease probability of block loss

[ https://issues.apache.org/jira/browse/HDFS-1094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12885849#action_12885849 ] Rodrigo Schmidt commented on HDFS-1094: --- The assumption that failures are independent

[jira] Commented: (HDFS-1094) Intelligent block placement policy to decrease probability of block loss

[ https://issues.apache.org/jira/browse/HDFS-1094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12885852#action_12885852 ] Joydeep Sen Sarma commented on HDFS-1094: - but map-reduce programs rarely read from

[jira] Commented: (HDFS-1094) Intelligent block placement policy to decrease probability of block loss

[ https://issues.apache.org/jira/browse/HDFS-1094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12886244#action_12886244 ] Hairong Kuang commented on HDFS-1094: - More questions... :) I understand that the

[jira] Commented: (HDFS-1094) Intelligent block placement policy to decrease probability of block loss

[ https://issues.apache.org/jira/browse/HDFS-1094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12886246#action_12886246 ] Rodrigo Schmidt commented on HDFS-1094: --- The higher the number of racks, the higher

[jira] Commented: (HDFS-1094) Intelligent block placement policy to decrease probability of block loss

[ https://issues.apache.org/jira/browse/HDFS-1094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12885711#action_12885711 ] Hairong Kuang commented on HDFS-1094: - This policy seems to decrease the aggregate read

[jira] Commented: (HDFS-1094) Intelligent block placement policy to decrease probability of block loss

[ https://issues.apache.org/jira/browse/HDFS-1094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12885765#action_12885765 ] Rodrigo Schmidt commented on HDFS-1094: --- @Hairong: I'm not sure if I understand your

[jira] Commented: (HDFS-1094) Intelligent block placement policy to decrease probability of block loss

[ https://issues.apache.org/jira/browse/HDFS-1094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12885837#action_12885837 ] dhruba borthakur commented on HDFS-1094: Hi hairong; does your comment about

[jira] Commented: (HDFS-1094) Intelligent block placement policy to decrease probability of block loss

[ https://issues.apache.org/jira/browse/HDFS-1094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12885836#action_12885836 ] Hairong Kuang commented on HDFS-1094: - Suppose a file has N blocks. The proposed policy

[jira] Commented: (HDFS-1094) Intelligent block placement policy to decrease probability of block loss

[ https://issues.apache.org/jira/browse/HDFS-1094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12882774#action_12882774 ] Tsz Wo (Nicholas), SZE commented on HDFS-1094: -- Quoted from prob.pdf: {quote}

[jira] Commented: (HDFS-1094) Intelligent block placement policy to decrease probability of block loss

[ https://issues.apache.org/jira/browse/HDFS-1094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12882788#action_12882788 ] Rodrigo Schmidt commented on HDFS-1094: --- There is a number of simplifications there,

[jira] Commented: (HDFS-1094) Intelligent block placement policy to decrease probability of block loss

[ https://issues.apache.org/jira/browse/HDFS-1094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12882794#action_12882794 ] Tsz Wo (Nicholas), SZE commented on HDFS-1094: -- There is a number of

[jira] Commented: (HDFS-1094) Intelligent block placement policy to decrease probability of block loss

[ https://issues.apache.org/jira/browse/HDFS-1094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12882795#action_12882795 ] Rodrigo Schmidt commented on HDFS-1094: --- That might make it hard to start reading,

[jira] Commented: (HDFS-1094) Intelligent block placement policy to decrease probability of block loss

[ https://issues.apache.org/jira/browse/HDFS-1094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12882798#action_12882798 ] Koji Noguchi commented on HDFS-1094: Just curious. Are you getting missing/corrupt

[jira] Commented: (HDFS-1094) Intelligent block placement policy to decrease probability of block loss

[ https://issues.apache.org/jira/browse/HDFS-1094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12878534#action_12878534 ] Rodrigo Schmidt commented on HDFS-1094: --- For this JIRA I'm implementing a general

[jira] Commented: (HDFS-1094) Intelligent block placement policy to decrease probability of block loss

[ https://issues.apache.org/jira/browse/HDFS-1094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12873238#action_12873238 ] dhruba borthakur commented on HDFS-1094: From all the above mathematical formulae,

[jira] Commented: (HDFS-1094) Intelligent block placement policy to decrease probability of block loss

[ https://issues.apache.org/jira/browse/HDFS-1094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12856510#action_12856510 ] Karthik Ranganathan commented on HDFS-1094: --- Hey Brian, Yes, that would help with

[jira] Commented: (HDFS-1094) Intelligent block placement policy to decrease probability of block loss

[ https://issues.apache.org/jira/browse/HDFS-1094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12856673#action_12856673 ] Konstantin Shvachko commented on HDFS-1094: --- P(of not losing a specific block) =

[jira] Commented: (HDFS-1094) Intelligent block placement policy to decrease probability of block loss

[ https://issues.apache.org/jira/browse/HDFS-1094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12856716#action_12856716 ] Jitendra Nath Pandey commented on HDFS-1094: I think you are throwing one

[jira] Commented: (HDFS-1094) Intelligent block placement policy to decrease probability of block loss

[ https://issues.apache.org/jira/browse/HDFS-1094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12855883#action_12855883 ] dhruba borthakur commented on HDFS-1094: This is more of a discussion on how to

[jira] Commented: (HDFS-1094) Intelligent block placement policy to decrease probability of block loss

[ https://issues.apache.org/jira/browse/HDFS-1094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12856139#action_12856139 ] Brian Bockelman commented on HDFS-1094: --- Hey Karthik, Let me play dumb (it might not

[jira] Commented: (HDFS-1094) Intelligent block placement policy to decrease probability of block loss

[ https://issues.apache.org/jira/browse/HDFS-1094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12856223#action_12856223 ] Brian Bockelman commented on HDFS-1094: --- A funny related anecdote that I've heard

[jira] Commented: (HDFS-1094) Intelligent block placement policy to decrease probability of block loss

[ https://issues.apache.org/jira/browse/HDFS-1094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12856269#action_12856269 ] Rodrigo Schmidt commented on HDFS-1094: --- Yes, if we want to minimize the chances of

[jira] Commented: (HDFS-1094) Intelligent block placement policy to decrease probability of block loss

[ https://issues.apache.org/jira/browse/HDFS-1094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12856271#action_12856271 ] Brian Bockelman commented on HDFS-1094: --- Hey Karthik, Why don't we take this to the