[jira] [Commented] (YARN-5667) Move HBase backend code in ATS v2 into its separate module
[ https://issues.apache.org/jira/browse/YARN-5667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15743286#comment-15743286 ] Haibo Chen commented on YARN-5667: -- [~andrew.wang] We are making progress in the subtasks. One more subtask to go. > Move HBase backend code in ATS v2 into its separate module > --- > > Key: YARN-5667 > URL: https://issues.apache.org/jira/browse/YARN-5667 > Project: Hadoop YARN > Issue Type: Task > Components: yarn >Affects Versions: 3.0.0-alpha1 >Reporter: Haibo Chen >Assignee: Haibo Chen >Priority: Blocker > Attachments: New module structure.png, part1.yarn5667.prelim.patch, > part2.yarn5667.prelim.patch, part3.yarn5667.prelim.patch, > part4.yarn5667.prelim.patch, part5.yarn5667.prelim.patch, > pt1.yarn5667.001.patch, pt2.yarn5667.001.patch, pt3.yarn5667.001.patch, > pt4.yarn5667.001.patch, pt5.yarn5667.001.patch, pt6.yarn5667.001.patch, > pt9.yarn5667.001.patch, yarn5667-001.tar.gz > > > The HBase backend code currently lives along with the core ATS v2 code in > hadoop-yarn-server-timelineservice module. Because Resource Manager depends > on hadoop-yarn-server-timelineservice, an unnecessary dependency of the RM > module on HBase modules is introduced (HBase backend is pluggable, so we do > not need to directly pull in HBase jars). > In our internal effort to try ATS v2 with HBase 2.0 which depends on Hadoop > 3, we encountered a circular dependency during our builds between HBase2.0 > and Hadoop3 artifacts. > {code} > hadoop-mapreduce-client-common, hadoop-yarn-client, > hadoop-yarn-server-resourcemanager, hadoop-yarn-server-timelineservice, > hbase-server, hbase-prefix-tree, hbase-hadoop2-compat, > hadoop-mapreduce-client-jobclient, hadoop-mapreduce-client-common] > {code} > This jira proposes we move all HBase-backend-related code from > hadoop-yarn-server-timelineservice into its own module (possible name is > yarn-server-timelineservice-storage) so that core RM modules do not depend on > HBase modules any more. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5667) Move HBase backend code in ATS v2 into its separate module
[ https://issues.apache.org/jira/browse/YARN-5667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15736883#comment-15736883 ] Andrew Wang commented on YARN-5667: --- Ping [~haibochen], any update on this JIRA? > Move HBase backend code in ATS v2 into its separate module > --- > > Key: YARN-5667 > URL: https://issues.apache.org/jira/browse/YARN-5667 > Project: Hadoop YARN > Issue Type: Task > Components: yarn >Affects Versions: 3.0.0-alpha1 >Reporter: Haibo Chen >Assignee: Haibo Chen >Priority: Blocker > Attachments: New module structure.png, part1.yarn5667.prelim.patch, > part2.yarn5667.prelim.patch, part3.yarn5667.prelim.patch, > part4.yarn5667.prelim.patch, part5.yarn5667.prelim.patch, > pt1.yarn5667.001.patch, pt2.yarn5667.001.patch, pt3.yarn5667.001.patch, > pt4.yarn5667.001.patch, pt5.yarn5667.001.patch, pt6.yarn5667.001.patch, > pt9.yarn5667.001.patch, yarn5667-001.tar.gz > > > The HBase backend code currently lives along with the core ATS v2 code in > hadoop-yarn-server-timelineservice module. Because Resource Manager depends > on hadoop-yarn-server-timelineservice, an unnecessary dependency of the RM > module on HBase modules is introduced (HBase backend is pluggable, so we do > not need to directly pull in HBase jars). > In our internal effort to try ATS v2 with HBase 2.0 which depends on Hadoop > 3, we encountered a circular dependency during our builds between HBase2.0 > and Hadoop3 artifacts. > {code} > hadoop-mapreduce-client-common, hadoop-yarn-client, > hadoop-yarn-server-resourcemanager, hadoop-yarn-server-timelineservice, > hbase-server, hbase-prefix-tree, hbase-hadoop2-compat, > hadoop-mapreduce-client-jobclient, hadoop-mapreduce-client-common] > {code} > This jira proposes we move all HBase-backend-related code from > hadoop-yarn-server-timelineservice into its own module (possible name is > yarn-server-timelineservice-storage) so that core RM modules do not depend on > HBase modules any more. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5667) Move HBase backend code in ATS v2 into its separate module
[ https://issues.apache.org/jira/browse/YARN-5667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15684390#comment-15684390 ] Sangjin Lee commented on YARN-5667: --- Sounds good. Hopefully the changes between the trunk and the feature branch are not big enough, or the rebase would be adventurous. > Move HBase backend code in ATS v2 into its separate module > --- > > Key: YARN-5667 > URL: https://issues.apache.org/jira/browse/YARN-5667 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Affects Versions: 3.0.0-alpha1 >Reporter: Haibo Chen >Assignee: Haibo Chen >Priority: Blocker > Attachments: New module structure.png, part1.yarn5667.prelim.patch, > part2.yarn5667.prelim.patch, part3.yarn5667.prelim.patch, > part4.yarn5667.prelim.patch, part5.yarn5667.prelim.patch, > pt1.yarn5667.001.patch, pt2.yarn5667.001.patch, pt3.yarn5667.001.patch, > pt4.yarn5667.001.patch, pt5.yarn5667.001.patch, pt6.yarn5667.001.patch, > pt9.yarn5667.001.patch, yarn5667-001.tar.gz > > > The HBase backend code currently lives along with the core ATS v2 code in > hadoop-yarn-server-timelineservice module. Because Resource Manager depends > on hadoop-yarn-server-timelineservice, an unnecessary dependency of the RM > module on HBase modules is introduced (HBase backend is pluggable, so we do > not need to directly pull in HBase jars). > In our internal effort to try ATS v2 with HBase 2.0 which depends on Hadoop > 3, we encountered a circular dependency during our builds between HBase2.0 > and Hadoop3 artifacts. > {code} > hadoop-mapreduce-client-common, hadoop-yarn-client, > hadoop-yarn-server-resourcemanager, hadoop-yarn-server-timelineservice, > hbase-server, hbase-prefix-tree, hbase-hadoop2-compat, > hadoop-mapreduce-client-jobclient, hadoop-mapreduce-client-common] > {code} > This jira proposes we move all HBase-backend-related code from > hadoop-yarn-server-timelineservice into its own module (possible name is > yarn-server-timelineservice-storage) so that core RM modules do not depend on > HBase modules any more. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5667) Move HBase backend code in ATS v2 into its separate module
[ https://issues.apache.org/jira/browse/YARN-5667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15677340#comment-15677340 ] Haibo Chen commented on YARN-5667: -- My first version of the patch is actually created for trunk. Based on that experience, it should be manageable to create one for trunk and another for yarn-5355. To make things easier and move quicker, I'll break this change into a few pieces and create a subtask for each of them. > Move HBase backend code in ATS v2 into its separate module > --- > > Key: YARN-5667 > URL: https://issues.apache.org/jira/browse/YARN-5667 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Affects Versions: 3.0.0-alpha1 >Reporter: Haibo Chen >Assignee: Haibo Chen >Priority: Blocker > Attachments: New module structure.png, part1.yarn5667.prelim.patch, > part2.yarn5667.prelim.patch, part3.yarn5667.prelim.patch, > part4.yarn5667.prelim.patch, part5.yarn5667.prelim.patch, > pt1.yarn5667.001.patch, pt2.yarn5667.001.patch, pt3.yarn5667.001.patch, > pt4.yarn5667.001.patch, pt5.yarn5667.001.patch, pt6.yarn5667.001.patch, > pt9.yarn5667.001.patch, yarn5667-001.tar.gz > > > The HBase backend code currently lives along with the core ATS v2 code in > hadoop-yarn-server-timelineservice module. Because Resource Manager depends > on hadoop-yarn-server-timelineservice, an unnecessary dependency of the RM > module on HBase modules is introduced (HBase backend is pluggable, so we do > not need to directly pull in HBase jars). > In our internal effort to try ATS v2 with HBase 2.0 which depends on Hadoop > 3, we encountered a circular dependency during our builds between HBase2.0 > and Hadoop3 artifacts. > {code} > hadoop-mapreduce-client-common, hadoop-yarn-client, > hadoop-yarn-server-resourcemanager, hadoop-yarn-server-timelineservice, > hbase-server, hbase-prefix-tree, hbase-hadoop2-compat, > hadoop-mapreduce-client-jobclient, hadoop-mapreduce-client-common] > {code} > This jira proposes we move all HBase-backend-related code from > hadoop-yarn-server-timelineservice into its own module (possible name is > yarn-server-timelineservice-storage) so that core RM modules do not depend on > HBase modules any more. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5667) Move HBase backend code in ATS v2 into its separate module
[ https://issues.apache.org/jira/browse/YARN-5667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15675617#comment-15675617 ] Sangjin Lee commented on YARN-5667: --- Originally this was targeted for our feature branch (YARN-5355) which isn't merging to 3.0.0-alpha2. We can pull this in for 3.0.0-alpha2 if needed. The implication then is, we need to prepare two separate patches for this: one for the trunk (for 3.0.0-alpha2) and another for the YARN-5355 branch. I expect both to be somewhat different from each other. [~haibochen], thoughts? Can you look into it? > Move HBase backend code in ATS v2 into its separate module > --- > > Key: YARN-5667 > URL: https://issues.apache.org/jira/browse/YARN-5667 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Affects Versions: 3.0.0-alpha1 >Reporter: Haibo Chen >Assignee: Haibo Chen >Priority: Blocker > Attachments: New module structure.png, part1.yarn5667.prelim.patch, > part2.yarn5667.prelim.patch, part3.yarn5667.prelim.patch, > part4.yarn5667.prelim.patch, part5.yarn5667.prelim.patch, > pt1.yarn5667.001.patch, pt2.yarn5667.001.patch, pt3.yarn5667.001.patch, > pt4.yarn5667.001.patch, pt5.yarn5667.001.patch, pt6.yarn5667.001.patch, > pt9.yarn5667.001.patch, yarn5667-001.tar.gz > > > The HBase backend code currently lives along with the core ATS v2 code in > hadoop-yarn-server-timelineservice module. Because Resource Manager depends > on hadoop-yarn-server-timelineservice, an unnecessary dependency of the RM > module on HBase modules is introduced (HBase backend is pluggable, so we do > not need to directly pull in HBase jars). > In our internal effort to try ATS v2 with HBase 2.0 which depends on Hadoop > 3, we encountered a circular dependency during our builds between HBase2.0 > and Hadoop3 artifacts. > {code} > hadoop-mapreduce-client-common, hadoop-yarn-client, > hadoop-yarn-server-resourcemanager, hadoop-yarn-server-timelineservice, > hbase-server, hbase-prefix-tree, hbase-hadoop2-compat, > hadoop-mapreduce-client-jobclient, hadoop-mapreduce-client-common] > {code} > This jira proposes we move all HBase-backend-related code from > hadoop-yarn-server-timelineservice into its own module (possible name is > yarn-server-timelineservice-storage) so that core RM modules do not depend on > HBase modules any more. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5667) Move HBase backend code in ATS v2 into its separate module
[ https://issues.apache.org/jira/browse/YARN-5667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15622760#comment-15622760 ] Haibo Chen commented on YARN-5667: -- Thanks for the update [~sjlee0]. Let's wait until they are finished. > Move HBase backend code in ATS v2 into its separate module > --- > > Key: YARN-5667 > URL: https://issues.apache.org/jira/browse/YARN-5667 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Reporter: Haibo Chen >Assignee: Haibo Chen > Attachments: New module structure.png, part1.yarn5667.prelim.patch, > part2.yarn5667.prelim.patch, part3.yarn5667.prelim.patch, > part4.yarn5667.prelim.patch, part5.yarn5667.prelim.patch, > pt1.yarn5667.001.patch, pt2.yarn5667.001.patch, pt3.yarn5667.001.patch, > pt4.yarn5667.001.patch, pt5.yarn5667.001.patch, pt6.yarn5667.001.patch, > pt9.yarn5667.001.patch, yarn5667-001.tar.gz > > > The HBase backend code currently lives along with the core ATS v2 code in > hadoop-yarn-server-timelineservice module. Because Resource Manager depends > on hadoop-yarn-server-timelineservice, an unnecessary dependency of the RM > module on HBase modules is introduced (HBase backend is pluggable, so we do > not need to directly pull in HBase jars). > In our internal effort to try ATS v2 with HBase 2.0 which depends on Hadoop > 3, we encountered a circular dependency during our builds between HBase2.0 > and Hadoop3 artifacts. > {code} > hadoop-mapreduce-client-common, hadoop-yarn-client, > hadoop-yarn-server-resourcemanager, hadoop-yarn-server-timelineservice, > hbase-server, hbase-prefix-tree, hbase-hadoop2-compat, > hadoop-mapreduce-client-jobclient, hadoop-mapreduce-client-common] > {code} > This jira proposes we move all HBase-backend-related code from > hadoop-yarn-server-timelineservice into its own module (possible name is > yarn-server-timelineservice-storage) so that core RM modules do not depend on > HBase modules any more. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5667) Move HBase backend code in ATS v2 into its separate module
[ https://issues.apache.org/jira/browse/YARN-5667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15622717#comment-15622717 ] Sangjin Lee commented on YARN-5667: --- There are a couple of JIRAs that are being worked on (YARN-5585 for example). It would be great if we can give it a little more time. Thanks! > Move HBase backend code in ATS v2 into its separate module > --- > > Key: YARN-5667 > URL: https://issues.apache.org/jira/browse/YARN-5667 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Reporter: Haibo Chen >Assignee: Haibo Chen > Attachments: New module structure.png, part1.yarn5667.prelim.patch, > part2.yarn5667.prelim.patch, part3.yarn5667.prelim.patch, > part4.yarn5667.prelim.patch, part5.yarn5667.prelim.patch, > pt1.yarn5667.001.patch, pt2.yarn5667.001.patch, pt3.yarn5667.001.patch, > pt4.yarn5667.001.patch, pt5.yarn5667.001.patch, pt6.yarn5667.001.patch, > pt9.yarn5667.001.patch, yarn5667-001.tar.gz > > > The HBase backend code currently lives along with the core ATS v2 code in > hadoop-yarn-server-timelineservice module. Because Resource Manager depends > on hadoop-yarn-server-timelineservice, an unnecessary dependency of the RM > module on HBase modules is introduced (HBase backend is pluggable, so we do > not need to directly pull in HBase jars). > In our internal effort to try ATS v2 with HBase 2.0 which depends on Hadoop > 3, we encountered a circular dependency during our builds between HBase2.0 > and Hadoop3 artifacts. > {code} > hadoop-mapreduce-client-common, hadoop-yarn-client, > hadoop-yarn-server-resourcemanager, hadoop-yarn-server-timelineservice, > hbase-server, hbase-prefix-tree, hbase-hadoop2-compat, > hadoop-mapreduce-client-jobclient, hadoop-mapreduce-client-common] > {code} > This jira proposes we move all HBase-backend-related code from > hadoop-yarn-server-timelineservice into its own module (possible name is > yarn-server-timelineservice-storage) so that core RM modules do not depend on > HBase modules any more. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5667) Move HBase backend code in ATS v2 into its separate module
[ https://issues.apache.org/jira/browse/YARN-5667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15622470#comment-15622470 ] Haibo Chen commented on YARN-5667: -- [~sjlee0] Is it OK for me to rebase the patch now (Sorry, I lost track of all other ongoing jira development)? If other jiras are still going to take some time, I can break it up into smaller patcher and create sub jiras for them. > Move HBase backend code in ATS v2 into its separate module > --- > > Key: YARN-5667 > URL: https://issues.apache.org/jira/browse/YARN-5667 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Reporter: Haibo Chen >Assignee: Haibo Chen > Attachments: New module structure.png, part1.yarn5667.prelim.patch, > part2.yarn5667.prelim.patch, part3.yarn5667.prelim.patch, > part4.yarn5667.prelim.patch, part5.yarn5667.prelim.patch, > pt1.yarn5667.001.patch, pt2.yarn5667.001.patch, pt3.yarn5667.001.patch, > pt4.yarn5667.001.patch, pt5.yarn5667.001.patch, pt6.yarn5667.001.patch, > pt9.yarn5667.001.patch, yarn5667-001.tar.gz > > > The HBase backend code currently lives along with the core ATS v2 code in > hadoop-yarn-server-timelineservice module. Because Resource Manager depends > on hadoop-yarn-server-timelineservice, an unnecessary dependency of the RM > module on HBase modules is introduced (HBase backend is pluggable, so we do > not need to directly pull in HBase jars). > In our internal effort to try ATS v2 with HBase 2.0 which depends on Hadoop > 3, we encountered a circular dependency during our builds between HBase2.0 > and Hadoop3 artifacts. > {code} > hadoop-mapreduce-client-common, hadoop-yarn-client, > hadoop-yarn-server-resourcemanager, hadoop-yarn-server-timelineservice, > hbase-server, hbase-prefix-tree, hbase-hadoop2-compat, > hadoop-mapreduce-client-jobclient, hadoop-mapreduce-client-common] > {code} > This jira proposes we move all HBase-backend-related code from > hadoop-yarn-server-timelineservice into its own module (possible name is > yarn-server-timelineservice-storage) so that core RM modules do not depend on > HBase modules any more. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5667) Move HBase backend code in ATS v2 into its separate module
[ https://issues.apache.org/jira/browse/YARN-5667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15572341#comment-15572341 ] Sangjin Lee commented on YARN-5667: --- {{mvn depdency:analyze}} should be reliable once you built it ({{mvn install}} or {{mvn package}}). There are two parts to this: used but undeclared dependencies and unused declared dependencies. I normally find used but undeclared dependencies to be accurate with few exceptions. I believe it is based on bytecode analysis, and it should be highly accurate. Sometimes the referred classes can be in multiple artifacts, and that might confuse the dependency plugin, but that's about it. On the other hand, unused declared dependencies are tricky. In most cases, these are runtime dependencies that are truly needed but escape bytecode analysis. Unfortunately one would just have to take a look at them one by one. > Move HBase backend code in ATS v2 into its separate module > --- > > Key: YARN-5667 > URL: https://issues.apache.org/jira/browse/YARN-5667 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Reporter: Haibo Chen >Assignee: Haibo Chen > Attachments: New module structure.png, part1.yarn5667.prelim.patch, > part2.yarn5667.prelim.patch, part3.yarn5667.prelim.patch, > part4.yarn5667.prelim.patch, part5.yarn5667.prelim.patch, > pt1.yarn5667.001.patch, pt2.yarn5667.001.patch, pt3.yarn5667.001.patch, > pt4.yarn5667.001.patch, pt5.yarn5667.001.patch, pt6.yarn5667.001.patch, > pt9.yarn5667.001.patch, yarn5667-001.tar.gz > > > The HBase backend code currently lives along with the core ATS v2 code in > hadoop-yarn-server-timelineservice module. Because Resource Manager depends > on hadoop-yarn-server-timelineservice, an unnecessary dependency of the RM > module on HBase modules is introduced (HBase backend is pluggable, so we do > not need to directly pull in HBase jars). > In our internal effort to try ATS v2 with HBase 2.0 which depends on Hadoop > 3, we encountered a circular dependency during our builds between HBase2.0 > and Hadoop3 artifacts. > {code} > hadoop-mapreduce-client-common, hadoop-yarn-client, > hadoop-yarn-server-resourcemanager, hadoop-yarn-server-timelineservice, > hbase-server, hbase-prefix-tree, hbase-hadoop2-compat, > hadoop-mapreduce-client-jobclient, hadoop-mapreduce-client-common] > {code} > This jira proposes we move all HBase-backend-related code from > hadoop-yarn-server-timelineservice into its own module (possible name is > yarn-server-timelineservice-storage) so that core RM modules do not depend on > HBase modules any more. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5667) Move HBase backend code in ATS v2 into its separate module
[ https://issues.apache.org/jira/browse/YARN-5667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15571044#comment-15571044 ] Haibo Chen commented on YARN-5667: -- Please leave out part 8 of the patch when you review. I will go over the dependency again to do the cleanup. Thinking of going over the dependency listed in maven dependency:analyze, and checking if the dependency is indeed used in source code (not rely on mvn, or compilation). Any suggestion? > Move HBase backend code in ATS v2 into its separate module > --- > > Key: YARN-5667 > URL: https://issues.apache.org/jira/browse/YARN-5667 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Reporter: Haibo Chen >Assignee: Haibo Chen > Attachments: New module structure.png, part1.yarn5667.prelim.patch, > part2.yarn5667.prelim.patch, part3.yarn5667.prelim.patch, > part4.yarn5667.prelim.patch, part5.yarn5667.prelim.patch, > pt1.yarn5667.001.patch, pt2.yarn5667.001.patch, pt3.yarn5667.001.patch, > pt4.yarn5667.001.patch, pt5.yarn5667.001.patch, pt6.yarn5667.001.patch, > pt9.yarn5667.001.patch, yarn5667-001.tar.gz > > > The HBase backend code currently lives along with the core ATS v2 code in > hadoop-yarn-server-timelineservice module. Because Resource Manager depends > on hadoop-yarn-server-timelineservice, an unnecessary dependency of the RM > module on HBase modules is introduced (HBase backend is pluggable, so we do > not need to directly pull in HBase jars). > In our internal effort to try ATS v2 with HBase 2.0 which depends on Hadoop > 3, we encountered a circular dependency during our builds between HBase2.0 > and Hadoop3 artifacts. > {code} > hadoop-mapreduce-client-common, hadoop-yarn-client, > hadoop-yarn-server-resourcemanager, hadoop-yarn-server-timelineservice, > hbase-server, hbase-prefix-tree, hbase-hadoop2-compat, > hadoop-mapreduce-client-jobclient, hadoop-mapreduce-client-common] > {code} > This jira proposes we move all HBase-backend-related code from > hadoop-yarn-server-timelineservice into its own module (possible name is > yarn-server-timelineservice-storage) so that core RM modules do not depend on > HBase modules any more. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5667) Move HBase backend code in ATS v2 into its separate module
[ https://issues.apache.org/jira/browse/YARN-5667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15570968#comment-15570968 ] Haibo Chen commented on YARN-5667: -- [~sjlee0] Sorry for the dependency problem in my patch (part_8). Mind sharing how you determined what modules are needed or not? I just ran mvn dependency:analyze again. The output seems to indicate that mvn is not reliable. > Move HBase backend code in ATS v2 into its separate module > --- > > Key: YARN-5667 > URL: https://issues.apache.org/jira/browse/YARN-5667 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Reporter: Haibo Chen >Assignee: Haibo Chen > Attachments: New module structure.png, part1.yarn5667.prelim.patch, > part2.yarn5667.prelim.patch, part3.yarn5667.prelim.patch, > part4.yarn5667.prelim.patch, part5.yarn5667.prelim.patch, > pt1.yarn5667.001.patch, pt2.yarn5667.001.patch, pt3.yarn5667.001.patch, > pt4.yarn5667.001.patch, pt5.yarn5667.001.patch, pt6.yarn5667.001.patch, > pt9.yarn5667.001.patch, yarn5667-001.tar.gz > > > The HBase backend code currently lives along with the core ATS v2 code in > hadoop-yarn-server-timelineservice module. Because Resource Manager depends > on hadoop-yarn-server-timelineservice, an unnecessary dependency of the RM > module on HBase modules is introduced (HBase backend is pluggable, so we do > not need to directly pull in HBase jars). > In our internal effort to try ATS v2 with HBase 2.0 which depends on Hadoop > 3, we encountered a circular dependency during our builds between HBase2.0 > and Hadoop3 artifacts. > {code} > hadoop-mapreduce-client-common, hadoop-yarn-client, > hadoop-yarn-server-resourcemanager, hadoop-yarn-server-timelineservice, > hbase-server, hbase-prefix-tree, hbase-hadoop2-compat, > hadoop-mapreduce-client-jobclient, hadoop-mapreduce-client-common] > {code} > This jira proposes we move all HBase-backend-related code from > hadoop-yarn-server-timelineservice into its own module (possible name is > yarn-server-timelineservice-storage) so that core RM modules do not depend on > HBase modules any more. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5667) Move HBase backend code in ATS v2 into its separate module
[ https://issues.apache.org/jira/browse/YARN-5667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15570498#comment-15570498 ] Sangjin Lee commented on YARN-5667: --- Took a closer look at the patch (mostly at the pom changes). I prefer that we are 100% accurate in terms of including all direct references (classes referring to classes in a dependency directly) as dependencies. I run mvn dependency:analyze and ideally there should be no "used undeclared" dependencies. That's just goo hygiene. I do see quite a few used undeclared dependencies in the timelineservice, timelineservice-hbase and timelineservice-hbase-tests modules. Could you please look into them and restore them accurately? In some more detail, (timelineservice) - yarn-api should not be removed (the entity API is all here) - yarn-server-common should not be removed - yarn-common should not be removed (e.g. YarnRPC) - jersey-core should not be removed (javax.ws.rs package) (timelineservice-hbase) - should add the license - yarn-api should be added (the entity API is all here) - yarn-server-applicationhistoryserver should be added (GenericObjectMapper) - hadoop-common should be added - hadoop- annotations should be added - hadoop-yarn-common should be added (timelineservice-hbase-tests) - similarly, we should not remove the dependencies here: they were carefully analyzed and added for direct dependencies - also, we still need to add timelineservice (there is a dependency to that) > Move HBase backend code in ATS v2 into its separate module > --- > > Key: YARN-5667 > URL: https://issues.apache.org/jira/browse/YARN-5667 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Reporter: Haibo Chen >Assignee: Haibo Chen > Attachments: New module structure.png, part1.yarn5667.prelim.patch, > part2.yarn5667.prelim.patch, part3.yarn5667.prelim.patch, > part4.yarn5667.prelim.patch, part5.yarn5667.prelim.patch, > pt1.yarn5667.001.patch, pt2.yarn5667.001.patch, pt3.yarn5667.001.patch, > pt4.yarn5667.001.patch, pt5.yarn5667.001.patch, pt6.yarn5667.001.patch, > pt9.yarn5667.001.patch, yarn5667-001.tar.gz > > > The HBase backend code currently lives along with the core ATS v2 code in > hadoop-yarn-server-timelineservice module. Because Resource Manager depends > on hadoop-yarn-server-timelineservice, an unnecessary dependency of the RM > module on HBase modules is introduced (HBase backend is pluggable, so we do > not need to directly pull in HBase jars). > In our internal effort to try ATS v2 with HBase 2.0 which depends on Hadoop > 3, we encountered a circular dependency during our builds between HBase2.0 > and Hadoop3 artifacts. > {code} > hadoop-mapreduce-client-common, hadoop-yarn-client, > hadoop-yarn-server-resourcemanager, hadoop-yarn-server-timelineservice, > hbase-server, hbase-prefix-tree, hbase-hadoop2-compat, > hadoop-mapreduce-client-jobclient, hadoop-mapreduce-client-common] > {code} > This jira proposes we move all HBase-backend-related code from > hadoop-yarn-server-timelineservice into its own module (possible name is > yarn-server-timelineservice-storage) so that core RM modules do not depend on > HBase modules any more. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5667) Move HBase backend code in ATS v2 into its separate module
[ https://issues.apache.org/jira/browse/YARN-5667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15570322#comment-15570322 ] Sangjin Lee commented on YARN-5667: --- Thanks for updating the patch. I'll take a closer look. We need to think about the sequence of some of the outstanding patches going in. Probably this should be done after the outstanding ones are committed? > Move HBase backend code in ATS v2 into its separate module > --- > > Key: YARN-5667 > URL: https://issues.apache.org/jira/browse/YARN-5667 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Reporter: Haibo Chen >Assignee: Haibo Chen > Attachments: New module structure.png, part1.yarn5667.prelim.patch, > part2.yarn5667.prelim.patch, part3.yarn5667.prelim.patch, > part4.yarn5667.prelim.patch, part5.yarn5667.prelim.patch, > pt1.yarn5667.001.patch, pt2.yarn5667.001.patch, pt3.yarn5667.001.patch, > pt4.yarn5667.001.patch, pt5.yarn5667.001.patch, pt6.yarn5667.001.patch, > pt9.yarn5667.001.patch, yarn5667-001.tar.gz > > > The HBase backend code currently lives along with the core ATS v2 code in > hadoop-yarn-server-timelineservice module. Because Resource Manager depends > on hadoop-yarn-server-timelineservice, an unnecessary dependency of the RM > module on HBase modules is introduced (HBase backend is pluggable, so we do > not need to directly pull in HBase jars). > In our internal effort to try ATS v2 with HBase 2.0 which depends on Hadoop > 3, we encountered a circular dependency during our builds between HBase2.0 > and Hadoop3 artifacts. > {code} > hadoop-mapreduce-client-common, hadoop-yarn-client, > hadoop-yarn-server-resourcemanager, hadoop-yarn-server-timelineservice, > hbase-server, hbase-prefix-tree, hbase-hadoop2-compat, > hadoop-mapreduce-client-jobclient, hadoop-mapreduce-client-common] > {code} > This jira proposes we move all HBase-backend-related code from > hadoop-yarn-server-timelineservice into its own module (possible name is > yarn-server-timelineservice-storage) so that core RM modules do not depend on > HBase modules any more. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5667) Move HBase backend code in ATS v2 into its separate module
[ https://issues.apache.org/jira/browse/YARN-5667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15564042#comment-15564042 ] Haibo Chen commented on YARN-5667: -- [~sjlee0] Thanks a lot for the suggestion! It is exactly as you pointed out. The hbase dependency needs to be included in hadoop-yarn-project pom file. Once I do that, the hbase dependency jars are copied into the lib directory. Will upload an addendum patch to address that. > Move HBase backend code in ATS v2 into its separate module > --- > > Key: YARN-5667 > URL: https://issues.apache.org/jira/browse/YARN-5667 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Reporter: Haibo Chen >Assignee: Haibo Chen > Attachments: New module structure.png, part1.yarn5667.prelim.patch, > part2.yarn5667.prelim.patch, part3.yarn5667.prelim.patch, > part4.yarn5667.prelim.patch, part5.yarn5667.prelim.patch, > pt1.yarn5667.001.patch, pt2.yarn5667.001.patch, pt3.yarn5667.001.patch, > pt4.yarn5667.001.patch, pt5.yarn5667.001.patch, pt6.yarn5667.001.patch, > yarn5667-001.tar.gz > > > The HBase backend code currently lives along with the core ATS v2 code in > hadoop-yarn-server-timelineservice module. Because Resource Manager depends > on hadoop-yarn-server-timelineservice, an unnecessary dependency of the RM > module on HBase modules is introduced (HBase backend is pluggable, so we do > not need to directly pull in HBase jars). > In our internal effort to try ATS v2 with HBase 2.0 which depends on Hadoop > 3, we encountered a circular dependency during our builds between HBase2.0 > and Hadoop3 artifacts. > {code} > hadoop-mapreduce-client-common, hadoop-yarn-client, > hadoop-yarn-server-resourcemanager, hadoop-yarn-server-timelineservice, > hbase-server, hbase-prefix-tree, hbase-hadoop2-compat, > hadoop-mapreduce-client-jobclient, hadoop-mapreduce-client-common] > {code} > This jira proposes we move all HBase-backend-related code from > hadoop-yarn-server-timelineservice into its own module (possible name is > yarn-server-timelineservice-storage) so that core RM modules do not depend on > HBase modules any more. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5667) Move HBase backend code in ATS v2 into its separate module
[ https://issues.apache.org/jira/browse/YARN-5667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15563948#comment-15563948 ] Sangjin Lee commented on YARN-5667: --- If I'm not mistaken, the dist profile (one that gets invoked in creating the tarball) copies things from {{hadoop-yarn-project/target}}. It appears that {{hadoop-yarn-project}} doesn't include the hbase dependencies given the way its pom is structured. You might want to look at that. > Move HBase backend code in ATS v2 into its separate module > --- > > Key: YARN-5667 > URL: https://issues.apache.org/jira/browse/YARN-5667 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Reporter: Haibo Chen >Assignee: Haibo Chen > Attachments: New module structure.png, part1.yarn5667.prelim.patch, > part2.yarn5667.prelim.patch, part3.yarn5667.prelim.patch, > part4.yarn5667.prelim.patch, part5.yarn5667.prelim.patch, > pt1.yarn5667.001.patch, pt2.yarn5667.001.patch, pt3.yarn5667.001.patch, > pt4.yarn5667.001.patch, pt5.yarn5667.001.patch, pt6.yarn5667.001.patch, > yarn5667-001.tar.gz > > > The HBase backend code currently lives along with the core ATS v2 code in > hadoop-yarn-server-timelineservice module. Because Resource Manager depends > on hadoop-yarn-server-timelineservice, an unnecessary dependency of the RM > module on HBase modules is introduced (HBase backend is pluggable, so we do > not need to directly pull in HBase jars). > In our internal effort to try ATS v2 with HBase 2.0 which depends on Hadoop > 3, we encountered a circular dependency during our builds between HBase2.0 > and Hadoop3 artifacts. > {code} > hadoop-mapreduce-client-common, hadoop-yarn-client, > hadoop-yarn-server-resourcemanager, hadoop-yarn-server-timelineservice, > hbase-server, hbase-prefix-tree, hbase-hadoop2-compat, > hadoop-mapreduce-client-jobclient, hadoop-mapreduce-client-common] > {code} > This jira proposes we move all HBase-backend-related code from > hadoop-yarn-server-timelineservice into its own module (possible name is > yarn-server-timelineservice-storage) so that core RM modules do not depend on > HBase modules any more. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5667) Move HBase backend code in ATS v2 into its separate module
[ https://issues.apache.org/jira/browse/YARN-5667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15563742#comment-15563742 ] Haibo Chen commented on YARN-5667: -- Sorry for being late to update the patch, [~sjlee0]. I have cleaned up some unused dependencies based on mvn's analysis. For some reason, mvn fails to detect a lot of dependencies that are actually used. The version specification of the dependency has been moved to hadoop-yarn as well. Uploading all 3 more parts. part 6: fixes the issue that for some classes, package names do not follow the directory path of the source file part 7: move version specification of timelineservice-hbase-test pom.xml to hadoop-project pom.xml part 8: clean up unused dependencies and declared dependencies that were not declared in timelineservice modules. One problem I still could solve though, is that the hbase dependencies are not copied into the lib directory when hadoop is packaged. Any pointer as to where I should be look at to fix this, [~sjlee0] > Move HBase backend code in ATS v2 into its separate module > --- > > Key: YARN-5667 > URL: https://issues.apache.org/jira/browse/YARN-5667 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Reporter: Haibo Chen >Assignee: Haibo Chen > Attachments: New module structure.png, part1.yarn5667.prelim.patch, > part2.yarn5667.prelim.patch, part3.yarn5667.prelim.patch, > part4.yarn5667.prelim.patch, part5.yarn5667.prelim.patch, > pt1.yarn5667.001.patch, pt2.yarn5667.001.patch, pt3.yarn5667.001.patch, > pt4.yarn5667.001.patch, pt5.yarn5667.001.patch, pt6.yarn5667.001.patch, > yarn5667-001.tar.gz > > > The HBase backend code currently lives along with the core ATS v2 code in > hadoop-yarn-server-timelineservice module. Because Resource Manager depends > on hadoop-yarn-server-timelineservice, an unnecessary dependency of the RM > module on HBase modules is introduced (HBase backend is pluggable, so we do > not need to directly pull in HBase jars). > In our internal effort to try ATS v2 with HBase 2.0 which depends on Hadoop > 3, we encountered a circular dependency during our builds between HBase2.0 > and Hadoop3 artifacts. > {code} > hadoop-mapreduce-client-common, hadoop-yarn-client, > hadoop-yarn-server-resourcemanager, hadoop-yarn-server-timelineservice, > hbase-server, hbase-prefix-tree, hbase-hadoop2-compat, > hadoop-mapreduce-client-jobclient, hadoop-mapreduce-client-common] > {code} > This jira proposes we move all HBase-backend-related code from > hadoop-yarn-server-timelineservice into its own module (possible name is > yarn-server-timelineservice-storage) so that core RM modules do not depend on > HBase modules any more. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5667) Move HBase backend code in ATS v2 into its separate module
[ https://issues.apache.org/jira/browse/YARN-5667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15552577#comment-15552577 ] Sangjin Lee commented on YARN-5667: --- It now builds cleanly. Thanks. I am still looking into the changes, but a couple of quick comments: (1) Could you please ensure the dependencies are as correct as they can be? Specifically run {{mvn dependency:analyze}} to see if there are used but undeclared dependencies or unused but declared dependencies and whether they need to be addressed. (2) In {{yarn-server-timelineservice-hbase-tests}}, I see a specific mention of the version {{3.0.0-alpha2-SNAPSHOT}} for the dependency. It is better to manage them in the main parent pom (I think {{hadoop-project/pom.xml}}) than doing it here. This makes it difficult to change versions across the entire hadoop repository. > Move HBase backend code in ATS v2 into its separate module > --- > > Key: YARN-5667 > URL: https://issues.apache.org/jira/browse/YARN-5667 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Reporter: Haibo Chen >Assignee: Haibo Chen > Attachments: New module structure.png, part1.yarn5667.prelim.patch, > part2.yarn5667.prelim.patch, part3.yarn5667.prelim.patch, > part4.yarn5667.prelim.patch, part5.yarn5667.prelim.patch, > pt1.yarn5667.001.patch, pt2.yarn5667.001.patch, pt3.yarn5667.001.patch, > pt4.yarn5667.001.patch, pt5.yarn5667.001.patch, pt6.yarn5667.001.patch > > > The HBase backend code currently lives along with the core ATS v2 code in > hadoop-yarn-server-timelineservice module. Because Resource Manager depends > on hadoop-yarn-server-timelineservice, an unnecessary dependency of the RM > module on HBase modules is introduced (HBase backend is pluggable, so we do > not need to directly pull in HBase jars). > In our internal effort to try ATS v2 with HBase 2.0 which depends on Hadoop > 3, we encountered a circular dependency during our builds between HBase2.0 > and Hadoop3 artifacts. > {code} > hadoop-mapreduce-client-common, hadoop-yarn-client, > hadoop-yarn-server-resourcemanager, hadoop-yarn-server-timelineservice, > hbase-server, hbase-prefix-tree, hbase-hadoop2-compat, > hadoop-mapreduce-client-jobclient, hadoop-mapreduce-client-common] > {code} > This jira proposes we move all HBase-backend-related code from > hadoop-yarn-server-timelineservice into its own module (possible name is > yarn-server-timelineservice-storage) so that core RM modules do not depend on > HBase modules any more. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5667) Move HBase backend code in ATS v2 into its separate module
[ https://issues.apache.org/jira/browse/YARN-5667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15550498#comment-15550498 ] Haibo Chen commented on YARN-5667: -- My bad. I was working on the trunk branch. I will update the patches for YARN-5355 specifically. > Move HBase backend code in ATS v2 into its separate module > --- > > Key: YARN-5667 > URL: https://issues.apache.org/jira/browse/YARN-5667 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Reporter: Haibo Chen >Assignee: Haibo Chen > Attachments: part1.yarn5667.prelim.patch, > part2.yarn5667.prelim.patch, part3.yarn5667.prelim.patch, > part4.yarn5667.prelim.patch, part5.yarn5667.prelim.patch > > > The HBase backend code currently lives along with the core ATS v2 code in > hadoop-yarn-server-timelineservice module. Because Resource Manager depends > on hadoop-yarn-server-timelineservice, an unnecessary dependency of the RM > module on HBase modules is introduced (HBase backend is pluggable, so we do > not need to directly pull in HBase jars). > In our internal effort to try ATS v2 with HBase 2.0 which depends on Hadoop > 3, we encountered a circular dependency during our builds between HBase2.0 > and Hadoop3 artifacts. > {code} > hadoop-mapreduce-client-common, hadoop-yarn-client, > hadoop-yarn-server-resourcemanager, hadoop-yarn-server-timelineservice, > hbase-server, hbase-prefix-tree, hbase-hadoop2-compat, > hadoop-mapreduce-client-jobclient, hadoop-mapreduce-client-common] > {code} > This jira proposes we move all HBase-backend-related code from > hadoop-yarn-server-timelineservice into its own module (possible name is > yarn-server-timelineservice-storage) so that core RM modules do not depend on > HBase modules any more. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5667) Move HBase backend code in ATS v2 into its separate module
[ https://issues.apache.org/jira/browse/YARN-5667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15550484#comment-15550484 ] Sangjin Lee commented on YARN-5667: --- [~haibochen], thanks for your patches! I just applied your patches in order, but am unable to get a clean build. Perhaps some files are missing in the patches? Could you try on a clean YARN-5355 branch and see if the patches are correct? > Move HBase backend code in ATS v2 into its separate module > --- > > Key: YARN-5667 > URL: https://issues.apache.org/jira/browse/YARN-5667 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Reporter: Haibo Chen >Assignee: Haibo Chen > Attachments: part1.yarn5667.prelim.patch, > part2.yarn5667.prelim.patch, part3.yarn5667.prelim.patch, > part4.yarn5667.prelim.patch, part5.yarn5667.prelim.patch > > > The HBase backend code currently lives along with the core ATS v2 code in > hadoop-yarn-server-timelineservice module. Because Resource Manager depends > on hadoop-yarn-server-timelineservice, an unnecessary dependency of the RM > module on HBase modules is introduced (HBase backend is pluggable, so we do > not need to directly pull in HBase jars). > In our internal effort to try ATS v2 with HBase 2.0 which depends on Hadoop > 3, we encountered a circular dependency during our builds between HBase2.0 > and Hadoop3 artifacts. > {code} > hadoop-mapreduce-client-common, hadoop-yarn-client, > hadoop-yarn-server-resourcemanager, hadoop-yarn-server-timelineservice, > hbase-server, hbase-prefix-tree, hbase-hadoop2-compat, > hadoop-mapreduce-client-jobclient, hadoop-mapreduce-client-common] > {code} > This jira proposes we move all HBase-backend-related code from > hadoop-yarn-server-timelineservice into its own module (possible name is > yarn-server-timelineservice-storage) so that core RM modules do not depend on > HBase modules any more. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5667) Move HBase backend code in ATS v2 into its separate module
[ https://issues.apache.org/jira/browse/YARN-5667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15550424#comment-15550424 ] Haibo Chen commented on YARN-5667: -- Thanks for your quick reviews [~vrushalic]. I did not change the hbase version, so I believe I must have been testing against hbase 1.1.3. In terms of testing done, I enabled coprocessor in my hbase cluster and ATS v2 in my yarn cluster, ran a few mapreduce example jobs, and then verified the reader through a few flow and application-level REST requests. Everthing seems to have worked for me. I could see the metrics for a flow run as well (Is this what you mean by " Did you recollect the counters aggregated to the flow level"?) I will update with a layout of what the file structure is like before and after my change. > Move HBase backend code in ATS v2 into its separate module > --- > > Key: YARN-5667 > URL: https://issues.apache.org/jira/browse/YARN-5667 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Reporter: Haibo Chen >Assignee: Haibo Chen > Attachments: part1.yarn5667.prelim.patch, > part2.yarn5667.prelim.patch, part3.yarn5667.prelim.patch, > part4.yarn5667.prelim.patch, part5.yarn5667.prelim.patch > > > The HBase backend code currently lives along with the core ATS v2 code in > hadoop-yarn-server-timelineservice module. Because Resource Manager depends > on hadoop-yarn-server-timelineservice, an unnecessary dependency of the RM > module on HBase modules is introduced (HBase backend is pluggable, so we do > not need to directly pull in HBase jars). > In our internal effort to try ATS v2 with HBase 2.0 which depends on Hadoop > 3, we encountered a circular dependency during our builds between HBase2.0 > and Hadoop3 artifacts. > {code} > hadoop-mapreduce-client-common, hadoop-yarn-client, > hadoop-yarn-server-resourcemanager, hadoop-yarn-server-timelineservice, > hbase-server, hbase-prefix-tree, hbase-hadoop2-compat, > hadoop-mapreduce-client-jobclient, hadoop-mapreduce-client-common] > {code} > This jira proposes we move all HBase-backend-related code from > hadoop-yarn-server-timelineservice into its own module (possible name is > yarn-server-timelineservice-storage) so that core RM modules do not depend on > HBase modules any more. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5667) Move HBase backend code in ATS v2 into its separate module
[ https://issues.apache.org/jira/browse/YARN-5667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15549768#comment-15549768 ] Vrushali C commented on YARN-5667: -- Thanks [~haibochen], I took a quick look and it seems to be good. It does not look like you have updated the hbase version in this patch, right? Since you had mentioned you wanted to build with hbase 2.0. Am I correct in thinking you tested this with hbase 1.1.3? Also, there are a lot of files moved around, so wanted to confirm, the classes being moved are the ones primarily under hadoop-yarn-server-timelineservice/src/main/java/org/apache/hadoop/yarn/server/timelineservice/storage/. If possible, could you share a listing of what remains in the hadoop-yarn-server-timelineservice/src/main? Also, did you enable the coprocessor on your pseudo cluster during your checks? Did you recollect the counters aggregated to the flow level? > Move HBase backend code in ATS v2 into its separate module > --- > > Key: YARN-5667 > URL: https://issues.apache.org/jira/browse/YARN-5667 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Reporter: Haibo Chen >Assignee: Haibo Chen > Attachments: part1.yarn5667.prelim.patch, > part2.yarn5667.prelim.patch, part3.yarn5667.prelim.patch, > part4.yarn5667.prelim.patch, part5.yarn5667.prelim.patch > > > The HBase backend code currently lives along with the core ATS v2 code in > hadoop-yarn-server-timelineservice module. Because Resource Manager depends > on hadoop-yarn-server-timelineservice, an unnecessary dependency of the RM > module on HBase modules is introduced (HBase backend is pluggable, so we do > not need to directly pull in HBase jars). > In our internal effort to try ATS v2 with HBase 2.0 which depends on Hadoop > 3, we encountered a circular dependency during our builds between HBase2.0 > and Hadoop3 artifacts. > {code} > hadoop-mapreduce-client-common, hadoop-yarn-client, > hadoop-yarn-server-resourcemanager, hadoop-yarn-server-timelineservice, > hbase-server, hbase-prefix-tree, hbase-hadoop2-compat, > hadoop-mapreduce-client-jobclient, hadoop-mapreduce-client-common] > {code} > This jira proposes we move all HBase-backend-related code from > hadoop-yarn-server-timelineservice into its own module (possible name is > yarn-server-timelineservice-storage) so that core RM modules do not depend on > HBase modules any more. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5667) Move HBase backend code in ATS v2 into its separate module
[ https://issues.apache.org/jira/browse/YARN-5667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15547279#comment-15547279 ] Haibo Chen commented on YARN-5667: -- Here is the description of what each part does: part1: remove direct references of HBaseTimelineWriter/Reader in core ATS classes that Yarn will need. part2: extract some methods that are used only by the HBase implementation in TimelineStorageUtil into a new util class HBaseTimelineStorageUtil part3: move all HBase related code into a new module that the hadoop-yarn-server-timelineservice module depends on only at runtime. part4: fix issues introduced in the hbase-test module part5: update the ATS v2 documentation. > Move HBase backend code in ATS v2 into its separate module > --- > > Key: YARN-5667 > URL: https://issues.apache.org/jira/browse/YARN-5667 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Reporter: Haibo Chen >Assignee: Haibo Chen > Attachments: part1.yarn5667.prelim.patch, > part2.yarn5667.prelim.patch, part3.yarn5667.prelim.patch, > part4.yarn5667.prelim.patch, part5.yarn5667.prelim.patch > > > The HBase backend code currently lives along with the core ATS v2 code in > hadoop-yarn-server-timelineservice module. Because Resource Manager depends > on hadoop-yarn-server-timelineservice, an unnecessary dependency of the RM > module on HBase modules is introduced (HBase backend is pluggable, so we do > not need to directly pull in HBase jars). > In our internal effort to try ATS v2 with HBase 2.0 which depends on Hadoop > 3, we encountered a circular dependency during our builds between HBase2.0 > and Hadoop3 artifacts. > {code} > hadoop-mapreduce-client-common, hadoop-yarn-client, > hadoop-yarn-server-resourcemanager, hadoop-yarn-server-timelineservice, > hbase-server, hbase-prefix-tree, hbase-hadoop2-compat, > hadoop-mapreduce-client-jobclient, hadoop-mapreduce-client-common] > {code} > This jira proposes we move all HBase-backend-related code from > hadoop-yarn-server-timelineservice into its own module (possible name is > yarn-server-timelineservice-storage) so that core RM modules do not depend on > HBase modules any more. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5667) Move HBase backend code in ATS v2 into its separate module
[ https://issues.apache.org/jira/browse/YARN-5667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15547147#comment-15547147 ] Haibo Chen commented on YARN-5667: -- I have run all ATS-v2 unit tests, and deployed this into a pseudo-distributed cluster to run an example mapreduce job to verify this change does not break functionalities. [~vrushalic], [~sjlee0], Do you have any suggestion on how to further test this change? However, there is still one piece that is missing here, that is, copying hbase jars into the share/hadoop/yarn/lib directory at packaging time. For some reason, the hadoop-specific maven assembly plugin (I believe it is hadoop-yarn-dist.xml) no longer detects the hbase dependency in the new module, which was previously in the timelineservice module. [~sjlee0] What I am missing? > Move HBase backend code in ATS v2 into its separate module > --- > > Key: YARN-5667 > URL: https://issues.apache.org/jira/browse/YARN-5667 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Reporter: Haibo Chen >Assignee: Haibo Chen > Attachments: part1.yarn5667.prelim.patch, > part2.yarn5667.prelim.patch, part3.yarn5667.prelim.patch, > part4.yarn5667.prelim.patch, part5.yarn5667.prelim.patch > > > The HBase backend code currently lives along with the core ATS v2 code in > hadoop-yarn-server-timelineservice module. Because Resource Manager depends > on hadoop-yarn-server-timelineservice, an unnecessary dependency of the RM > module on HBase modules is introduced (HBase backend is pluggable, so we do > not need to directly pull in HBase jars). > In our internal effort to try ATS v2 with HBase 2.0 which depends on Hadoop > 3, we encountered a circular dependency during our builds between HBase2.0 > and Hadoop3 artifacts. > {code} > hadoop-mapreduce-client-common, hadoop-yarn-client, > hadoop-yarn-server-resourcemanager, hadoop-yarn-server-timelineservice, > hbase-server, hbase-prefix-tree, hbase-hadoop2-compat, > hadoop-mapreduce-client-jobclient, hadoop-mapreduce-client-common] > {code} > This jira proposes we move all HBase-backend-related code from > hadoop-yarn-server-timelineservice into its own module (possible name is > yarn-server-timelineservice-storage) so that core RM modules do not depend on > HBase modules any more. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5667) Move HBase backend code in ATS v2 into its separate module
[ https://issues.apache.org/jira/browse/YARN-5667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15534218#comment-15534218 ] Haibo Chen commented on YARN-5667: -- Thanks [~vrushalic] for the info! I will then leave the moving of coprocessor to YARN-4985. As discussed in the meeting, I am planning to extract all HBase-related code from the core ATS into a separate module first. Then we can continue to extract the coprocessor code as a follow-up. Per previous discussion with Sangjin, the [coprocessor ---> HBase client code in ATS] dependency is not necessary. But looking at the code, the coprocessor does need to depend on the HBase table definition code, so I figure if it is OK to extract the schema code out and then have the coprocessor and HBaseWriter/Reader both depend on the schema module. We can always defer the discussion to YARN-4985 when we get there, though. > Move HBase backend code in ATS v2 into its separate module > --- > > Key: YARN-5667 > URL: https://issues.apache.org/jira/browse/YARN-5667 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Reporter: Haibo Chen >Assignee: Haibo Chen > > The HBase backend code currently lives along with the core ATS v2 code in > hadoop-yarn-server-timelineservice module. Because Resource Manager depends > on hadoop-yarn-server-timelineservice, an unnecessary dependency of the RM > module on HBase modules is introduced (HBase backend is pluggable, so we do > not need to directly pull in HBase jars). > In our internal effort to try ATS v2 with HBase 2.0 which depends on Hadoop > 3, we encountered a circular dependency during our builds between HBase2.0 > and Hadoop3 artifacts. > {code} > hadoop-mapreduce-client-common, hadoop-yarn-client, > hadoop-yarn-server-resourcemanager, hadoop-yarn-server-timelineservice, > hbase-server, hbase-prefix-tree, hbase-hadoop2-compat, > hadoop-mapreduce-client-jobclient, hadoop-mapreduce-client-common] > {code} > This jira proposes we move all HBase-backend-related code from > hadoop-yarn-server-timelineservice into its own module (possible name is > yarn-server-timelineservice-storage) so that core RM modules do not depend on > HBase modules any more. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5667) Move HBase backend code in ATS v2 into its separate module
[ https://issues.apache.org/jira/browse/YARN-5667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15533450#comment-15533450 ] Vrushali C commented on YARN-5667: -- Yes, we are considering moving the coprocessor related code into a separate package in YARN-4985 (not much progress there yet). I am also considering moving the coprocessor related functionality (ie aggregation) into hbase itself. That way ATS v2 can invoke this aggregation via hbase api calls which might make the dependency dags "easier" to manage. But that remains TBD, so till then we should proceed with moving the coproc related code into a separate package. > Move HBase backend code in ATS v2 into its separate module > --- > > Key: YARN-5667 > URL: https://issues.apache.org/jira/browse/YARN-5667 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Reporter: Haibo Chen >Assignee: Haibo Chen > > The HBase backend code currently lives along with the core ATS v2 code in > hadoop-yarn-server-timelineservice module. Because Resource Manager depends > on hadoop-yarn-server-timelineservice, an unnecessary dependency of the RM > module on HBase modules is introduced (HBase backend is pluggable, so we do > not need to directly pull in HBase jars). > In our internal effort to try ATS v2 with HBase 2.0 which depends on Hadoop > 3, we encountered a circular dependency during our builds between HBase2.0 > and Hadoop3 artifacts. > {code} > hadoop-mapreduce-client-common, hadoop-yarn-client, > hadoop-yarn-server-resourcemanager, hadoop-yarn-server-timelineservice, > hbase-server, hbase-prefix-tree, hbase-hadoop2-compat, > hadoop-mapreduce-client-jobclient, hadoop-mapreduce-client-common] > {code} > This jira proposes we move all HBase-backend-related code from > hadoop-yarn-server-timelineservice into its own module (possible name is > yarn-server-timelineservice-storage) so that core RM modules do not depend on > HBase modules any more. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5667) Move HBase backend code in ATS v2 into its separate module
[ https://issues.apache.org/jira/browse/YARN-5667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15526858#comment-15526858 ] Haibo Chen commented on YARN-5667: -- Thanks for the great in-depth explanation, [~sjlee0]! Every question I had now goes away. I'll start looking into extracting coprocessor code out into a separate module. > Move HBase backend code in ATS v2 into its separate module > --- > > Key: YARN-5667 > URL: https://issues.apache.org/jira/browse/YARN-5667 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Reporter: Haibo Chen >Assignee: Haibo Chen > > The HBase backend code currently lives along with the core ATS v2 code in > hadoop-yarn-server-timelineservice module. Because Resource Manager depends > on hadoop-yarn-server-timelineservice, an unnecessary dependency of the RM > module on HBase modules is introduced (HBase backend is pluggable, so we do > not need to directly pull in HBase jars). > In our internal effort to try ATS v2 with HBase 2.0 which depends on Hadoop > 3, we encountered a circular dependency during our builds between HBase2.0 > and Hadoop3 artifacts. > {code} > hadoop-mapreduce-client-common, hadoop-yarn-client, > hadoop-yarn-server-resourcemanager, hadoop-yarn-server-timelineservice, > hbase-server, hbase-prefix-tree, hbase-hadoop2-compat, > hadoop-mapreduce-client-jobclient, hadoop-mapreduce-client-common] > {code} > This jira proposes we move all HBase-backend-related code from > hadoop-yarn-server-timelineservice into its own module (possible name is > yarn-server-timelineservice-storage) so that core RM modules do not depend on > HBase modules any more. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5667) Move HBase backend code in ATS v2 into its separate module
[ https://issues.apache.org/jira/browse/YARN-5667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15524746#comment-15524746 ] Sangjin Lee commented on YARN-5667: --- Those are great questions. The diamond dependency (where there are more than one version of a given library in the dependency graph) happens because the hadoop code uses hadoop-common 3.0.0-alpha1 directly for example, and also 2.5.1 via indirect dependency via hbase 1.1.3. Due to hadoop's version management, 3.0.0-alpha1 is picked. The implication of this is that we build and test hbase code in the context of timeline service *diffrent than* the declared hbase's hadoop dependencies. Now if we think about hbase client code and hbase coprocessor code separately, we see that the runtime for both pieces of code is different. The code that uses hbase client runs on YARN (and therefore hadoop 3.0.0). In that environment, we need to ensure the hbase client itself (not our code that uses hbase client) works correctly against the trunk version of hadoop. On the other hand, the hbase coprocessor code runs on hbase. Therefore, it is now timeline service coprocessor code that needs to run under hadoop 2.5.1 (until/unless we upgrade hbase). These both aspects need to be verified if we decide to split the code into separate modules. That would be made easier by having them in separate modules. If we have an hbase version that depends on the trunk, these problems would go away. And I understand that the hbase folks are making effort to ensure the latest hbase version works against the hadoop trunk version. That said, hbase officially can depend only on released versions, and there will always be lags. As for the reason that the coprocessor depends on the hbase-client-related code, there is no strong reason that should be the case. It's just the way the code evolved. Actually it would be good to refactor the code so that the coprocessor code has minimal dependencies. It's worth looking into. > Move HBase backend code in ATS v2 into its separate module > --- > > Key: YARN-5667 > URL: https://issues.apache.org/jira/browse/YARN-5667 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Reporter: Haibo Chen >Assignee: Haibo Chen > > The HBase backend code currently lives along with the core ATS v2 code in > hadoop-yarn-server-timelineservice module. Because Resource Manager depends > on hadoop-yarn-server-timelineservice, an unnecessary dependency of the RM > module on HBase modules is introduced (HBase backend is pluggable, so we do > not need to directly pull in HBase jars). > In our internal effort to try ATS v2 with HBase 2.0 which depends on Hadoop > 3, we encountered a circular dependency during our builds between HBase2.0 > and Hadoop3 artifacts. > {code} > hadoop-mapreduce-client-common, hadoop-yarn-client, > hadoop-yarn-server-resourcemanager, hadoop-yarn-server-timelineservice, > hbase-server, hbase-prefix-tree, hbase-hadoop2-compat, > hadoop-mapreduce-client-jobclient, hadoop-mapreduce-client-common] > {code} > This jira proposes we move all HBase-backend-related code from > hadoop-yarn-server-timelineservice into its own module (possible name is > yarn-server-timelineservice-storage) so that core RM modules do not depend on > HBase modules any more. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5667) Move HBase backend code in ATS v2 into its separate module
[ https://issues.apache.org/jira/browse/YARN-5667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15524253#comment-15524253 ] Haibo Chen commented on YARN-5667: -- Thanks very much for the explanation [~sjlee0]! Trying to understand this a bit more (Sorry for my limited understanding of this issue in general). Is the diamond dependency introduced because HBase client depends on hadoop yarn in trunk whereas HBase coprocessor depends on an earlier version of hadoop? . If so, I'd guess in the future when we move to HBase 2.0 in ATS v2 (IIUC, this will not happen soon because of Phoenix), the diamond dependency can be resolved with the upgrade. Also, can elaborate on why it is necessary to have coprocessor depend on HBase-client-related code in ATS? Thanks again for your help, Sangjin. > Move HBase backend code in ATS v2 into its separate module > --- > > Key: YARN-5667 > URL: https://issues.apache.org/jira/browse/YARN-5667 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Reporter: Haibo Chen >Assignee: Haibo Chen > > The HBase backend code currently lives along with the core ATS v2 code in > hadoop-yarn-server-timelineservice module. Because Resource Manager depends > on hadoop-yarn-server-timelineservice, an unnecessary dependency of the RM > module on HBase modules is introduced (HBase backend is pluggable, so we do > not need to directly pull in HBase jars). > In our internal effort to try ATS v2 with HBase 2.0 which depends on Hadoop > 3, we encountered a circular dependency during our builds between HBase2.0 > and Hadoop3 artifacts. > {code} > hadoop-mapreduce-client-common, hadoop-yarn-client, > hadoop-yarn-server-resourcemanager, hadoop-yarn-server-timelineservice, > hbase-server, hbase-prefix-tree, hbase-hadoop2-compat, > hadoop-mapreduce-client-jobclient, hadoop-mapreduce-client-common] > {code} > This jira proposes we move all HBase-backend-related code from > hadoop-yarn-server-timelineservice into its own module (possible name is > yarn-server-timelineservice-storage) so that core RM modules do not depend on > HBase modules any more. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5667) Move HBase backend code in ATS v2 into its separate module
[ https://issues.apache.org/jira/browse/YARN-5667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15523418#comment-15523418 ] Sangjin Lee commented on YARN-5667: --- In principle, no. You'll find that we've already isolated the hbase test code into a separate module (hadoop-yarn-server-timelineservice-hbase-tests). I considered moving the non-test hbase code there too, but stopped. The main reason is bit complicated. There are two pieces of non-test hbase code that is under timeline service: hbase client code and hbase co-processor code. The major difference between them is that the hbase client code needs to run under the hadoop yarn trunk dependencies (as that's where it needs to run) whereas the co-processor code needs to run under the hbase dependencies. The hbase tests require the hbase dependencies because of the testing utility dependency. So there is not only a potential for circular dependency, but also a diamond dependency. If we were to separate the hbase code out of the timeline service module, then we may need to create 2 more modules, one for hbase client and another for hbase co-processor. But they have dependencies between them still. So we'd need to untangle them. I know it's bit dense, but please feel free to reach out to me so we can discuss more. > Move HBase backend code in ATS v2 into its separate module > --- > > Key: YARN-5667 > URL: https://issues.apache.org/jira/browse/YARN-5667 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Reporter: Haibo Chen >Assignee: Haibo Chen > > The HBase backend code currently lives along with the core ATS v2 code in > hadoop-yarn-server-timelineservice module. Because Resource Manager depends > on hadoop-yarn-server-timelineservice, an unnecessary dependency of the RM > module on HBase modules is introduced (HBase backend is pluggable, so we do > not need to directly pull in HBase jars). > In our internal effort to try ATS v2 with HBase 2.0 which depends on Hadoop > 3, we encountered a circular dependency during our builds between HBase2.0 > and Hadoop3 artifacts. > {code} > hadoop-mapreduce-client-common, hadoop-yarn-client, > hadoop-yarn-server-resourcemanager, hadoop-yarn-server-timelineservice, > hbase-server, hbase-prefix-tree, hbase-hadoop2-compat, > hadoop-mapreduce-client-jobclient, hadoop-mapreduce-client-common] > {code} > This jira proposes we move all HBase-backend-related code from > hadoop-yarn-server-timelineservice into its own module (possible name is > yarn-server-timelineservice-storage) so that core RM modules do not depend on > HBase modules any more. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5667) Move HBase backend code in ATS v2 into its separate module
[ https://issues.apache.org/jira/browse/YARN-5667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15521921#comment-15521921 ] Haibo Chen commented on YARN-5667: -- [~sjlee0] Any reason that I am unaware of, but makes it necessary to have HBase code stay within hadoop-yarn-server-timelineservice module? > Move HBase backend code in ATS v2 into its separate module > --- > > Key: YARN-5667 > URL: https://issues.apache.org/jira/browse/YARN-5667 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Reporter: Haibo Chen >Assignee: Haibo Chen > > The HBase backend code currently lives along with the core ATS v2 code in > hadoop-yarn-server-timelineservice module. Because Resource Manager depends > on hadoop-yarn-server-timelineservice, an unnecessary dependency of the RM > module on HBase modules is introduced (HBase backend is pluggable, so we do > not need to directly pull in HBase jars). > In our internal effort to try ATS v2 with HBase 2.0 which depends on Hadoop > 3, we encountered a circular dependency during our builds between HBase2.0 > and Hadoop3 artifacts. > {code} > hadoop-mapreduce-client-common, hadoop-yarn-client, > hadoop-yarn-server-resourcemanager, hadoop-yarn-server-timelineservice, > hbase-server, hbase-prefix-tree, hbase-hadoop2-compat, > hadoop-mapreduce-client-jobclient, hadoop-mapreduce-client-common] > {code} > This jira proposes we move all HBase-backend-related code from > hadoop-yarn-server-timelineservice into its own module (possible name is > yarn-server-timelineservice-storage) so that core RM modules do not depend on > HBase modules any more. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org