Hi Gunnar, Agree with everything you state above, except "However, YARN is required for some of the backup/restore..." I am sorry I said the wrong thing way up in this thread. backup/restore uses map reduce to copy files. I think it will work with both MRv1 and MRv2. So we should be good with this line alone "you install Hive and whatever version of MapReduce you want to use for Hive". backup/restore should be able to use the same version of MapReduce as the one Hive is using.
However there is a caveat that Trafodion backup/restore use HBase's ExportSnapshot java class. If this class is changed such that it can use only MRv2 then we will have the same dependency. If we are looking for simple install instructions then maybe we should just say that MRv2 (YARN) is required and can be used for both Hive and backup/restore? Thanks Suresh On Tue, Feb 2, 2016 at 6:20 PM, Gunnar Tapper <[email protected]> wrote: > Hi, > > I am a bit lost. Per previous messages, Hive requires MapReduce. So, > MapReduce must be required for full function. I can see that MapReduce is > not required if you don't use the Hive functionality. > > The Jira Hans pointed to seems to suggest to use MapReduce in lieu of > YARN, which must mean MRv1 since MRv2 is part of YARN. > > From what I now understand, you install Hive and whatever version of > MapReduce you want to use for Hive. However, YARN is required for some of > the backup/restore capabilities so you always need to install MRv2 (since > its part of YARN). So, MRv1 is relevant ONLY IF your installation is using > MRv1 for Hive processing. > > Did I get that right? > > I don't think it's wise to discuss exceptions such as "you don't need > MapReduce if you don't plan to use Hive via Trafodion" in the first > revision of the Trafodion Provisioning Guide. Too many angles dancing on a > needle's head. Instead, let's keep the requirements as simple as we can. > > Thanks, > > Gunnar > > On Tue, Feb 2, 2016 at 3:44 PM, Amanda Moran <[email protected]> > wrote: > >> I have done many installs without Yarn or MapReduce installed at all. >> Trafodion runs fine :) >> >> On Tue, Feb 2, 2016 at 2:39 PM, Hans Zeller <[email protected]> >> wrote: >> >>> No, it is not required in the build environment. >>> >>> Hans >>> >>> On Tue, Feb 2, 2016 at 1:54 PM, Gunnar Tapper <[email protected]> >>> wrote: >>> >>>> Does this mean that MRv1 is now required in the build environment? >>>> >>>> On Tue, Feb 2, 2016 at 2:12 PM, Hans Zeller <[email protected]> >>>> wrote: >>>> >>>>> Hi, >>>>> >>>>> That decision would be made by Hive, not Trafodion. For people who use >>>>> install_local_hadoop, we recently changed that setup to use local >>>>> MapReduce, not YARN, see >>>>> https://issues.apache.org/jira/browse/TRAFODION-1781. >>>>> >>>>> Hans >>>>> >>>>> On Tue, Feb 2, 2016 at 12:58 PM, Gunnar Tapper < >>>>> [email protected]> wrote: >>>>> >>>>>> Hi Suresh: >>>>>> >>>>>> Thanks for the information. >>>>>> >>>>>> Given from what you write, it seems that YARN with MRv2 is required >>>>>> for full functionality. >>>>>> >>>>>> MRv1 is a separate install in current distributions, which is why I >>>>>> am asking about it. How does Trafodion decide to run the MapReduce job as >>>>>> MRv1 vs. MRv2 if both are installed? >>>>>> >>>>>> Thanks, >>>>>> >>>>>> Gunnar >>>>>> >>>>>> >>>>>> >>>>>> On Tue, Feb 2, 2016 at 1:50 PM, Suresh Subbiah < >>>>>> [email protected]> wrote: >>>>>> >>>>>>> Hi, >>>>>>> >>>>>>> I don't think Trafodion requires YARN for most activity. >>>>>>> >>>>>>> For Hive table access, Trafodion uses Hive metadata access Java API >>>>>>> and libhdfs to actually scan the data file. Therefore YARN is not needed >>>>>>> for Hive access. >>>>>>> YARN is not needed for native Hbase or Trafodion table access too. >>>>>>> YARN is needed for backup/restore, since the HBase exportSnapshot >>>>>>> class Trafodion calls, used MapReduce to copy large snapshot files >>>>>>> to/from >>>>>>> the backup location. >>>>>>> YARN is also needed for developer regressions as some vanilla Hive >>>>>>> commands are executed during the regression run. >>>>>>> For the last 2 lines I think both MRv1 and MRv2 is supported. >>>>>>> >>>>>>> Thanks >>>>>>> Suresh >>>>>>> >>>>>>> >>>>>>> On Tue, Feb 2, 2016 at 2:36 PM, Gunnar Tapper < >>>>>>> [email protected]> wrote: >>>>>>> >>>>>>>> Hi, >>>>>>>> >>>>>>>> Does Trafodion require YARN with MRv2 or is MRv1 supported, too? >>>>>>>> >>>>>>>> -- >>>>>>>> Thanks, >>>>>>>> >>>>>>>> Gunnar >>>>>>>> *If you think you can you can, if you think you can't you're right.* >>>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Thanks, >>>>>> >>>>>> Gunnar >>>>>> *If you think you can you can, if you think you can't you're right.* >>>>>> >>>>> >>>>> >>>> >>>> >>>> -- >>>> Thanks, >>>> >>>> Gunnar >>>> *If you think you can you can, if you think you can't you're right.* >>>> >>> >>> >> >> >> -- >> Thanks, >> >> Amanda Moran >> > > > > -- > Thanks, > > Gunnar > *If you think you can you can, if you think you can't you're right.* >
