Abhishek, next time you do a bulk change, please remember to tell it not to send out an email for every change :)
On Tue, Aug 2, 2016 at 4:28 PM, Robert Kanter <[email protected]> wrote: > Rohini, that sounds good to me. Especially because we're on "master" > branch but "trunk" version in JIRA and it gets a little confusing. Instead > of deleting "trunk", I think we're better off just renaming it to the > release number. Otherwise, we'll have to re-label any current "trunk" > fields. We can use 4.4.0 for now, but I imagine that the next release > might end up being 5.0.0 (with Oozie on Yarn); we can always rename it > later. > > I'm not sure if there's a specific rule here, but I suppose we should > probably have a vote in the private list to make things official as this is > a release-management-related change. Can you send an email? > > Abhishek, thanks for putting this all together. I think Rohini's > suggestion of updating the JIRAs is the right next step. You're probably > best off using the bulk change feature instead of doing each JIRA > individually. I should have time to do a bunch of reviews this week, I've > actually already started. I should have some time to continue doing more > reviews. As a general thing, we've been pretty slow about getting these > done :( > > - Robert > > On Tue, Aug 2, 2016 at 4:25 PM, Rohini Palaniswamy < > [email protected]> wrote: > >> I also did mark some of the enhancement jiras that Satish was working on >> and which might take time to review to 4.4.0 >> >> Regards, >> Rohini >> >> On Tue, Aug 2, 2016 at 3:01 PM, Rohini Palaniswamy < >> [email protected]> >> wrote: >> >> > Abhishek, >> > Can you mark all jiras already committed to trunk as 4.3.0 and those >> > planned in the scope document as 4.3.0? This will help filter on those >> > and review instead of referring to a google doc. I can pick up 10-15 >> > jiras for review end of this week. We should try to get majority of >> patches >> > in the next 10 days and branch by Aug 12th. You can then use the 3rd >> week >> > for any pending critical issues and rolling the release candidate. >> > >> > Robert, >> > I am thinking, that we should get rid of the trunk version and use >> > release numbers when committing from the beginning like all other >> projects >> > do. Unnecessary step to change from trunk to release version for every >> > release. If folks are in agreement, then will delete the trunk version >> > altogether. Does that require a mail to private? >> > >> > Regards, >> > Rohini >> > >> > On Fri, Jul 29, 2016 at 3:38 AM, Abhishek Bafna <[email protected]> >> > wrote: >> > >> >> Hi All, >> >> >> >> As we prepared a Oozie 4.3.0 scope document < >> >> >> https://docs.google.com/document/d/1E1vGTul0hrWC4R8EgCRkO3ZmGNT5GmFsuHQZmrT63Ow/edit#heading=h.w9ycxfry7sfu >> > >> >> and listed all the things which we wanted to include in the release. >> In the >> >> last few days, I spent some time and reviewed some of the patches and >> >> provided my feedback. I will continue my efforts to review more >> patches. >> >> These patches needs to be reviewed by committers and get committed. I >> >> request other contributors also, review some of the patches and provide >> >> their feedback. >> >> >> >> I am thinking, may be, we can decide upon some fixed (approximate) >> >> release date and start working accordingly. Because, If we wait for >> all the >> >> listed items to be reviewed and committed, then it might take a lot >> more >> >> time. I am thinking we should target third week of August to complete >> and >> >> release. >> >> >> >> Feel free to provide your feedback and suggestions. >> >> >> >> Thanks, >> >> Abhishek >> >> > On Jul 21, 2016, at 4:39 PM, Abhishek Bafna <[email protected]> >> >> wrote: >> >> > >> >> > Hi, >> >> > >> >> > I looked into the available patches and prepared the list. I have >> put, >> >> scope content which we have discussed till now, into a word document. >> (For >> >> the better formatting and easy to read and comment). >> >> > >> >> > I have noted the additional Jira into the list. Added two more Jira >> >> related to spark (Scala version and Spark 2.0 property). >> >> > >> >> > Please take a look at the document (link below) and provide your >> >> comments. >> >> > >> >> > If this looks good, we can start the release process. >> >> > >> >> > Scope Document: >> >> >> https://docs.google.com/document/d/1E1vGTul0hrWC4R8EgCRkO3ZmGNT5GmFsuHQZmrT63Ow/edit# >> >> > >> >> > Thanks, >> >> > Abhishek >> >> > >> >> >> On Jul 21, 2016, at 5:07 AM, Robert Kanter <[email protected]> >> >> wrote: >> >> >> >> >> >> That list looks good to me. The only one I'm not sure of is >> OOZIE-2306 >> >> >> (Oozie Health Check). It's a new feature and it might make sense to >> >> let it >> >> >> "bake" a while first? I'm also not sure it's quite ready yet. >> >> >> >> >> >> I'd also like to add a few others to the list: >> >> >> - OOZIE-2273 (MiniOozie does not work outside of Oozie) >> >> >> - The subtasks under OOZIE-2036 (Drop support for Java 1.6) >> >> >> --- OOZIE-2037 (Add TLSv1.1,TLSv1.2) >> >> >> --- OOZIE-1793 (Improve find bugs reporting for Oozie) >> >> >> --- OOZIE-2552 (Update ActiveMQ version for security and other >> fixes) >> >> >> >> >> >> There's a number of other Patch Available JIRAs not on your list >> >> (including >> >> >> the ones I mentioned above). While we don't need to include >> everything >> >> >> here (and we probably shouldn't nor do we have the review >> bandwidth), >> >> did >> >> >> you consider each of them when creating the list? I currently see >> 97 >> >> >> JIRAs: https://s.apache.org/oozie-patch-available >> >> >> I think we should include all Spark Action fixes/improvements, given >> >> how >> >> >> popular it seems to be. >> >> >> >> >> >> thanks >> >> >> - Robert >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> On Wed, Jul 20, 2016 at 5:00 AM, Abhishek Bafna < >> [email protected]> >> >> >> wrote: >> >> >> >> >> >>> Hi All, >> >> >>> >> >> >>> Below is the list of issues, picked from currently available >> patches. >> >> I >> >> >>> could not decide up any 'open' issue, If there is anything, please >> >> reply >> >> >>> here. >> >> >>> >> >> >>> We can 'drop/add' more if required based on the feedback. Important >> >> is, >> >> >>> how many of these we will be able to review and commit. >> >> >>> >> >> >>> I will starting looking into these issues and provide feedback. >> >> >>> >> >> >> --------------------------------------------------------------------------- >> >> >>> OOZIE-1978 Forkjoin validation code is ridiculously slow in >> some >> >> cases >> >> >>> OOZIE-2508 Documentation change for Coord action rerun >> >> [OOZIE-1735] >> >> >>> OOZIE-2583 oozie throws EL Exception when reference variable >> name >> >> >>> containing dot >> >> >>> OOZIE-2582 Populating external child Ids for action failures >> >> >>> OOZIE-2517 Add support for startCreatedTime and endCreatedTime >> >> >>> filters for coord and bundles >> >> >>> OOZIE-2512 ShareLibservice returns incorrect path for jar >> >> >>> OOZIE-2572 SLA DURATION miss not shown when job is running for >> >> longer >> >> >>> than expected time >> >> >>> OOZIE-2457 Oozie log parsing regex consume more than 90% cpu >> >> >>> OOZIE-2574 Oozie to support replication-enabled mysql urls >> >> >>> OOZIE-2506 Add logs into RecoverService for logging >> information >> >> about >> >> >>> queued commnads >> >> >>> OOZIE-2573 dataset url which contains spaces can not be >> handled >> >> >>> rightly >> >> >>> OOZIE-2569 Adding yarn-site, core-site, hdfs-site and >> mapred-site >> >> >>> into spark launcher >> >> >>> OOZIE-2491 oozie acl cannot specify group,it does`t work >> >> >>> OOZIE-2539 Incorrect property key is used for 'hive log4j >> >> >>> configuration file for execution mode' >> >> >>> OOZIE-2555 Oozie SSL enable setup does not return port for >> admin >> >> >>> -servers >> >> >>> OOZIE-2515 Duplicate information for "Changing >> endtime/pausetime >> >> of a >> >> >>> Bundle Job" in CommandLineTool wiki >> >> >>> OOZIE-2498 Oozie CallerId configuration for downstream >> components >> >> >>> OOZIE-2516 Update web service documentation for jobs API >> >> >>> OOZIE-2500 -DtestJarSimple option mentioned in minioozie doc >> >> does not >> >> >>> work >> >> >>> OOZIE-2513 log.scan.duration should not be used for error and >> >> audit >> >> >>> logs >> >> >>> OOZIE-2461 Workflow, Coordinator and Bundle job querying >> should >> >> have >> >> >>> last modified filter >> >> >>> OOZIE-2306 Oozie health check >> >> >>> OOZIE-2518 Oozie is creating many files and directories in >> /tmp >> >> per >> >> >>> day >> >> >>> OOZIE-2409 hive2 action with hive 1.2.1 failed >> >> >>> OOZIE-2473 Connection pool for SMTP connection >> >> >>> OOZIE-2440 Exponential re-try policy for workflow action >> >> >>> OOZIE-2458 'oozie-setup.sh sharelib create' should ensure >> >> uploaded >> >> >>> jars are world readable >> >> >>> OOZIE-2390 Rerun with failed option removing completed output >> >> data >> >> >>> OOZIE-1810 Workflow cannot get into Failed state when kill >> >> control >> >> >>> node cannot resolve variable in message >> >> >>> OOZIE-2244 Oozie should mask passwords in the logs when >> logging >> >> >>> command arguments >> >> >>> >> >> >> --------------------------------------------------------------------------- >> >> >>> Thanks, >> >> >>> Abhishek >> >> >>> >> >> >>>> On Jul 18, 2016, at 6:35 PM, Abhishek Bafna <[email protected] >> > >> >> >>> wrote: >> >> >>>> >> >> >>>> Hi All, >> >> >>>> >> >> >>>> I looked into the commits (approx. 160), all looked good to me and >> >> >>> should be included into the release. If there are things, which >> needs >> >> to be >> >> >>> excluded, provide your comments. >> >> >>>> >> >> >>>> We still need to decide on the patch which are in progress status. >> >> >>>> >> >> >>>> On the upgrade side: >> >> >>>> Java version 1.6 to 1.7 >> >> >>>> OpenJPA version 2.2.2 to 2.4.1 >> >> >>>> Curator version 2.5.0 to 2.11.0 >> >> >>>> Pig version 0.12.1 to 0.16.0 >> >> >>>> Hive version 0.13.1 to 1.2.0 >> >> >>>> HttpClient version 4.3 to 4.3.6 >> >> >>>> commons-httpclient version 3.1 to 4.3.6 (discontinued) >> >> >>>> >> >> >>>> Please provide your feedback on the above upgrades. >> >> >>>> >> >> >>>> Once we decide on the scope of the release. We will start working >> on >> >> the >> >> >>> completing/testing the release. >> >> >>>> >> >> >>>> Thanks, >> >> >>>> Abhishek >> >> >>>> >> >> >>>>> On Jul 16, 2016, at 2:40 AM, Abhishek Bafna < >> [email protected] >> >> >>> <mailto:[email protected]>> wrote: >> >> >>>>> >> >> >>>>> Okay. We will make the Java version to 1.7. >> >> >>>>> >> >> >>>>> There are upgrades we can do like OpenJPA, Curator, HttpClient, >> Pig >> >> etc. >> >> >>>>> >> >> >>>>> HttpClient, Pig (0.16) version upgrade we have done internally as >> >> well. >> >> >>>>> >> >> >>>>> As per the comments on the OOZIE-2231 < >> >> >>> https://issues.apache.org/jira/browse/OOZIE-2231>, curator version >> >> also >> >> >>> can be upgraded to 2.11. >> >> >>>>> >> >> >>>>> Thanks, >> >> >>>>> Abhishek >> >> >>>>>> On Jul 16, 2016, at 2:06 AM, Robert Kanter < >> [email protected] >> >> >>> <mailto:[email protected]>> wrote: >> >> >>>>>> >> >> >>>>>> We've been using a version of Hive/HCat based on 1.1.0 for over >> a >> >> year >> >> >>> now >> >> >>>>>> with no problems. We did have to comment out a few lines in >> >> >>> MiniHCatServer >> >> >>>>>> and MiniHS2 that were no longer needed: >> >> >>>>>> >> >> >>>>>> diff --git >> >> >>> a/core/src/test/java/org/apache/oozie/test/MiniHCatServer.java >> >> >>>>>> b/core/src/test/java/org/apache/oozie/test/MiniHCatServer.java >> >> >>>>>> index 8699ff8..d694977 100644 >> >> >>>>>> --- >> a/core/src/test/java/org/apache/oozie/test/MiniHCatServer.java >> >> >>>>>> +++ >> b/core/src/test/java/org/apache/oozie/test/MiniHCatServer.java >> >> >>>>>> @@ -150,9 +150,10 @@ public class MiniHCatServer { >> >> >>>>>> public static void resetDefaultDBCreation() throws Exception { >> >> >>>>>> // Need to do this, else default db will not be created >> for >> >> >>> local >> >> >>>>>> metastores. >> >> >>>>>> // TestHiveMain will fail with >> >> >>>>>> InvalidObjectException(message:There is no database named >> default) >> >> >>>>>> - Field declaredField = >> >> >>>>>> HMSHandler.class.getDeclaredField("createDefaultDB"); >> >> >>>>>> - declaredField.setAccessible(true); >> >> >>>>>> - declaredField.set(null, false); >> >> >>>>>> + // CLOUDERA-BUILD: Don't need this for Hive 0.14 and >> later >> >> >>>>>> + //Field declaredField = >> >> >>>>>> HMSHandler.class.getDeclaredField("createDefaultDB"); >> >> >>>>>> + //declaredField.setAccessible(true); >> >> >>>>>> + //declaredField.set(null, false); >> >> >>>>>> } >> >> >>>>>> >> >> >>>>>> public static void resetHiveConfStaticVariables() throws >> >> Exception >> >> >>> { >> >> >>>>>> diff --git >> >> a/core/src/test/java/org/apache/oozie/test/hive/MiniHS2.java >> >> >>>>>> b/core/src/test/java/org/apache/oozie/test/hive/MiniHS2.java >> >> >>>>>> index c9dfd56..505dc55 100644 >> >> >>>>>> --- a/core/src/test/java/org/apache/oozie/test/hive/MiniHS2.java >> >> >>>>>> +++ b/core/src/test/java/org/apache/oozie/test/hive/MiniHS2.java >> >> >>>>>> @@ -84,7 +84,8 @@ public class MiniHS2 extends >> AbstractHiveService >> >> { >> >> >>>>>> hiveConf.setVar(ConfVars.HIVE_SERVER2_THRIFT_BIND_HOST, >> >> getHost()); >> >> >>>>>> hiveConf.setIntVar(ConfVars.HIVE_SERVER2_THRIFT_PORT, >> >> >>> getBinaryPort()); >> >> >>>>>> hiveConf.setIntVar(ConfVars.HIVE_SERVER2_THRIFT_HTTP_PORT, >> >> >>>>>> getHttpPort()); >> >> >>>>>> - HiveMetaStore.HMSHandler.resetDefaultDBFlag(); >> >> >>>>>> + // CLOUDERA-BUILD: Don't need this for Hive 0.14 and later >> >> >>>>>> + //HiveMetaStore.HMSHandler.resetDefaultDBFlag(); >> >> >>>>>> >> >> >>>>>> Path scratchDir = new Path(baseDfsDir, "scratch"); >> >> >>>>>> fs.mkdirs(scratchDir); >> >> >>>>>> >> >> >>>>>> On Fri, Jul 15, 2016 at 1:29 PM, Venkat Ranganathan < >> >> >>>>>> [email protected] <mailto: >> [email protected] >> >> >> >> >> >>> wrote: >> >> >>>>>> >> >> >>>>>>> Let us explore changing the default hive version also to 1.x. >> >> 0.13 >> >> >>> is >> >> >>>>>>> old >> >> >>>>>>> >> >> >>>>>>> Thanks >> >> >>>>>>> >> >> >>>>>>> Venkat >> >> >>>>>>> >> >> >>>>>>> >> >> >>>>>>> On 7/15/16, 11:31 AM, "Robert Kanter" <[email protected] >> >> <mailto: >> >> >>> [email protected]>> wrote: >> >> >>>>>>> >> >> >>>>>>> I don't think we have any guidelines defined anywhere about >> when >> >> >>> it's >> >> >>>>>>> okay >> >> >>>>>>> to change the minimum Java version, but perhaps we need to >> wait >> >> >>> until >> >> >>>>>>> the >> >> >>>>>>> next major release (5.0)? That said, Hadoop did this in a >> minor >> >> >>>>>>> release >> >> >>>>>>> and Java 6 was EoL in 2013. So I'm fine with doing it in 4.3 >> if >> >> >>>>>>> nobody has >> >> >>>>>>> an objections; we should just make sure to call that out in >> our >> >> >>> release >> >> >>>>>>> notes. >> >> >>>>>>> >> >> >>>>>>> Otherwise, nothing comes to mind for anything we should leave >> out >> >> >>> other >> >> >>>>>>> than the Oozie on Yarn stuff, which is not currently in trunk >> >> >>> anyway. >> >> >>>>>>> >> >> >>>>>>> I know there's a number of Spark Action JIRAs; we should make >> >> sure >> >> >>> to >> >> >>>>>>> include all of these and any pending ones. >> >> >>>>>>> @Rohini, @Puru, @Satish, anything you think needs more time to >> >> bake >> >> >>>>>>> that we >> >> >>>>>>> should hold off on including in the release? >> >> >>>>>>> >> >> >>>>>>> >> >> >>>>>>> thanks >> >> >>>>>>> - Robert >> >> >>>>>>> >> >> >>>>>>> On Fri, Jul 15, 2016 at 8:35 AM, Abhishek Bafna < >> >> >>> [email protected] <mailto:[email protected]>> >> >> >>>>>>> wrote: >> >> >>>>>>> >> >> >>>>>>>> Hi All, >> >> >>>>>>>> >> >> >>>>>>>> Any suggestions or comments on the scope for the release? >> >> >>>>>>>> >> >> >>>>>>>> Thanks, >> >> >>>>>>>> Abhishek >> >> >>>>>>>> >> >> >>>>>>>>> On Jul 13, 2016, at 3:07 AM, Abhishek Bafna < >> >> [email protected] >> >> >>> <mailto:[email protected]>> >> >> >>>>>>>> wrote: >> >> >>>>>>>>> >> >> >>>>>>>>> Hi All, >> >> >>>>>>>>> >> >> >>>>>>>>> Oozie is starting the release process for Oozie 4.3.0. This >> >> thread >> >> >>>>>>> is >> >> >>>>>>>> for discussion about the scope of the release. Please provide >> >> your >> >> >>>>>>>> suggestion about what should be included (or any exclusions) >> in >> >> the >> >> >>>>>>> release. >> >> >>>>>>>>> >> >> >>>>>>>>> Currently, we have around 158 patches submitted to trunk. >> Based >> >> on >> >> >>>>>>> the >> >> >>>>>>>> scope, we will include (exclude) more if required. >> >> >>>>>>>>> Version Upgrade >> >> >>>>>>>>> Java 1.7 (OOZIE-2036 < >> >> >>>>>>> https://issues.apache.org/jira/browse/OOZIE-2036 < >> >> >>> https://issues.apache.org/jira/browse/OOZIE-2036>>) >> >> >>>>>>>>> OpenJPA (OOZIE-2488 < >> >> >>>>>>> https://issues.apache.org/jira/browse/OOZIE-2488 < >> >> >>> https://issues.apache.org/jira/browse/OOZIE-2488>>), >> >> >>>>>>>>> Curator (OOZIE-2231 < >> >> >>>>>>> https://issues.apache.org/jira/browse/OOZIE-2231 < >> >> >>> https://issues.apache.org/jira/browse/OOZIE-2231> >> >> >>>>>>>>> ), >> >> >>>>>>>>> HttpClient (OOZIE-2538 < >> >> >>>>>>>> https://issues.apache.org/jira/browse/OOZIE-2538 < >> >> >>> https://issues.apache.org/jira/browse/OOZIE-2538>>), >> >> >>>>>>>>> Pig (OOZIE-2602 < >> >> https://issues.apache.org/jira/browse/OOZIE-2602 < >> >> >>> https://issues.apache.org/jira/browse/OOZIE-2602> >> >> >>>>>>>> ). >> >> >>>>>>>>> New Features/Improvements: Add default JT and NN, Database >> >> schema >> >> >>>>>>>> service check, instrumental metrics, Tomcat upgrade etc. >> >> >>>>>>>>> Bug Fixes. >> >> >>>>>>>>> Provide your opinion on inclusion/exclusion of the above >> points >> >> or >> >> >>>>>>> more >> >> >>>>>>>> for the scope of the release. We can also, take some work >> which >> >> is >> >> >>>>>>> already >> >> >>>>>>>> done and review is pending for them (depending on the >> >> availability >> >> >>>>>>> of the >> >> >>>>>>>> reviewers). >> >> >>>>>>>>> >> >> >>>>>>>>> Thanks, >> >> >>>>>>>>> Abhishek >> >> >>>>>>>> >> >> >>>>>>>> >> >> >>>>>>> >> >> >>>>>>> >> >> >>>>>>> >> >> >>>>> >> >> >>>> >> >> >>> >> >> >>> >> >> > >> >> >> >> >> > >> > >
