Hi, I looked into the available patches and prepared the list. I have put, scope content which we have discussed till now, into a word document. (For the better formatting and easy to read and comment).
I have noted the additional Jira into the list. Added two more Jira related to spark (Scala version and Spark 2.0 property). Please take a look at the document (link below) and provide your comments. If this looks good, we can start the release process. Scope Document: https://docs.google.com/document/d/1E1vGTul0hrWC4R8EgCRkO3ZmGNT5GmFsuHQZmrT63Ow/edit# Thanks, Abhishek > On Jul 21, 2016, at 5:07 AM, Robert Kanter <[email protected]> wrote: > > That list looks good to me. The only one I'm not sure of is OOZIE-2306 > (Oozie Health Check). It's a new feature and it might make sense to let it > "bake" a while first? I'm also not sure it's quite ready yet. > > I'd also like to add a few others to the list: > - OOZIE-2273 (MiniOozie does not work outside of Oozie) > - The subtasks under OOZIE-2036 (Drop support for Java 1.6) > --- OOZIE-2037 (Add TLSv1.1,TLSv1.2) > --- OOZIE-1793 (Improve find bugs reporting for Oozie) > --- OOZIE-2552 (Update ActiveMQ version for security and other fixes) > > There's a number of other Patch Available JIRAs not on your list (including > the ones I mentioned above). While we don't need to include everything > here (and we probably shouldn't nor do we have the review bandwidth), did > you consider each of them when creating the list? I currently see 97 > JIRAs: https://s.apache.org/oozie-patch-available > I think we should include all Spark Action fixes/improvements, given how > popular it seems to be. > > thanks > - Robert > > > > > On Wed, Jul 20, 2016 at 5:00 AM, Abhishek Bafna <[email protected]> > wrote: > >> Hi All, >> >> Below is the list of issues, picked from currently available patches. I >> could not decide up any 'open' issue, If there is anything, please reply >> here. >> >> We can 'drop/add' more if required based on the feedback. Important is, >> how many of these we will be able to review and commit. >> >> I will starting looking into these issues and provide feedback. >> --------------------------------------------------------------------------- >> OOZIE-1978 Forkjoin validation code is ridiculously slow in some cases >> OOZIE-2508 Documentation change for Coord action rerun [OOZIE-1735] >> OOZIE-2583 oozie throws EL Exception when reference variable name >> containing dot >> OOZIE-2582 Populating external child Ids for action failures >> OOZIE-2517 Add support for startCreatedTime and endCreatedTime >> filters for coord and bundles >> OOZIE-2512 ShareLibservice returns incorrect path for jar >> OOZIE-2572 SLA DURATION miss not shown when job is running for longer >> than expected time >> OOZIE-2457 Oozie log parsing regex consume more than 90% cpu >> OOZIE-2574 Oozie to support replication-enabled mysql urls >> OOZIE-2506 Add logs into RecoverService for logging information about >> queued commnads >> OOZIE-2573 dataset url which contains spaces can not be handled >> rightly >> OOZIE-2569 Adding yarn-site, core-site, hdfs-site and mapred-site >> into spark launcher >> OOZIE-2491 oozie acl cannot specify group,it does`t work >> OOZIE-2539 Incorrect property key is used for 'hive log4j >> configuration file for execution mode' >> OOZIE-2555 Oozie SSL enable setup does not return port for admin >> -servers >> OOZIE-2515 Duplicate information for "Changing endtime/pausetime of a >> Bundle Job" in CommandLineTool wiki >> OOZIE-2498 Oozie CallerId configuration for downstream components >> OOZIE-2516 Update web service documentation for jobs API >> OOZIE-2500 -DtestJarSimple option mentioned in minioozie doc does not >> work >> OOZIE-2513 log.scan.duration should not be used for error and audit >> logs >> OOZIE-2461 Workflow, Coordinator and Bundle job querying should have >> last modified filter >> OOZIE-2306 Oozie health check >> OOZIE-2518 Oozie is creating many files and directories in /tmp per >> day >> OOZIE-2409 hive2 action with hive 1.2.1 failed >> OOZIE-2473 Connection pool for SMTP connection >> OOZIE-2440 Exponential re-try policy for workflow action >> OOZIE-2458 'oozie-setup.sh sharelib create' should ensure uploaded >> jars are world readable >> OOZIE-2390 Rerun with failed option removing completed output data >> OOZIE-1810 Workflow cannot get into Failed state when kill control >> node cannot resolve variable in message >> OOZIE-2244 Oozie should mask passwords in the logs when logging >> command arguments >> --------------------------------------------------------------------------- >> Thanks, >> Abhishek >> >>> On Jul 18, 2016, at 6:35 PM, Abhishek Bafna <[email protected]> >> wrote: >>> >>> Hi All, >>> >>> I looked into the commits (approx. 160), all looked good to me and >> should be included into the release. If there are things, which needs to be >> excluded, provide your comments. >>> >>> We still need to decide on the patch which are in progress status. >>> >>> On the upgrade side: >>> Java version 1.6 to 1.7 >>> OpenJPA version 2.2.2 to 2.4.1 >>> Curator version 2.5.0 to 2.11.0 >>> Pig version 0.12.1 to 0.16.0 >>> Hive version 0.13.1 to 1.2.0 >>> HttpClient version 4.3 to 4.3.6 >>> commons-httpclient version 3.1 to 4.3.6 (discontinued) >>> >>> Please provide your feedback on the above upgrades. >>> >>> Once we decide on the scope of the release. We will start working on the >> completing/testing the release. >>> >>> Thanks, >>> Abhishek >>> >>>> On Jul 16, 2016, at 2:40 AM, Abhishek Bafna <[email protected] >> <mailto:[email protected]>> wrote: >>>> >>>> Okay. We will make the Java version to 1.7. >>>> >>>> There are upgrades we can do like OpenJPA, Curator, HttpClient, Pig etc. >>>> >>>> HttpClient, Pig (0.16) version upgrade we have done internally as well. >>>> >>>> As per the comments on the OOZIE-2231 < >> https://issues.apache.org/jira/browse/OOZIE-2231>, curator version also >> can be upgraded to 2.11. >>>> >>>> Thanks, >>>> Abhishek >>>>> On Jul 16, 2016, at 2:06 AM, Robert Kanter <[email protected] >> <mailto:[email protected]>> wrote: >>>>> >>>>> We've been using a version of Hive/HCat based on 1.1.0 for over a year >> now >>>>> with no problems. We did have to comment out a few lines in >> MiniHCatServer >>>>> and MiniHS2 that were no longer needed: >>>>> >>>>> diff --git >> a/core/src/test/java/org/apache/oozie/test/MiniHCatServer.java >>>>> b/core/src/test/java/org/apache/oozie/test/MiniHCatServer.java >>>>> index 8699ff8..d694977 100644 >>>>> --- a/core/src/test/java/org/apache/oozie/test/MiniHCatServer.java >>>>> +++ b/core/src/test/java/org/apache/oozie/test/MiniHCatServer.java >>>>> @@ -150,9 +150,10 @@ public class MiniHCatServer { >>>>> public static void resetDefaultDBCreation() throws Exception { >>>>> // Need to do this, else default db will not be created for >> local >>>>> metastores. >>>>> // TestHiveMain will fail with >>>>> InvalidObjectException(message:There is no database named default) >>>>> - Field declaredField = >>>>> HMSHandler.class.getDeclaredField("createDefaultDB"); >>>>> - declaredField.setAccessible(true); >>>>> - declaredField.set(null, false); >>>>> + // CLOUDERA-BUILD: Don't need this for Hive 0.14 and later >>>>> + //Field declaredField = >>>>> HMSHandler.class.getDeclaredField("createDefaultDB"); >>>>> + //declaredField.setAccessible(true); >>>>> + //declaredField.set(null, false); >>>>> } >>>>> >>>>> public static void resetHiveConfStaticVariables() throws Exception >> { >>>>> diff --git a/core/src/test/java/org/apache/oozie/test/hive/MiniHS2.java >>>>> b/core/src/test/java/org/apache/oozie/test/hive/MiniHS2.java >>>>> index c9dfd56..505dc55 100644 >>>>> --- a/core/src/test/java/org/apache/oozie/test/hive/MiniHS2.java >>>>> +++ b/core/src/test/java/org/apache/oozie/test/hive/MiniHS2.java >>>>> @@ -84,7 +84,8 @@ public class MiniHS2 extends AbstractHiveService { >>>>> hiveConf.setVar(ConfVars.HIVE_SERVER2_THRIFT_BIND_HOST, getHost()); >>>>> hiveConf.setIntVar(ConfVars.HIVE_SERVER2_THRIFT_PORT, >> getBinaryPort()); >>>>> hiveConf.setIntVar(ConfVars.HIVE_SERVER2_THRIFT_HTTP_PORT, >>>>> getHttpPort()); >>>>> - HiveMetaStore.HMSHandler.resetDefaultDBFlag(); >>>>> + // CLOUDERA-BUILD: Don't need this for Hive 0.14 and later >>>>> + //HiveMetaStore.HMSHandler.resetDefaultDBFlag(); >>>>> >>>>> Path scratchDir = new Path(baseDfsDir, "scratch"); >>>>> fs.mkdirs(scratchDir); >>>>> >>>>> On Fri, Jul 15, 2016 at 1:29 PM, Venkat Ranganathan < >>>>> [email protected] <mailto:[email protected]>> >> wrote: >>>>> >>>>>> Let us explore changing the default hive version also to 1.x. 0.13 >> is >>>>>> old >>>>>> >>>>>> Thanks >>>>>> >>>>>> Venkat >>>>>> >>>>>> >>>>>> On 7/15/16, 11:31 AM, "Robert Kanter" <[email protected] <mailto: >> [email protected]>> wrote: >>>>>> >>>>>> I don't think we have any guidelines defined anywhere about when >> it's >>>>>> okay >>>>>> to change the minimum Java version, but perhaps we need to wait >> until >>>>>> the >>>>>> next major release (5.0)? That said, Hadoop did this in a minor >>>>>> release >>>>>> and Java 6 was EoL in 2013. So I'm fine with doing it in 4.3 if >>>>>> nobody has >>>>>> an objections; we should just make sure to call that out in our >> release >>>>>> notes. >>>>>> >>>>>> Otherwise, nothing comes to mind for anything we should leave out >> other >>>>>> than the Oozie on Yarn stuff, which is not currently in trunk >> anyway. >>>>>> >>>>>> I know there's a number of Spark Action JIRAs; we should make sure >> to >>>>>> include all of these and any pending ones. >>>>>> @Rohini, @Puru, @Satish, anything you think needs more time to bake >>>>>> that we >>>>>> should hold off on including in the release? >>>>>> >>>>>> >>>>>> thanks >>>>>> - Robert >>>>>> >>>>>> On Fri, Jul 15, 2016 at 8:35 AM, Abhishek Bafna < >> [email protected] <mailto:[email protected]>> >>>>>> wrote: >>>>>> >>>>>>> Hi All, >>>>>>> >>>>>>> Any suggestions or comments on the scope for the release? >>>>>>> >>>>>>> Thanks, >>>>>>> Abhishek >>>>>>> >>>>>>>> On Jul 13, 2016, at 3:07 AM, Abhishek Bafna <[email protected] >> <mailto:[email protected]>> >>>>>>> wrote: >>>>>>>> >>>>>>>> Hi All, >>>>>>>> >>>>>>>> Oozie is starting the release process for Oozie 4.3.0. This thread >>>>>> is >>>>>>> for discussion about the scope of the release. Please provide your >>>>>>> suggestion about what should be included (or any exclusions) in the >>>>>> release. >>>>>>>> >>>>>>>> Currently, we have around 158 patches submitted to trunk. Based on >>>>>> the >>>>>>> scope, we will include (exclude) more if required. >>>>>>>> Version Upgrade >>>>>>>> Java 1.7 (OOZIE-2036 < >>>>>> https://issues.apache.org/jira/browse/OOZIE-2036 < >> https://issues.apache.org/jira/browse/OOZIE-2036>>) >>>>>>>> OpenJPA (OOZIE-2488 < >>>>>> https://issues.apache.org/jira/browse/OOZIE-2488 < >> https://issues.apache.org/jira/browse/OOZIE-2488>>), >>>>>>>> Curator (OOZIE-2231 < >>>>>> https://issues.apache.org/jira/browse/OOZIE-2231 < >> https://issues.apache.org/jira/browse/OOZIE-2231> >>>>>>>> ), >>>>>>>> HttpClient (OOZIE-2538 < >>>>>>> https://issues.apache.org/jira/browse/OOZIE-2538 < >> https://issues.apache.org/jira/browse/OOZIE-2538>>), >>>>>>>> Pig (OOZIE-2602 <https://issues.apache.org/jira/browse/OOZIE-2602 < >> https://issues.apache.org/jira/browse/OOZIE-2602> >>>>>>> ). >>>>>>>> New Features/Improvements: Add default JT and NN, Database schema >>>>>>> service check, instrumental metrics, Tomcat upgrade etc. >>>>>>>> Bug Fixes. >>>>>>>> Provide your opinion on inclusion/exclusion of the above points or >>>>>> more >>>>>>> for the scope of the release. We can also, take some work which is >>>>>> already >>>>>>> done and review is pending for them (depending on the availability >>>>>> of the >>>>>>> reviewers). >>>>>>>> >>>>>>>> Thanks, >>>>>>>> Abhishek >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> >>>> >>> >> >>
