Abhishek,
Can you mark all jiras already committed to trunk as 4.3.0 and those
planned in the scope document as 4.3.0? This will help filter on those and
review instead of referring to a google doc. I can pick up 10-15 jiras for
review end of this week. We should try to get majority of patches in the
next 10 days and branch by Aug 12th. You can then use the 3rd week for any
pending critical issues and rolling the release candidate.
Robert,
I am thinking, that we should get rid of the trunk version and use
release numbers when committing from the beginning like all other projects
do. Unnecessary step to change from trunk to release version for every
release. If folks are in agreement, then will delete the trunk version
altogether. Does that require a mail to private?
Regards,
Rohini
On Fri, Jul 29, 2016 at 3:38 AM, Abhishek Bafna <[email protected]>
wrote:
> Hi All,
>
> As we prepared a Oozie 4.3.0 scope document <
> https://docs.google.com/document/d/1E1vGTul0hrWC4R8EgCRkO3ZmGNT5GmFsuHQZmrT63Ow/edit#heading=h.w9ycxfry7sfu>
> and listed all the things which we wanted to include in the release. In the
> last few days, I spent some time and reviewed some of the patches and
> provided my feedback. I will continue my efforts to review more patches.
> These patches needs to be reviewed by committers and get committed. I
> request other contributors also, review some of the patches and provide
> their feedback.
>
> I am thinking, may be, we can decide upon some fixed (approximate) release
> date and start working accordingly. Because, If we wait for all the listed
> items to be reviewed and committed, then it might take a lot more time. I
> am thinking we should target third week of August to complete and release.
>
> Feel free to provide your feedback and suggestions.
>
> Thanks,
> Abhishek
> > On Jul 21, 2016, at 4:39 PM, Abhishek Bafna <[email protected]>
> wrote:
> >
> > Hi,
> >
> > I looked into the available patches and prepared the list. I have put,
> scope content which we have discussed till now, into a word document. (For
> the better formatting and easy to read and comment).
> >
> > I have noted the additional Jira into the list. Added two more Jira
> related to spark (Scala version and Spark 2.0 property).
> >
> > Please take a look at the document (link below) and provide your
> comments.
> >
> > If this looks good, we can start the release process.
> >
> > Scope Document:
> https://docs.google.com/document/d/1E1vGTul0hrWC4R8EgCRkO3ZmGNT5GmFsuHQZmrT63Ow/edit#
> >
> > Thanks,
> > Abhishek
> >
> >> On Jul 21, 2016, at 5:07 AM, Robert Kanter <[email protected]>
> wrote:
> >>
> >> That list looks good to me. The only one I'm not sure of is OOZIE-2306
> >> (Oozie Health Check). It's a new feature and it might make sense to
> let it
> >> "bake" a while first? I'm also not sure it's quite ready yet.
> >>
> >> I'd also like to add a few others to the list:
> >> - OOZIE-2273 (MiniOozie does not work outside of Oozie)
> >> - The subtasks under OOZIE-2036 (Drop support for Java 1.6)
> >> --- OOZIE-2037 (Add TLSv1.1,TLSv1.2)
> >> --- OOZIE-1793 (Improve find bugs reporting for Oozie)
> >> --- OOZIE-2552 (Update ActiveMQ version for security and other fixes)
> >>
> >> There's a number of other Patch Available JIRAs not on your list
> (including
> >> the ones I mentioned above). While we don't need to include everything
> >> here (and we probably shouldn't nor do we have the review bandwidth),
> did
> >> you consider each of them when creating the list? I currently see 97
> >> JIRAs: https://s.apache.org/oozie-patch-available
> >> I think we should include all Spark Action fixes/improvements, given how
> >> popular it seems to be.
> >>
> >> thanks
> >> - Robert
> >>
> >>
> >>
> >>
> >> On Wed, Jul 20, 2016 at 5:00 AM, Abhishek Bafna <[email protected]>
> >> wrote:
> >>
> >>> Hi All,
> >>>
> >>> Below is the list of issues, picked from currently available patches. I
> >>> could not decide up any 'open' issue, If there is anything, please
> reply
> >>> here.
> >>>
> >>> We can 'drop/add' more if required based on the feedback. Important is,
> >>> how many of these we will be able to review and commit.
> >>>
> >>> I will starting looking into these issues and provide feedback.
> >>>
> ---------------------------------------------------------------------------
> >>> OOZIE-1978 Forkjoin validation code is ridiculously slow in some
> cases
> >>> OOZIE-2508 Documentation change for Coord action rerun
> [OOZIE-1735]
> >>> OOZIE-2583 oozie throws EL Exception when reference variable name
> >>> containing dot
> >>> OOZIE-2582 Populating external child Ids for action failures
> >>> OOZIE-2517 Add support for startCreatedTime and endCreatedTime
> >>> filters for coord and bundles
> >>> OOZIE-2512 ShareLibservice returns incorrect path for jar
> >>> OOZIE-2572 SLA DURATION miss not shown when job is running for
> longer
> >>> than expected time
> >>> OOZIE-2457 Oozie log parsing regex consume more than 90% cpu
> >>> OOZIE-2574 Oozie to support replication-enabled mysql urls
> >>> OOZIE-2506 Add logs into RecoverService for logging information
> about
> >>> queued commnads
> >>> OOZIE-2573 dataset url which contains spaces can not be handled
> >>> rightly
> >>> OOZIE-2569 Adding yarn-site, core-site, hdfs-site and mapred-site
> >>> into spark launcher
> >>> OOZIE-2491 oozie acl cannot specify group,it does`t work
> >>> OOZIE-2539 Incorrect property key is used for 'hive log4j
> >>> configuration file for execution mode'
> >>> OOZIE-2555 Oozie SSL enable setup does not return port for admin
> >>> -servers
> >>> OOZIE-2515 Duplicate information for "Changing endtime/pausetime
> of a
> >>> Bundle Job" in CommandLineTool wiki
> >>> OOZIE-2498 Oozie CallerId configuration for downstream components
> >>> OOZIE-2516 Update web service documentation for jobs API
> >>> OOZIE-2500 -DtestJarSimple option mentioned in minioozie doc does
> not
> >>> work
> >>> OOZIE-2513 log.scan.duration should not be used for error and
> audit
> >>> logs
> >>> OOZIE-2461 Workflow, Coordinator and Bundle job querying should
> have
> >>> last modified filter
> >>> OOZIE-2306 Oozie health check
> >>> OOZIE-2518 Oozie is creating many files and directories in /tmp
> per
> >>> day
> >>> OOZIE-2409 hive2 action with hive 1.2.1 failed
> >>> OOZIE-2473 Connection pool for SMTP connection
> >>> OOZIE-2440 Exponential re-try policy for workflow action
> >>> OOZIE-2458 'oozie-setup.sh sharelib create' should ensure uploaded
> >>> jars are world readable
> >>> OOZIE-2390 Rerun with failed option removing completed output data
> >>> OOZIE-1810 Workflow cannot get into Failed state when kill control
> >>> node cannot resolve variable in message
> >>> OOZIE-2244 Oozie should mask passwords in the logs when logging
> >>> command arguments
> >>>
> ---------------------------------------------------------------------------
> >>> Thanks,
> >>> Abhishek
> >>>
> >>>> On Jul 18, 2016, at 6:35 PM, Abhishek Bafna <[email protected]>
> >>> wrote:
> >>>>
> >>>> Hi All,
> >>>>
> >>>> I looked into the commits (approx. 160), all looked good to me and
> >>> should be included into the release. If there are things, which needs
> to be
> >>> excluded, provide your comments.
> >>>>
> >>>> We still need to decide on the patch which are in progress status.
> >>>>
> >>>> On the upgrade side:
> >>>> Java version 1.6 to 1.7
> >>>> OpenJPA version 2.2.2 to 2.4.1
> >>>> Curator version 2.5.0 to 2.11.0
> >>>> Pig version 0.12.1 to 0.16.0
> >>>> Hive version 0.13.1 to 1.2.0
> >>>> HttpClient version 4.3 to 4.3.6
> >>>> commons-httpclient version 3.1 to 4.3.6 (discontinued)
> >>>>
> >>>> Please provide your feedback on the above upgrades.
> >>>>
> >>>> Once we decide on the scope of the release. We will start working on
> the
> >>> completing/testing the release.
> >>>>
> >>>> Thanks,
> >>>> Abhishek
> >>>>
> >>>>> On Jul 16, 2016, at 2:40 AM, Abhishek Bafna <[email protected]
> >>> <mailto:[email protected]>> wrote:
> >>>>>
> >>>>> Okay. We will make the Java version to 1.7.
> >>>>>
> >>>>> There are upgrades we can do like OpenJPA, Curator, HttpClient, Pig
> etc.
> >>>>>
> >>>>> HttpClient, Pig (0.16) version upgrade we have done internally as
> well.
> >>>>>
> >>>>> As per the comments on the OOZIE-2231 <
> >>> https://issues.apache.org/jira/browse/OOZIE-2231>, curator version
> also
> >>> can be upgraded to 2.11.
> >>>>>
> >>>>> Thanks,
> >>>>> Abhishek
> >>>>>> On Jul 16, 2016, at 2:06 AM, Robert Kanter <[email protected]
> >>> <mailto:[email protected]>> wrote:
> >>>>>>
> >>>>>> We've been using a version of Hive/HCat based on 1.1.0 for over a
> year
> >>> now
> >>>>>> with no problems. We did have to comment out a few lines in
> >>> MiniHCatServer
> >>>>>> and MiniHS2 that were no longer needed:
> >>>>>>
> >>>>>> diff --git
> >>> a/core/src/test/java/org/apache/oozie/test/MiniHCatServer.java
> >>>>>> b/core/src/test/java/org/apache/oozie/test/MiniHCatServer.java
> >>>>>> index 8699ff8..d694977 100644
> >>>>>> --- a/core/src/test/java/org/apache/oozie/test/MiniHCatServer.java
> >>>>>> +++ b/core/src/test/java/org/apache/oozie/test/MiniHCatServer.java
> >>>>>> @@ -150,9 +150,10 @@ public class MiniHCatServer {
> >>>>>> public static void resetDefaultDBCreation() throws Exception {
> >>>>>> // Need to do this, else default db will not be created for
> >>> local
> >>>>>> metastores.
> >>>>>> // TestHiveMain will fail with
> >>>>>> InvalidObjectException(message:There is no database named default)
> >>>>>> - Field declaredField =
> >>>>>> HMSHandler.class.getDeclaredField("createDefaultDB");
> >>>>>> - declaredField.setAccessible(true);
> >>>>>> - declaredField.set(null, false);
> >>>>>> + // CLOUDERA-BUILD: Don't need this for Hive 0.14 and later
> >>>>>> + //Field declaredField =
> >>>>>> HMSHandler.class.getDeclaredField("createDefaultDB");
> >>>>>> + //declaredField.setAccessible(true);
> >>>>>> + //declaredField.set(null, false);
> >>>>>> }
> >>>>>>
> >>>>>> public static void resetHiveConfStaticVariables() throws Exception
> >>> {
> >>>>>> diff --git
> a/core/src/test/java/org/apache/oozie/test/hive/MiniHS2.java
> >>>>>> b/core/src/test/java/org/apache/oozie/test/hive/MiniHS2.java
> >>>>>> index c9dfd56..505dc55 100644
> >>>>>> --- a/core/src/test/java/org/apache/oozie/test/hive/MiniHS2.java
> >>>>>> +++ b/core/src/test/java/org/apache/oozie/test/hive/MiniHS2.java
> >>>>>> @@ -84,7 +84,8 @@ public class MiniHS2 extends AbstractHiveService {
> >>>>>> hiveConf.setVar(ConfVars.HIVE_SERVER2_THRIFT_BIND_HOST,
> getHost());
> >>>>>> hiveConf.setIntVar(ConfVars.HIVE_SERVER2_THRIFT_PORT,
> >>> getBinaryPort());
> >>>>>> hiveConf.setIntVar(ConfVars.HIVE_SERVER2_THRIFT_HTTP_PORT,
> >>>>>> getHttpPort());
> >>>>>> - HiveMetaStore.HMSHandler.resetDefaultDBFlag();
> >>>>>> + // CLOUDERA-BUILD: Don't need this for Hive 0.14 and later
> >>>>>> + //HiveMetaStore.HMSHandler.resetDefaultDBFlag();
> >>>>>>
> >>>>>> Path scratchDir = new Path(baseDfsDir, "scratch");
> >>>>>> fs.mkdirs(scratchDir);
> >>>>>>
> >>>>>> On Fri, Jul 15, 2016 at 1:29 PM, Venkat Ranganathan <
> >>>>>> [email protected] <mailto:[email protected]>>
> >>> wrote:
> >>>>>>
> >>>>>>> Let us explore changing the default hive version also to 1.x.
> 0.13
> >>> is
> >>>>>>> old
> >>>>>>>
> >>>>>>> Thanks
> >>>>>>>
> >>>>>>> Venkat
> >>>>>>>
> >>>>>>>
> >>>>>>> On 7/15/16, 11:31 AM, "Robert Kanter" <[email protected]
> <mailto:
> >>> [email protected]>> wrote:
> >>>>>>>
> >>>>>>> I don't think we have any guidelines defined anywhere about when
> >>> it's
> >>>>>>> okay
> >>>>>>> to change the minimum Java version, but perhaps we need to wait
> >>> until
> >>>>>>> the
> >>>>>>> next major release (5.0)? That said, Hadoop did this in a minor
> >>>>>>> release
> >>>>>>> and Java 6 was EoL in 2013. So I'm fine with doing it in 4.3 if
> >>>>>>> nobody has
> >>>>>>> an objections; we should just make sure to call that out in our
> >>> release
> >>>>>>> notes.
> >>>>>>>
> >>>>>>> Otherwise, nothing comes to mind for anything we should leave out
> >>> other
> >>>>>>> than the Oozie on Yarn stuff, which is not currently in trunk
> >>> anyway.
> >>>>>>>
> >>>>>>> I know there's a number of Spark Action JIRAs; we should make sure
> >>> to
> >>>>>>> include all of these and any pending ones.
> >>>>>>> @Rohini, @Puru, @Satish, anything you think needs more time to
> bake
> >>>>>>> that we
> >>>>>>> should hold off on including in the release?
> >>>>>>>
> >>>>>>>
> >>>>>>> thanks
> >>>>>>> - Robert
> >>>>>>>
> >>>>>>> On Fri, Jul 15, 2016 at 8:35 AM, Abhishek Bafna <
> >>> [email protected] <mailto:[email protected]>>
> >>>>>>> wrote:
> >>>>>>>
> >>>>>>>> Hi All,
> >>>>>>>>
> >>>>>>>> Any suggestions or comments on the scope for the release?
> >>>>>>>>
> >>>>>>>> Thanks,
> >>>>>>>> Abhishek
> >>>>>>>>
> >>>>>>>>> On Jul 13, 2016, at 3:07 AM, Abhishek Bafna <
> [email protected]
> >>> <mailto:[email protected]>>
> >>>>>>>> wrote:
> >>>>>>>>>
> >>>>>>>>> Hi All,
> >>>>>>>>>
> >>>>>>>>> Oozie is starting the release process for Oozie 4.3.0. This
> thread
> >>>>>>> is
> >>>>>>>> for discussion about the scope of the release. Please provide your
> >>>>>>>> suggestion about what should be included (or any exclusions) in
> the
> >>>>>>> release.
> >>>>>>>>>
> >>>>>>>>> Currently, we have around 158 patches submitted to trunk. Based
> on
> >>>>>>> the
> >>>>>>>> scope, we will include (exclude) more if required.
> >>>>>>>>> Version Upgrade
> >>>>>>>>> Java 1.7 (OOZIE-2036 <
> >>>>>>> https://issues.apache.org/jira/browse/OOZIE-2036 <
> >>> https://issues.apache.org/jira/browse/OOZIE-2036>>)
> >>>>>>>>> OpenJPA (OOZIE-2488 <
> >>>>>>> https://issues.apache.org/jira/browse/OOZIE-2488 <
> >>> https://issues.apache.org/jira/browse/OOZIE-2488>>),
> >>>>>>>>> Curator (OOZIE-2231 <
> >>>>>>> https://issues.apache.org/jira/browse/OOZIE-2231 <
> >>> https://issues.apache.org/jira/browse/OOZIE-2231>
> >>>>>>>>> ),
> >>>>>>>>> HttpClient (OOZIE-2538 <
> >>>>>>>> https://issues.apache.org/jira/browse/OOZIE-2538 <
> >>> https://issues.apache.org/jira/browse/OOZIE-2538>>),
> >>>>>>>>> Pig (OOZIE-2602 <
> https://issues.apache.org/jira/browse/OOZIE-2602 <
> >>> https://issues.apache.org/jira/browse/OOZIE-2602>
> >>>>>>>> ).
> >>>>>>>>> New Features/Improvements: Add default JT and NN, Database schema
> >>>>>>>> service check, instrumental metrics, Tomcat upgrade etc.
> >>>>>>>>> Bug Fixes.
> >>>>>>>>> Provide your opinion on inclusion/exclusion of the above points
> or
> >>>>>>> more
> >>>>>>>> for the scope of the release. We can also, take some work which is
> >>>>>>> already
> >>>>>>>> done and review is pending for them (depending on the availability
> >>>>>>> of the
> >>>>>>>> reviewers).
> >>>>>>>>>
> >>>>>>>>> Thanks,
> >>>>>>>>> Abhishek
> >>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>
> >>>>
> >>>
> >>>
> >
>
>
