I opened following PRs for the package change:
https://github.com/apache/apex-malhar/pull/662
Moves all classes with history retained (hence 2 commits). Also
contains
checkstyle and other mechanical changes.
https://github.com/apache/apex-malhar/pull/664
Adds backward compatibility jar.
Once above PRs are merged the new artifact can be deployed and
introduced
as dependency in malhar-library.
Please review.
Thanks,
Thomas
On Sun, Jul 16, 2017 at 7:04 AM, Thomas Weise <t...@apache.org> wrote:
My original list of work items contained the b/w compatibility aspect,
I
don't think there should be any confusion of whether it will be
covered
here or not.
The proposed shading will provide the old classes and they will be
frozen
as of release 3.7. That's the same as making a copy of the code and
never
again making changes to the original classes. This cannot be
accomplished
by using the older 3.7 release in your project because you cannot use
2
different versions of Malhar in parallel unless you apply shading.
The shaded artifact will only expose the com.datatorrent classes, and
they
will be self-contained as the rest of the classes that they may depend
on
are shaded. The shaded artifact does not evolve, there are not more
changes
to com.datatorrent classes after they are relocated in master.
Thanks,
Thomas
On Sun, Jul 16, 2017 at 2:00 AM, Pramod Immaneni <
pra...@datatorrent.com
wrote:
I don't think we can limit the topic strictly to relocation without
having
a good b/w compatibility story or at least one that goes far enough.
The shading idea sounds interesting. Why not let the shaded version
move
forward with each release till we hit a major release. If it is going
to
remain pegged at 3.7.0, why shade in the first place as the regular
3.7.0
release would do the same job and it would be same as the loss of b/w
compatibility with newer releases.
Thanks
On Sat, Jul 15, 2017 at 7:57 AM, Thomas Weise <t...@apache.org>
wrote:
Discussing what in the future might become stable needs to be a
separate
thread, it will be a much bigger discussion.
The topic here is to relocate the packages. With a few exceptions
relocation won't affect the semantic versioning. Semantic
versioning
is
essentially not effective for Malhar because almost everything is
@Evolving
(and there are reasons for that.. -> separate topic)
I don't really like the idea of creating bw compatibility stubs for
the
follow up PR. It creates even more clutter in the source tree than
there
already is and so here is an alternative suggestion:
https://github.com/tweise/apex-malhar/blob/malhar37-
compat/shaded-malhar37/pom.xml
Create a shaded artifact that provides the old com.datatorrent.*
classes as
of release 3.7. Users can include that artifact if they don't want
to
change import statements. At the same time they have an incentive to
switch
to the relocated classes to take advantage of bug fixes and new
functionality.
I will work on the first PR that does the relocate. In the
meantime,
we
can
finalize what backward compatibility support we want to provide and
how.
Thanks,
Thomas
On Fri, Jul 14, 2017 at 11:33 AM, Pramod Immaneni <
pra...@datatorrent.com>
wrote:
How about coming up with a list of @Evolving operators that we
would
like
to see in the eventual stable list and move those along with the
not
@Evolving ones in org.apache.apex with b/w stubs and leave the
rest
as
they
are. Then have a follow up JIRA for the rest to be moved over to
contrib
and be deprecated.
Thanks
On Fri, Jul 14, 2017 at 10:37 AM, Thomas Weise <
thomas.we...@gmail.com>
wrote:
We need to keep the discussion here on topic, if other things
are
piled
on
top then nothing gets done.
Most operators today are not stable, they are @Evolving. So
perhaps
it
is
necessary to have a separate discussion about that, outside of
this
thread.
My guess is that there are only a few operators that we could
declare
stable. Specifically, under contrib the closest one might have
been
Kafka,
but that is already superseded by the newer versions.
Thomas
On Fri, Jul 14, 2017 at 10:21 AM, Pramod Immaneni <
pra...@datatorrent.com>
wrote:
We had a discussion a while back, agreed to relegate
non-stable
and
experimental operators to contrib and also added this to
contribution
guidelines. We aexecuted on this and cleaned up the repo by
moving
operators deemed non-suitable for production use at that time
to
contrib.
So I wouldn't say the operators that are at the top level
today
or
the
ones
in library are 0.x.x quality. Granted, we may need to do one
more
pass
as
some of the operators may no longer be considered the right
implementations
with the advent of the windowed operator.
Since contrib used to be the place that housed operators that
required
third party libraries, there are some production quality
operators in
there
that need to be pulled out into top level modules.
I think we are in agreement that for operators that we
consider
stable,
we
should provide a b/w stub. I would add that we strongly
consider
creating
the org.apache.apex counterparts of any stable operators that
are
in
contrib out in top level modules and have the com.datatorrent
stubs
in
contrib.
For the operators not considered stable, I would prefer we
either
leave
the
current package path as is or if we change the package path
then
create
the
b/w stub as I am not sure how widely they are in use (so, in
essence,
preserve semantic versioning). It would be good if there is a
followup
JIRA
that takes a look at what other operators can be moved to
contrib
with
the
advent of the newer frameworks and understanding.
Thanks
On Fri, Jul 14, 2017 at 9:44 AM, Thomas Weise <t...@apache.org
wrote:
Most of the operators evolve, as is quite clearly indicated
by
@Evolving
annotations. So while perhaps 0.x.x would be a more
appropriate
version
number, I don't think you can change that.
Thomas
On Fri, Jul 14, 2017 at 9:35 AM, Vlad Rozov <
v.ro...@datatorrent.com
wrote:
If entire library is not stable, its version should be
0.x.x
and
there
should not be any semantic versioning enabled or implied.
It
evolves.
If
some operators are stable as 3.8.x version suggests, the
library
should
follow semantic versioning and it is not OK to make a
major
change
that
breaks semantic versioning across the entire library. It
is
not a
finding
an excuse not to make a change. For me semantic versioning
and
backward
compatibility is more important than a proper package
name.
Thank you,
Vlad
On 7/14/17 09:11, Thomas Weise wrote:
Semantic versioning makes sense when you have a stable
baseline.
As
long
as
frequent fixes need to be made to address basic issues,
it
makes
no
sense
to declare operators stable, which is why they are marked
evolving.
Instead of finding reasons to avoid changes that should
have
been
made a
long time ago, why not discuss what a "stable operator"
is
and
which
ones
deserve to be in that category. Those are the ones that
will
get
the
backward compatibility stub.
Thanks,
Thomas
On Fri, Jul 14, 2017 at 8:34 AM, Vlad Rozov <
v.ro...@datatorrent.com>
wrote:
My preference is to agree that the next Malhar release is
4.0.0
and
make
the proposed changes when 3.8.0 goes out. Otherwise why
to
keep
semantic
versioning checks on Malhar in case there is no version
compatibility.
Thank you,
Vlad
On 7/14/17 08:11, Thomas Weise wrote:
next release is 3.8.0
On Fri, Jul 14, 2017 at 7:55 AM, Vlad Rozov <
v.ro...@datatorrent.com>
wrote:
next release - 3.9.0 or 4.0.0?
Thank you,
Vlad
On 7/13/17 22:25, Thomas Weise wrote:
It is time to resurrect this thread and get going with
the
work.
For the next release, I will sign up to do the
package
move
in
Malhar:
https://issues.apache.org/
jira/browse/APEXMALHAR-2517
In general this will be straightforward; most classes
in
Malhar
are
marked
evolving and it is trivial for users to change import
statements.
However,
I would suggest to discuss if there are selected
operators
that
are
worth
keeping a backward compatibility stub in the old
location.
Here is my plan:
1. Relocate all classes in *lib* and *contrib* within
one
PR -
this
is
all
IDE automated work
2. Add backward compatibility classes, if, any in
separate
PR
3. Create PR for Megh library to reflect moved
classes
Thanks,
Thomas
On Wed, Mar 1, 2017 at 2:24 PM, Pramod Immaneni <
pra...@datatorrent.com
wrote:
Inline
On Mon, Feb 27, 2017 at 11:03 PM, Thomas Weise <
t...@apache.org>
wrote:
-->
On Mon, Feb 27, 2017 at 1:27 PM, Pramod Immaneni <
pra...@datatorrent.com
wrote:
For malhar, for existing operators, I prefer we do
this as
part
of
the
planned refactoring for breaking the monolith
modules
into
baby
packages
and would also prefer deprecating the existing
operators
in
place.
Refactor into smaller modules was discussed for
malhar-contrib
and
given
the overall state of that module I think it is OK
to
defer
package
renaming
there. I do however prefer to see the package rename
addressed
for
other
modules, especially for the main library module.
Should we consider breaking the library into
smaller
modules
as
well,
the
file/block operators for example probably can be in
their
own
module
from
just an organizational perspective.
This
will help us achieve two things. First, the user
will
see
all
the
new
changes at once as opposed to dealing with it
twice
(with
package
rename
and dependency changes) and second it will allow
for
a
smoother
transition
as the existing code will still work in a
deprecated
state.
It
will
also
give a more consistent structure to malhar. For new
operators,
we
can
go
with the new package path but we need to ensure
they
will
get
moved
into
the baby packages as well.
I think existing operators should be renamed so
that
git
history
remains. A
possible solution for backward compatibility could
be
to
subsequently
add
empty subclasses in the previous location (for
existing
