incubator-distributedlog git commit: DL-176: Rename the DL artifact from com.twitter to org.apache.distributedlog

2017-01-06 Thread sijie 
Repository: incubator-distributedlog
Updated Branches:
  refs/heads/master 859b342b2 -> 1bb44e776


DL-176: Rename the DL artifact from com.twitter to org.apache.distributedlog

Author: Sijie Guo 

Reviewers: Dave Rusek , Leigh Stewart 

Closes #102 from sijie/sijie/fix_pom_file_layout


Project: http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/repo
Commit: 
http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/commit/1bb44e77
Tree: 
http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/tree/1bb44e77
Diff: 
http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/diff/1bb44e77

Branch: refs/heads/master
Commit: 1bb44e776775636fdc34b5e88aad2a93651f4fe3
Parents: 859b342
Author: Sijie Guo 
Authored: Fri Jan 6 16:51:06 2017 -0800
Committer: Sijie Guo 
Committed: Fri Jan 6 16:51:06 2017 -0800

--
 distributedlog-benchmark/bin/dbench |  6 +++---
 distributedlog-benchmark/pom.xml| 10 +-
 distributedlog-build-tools/pom.xml  |  2 +-
 distributedlog-client/pom.xml   |  8 
 distributedlog-core/bin/dlog|  6 +++---
 distributedlog-core/pom.xml |  6 +++---
 distributedlog-protocol/pom.xml |  4 ++--
 distributedlog-service/bin/dlog | 10 +-
 distributedlog-service/pom.xml  | 12 ++--
 distributedlog-tutorials/distributedlog-basic/pom.xml   |  8 
 distributedlog-tutorials/distributedlog-kafka/pom.xml   |  6 +++---
 .../distributedlog-mapreduce/pom.xml|  8 
 .../distributedlog-messaging/pom.xml|  8 
 distributedlog-tutorials/pom.xml|  2 +-
 pom.xml |  2 +-
 15 files changed, 49 insertions(+), 49 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/blob/1bb44e77/distributedlog-benchmark/bin/dbench
--
diff --git a/distributedlog-benchmark/bin/dbench 
b/distributedlog-benchmark/bin/dbench
index 1baa21c..afcca62 100755
--- a/distributedlog-benchmark/bin/dbench
+++ b/distributedlog-benchmark/bin/dbench
@@ -76,7 +76,7 @@ case "${COMMAND}" in
 --change-interval ${CHANGE_RATE_INTERVAL} \\
 """
 BENCH_ARGS="${BENCH_ARGS} \\ ${BENCH_WRITE_ARGS} \\ --mode dlwrite \\"
-exec java $OPTS $JMX_ARGS com.twitter.distributedlog.benchmark.Benchmarker 
$BENCH_ARGS $@
+exec java $OPTS $JMX_ARGS org.apache.distributedlog.benchmark.Benchmarker 
$BENCH_ARGS $@
 ;;
   write)
 BENCH_WRITE_ARGS="""
@@ -88,7 +88,7 @@ case "${COMMAND}" in
 --finagle-name inet!localhost:8000 \\
 """
 BENCH_ARGS="${BENCH_ARGS} \\ ${BENCH_WRITE_ARGS} \\ --mode write \\"
-exec java $OPTS $JMX_ARGS com.twitter.distributedlog.benchmark.Benchmarker 
$BENCH_ARGS $@
+exec java $OPTS $JMX_ARGS org.apache.distributedlog.benchmark.Benchmarker 
$BENCH_ARGS $@
 ;;
   read)
 BENCH_READ_ARGS="""
@@ -98,7 +98,7 @@ case "${COMMAND}" in
 --finagle-name inet!localhost:8000 \\
 """
 BENCH_ARGS="${BENCH_ARGS} \\ ${BENCH_READ_ARGS} \\ --mode read \\"
-exec java $OPTS $JMX_ARGS com.twitter.distributedlog.benchmark.Benchmarker 
$BENCH_ARGS $@
+exec java $OPTS $JMX_ARGS org.apache.distributedlog.benchmark.Benchmarker 
$BENCH_ARGS $@
 ;;
   help)
 usage

http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/blob/1bb44e77/distributedlog-benchmark/pom.xml
--
diff --git a/distributedlog-benchmark/pom.xml b/distributedlog-benchmark/pom.xml
index bb4d3b9..b3ed0f0 100644
--- a/distributedlog-benchmark/pom.xml
+++ b/distributedlog-benchmark/pom.xml
@@ -19,7 +19,7 @@
  xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 
http://maven.apache.org/maven-v4_0_0.xsd;>
   4.0.0
   
-com.twitter
+org.apache.distributedlog
 distributedlog
 0.4.0-incubating-SNAPSHOT
   
@@ -27,12 +27,12 @@
   Apache DistributedLog :: Benchmark
   
 
-  com.twitter
+  org.apache.distributedlog
   distributedlog-client
   ${project.parent.version}
 
 
-  com.twitter
+  org.apache.distributedlog
   distributedlog-service
   ${project.parent.version}
 
@@ -42,7 +42,7 @@
   ${birdcage.sha}
 
 
-  com.twitter
+  org.apache.distributedlog
   distributedlog-core
   ${project.parent.version}
   test
@@ -123,7 +123,7 @@
 6.19
   
   
-com.twitter
+org.apache.distributedlog

[GitHub] incubator-distributedlog pull request #102: DL-176: Rename the DL artifact f...

2017-01-06 Thread asfgit 
Github user asfgit closed the pull request at:

https://github.com/apache/incubator-distributedlog/pull/102


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: [Discuss] Transaction Support

2017-01-06 Thread Sijie Guo 
On Thu, Jan 5, 2017 at 10:56 PM, Xi Liu  wrote:

> Asko and Sijie,
>
> Thank you so much for your feedbacks.
>
> We are not targeting at building a general XA transaction coordinator. The
> feature we want is be able to write data to multiple log streams in an
> atomic way.
>

So in other words, that means this feature is 'atomic writes across log
streams', no?


>
> I totally agreed with you about building minimal logic. We also don't want
> to enforce this feature to all the users of DL. Building the TC as a
> separated service sounds clear to me. We will do it follow your suggestion.
>
> I am also replying the comments to you and Leigh on the doc. Hopefully we
> can come to an agreement so that our changes can be accepted.
>
> - Xi
>
> On Wed, Jan 4, 2017 at 1:14 AM, Asko Kauppi 
> wrote:
>
> > > Beside that, I have one general question - What is the major goal for
> > this
> > > feature? Are you targeting on building a general XA transaction
> > coordinator
> > > or just for supporting things like `copy-modify-write' style workflow?
> >
> > The use case I would have for transactions - at some level of the stack -
> > is supporting dynamic configurations.
> >
> > If a config changes in e.g. three lines, some of the changes may
> logically
> > belong together. E.g. changing both “host” and “port” (if separate
> > entries). One shouldn’t be able to read a state, even temporarily, that
> has
> > new host but old port.
> >
> > I can do this in the application level - it does not need to be part of
> > the DL protocol.
> >
> >
> > Asko Kauppi
> > Zalando Tech Helsinki
> >
> > > On 4 Jan 2017, at 9.18, Sijie Guo  wrote:
> > >
> > > Sorry for late response. I think Leigh and you already had some very
> > > valuable discussions in the doc. I will try to add some of my questions
> > to
> > > the discussion.
> > >
> > > Beside that, I had a discussion with Leigh today about this. first of
> > all,
> > > I think it is very good to add transaction support in distributedlog.
> It
> > is
> > > one of the primitives that would help building distributed service. But
> > we
> > > have a concern about making this system become complicated and
> introduce
> > > operational overhead when it runs in the large scale system on
> > production.
> > > There are two major suggestions that I have for this feature -
> > >
> > > Build the 'minimum' logic in core - I think the minimum logic that need
> > to
> > > be added to the core is -  the special control records (begin, commit
> and
> > > abort) and make the reader be able to detect those special control
> > records
> > > and know what do they mean and how to interrupt with them. Since they
> are
> > > special control records, there is not overhead to other readers that
> > > doesn't require this feature.
> > >
> > > Build the transaction coordinator as a separated proxy service  - I
> think
> > > the major concern that we have is putting more complexities into the
> > 'write
> > > proxy' service. We architected distributedlog in a more
> microservice-like
> > > way - we have the core as the stream store, the proxy for serving write
> > and
> > > read traffic. It would be good that the transaction feature can be done
> > in
> > > a similar way. So the architecture would be like this -
> > >
> > > *[ write service ] [ read service ] [ transaction coordinator ]*
> > > *[ stream store
> > >]*
> > >
> > > if people doesn't need the transaction feature, they can turn if off
> > > completely without any operational overhead.
> > >
> > > Beside that, I have one general question - What is the major goal for
> > this
> > > feature? Are you targeting on building a general XA transaction
> > coordinator
> > > or just for supporting things like `copy-modify-write' style workflow?
> > >
> > >
> > > Thanks,
> > > Sijie
> > >
> > >
> > >
> > >
> > >
> > > On Wed, Dec 28, 2016 at 1:12 PM, Xi Liu  wrote:
> > >
> > >> Ping?
> > >>
> > >> On Mon, Dec 19, 2016 at 8:28 AM, Xi Liu  wrote:
> > >>
> > >>> Sijie,
> > >>>
> > >>> No. I thought it might be easier for people to comment on a google
> doc
> > to
> > >>> gather the initial feedback. I will put the content back to wiki page
> > >> once
> > >>> addressing the comments. Does that sound good to you?
> > >>>
> > >>> And thank you in advance.
> > >>>
> > >>> - Xi
> > >>>
> > >>>
> > >>>
> > >>> On Sun, Dec 18, 2016 at 8:48 AM, Sijie Guo  wrote:
> > >>>
> >  Hi Xi,
> > 
> >  sorry for late response. I will review it soon.
> > 
> >  regarding this, a separate question "are we going to use google doc
> >  instead
> >  of email thread for any discussion"? I am a bit worried that the
> >  discussion
> >  will become lost after moving to google doc. No idea on how other
> > apache
> >  projects are doing.
> > 
> >  - Sijie
> > 
> >  On Wed, Dec 14, 2016 at 11:41

Re: [Review] The first release of Apache DistributedLog

2017-01-06 Thread Sijie Guo 
Yes. it is part of the plan.

On Thu, Jan 5, 2017 at 11:27 PM, Xi Liu  wrote:

> It would be great to include any performance numbers.
>
> On Thu, Jan 5, 2017 at 5:51 PM, Sijie Guo  wrote:
>
> > Cool to see you here, Enrico. And thank you for your suggestion.
> >
> > I will try to write a separate one for DL and BK. Try to keep this one
> > focus on a short release post.
> >
> > - Sijie
> >
> > On Thu, Jan 5, 2017 at 1:43 PM, Enrico Olivelli 
> > wrote:
> >
> > > Hi Sijie,
> > > I am following this release and the great work DL comunity is doing.
> > > Maybe it would be worth to write some paragraph about the difference
> from
> > > BookKeeper and/or the relation with it.
> > >
> > >
> > > Enrico
> > >
> > > Il gio 5 gen 2017, 19:13 Asko Kauppi  ha
> > scritto:
> > >
> > > > Hi Sijie,
> > > >
> > > > most readers will likely not know about DistributedLog. A short
> > > comparison
> > > > - or a link to one - e.g. with Kafka might help set the stage for
> them.
> > > > i.e. why does it exist?
> > > >
> > > > This is even more important since Uber recently also publicized their
> > > > persistent message bus solution. If these start dropping down, there
> > > needs
> > > > to be more (technical) reason than just another vendor opening their
> > > chest.
> > > > :)
> > > >
> > > > Other than that, the structure seemed nice but it can be slightly
> > > shorter.
> > > >
> > > > Just my 2c
> > > > - asko
> > > >
> > > >
> > > > On 4 January 2017 at 10:38, Sijie Guo  wrote:
> > > >
> > > > > I drafted a blog post for announcing the first apache release. Here
> > is
> > > > the
> > > > > draft. Please help review it :D
> > > > >
> > > > > https://docs.google.com/document/d/1IXVmP2cHkf4ydeUHUJN9p5ZWTpA1a
> > > > > uwBhfqMnYBu4A0/edit
> > > > >
> > > > > - Sijie
> > > > >
> > > >
> > > --
> > >
> > >
> > > -- Enrico Olivelli
> > >
> >
>

Re: vote process for proposals?

2017-01-06 Thread Sijie Guo 
seems reasonable to me. do you want to update the wiki to reflect it?

On Thu, Jan 5, 2017 at 11:32 PM, Xi Liu  wrote:

> I think there is not a lot of activities on proposals. A 'lazy approval'
> might be just good enough - a proposal with lazy approval  is implicitly
> allowed/accepted unless a  -1 vote is received. That's probably the best
> for now. what do you think?
>
> On Wed, Jan 4, 2017 at 12:50 AM, Sijie Guo  wrote:
>
> > Ping?
> >
> > Xi, Jon, any updates about this? Do any of you want to drive this?
> >
> > - Sijie
> >
> >
> >
> > On Thu, Dec 15, 2016 at 11:18 AM, Sijie Guo 
> > wrote:
> >
> > > Xi, Jon, are any of you interested in making a draft about about the
> > > proposal workflow?
> > >
> > > On Wed, Dec 14, 2016 at 6:14 PM, Jon Derrick <
> > jonathan.derri...@gmail.com>
> > > wrote:
> > >
> > > > I think it really worth having a voting proposal, as sometime I might
> > > lose
> > > > track of if a proposal is accepted or not and whether it is under
> > > > development.
> > > >
> > > > Beam's process looks promising. You can try to start with that.
> > > >
> > > > Another suggestion is it would be awesome if the DL jira queue can
> have
> > > new
> > > > type, called 'Proposal'. Then we can enforce the proposal workflow in
> > the
> > > > jira.
> > > >
> > >
> > > I think it is possible to ask INFRA team to create a new jira
> > type/workflow
> > > for us, if we can come up with more details. Can you tell us more about
> > > your thoughts?
> > >
> > > - Sijie
> > >
> > >
> > > >
> > > > On Tue, Dec 13, 2016 at 1:10 AM, Xi Liu 
> wrote:
> > > >
> > > > > Thank you Sijie. I feel it is good to have a voting process, so
> that
> > it
> > > > > would be good to track if a proposal is accepted for developing or
> > > > > discarded due to any reasons. I will start with my proposal and see
> > how
> > > > it
> > > > > is going with the community.
> > > > >
> > > > > - Xi
> > > > >
> > > > > On Thu, Dec 8, 2016 at 9:11 PM, Sijie Guo 
> wrote:
> > > > >
> > > > > > Xi, thank you for raising this up. I don't think we have a formal
> > > > process
> > > > > > for track proposals. I think we can learn the proposals from
> other
> > > > apache
> > > > > > projects. For example, beam has very nice documentation on
> > > contribution
> > > > > > guide (http://beam.incubator.apache.org/contribute/contribution-
> > > guide/
> > > > ).
> > > > > > We
> > > > > > probably can adopt it.
> > > > > >
> > > > > > I don't feel strong about the voting process. If it is easier for
> > > > making
> > > > > > conclusion on the proposal discussion, let's vote for any
> discussed
> > > > > > proposal.
> > > > > >
> > > > > > - Sijie
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Thu, Dec 8, 2016 at 9:10 AM, Xi Liu 
> > wrote:
> > > > > >
> > > > > > > Hi all,
> > > > > > >
> > > > > > > It is great that we have a process to track/discuss proposals.
> > but
> > > > the
> > > > > > > process is still a bit unclear to me. do we need a vote phase
> to
> > > > adopt
> > > > > > the
> > > > > > > proposals? and shall we document the process in wiki page?
> > > > > > >
> > > > > > > my basic understand about the process is:
> > > > > > >
> > > > > > > - create a proposal wiki page to describe the proposal
> > > > > > > - start the '[discussion]' email thread for the proposal
> > > > > > > - conversation will happen in the '[discussion]' email thread
> and
> > > the
> > > > > > wiki
> > > > > > > page will be refined
> > > > > > >
> > > > > > > I feel there will be a phase to decide whether this proposal
> will
> > > be
> > > > > > > accepted or discarded and update the state of the proposals.
> > shall
> > > I
> > > > > vote
> > > > > > > DP-2?
> > > > > > >
> > > > > > > - Xi
> > > > > > >
> > > > > >
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > - jderrick
> > > >
> > >
> >
>

[GitHub] incubator-distributedlog issue #102: DL-176: Rename the DL artifact from com...

2017-01-06 Thread sijie 
Github user sijie commented on the issue:

https://github.com/apache/incubator-distributedlog/pull/102
  
@mgodave for merging the apache pull request, we have to run the script 
under scripts/dev/dl-merge-pr.py


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

incubator-distributedlog git commit: DL-171: adding a short sleep to let the WriteCompleteListener have time to run before the final position be requested

2017-01-06 Thread sijie 
Repository: incubator-distributedlog
Updated Branches:
  refs/heads/master a31782093 -> 859b342b2


DL-171: adding a short sleep to let the WriteCompleteListener have time to run 
before the final position be requested

once the "writer.write" is done, if "writer.position()" be invoked easier than 
the WriteCompleteListener onSuccess callback, due to the "synchronized", the 
position result will be 0, not the expected 33. we can just add a short sleep 
to avoid this test issue.

Author: xieliang 

Reviewers: Sijie Guo 

Closes #98 from xieliang/DL-171-Fix-TestAppendOnlyStreamWriter


Project: http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/repo
Commit: 
http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/commit/859b342b
Tree: 
http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/tree/859b342b
Diff: 
http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/diff/859b342b

Branch: refs/heads/master
Commit: 859b342b25dcac3cf4c2353c2d2ff4994d4bcaba
Parents: a317820
Author: xieliang 
Authored: Fri Jan 6 16:30:32 2017 -0800
Committer: Sijie Guo 
Committed: Fri Jan 6 16:30:32 2017 -0800

--
 .../java/org/apache/distributedlog/TestAppendOnlyStreamWriter.java  | 1 +
 1 file changed, 1 insertion(+)
--


http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/blob/859b342b/distributedlog-core/src/test/java/org/apache/distributedlog/TestAppendOnlyStreamWriter.java
--
diff --git 
a/distributedlog-core/src/test/java/org/apache/distributedlog/TestAppendOnlyStreamWriter.java
 
b/distributedlog-core/src/test/java/org/apache/distributedlog/TestAppendOnlyStreamWriter.java
index d095af1..b5498ba 100644
--- 
a/distributedlog-core/src/test/java/org/apache/distributedlog/TestAppendOnlyStreamWriter.java
+++ 
b/distributedlog-core/src/test/java/org/apache/distributedlog/TestAppendOnlyStreamWriter.java
@@ -189,6 +189,7 @@ public class TestAppendOnlyStreamWriter extends 
TestDistributedLogBase {
 assertEquals(0, writer.position());
 
 Await.result(writer.write(byteStream));
+Thread.sleep(100); // let WriteCompleteListener have time to run
 assertEquals(33, writer.position());
 
 writer.close();

[GitHub] incubator-distributedlog pull request #98: DL-171 : adding a short sleep to ...

2017-01-06 Thread asfgit 
Github user asfgit closed the pull request at:

https://github.com/apache/incubator-distributedlog/pull/98


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-distributedlog issue #102: DL-176: Rename the DL artifact from com...

2017-01-06 Thread mgodave 
Github user mgodave commented on the issue:

https://github.com/apache/incubator-distributedlog/pull/102
  
I don't have write access to merge but lgtm #shipit


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-distributedlog pull request #106: DL-179: Support ZStandard Compre...

2017-01-06 Thread khurrumnasimm 
GitHub user khurrumnasimm opened a pull request:

https://github.com/apache/incubator-distributedlog/pull/106

DL-179: Support ZStandard Compression

This PR is to add [Zstd](https://github.com/facebook/zstd) compression 
support.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/khurrumnasimm/incubator-distributedlog kn/zstd

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-distributedlog/pull/106.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #106


commit 618cf56552e3bcb6660eb42307f0f1285f7b060a
Author: Khurrum Nasim 
Date:   2017-01-06T09:42:38Z

DL-179: Support ZStandard Compression




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: [DISCUSS] using protobuf than thrift

2017-01-06 Thread Gerrit Sundaram 
Khurrum,

As far as I know, the protobuf package was shaded in bookkeeper. there will
no backward compatible. At this point, I am not interested in bumping bc's
protobuf, especially the bk version is still twitter's branch.

On Thu, Jan 5, 2017 at 11:49 PM, Khurrum Nasim 
wrote:

> one question - bk is using protobuf 2.x while gRPC is using 3.x. IMO, they
> are not backward compatible. Are you also considering moving bk's protobuf
> to 3.x?
>
> On Thu, Jan 5, 2017 at 10:29 PM, Gerrit Sundaram  >
> wrote:
>
> > Hello all,
> >
> > for the comment in
> > https://github.com/apache/incubator-distributedlog/pull/99, I am
> starting
> > this email thread for discussing using protobuf to store metadata for
> ease
> > extension.
> >
> > I have a few reasons for using protobuf rather than using thrift:
> >
> > - bookkeeper is using protobuf for storing metadata. so there is no extra
> > dependency.   and it will make things consistent.
> > - the thrift version that DL is using now is 0.5.0-1, which is an
> > out-of-date thrift version and seems to be a special version that Twitter
> > customized for finagle. it makes me impossible to build a c++ client to
> > access DL.
> > - using protobuf, I can easily write a gRPC request handler for current
> > proxy service to support c++.
> >
> > Any thoughts?
> >
> > - Sijie
> >
>