Re: Pulsar meetup in June/July

[Sijie Guo]

That's great! Thank you, Joe!

Anyone wanna contribute the talks?

So we can start finalizing an agenda, and send out meetup invitations?

On Fri, Jun 8, 2018 at 10:39 AM Joe F  wrote:

> I have reserved the hall at Bldg C,  701, First Ave, Sunnyvale . This is in
> Oath Campus.
>
> Date is July 10th
>
> Joe
>
>
>
> On Thu, Jun 7, 2018 at 4:55 PM, Sijie Guo  wrote:
>
> > +1 go pulsar!
> >
> > - Sijie
> >
> > On Thu, Jun 7, 2018 at 4:49 PM Sanjeev Kulkarni 
> > wrote:
> >
> > > July 10th works for me as well!
> > >
> > > On Tue, Jun 5, 2018 at 11:21 AM Joe F  wrote:
> > >
> > > > All,
> > > >
> > > > I got the availability for the venue in the 2nd week of July  (50
> > > people),
> > > > and its available on  July 9th, 10th, or 13th .  How does that sound?
> > I
> > > > need to reserve a firm date...
> > > >
> > > > My preference is for 10th  [since its not a Monday or Friday :-) ]
> but
> > > I'm
> > > > fine with a whatever is convenient for all ..
> > > >
> > > >
> > > > Joe
> > > >
> > > > On Sun, Jun 3, 2018 at 7:21 PM, Sijie Guo 
> wrote:
> > > >
> > > > > Any updates?
> > > > >
> > > > > - Sijie
> > > > >
> > > > > On Mon, Apr 30, 2018 at 7:29 PM, Joe F  wrote:
> > > > >
> > > > > > I would expect people to volunteer and sign up... that's why I
> said
> > > > > "from
> > > > > > Pulsar users and developers".
> > > > > >
> > > > > >
> > > > > > Joe
> > > > > >
> > > > > > On Mon, Apr 30, 2018 at 6:44 PM, Dave Fisher <
> > dave2w...@comcast.net>
> > > > > > wrote:
> > > > > >
> > > > > > > Hi -
> > > > > > >
> > > > > > > What would be the agenda?
> > > > > > >
> > > > > > > How can this event be promoted to help grow the community?
> > > > > > >
> > > > > > > Regards,
> > > > > > > Dave
> > > > > > >
> > > > > > > Sent from my iPhone
> > > > > > >
> > > > > > > > On Apr 30, 2018, at 11:29 AM, Joe F 
> > > wrote:
> > > > > > > >
> > > > > > > > In June, it going to be a year since Pulsar was accepted into
> > the
> > > > > > > incubator.
> > > > > > > >
> > > > > > > > I'm thinking of a meetup, with a few in-depth
> > > talks/presentations,
> > > > > > what's
> > > > > > > > new in Pulsar 2.0 etc,   from Pulsar users and developers.
> Are
> > > > > people
> > > > > > > > interested in a 1/2day event?   I can arrange for the venue
> and
> > > > > > > logistics.
> > > > > > > >
> > > > > > > >
> > > > > > > > Joe
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: Pulsar meetup in June/July

[Joe F]

I have reserved the hall at Bldg C,  701, First Ave, Sunnyvale . This is in
Oath Campus.

Date is July 10th

Joe



On Thu, Jun 7, 2018 at 4:55 PM, Sijie Guo  wrote:

> +1 go pulsar!
>
> - Sijie
>
> On Thu, Jun 7, 2018 at 4:49 PM Sanjeev Kulkarni 
> wrote:
>
> > July 10th works for me as well!
> >
> > On Tue, Jun 5, 2018 at 11:21 AM Joe F  wrote:
> >
> > > All,
> > >
> > > I got the availability for the venue in the 2nd week of July  (50
> > people),
> > > and its available on  July 9th, 10th, or 13th .  How does that sound?
> I
> > > need to reserve a firm date...
> > >
> > > My preference is for 10th  [since its not a Monday or Friday :-) ] but
> > I'm
> > > fine with a whatever is convenient for all ..
> > >
> > >
> > > Joe
> > >
> > > On Sun, Jun 3, 2018 at 7:21 PM, Sijie Guo  wrote:
> > >
> > > > Any updates?
> > > >
> > > > - Sijie
> > > >
> > > > On Mon, Apr 30, 2018 at 7:29 PM, Joe F  wrote:
> > > >
> > > > > I would expect people to volunteer and sign up... that's why I said
> > > > "from
> > > > > Pulsar users and developers".
> > > > >
> > > > >
> > > > > Joe
> > > > >
> > > > > On Mon, Apr 30, 2018 at 6:44 PM, Dave Fisher <
> dave2w...@comcast.net>
> > > > > wrote:
> > > > >
> > > > > > Hi -
> > > > > >
> > > > > > What would be the agenda?
> > > > > >
> > > > > > How can this event be promoted to help grow the community?
> > > > > >
> > > > > > Regards,
> > > > > > Dave
> > > > > >
> > > > > > Sent from my iPhone
> > > > > >
> > > > > > > On Apr 30, 2018, at 11:29 AM, Joe F 
> > wrote:
> > > > > > >
> > > > > > > In June, it going to be a year since Pulsar was accepted into
> the
> > > > > > incubator.
> > > > > > >
> > > > > > > I'm thinking of a meetup, with a few in-depth
> > talks/presentations,
> > > > > what's
> > > > > > > new in Pulsar 2.0 etc,   from Pulsar users and developers.  Are
> > > > people
> > > > > > > interested in a 1/2day event?   I can arrange for the venue and
> > > > > > logistics.
> > > > > > >
> > > > > > >
> > > > > > > Joe
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: Non-serializable builders

[Sijie Guo]

Hi Geoffroy,

I created an issue for
https://github.com/apache/incubator-pulsar/issues/1943.

We will address it for the upcoming 2.1 release.

- Sijie

On Fri, Jun 8, 2018 at 6:39 AM Geoffroy Fouquier <
geoffroy.fouqu...@exensa.com> wrote:

> On 08/06/2018 12:01, Sijie Guo wrote:
> > Geoffroy,
> >
> > Thank you for reporting this. The change was made to evolve pulsar into a
> > type-safe client with schema and more validations. Along with the idea,
> > we changed the client to become a more fluent style builder and hide the
> > configuration data as an implementation detail to the pulsar client
> > implementation.
> >
> > However we missed the use cases in data computing world, people tends to
> > use java Serializable on sharing configuration.
> >
> > We would incorporate your use cases and try to bring the configuration
> > object back.
> >
> > - Sijie
>
> That would be great, thanks!
>
> >
> >
> > On Fri, Jun 8, 2018 at 1:09 AM Geoffroy Fouquier <
> > geoffroy.fouqu...@exensa.com> wrote:
> >
> >> We are using pulsar 1.22 within a spark framework and I am currently
> >> upgrading my cluster to pulsar 2.0. One of the main change concerns the
> >> configuration classes, replace by builders. Although ProducerBuilder and
> >> ConsumerBuilder interfaces implement Serializable, corresponding
> >> implementations aren't, because a pulsar client object is embedded. The
> >> schema object is also a problem for serialization.
> >>
> >>
> >> Basically, with 1.22 my producer app. instanciate a client and a
> >> producer on each executor and only configurations classes need to be
> >> serialized
> >>
> >> ---8<---
> >>
> >> WarcRecordExtractor
> >>   .load(sc, input)
> >>   .foreachPartition{ ite =>
> >>
> >>   val client = PulsarClient.create(ServiceUrl, clientConf)
> >>   val producer = client.createProducer(topic, producerConf)
> >>   ite.foreach{ x => producer.sendAsync(x._2.content) }
> >>   producer.close()
> >>
> >>   client.close()
> >>
> >>   }
> >>
> >> ---8<---
> >>
> >>
> >> With 2.0 I try to replace the configuration classes with the
> >> corresponding builders, but this setting doesn't work since the
> >> producerBuilder isn't serializable :
> >>
> >> ---8<---
> >>
> >>   [...]
> >>
> >>   val client = clientBuilder.serviceUrl(serviceUrl).build()
> >>   val producer = producerBuilder.topic(topicName).create
> >>
> >>   [...]
> >>
> >> ---8<---
> >>
> >>
> >> My actual workaround is to rewrite my own ProducerBuilderImpl without
> >> pulsar client or schemas (the createAsync method still exists but throw
> >> an exception). And then to instanciate a producer like this :
> >>
> >> ---8<---
> >>
> >>   [...]
> >>
> >>   val client : PulsarClientImpl =
> >> clientBuilder.serviceUrl(broker).build().asInstanceOf[PulsarClientImpl]
> >>   val producer =
> >> client.createProducerAsync(producerBuilder.topic(topic).getConf(),
> >> org.apache.pulsar.client.api.Schema.BYTES).get()
> >>
> >>   [...]
> >>
> >> ---8<---
> >>
> >> where producerBuilder is my own implementation which implements
> >> ProducerBuilder interface.
> >>
> >>
> >> I did the same thing for ConsumerBuilderImpl and also rewrite
> >> SparkStreamingPulsarReceiver with the same kind of workaround. I don't
> >> know why the configuration classes have been replaced by builders, but
> >> it became more difficult to use it with spark (and probably with other
> >> distributed framework), but maybe i did it wrong.
> >>
> >>
> >>
> >>
>
>

Re: Non-serializable builders

[Geoffroy Fouquier]

On 08/06/2018 12:01, Sijie Guo wrote:

Geoffroy,

Thank you for reporting this. The change was made to evolve pulsar into a
type-safe client with schema and more validations. Along with the idea,
we changed the client to become a more fluent style builder and hide the
configuration data as an implementation detail to the pulsar client
implementation.

However we missed the use cases in data computing world, people tends to
use java Serializable on sharing configuration.

We would incorporate your use cases and try to bring the configuration
object back.

- Sijie


That would be great, thanks!




On Fri, Jun 8, 2018 at 1:09 AM Geoffroy Fouquier <
geoffroy.fouqu...@exensa.com> wrote:


We are using pulsar 1.22 within a spark framework and I am currently
upgrading my cluster to pulsar 2.0. One of the main change concerns the
configuration classes, replace by builders. Although ProducerBuilder and
ConsumerBuilder interfaces implement Serializable, corresponding
implementations aren't, because a pulsar client object is embedded. The
schema object is also a problem for serialization.


Basically, with 1.22 my producer app. instanciate a client and a
producer on each executor and only configurations classes need to be
serialized

---8<---

WarcRecordExtractor
  .load(sc, input)
  .foreachPartition{ ite =>

  val client = PulsarClient.create(ServiceUrl, clientConf)
  val producer = client.createProducer(topic, producerConf)
  ite.foreach{ x => producer.sendAsync(x._2.content) }
  producer.close()

  client.close()

  }

---8<---


With 2.0 I try to replace the configuration classes with the
corresponding builders, but this setting doesn't work since the
producerBuilder isn't serializable :

---8<---

  [...]

  val client = clientBuilder.serviceUrl(serviceUrl).build()
  val producer = producerBuilder.topic(topicName).create

  [...]

---8<---


My actual workaround is to rewrite my own ProducerBuilderImpl without
pulsar client or schemas (the createAsync method still exists but throw
an exception). And then to instanciate a producer like this :

---8<---

  [...]

  val client : PulsarClientImpl =
clientBuilder.serviceUrl(broker).build().asInstanceOf[PulsarClientImpl]
  val producer =
client.createProducerAsync(producerBuilder.topic(topic).getConf(),
org.apache.pulsar.client.api.Schema.BYTES).get()

  [...]

---8<---

where producerBuilder is my own implementation which implements
ProducerBuilder interface.


I did the same thing for ConsumerBuilderImpl and also rewrite
SparkStreamingPulsarReceiver with the same kind of workaround. I don't
know why the configuration classes have been replaced by builders, but
it became more difficult to use it with spark (and probably with other
distributed framework), but maybe i did it wrong.

Re: Non-serializable builders

[Sijie Guo]

Geoffroy,

Thank you for reporting this. The change was made to evolve pulsar into a
type-safe client with schema and more validations. Along with the idea,
we changed the client to become a more fluent style builder and hide the
configuration data as an implementation detail to the pulsar client
implementation.

However we missed the use cases in data computing world, people tends to
use java Serializable on sharing configuration.

We would incorporate your use cases and try to bring the configuration
object back.

- Sijie



On Fri, Jun 8, 2018 at 1:09 AM Geoffroy Fouquier <
geoffroy.fouqu...@exensa.com> wrote:

>
> We are using pulsar 1.22 within a spark framework and I am currently
> upgrading my cluster to pulsar 2.0. One of the main change concerns the
> configuration classes, replace by builders. Although ProducerBuilder and
> ConsumerBuilder interfaces implement Serializable, corresponding
> implementations aren't, because a pulsar client object is embedded. The
> schema object is also a problem for serialization.
>
>
> Basically, with 1.22 my producer app. instanciate a client and a
> producer on each executor and only configurations classes need to be
> serialized
>
> ---8<---
>
> WarcRecordExtractor
>  .load(sc, input)
>  .foreachPartition{ ite =>
>
>  val client = PulsarClient.create(ServiceUrl, clientConf)
>  val producer = client.createProducer(topic, producerConf)
>  ite.foreach{ x => producer.sendAsync(x._2.content) }
>  producer.close()
>
>  client.close()
>
>  }
>
> ---8<---
>
>
> With 2.0 I try to replace the configuration classes with the
> corresponding builders, but this setting doesn't work since the
> producerBuilder isn't serializable :
>
> ---8<---
>
>  [...]
>
>  val client = clientBuilder.serviceUrl(serviceUrl).build()
>  val producer = producerBuilder.topic(topicName).create
>
>  [...]
>
> ---8<---
>
>
> My actual workaround is to rewrite my own ProducerBuilderImpl without
> pulsar client or schemas (the createAsync method still exists but throw
> an exception). And then to instanciate a producer like this :
>
> ---8<---
>
>  [...]
>
>  val client : PulsarClientImpl =
> clientBuilder.serviceUrl(broker).build().asInstanceOf[PulsarClientImpl]
>  val producer =
> client.createProducerAsync(producerBuilder.topic(topic).getConf(),
> org.apache.pulsar.client.api.Schema.BYTES).get()
>
>  [...]
>
> ---8<---
>
> where producerBuilder is my own implementation which implements
> ProducerBuilder interface.
>
>
> I did the same thing for ConsumerBuilderImpl and also rewrite
> SparkStreamingPulsarReceiver with the same kind of workaround. I don't
> know why the configuration classes have been replaced by builders, but
> it became more difficult to use it with spark (and probably with other
> distributed framework), but maybe i did it wrong.
>
>
>
>

Slack digest for #dev - 2018-06-08

[Apache Pulsar Slack]

2018-06-07 13:52:50 UTC - Chris Miller: @Chris Miller has joined the channel

Build failed in Jenkins: pulsar-website-build #269

[Apache Jenkins Server]

See 


Changes:

[sanjeevrk] Make Python code work with Pulsar Function localrun (#1930)

[github] Introduce kinesis sink on function (#1904)

[github] Only set parameters when things are not null (#1931)

[github] The check should be for null (#1932)

--
[...truncated 713.87 KB...]
import org.apache.commons.lang3.StringUtils;
   ^
pulsar-client-admin/src/main/java/org/apache/pulsar/client/admin/PulsarAdmin.java:33:
 error: package org.apache.http.conn.ssl does not exist
import org.apache.http.conn.ssl.DefaultHostnameVerifier;
   ^
pulsar-client-admin/src/main/java/org/apache/pulsar/client/admin/PulsarAdmin.java:34:
 error: package org.apache.http.conn.ssl does not exist
import org.apache.http.conn.ssl.NoopHostnameVerifier;
   ^
pulsar-client-admin/src/main/java/org/apache/pulsar/client/admin/PulsarAdmin.java:35:
 error: package org.apache.pulsar.client.admin.internal does not exist
import org.apache.pulsar.client.admin.internal.BrokerStatsImpl;
  ^
pulsar-client-admin/src/main/java/org/apache/pulsar/client/admin/PulsarAdmin.java:36:
 error: package org.apache.pulsar.client.admin.internal does not exist
import org.apache.pulsar.client.admin.internal.BrokersImpl;
  ^
pulsar-client-admin/src/main/java/org/apache/pulsar/client/admin/PulsarAdmin.java:37:
 error: package org.apache.pulsar.client.admin.internal does not exist
import org.apache.pulsar.client.admin.internal.ClustersImpl;
  ^
pulsar-client-admin/src/main/java/org/apache/pulsar/client/admin/PulsarAdmin.java:38:
 error: package org.apache.pulsar.client.admin.internal does not exist
import org.apache.pulsar.client.admin.internal.FunctionsImpl;
  ^
pulsar-client-admin/src/main/java/org/apache/pulsar/client/admin/PulsarAdmin.java:39:
 error: package org.apache.pulsar.client.admin.internal does not exist
import org.apache.pulsar.client.admin.internal.JacksonConfigurator;
  ^
pulsar-client-admin/src/main/java/org/apache/pulsar/client/admin/PulsarAdmin.java:40:
 error: package org.apache.pulsar.client.admin.internal does not exist
import org.apache.pulsar.client.admin.internal.LookupImpl;
  ^
pulsar-client-admin/src/main/java/org/apache/pulsar/client/admin/PulsarAdmin.java:41:
 error: package org.apache.pulsar.client.admin.internal does not exist
import org.apache.pulsar.client.admin.internal.NamespacesImpl;
  ^
pulsar-client-admin/src/main/java/org/apache/pulsar/client/admin/PulsarAdmin.java:42:
 error: package org.apache.pulsar.client.admin.internal does not exist
import org.apache.pulsar.client.admin.internal.NonPersistentTopicsImpl;
  ^
pulsar-client-admin/src/main/java/org/apache/pulsar/client/admin/PulsarAdmin.java:43:
 error: package org.apache.pulsar.client.admin.internal does not exist
import org.apache.pulsar.client.admin.internal.TopicsImpl;
  ^
pulsar-client-admin/src/main/java/org/apache/pulsar/client/admin/PulsarAdmin.java:44:
 error: package org.apache.pulsar.client.admin.internal does not exist
import org.apache.pulsar.client.admin.internal.TenantsImpl;
  ^
pulsar-client-admin/src/main/java/org/apache/pulsar/client/admin/PulsarAdmin.java:45:
 error: package org.apache.pulsar.client.admin.internal does not exist
import org.apache.pulsar.client.admin.internal.PulsarAdminBuilderImpl;
  ^
pulsar-client-admin/src/main/java/org/apache/pulsar/client/admin/PulsarAdmin.java:46:
 error: package org.apache.pulsar.client.admin.internal does not exist
import org.apache.pulsar.client.admin.internal.ResourceQuotasImpl;
  ^
pulsar-client-admin/src/main/java/org/apache/pulsar/client/admin/PulsarAdmin.java:47:
 error: package org.apache.pulsar.client.api does not exist
import org.apache.pulsar.client.api.Authentication;
   ^
pulsar-client-admin/src/main/java/org/apache/pulsar/client/admin/PulsarAdmin.java:48:
 error: package org.apache.pulsar.client.api does not exist
import org.apache.pulsar.client.api.AuthenticationDataProvider;
   ^
pulsar-client-admin/src/main/java/org/apache/pulsar/client/admin/PulsarAdmin.java:49:
 error: package org.apache.pulsar.client.api does not exist
import org.apache.pulsar.client.api.AuthenticationFactory;
   ^
pulsar-client-admin/src/main/java/org/apache/pulsar/client/admin/PulsarAdmin.java:50:
 error: packag

Non-serializable builders

[Geoffroy Fouquier]

We are using pulsar 1.22 within a spark framework and I am currently 
upgrading my cluster to pulsar 2.0. One of the main change concerns the 
configuration classes, replace by builders. Although ProducerBuilder and 
ConsumerBuilder interfaces implement Serializable, corresponding 
implementations aren't, because a pulsar client object is embedded. The 
schema object is also a problem for serialization.



Basically, with 1.22 my producer app. instanciate a client and a 
producer on each executor and only configurations classes need to be 
serialized


---8<---

WarcRecordExtractor
    .load(sc, input)
    .foreachPartition{ ite =>

            val client = PulsarClient.create(ServiceUrl, clientConf)
            val producer = client.createProducer(topic, producerConf)
        ite.foreach{ x => producer.sendAsync(x._2.content) }
    producer.close()

    client.close()

    }

---8<---


With 2.0 I try to replace the configuration classes with the 
corresponding builders, but this setting doesn't work since the 
producerBuilder isn't serializable :


---8<---

            [...]

            val client = clientBuilder.serviceUrl(serviceUrl).build()
            val producer = producerBuilder.topic(topicName).create

            [...]

---8<---


My actual workaround is to rewrite my own ProducerBuilderImpl without  
pulsar client or schemas (the createAsync method still exists but throw 
an exception). And then to instanciate a producer like this :


---8<---

            [...]

            val client : PulsarClientImpl = 
clientBuilder.serviceUrl(broker).build().asInstanceOf[PulsarClientImpl]
            val producer = 
client.createProducerAsync(producerBuilder.topic(topic).getConf(), 
org.apache.pulsar.client.api.Schema.BYTES).get()


            [...]

---8<---

where producerBuilder is my own implementation which implements 
ProducerBuilder interface.



I did the same thing for ConsumerBuilderImpl and also rewrite 
SparkStreamingPulsarReceiver with the same kind of workaround. I don't 
know why the configuration classes have been replaced by builders, but 
it became more difficult to use it with spark (and probably with other 
distributed framework), but maybe i did it wrong.

Re: [VOTE] Pulsar Release 2.0.1-incubating Candidate 2

[Hiroyuki Sakai]

+1 (binding)

Environment: MacOS 10.13.4

Checked:
- sha1, sha512, asc
- start standalone service and producer/consumer client test (Src/Bin
distribution)
- rat check on Src distribution
- compile and unit tests on Src distribution

Regards,
Hiroyuki 

-Original Message-
From: Matteo Merli 
Reply-To: "dev@pulsar.incubator.apache.org" 
Date: Friday, June 8, 2018 15:35
To: "dev@pulsar.incubator.apache.org" 
Subject: [VOTE] Pulsar Release 2.0.1-incubating Candidate 2

This is the second release candidate for Apache Pulsar, version
2.0.1-incubating.

This is a patch release that fixes few issues identified in previous
release 2.0.0-rc1-incubating.

It fixes the following issues:
https://github.com/apache/incubator-pulsar/milestone/14?closed=1

*** Please download, test and vote on this release. This vote will stay
open
for at least 72 hours ***

Note that we are voting upon the source (tag), binaries are provided for
convenience.

Source and binary files:

https://dist.apache.org/repos/dist/dev/incubator/pulsar/pulsar-2.0.1-incubating-candidate-2/

SHA-1 checksums:

5acd63f8aa1fa75f386df7b9fd803a943978e609
apache-pulsar-2.0.1-incubating-bin.tar.gz
312bc0f1ae170c1d2dc38b1e362fb50f82c00525
apache-pulsar-2.0.1-incubating-src.tar.gz

Maven staging repo:
https://repository.apache.org/content/repositories/orgapachepulsar-1020


The tag to be voted upon:
v2.0.1-incubating-candidate-2 (fd47532380d770e4fd78cabe71dea293fc2f0e06)

https://github.com/apache/incubator-pulsar/releases/tag/v2.0.1-incubating-candidate-2



Pulsar's KEYS file containing PGP keys we use to sign the release:
https://dist.apache.org/repos/dist/release/incubator/pulsar/KEYS

Please download the the source package, and follow the README to build
and run the Pulsar standalone service.
-- 
Matteo Merli