Any stream API for pulsar apps?

[jun aoki]

I did a 10 min google search for pulsar stream API, something comparable to
kafka stream API. (https://kafka.apache.org/documentation/streams/)
but I haven't been able to find any. Is there such thing exists? Or am I
conceptually mistaken?

-- 
-jun

Re: Embedded pulsar cluster?

[jun aoki]

Matteo, It was nice meeting your earlier and thank you for letting me those
options.
I am new so I can't picture what form of embedded/single class mini-pulsar
cluster would be ideal for our unit tests yet, so I'd have to try them and
get a feel of it first. I'm also looking into pulsar docker image and it
seems I can quickly make up some local integration with our services. (
https://hub.docker.com/r/apachepulsar/pulsar/tags/)

On Fri, May 4, 2018 at 4:50 PM, Matteo Merli  wrote:

> Hi Jun,
>
> there are few options:
>
>  1. We have a MockedBrokerService that we use in our unit tests. It's not
> super-fancy but it might work:
> https://github.com/apache/incubator-pulsar/blob/master/
> pulsar-broker/src/test/java/org/apache/pulsar/broker/auth/
> MockedPulsarServiceBaseTest.java
>That starts a "real" broker instance with mocked ZooKeeper and
> BookKeeper to keep data in memory. The way we typically use this is by
> extending from that class and create tests.
>
> 2. There is a standalone Pulsar service ("bin/pulsar standalone") that
> includes broker, ZK and BK service in a single JVM process. It uses the
> "real" implementation of each component and stores the data on disk. It was
> not initially meant to be embedded but it might actually very easy to fix.
>
> 3. There's a LocalBookkeeperEnsemble class that creates a ZK + BK service
> and, combined with a PulsarService instance, could get the whole service
> up. I agree that a single class "service" would be a nicer option.
>
> Matteo
>
> On Fri, May 4, 2018 at 12:55 PM jun aoki  wrote:
>
> > do you guys have embedded pulsar I can use for unit tests?
> > Something similar embedded kafka to
> >
> > https://github.com/apache/kafka/blob/trunk/streams/src/
> test/java/org/apache/kafka/streams/integration/utils/
> EmbeddedKafkaCluster.java
> >
> > I'd like to play with pulsar by replacing our kafka stream app with it.
> >
> > --
> > -jun
> >
> --
> Matteo Merli
> 
>



-- 
-jun

Build failed in Jenkins: pulsar-master #455

[Apache Jenkins Server]

See 


Changes:

[guosijie] Make Function port part of InstanceConfig (#1731)

--
[...truncated 2.79 MB...]
[INFO] Ign:2 http://deb.debian.org/debian stretch InRelease
[INFO] Get:3 http://deb.debian.org/debian stretch-updates InRelease [91.0 kB]
[INFO] Get:4 http://deb.debian.org/debian stretch Release [118 kB]
[INFO] Get:5 http://deb.debian.org/debian stretch Release.gpg [2434 B]
[INFO] Get:6 http://security.debian.org/debian-security stretch/updates/main 
amd64 Packages [471 kB]
[INFO] Get:7 http://deb.debian.org/debian stretch-updates/main amd64 Packages 
[12.1 kB]
[INFO] Get:8 http://deb.debian.org/debian stretch/main amd64 Packages [9530 kB]
[INFO] Fetched 10.3 MB in 1s (5893 kB/s)
[INFO] Reading package lists...
[INFO] 
[INFO] Reading package lists...
[INFO] 
[INFO] Building dependency tree...
[INFO] 
[INFO] Reading state information...
[INFO] 
[INFO] The following additional packages will be installed:
[INFO]   bind9-host geoip-database libbind9-140 libdns162 libgeoip1 libisc160
[INFO]   libisccc140 libisccfg140 liblwres141 netcat-traditional
[INFO] Suggested packages:
[INFO]   rblcheck geoip-bin python-kazoo-doc
[INFO] The following NEW packages will be installed:
[INFO]   bind9-host dnsutils geoip-database libbind9-140 libdns162 libgeoip1
[INFO]   libisc160 libisccc140 libisccfg140 liblwres141 netcat 
netcat-traditional
[INFO]   python-kazoo
[INFO] 0 upgraded, 13 newly installed, 0 to remove and 0 not upgraded.
[INFO] Need to get 5196 kB of archives.
[INFO] After this operation, 15.4 MB of additional disk space will be used.
[INFO] Get:1 http://deb.debian.org/debian stretch/main amd64 libgeoip1 amd64 
1.6.9-4 [90.5 kB]
[INFO] Get:2 http://deb.debian.org/debian stretch/main amd64 libisc160 amd64 
1:9.10.3.dfsg.P4-12.3+deb9u4 [398 kB]
[INFO] Get:3 http://deb.debian.org/debian stretch/main amd64 libdns162 amd64 
1:9.10.3.dfsg.P4-12.3+deb9u4 [1077 kB]
[INFO] Get:4 http://deb.debian.org/debian stretch/main amd64 libisccc140 amd64 
1:9.10.3.dfsg.P4-12.3+deb9u4 [198 kB]
[INFO] Get:5 http://deb.debian.org/debian stretch/main amd64 libisccfg140 amd64 
1:9.10.3.dfsg.P4-12.3+deb9u4 [223 kB]
[INFO] Get:6 http://deb.debian.org/debian stretch/main amd64 libbind9-140 amd64 
1:9.10.3.dfsg.P4-12.3+deb9u4 [206 kB]
[INFO] Get:7 http://deb.debian.org/debian stretch/main amd64 liblwres141 amd64 
1:9.10.3.dfsg.P4-12.3+deb9u4 [214 kB]
[INFO] Get:8 http://deb.debian.org/debian stretch/main amd64 bind9-host amd64 
1:9.10.3.dfsg.P4-12.3+deb9u4 [231 kB]
[INFO] Get:9 http://deb.debian.org/debian stretch/main amd64 netcat-traditional 
amd64 1.10-41+b1 [67.0 kB]
[INFO] Get:10 http://deb.debian.org/debian stretch/main amd64 dnsutils amd64 
1:9.10.3.dfsg.P4-12.3+deb9u4 [283 kB]
[INFO] Get:11 http://deb.debian.org/debian stretch/main amd64 geoip-database 
all 20170512-1 [2112 kB]
[INFO] Get:12 http://deb.debian.org/debian stretch/main amd64 python-kazoo all 
2.2.1-1 [88.2 kB]
[INFO] Get:13 http://deb.debian.org/debian stretch/main amd64 netcat all 
1.10-41 [8962 B]
[INFO] debconf: delaying package configuration, since apt-utils is not installed
[INFO] 
[INFO] Fetched 5196 kB in 0s (21.7 MB/s)
[INFO] Selecting previously unselected package libgeoip1:amd64.
[INFO] (Reading database ... 
[INFO] (Reading database ... 5%
[INFO] (Reading database ... 10%
[INFO] (Reading database ... 15%
[INFO] (Reading database ... 20%
[INFO] (Reading database ... 25%
[INFO] (Reading database ... 30%
[INFO] (Reading database ... 35%(Reading database ... 40%(Reading database ... 
45%(Reading database ... 50%(Reading database ... 55%(Reading database ... 
60%(Reading database ... 65%(Reading database ... 70%(Reading database ... 
75%(Reading database ... 80%(Reading database ... 85%(Reading database ... 
90%(Reading database ... 95%(Reading database ... 100%(Reading database ... 
23467 files and directories currently installed.)
[INFO] Preparing to unpack .../00-libgeoip1_1.6.9-4_amd64.deb ...
[INFO] Unpacking libgeoip1:amd64 (1.6.9-4) ...
[INFO] Selecting previously unselected package libisc160:amd64.
[INFO] Preparing to unpack 
.../01-libisc160_1%3a9.10.3.dfsg.P4-12.3+deb9u4_amd64.deb ...
[INFO] Unpacking libisc160:amd64 (1:9.10.3.dfsg.P4-12.3+deb9u4) ...
[INFO] Selecting previously unselected package libdns162:amd64.
[INFO] Preparing to unpack 
.../02-libdns162_1%3a9.10.3.dfsg.P4-12.3+deb9u4_amd64.deb ...
[INFO] Unpacking libdns162:amd64 (1:9.10.3.dfsg.P4-12.3+deb9u4) ...
[INFO] Selecting previously unselected package libisccc140:amd64.
[INFO] Preparing to unpack 
.../03-libisccc140_1%3a9.10.3.dfsg.P4-12.3+deb9u4_amd64.deb ...
[INFO] Unpacking libisccc140:amd64 (1:9.10.3.dfsg.P4-12.3+deb9u4) ...
[INFO] Selecting previously unselected package libisccfg140:amd64.
[INFO] Preparing to unpack 
.../04-libisccfg140_1%3a9.10.3.dfsg.P4-12.3+deb9u4_amd64.deb ...
[INFO] Unpacking libisccfg140:amd64 (1:9.10.3.dfsg.P4-12.3+deb9u4) ...
[INFO] Selec

Re: Embedded pulsar cluster?

[Matteo Merli]

Hi Jun,

there are few options:

 1. We have a MockedBrokerService that we use in our unit tests. It's not
super-fancy but it might work:
https://github.com/apache/incubator-pulsar/blob/master/pulsar-broker/src/test/java/org/apache/pulsar/broker/auth/MockedPulsarServiceBaseTest.java
   That starts a "real" broker instance with mocked ZooKeeper and
BookKeeper to keep data in memory. The way we typically use this is by
extending from that class and create tests.

2. There is a standalone Pulsar service ("bin/pulsar standalone") that
includes broker, ZK and BK service in a single JVM process. It uses the
"real" implementation of each component and stores the data on disk. It was
not initially meant to be embedded but it might actually very easy to fix.

3. There's a LocalBookkeeperEnsemble class that creates a ZK + BK service
and, combined with a PulsarService instance, could get the whole service
up. I agree that a single class "service" would be a nicer option.

Matteo

On Fri, May 4, 2018 at 12:55 PM jun aoki  wrote:

> do you guys have embedded pulsar I can use for unit tests?
> Something similar embedded kafka to
>
> https://github.com/apache/kafka/blob/trunk/streams/src/test/java/org/apache/kafka/streams/integration/utils/EmbeddedKafkaCluster.java
>
> I'd like to play with pulsar by replacing our kafka stream app with it.
>
> --
> -jun
>
-- 
Matteo Merli

Embedded pulsar cluster?

[jun aoki]

do you guys have embedded pulsar I can use for unit tests?
Something similar embedded kafka to
https://github.com/apache/kafka/blob/trunk/streams/src/test/java/org/apache/kafka/streams/integration/utils/EmbeddedKafkaCluster.java

I'd like to play with pulsar by replacing our kafka stream app with it.

-- 
-jun

Slack digest for #dev - 2018-05-04

[Apache Pulsar Slack]

2018-05-03 11:08:13 UTC - haphut: @haphut has joined the channel