date:20190923

[GitHub] [pulsar] sijie commented on issue #5254: Expose bookkeeperDiskWeightBasedPlacementEnabled in broker.conf

2019-09-23 Thread GitBox

sijie commented on issue #5254: Expose 
bookkeeperDiskWeightBasedPlacementEnabled in broker.conf
URL: https://github.com/apache/pulsar/pull/5254#issuecomment-534401324
 
 
   @codelipenghui weight based placement policy requires configuring bookies to 
use v3 protocol. there are two options: 
   
   1) make changes to managed ledger to allow using v3 protocol. so that you 
can use both.
   
   or 
   
   2) wait until we release bookkeeper 4.10. 4.10 will support a mixed mode of 
both v2 and v3. the release of bookkeeper 4.10 is under discussion.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [pulsar] yittg commented on issue #5165: [PIP-43] Support producer to send msg with different schema

2019-09-23 Thread GitBox

yittg commented on issue #5165: [PIP-43] Support producer to send msg with 
different schema
URL: https://github.com/apache/pulsar/pull/5165#issuecomment-534399428
 
 
   @sijie Does the new implementation make sense?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [pulsar] jerrypeng commented on issue #5261: Fix client backoff

2019-09-23 Thread GitBox

jerrypeng commented on issue #5261: Fix client backoff
URL: https://github.com/apache/pulsar/pull/5261#issuecomment-534395787
 
 
   run integration tests


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [pulsar] jerrypeng commented on issue #5243: Bug in Message Deduplication that may cause incorrect client/broker interaction

2019-09-23 Thread GitBox

jerrypeng commented on issue #5243: Bug in Message Deduplication that may cause 
incorrect client/broker interaction
URL: https://github.com/apache/pulsar/pull/5243#issuecomment-534395663
 
 
   run integration tests


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[pulsar.wiki] branch master updated: Updated PIP 33: Replicated subscriptions (markdown)

2019-09-23 Thread hjf

This is an automated email from the ASF dual-hosted git repository.

hjf pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/pulsar.wiki.git


The following commit(s) were added to refs/heads/master by this push:
 new a8bc8fa  Updated PIP 33: Replicated subscriptions (markdown)
a8bc8fa is described below

commit a8bc8fae634f6c025300924cc4731ddfe3aac591
Author: Jennifer Huang <47805623+jennifer88hu...@users.noreply.github.com>
AuthorDate: Tue Sep 24 12:00:01 2019 +0800

Updated PIP 33: Replicated subscriptions (markdown)
---
 PIP-33:-Replicated-subscriptions.md | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/PIP-33:-Replicated-subscriptions.md 
b/PIP-33:-Replicated-subscriptions.md
index c9799ad..95afdd4 100644
--- a/PIP-33:-Replicated-subscriptions.md
+++ b/PIP-33:-Replicated-subscriptions.md
@@ -13,18 +13,18 @@ asynchronously replicated across multiple geographical 
regions.
 
 ## Current state of affairs
 
-Pulsar support geo-replication feature, in which a topic can be
+Pulsar supports geo-replication feature, in which a topic can be
 configured to be replicated across N regions, (eg: `us-west`, `us-east`
 and `eu-central`).
 
 The topic is presented as a virtual "global" entity in which messages
-can be published and consumer from any of the configured cluster.
+can be published and consumed from any of the configured cluster.
 
 The only limitation is that subscriptions are currently "local" to the
 cluster in which they are created. That is, no state for the subscription
 is transferred across regions.
 
-If a consumer reconnects to a new region it will trigger the creation of a
+If a consumer reconnects to a new region, it will trigger the creation of a
 new unrelated subscription, albeit with the same name. This subscription
 will be created at the end of the topic in the new region (or at the
 beginning, depending on configuration) and at the same time, the

[GitHub] [pulsar] aahmed-se commented on issue #5263: Fix Cmake to build _pulsar for osx

2019-09-23 Thread GitBox

aahmed-se commented on issue #5263: Fix Cmake to build _pulsar for osx
URL: https://github.com/apache/pulsar/pull/5263#issuecomment-534376983
 
 
   run java8 tests
   run integration tests


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [pulsar] Monica-zy removed a comment on issue #5221: [doc] Improve Pulsar Administration Pulsar dashboard

2019-09-23 Thread GitBox

Monica-zy removed a comment on issue #5221: [doc] Improve Pulsar Administration 
Pulsar dashboard
URL: https://github.com/apache/pulsar/pull/5221#issuecomment-534363789
 
 
   run java8 tests


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [pulsar] jiazhai commented on issue #5243: Bug in Message Deduplication that may cause incorrect client/broker interaction

2019-09-23 Thread GitBox

jiazhai commented on issue #5243: Bug in Message Deduplication that may cause 
incorrect client/broker interaction
URL: https://github.com/apache/pulsar/pull/5243#issuecomment-534367523
 
 
   run integration tests
   
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [pulsar] Monica-zy commented on issue #5221: [doc] Improve Pulsar Administration Pulsar dashboard

2019-09-23 Thread GitBox

Monica-zy commented on issue #5221: [doc] Improve Pulsar Administration Pulsar 
dashboard
URL: https://github.com/apache/pulsar/pull/5221#issuecomment-534363789
 
 
   run java8 tests


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [pulsar] codelipenghui edited a comment on issue #5254: Expose bookkeeperDiskWeightBasedPlacementEnabled in broker.conf

2019-09-23 Thread GitBox

codelipenghui edited a comment on issue #5254: Expose 
bookkeeperDiskWeightBasedPlacementEnabled in broker.conf
URL: https://github.com/apache/pulsar/pull/5254#issuecomment-534360264
 
 
   When i test enable diskWeightBasedPlacement locally, standalone can't 
startup successful, the output log stoped at:
   ```
   10:22:07.615 [main-EventThread] INFO  
org.apache.bookkeeper.zookeeper.ZooKeeperWatcherBase - ZooKeeper client is 
connected now.
   10:22:07.618 [main] INFO  org.apache.distributedlog.impl.BKNamespaceDriver - 
Created shared client builder 
bk:distributedlog://127.0.0.1:2181/stream/storage:factory_writer_shared : 
zkServers = 127.0.0.1:2181, ledgersPath = /ledgers, numIOThreads = 6
   10:22:07.623 [main] INFO  org.apache.distributedlog.impl.metadata.BKDLConfig 
- Propagate BKDLConfig to DLConfig : encodeRegionID = false, 
firstLogSegmentSequenceNumber = 1, createStreamIfNotExists = true, isFederated 
= false.
   10:22:07.632 [main] INFO  org.apache.distributedlog.impl.BKNamespaceDriver - 
Initialized BK namespace driver: clientId = storage-server, regionId = 0, 
federated = false.
   10:22:07.652 [main] INFO  
org.apache.distributedlog.logsegment.LogSegmentMetadataCache - Log segment 
cache is enabled = true
   10:22:07.652 [main] INFO  
org.apache.bookkeeper.stream.server.service.DLNamespaceProviderService - 
Provided distributedlog namespace at 
distributedlog://127.0.0.1:2181/stream/storage.
   10:22:07.801 [main] INFO  
org.apache.bookkeeper.stream.storage.impl.sc.ZkStorageContainerManager - 
Watched cluster assignment data.
   10:22:07.801 [main] INFO  
org.apache.bookkeeper.stream.storage.impl.sc.ZkStorageContainerManager - 
Scheduled storage container probe task at every 15000 ms
   10:22:07.822 [zk-storage-container-manager] INFO  
org.apache.bookkeeper.stream.storage.impl.sc.ZkStorageContainerManager - 
Servers joined : [hostname: "10.2.4.175"
   port: 4181
   ]
   10:22:07.833 [main] INFO  
org.apache.bookkeeper.stream.storage.impl.grpc.GrpcMetaRangeService - Created 
MetaRange service
   10:22:07.838 [main] INFO  
org.apache.bookkeeper.stream.storage.impl.grpc.GrpcTableService - Created Table 
service
   10:22:07.871 [main] INFO  org.apache.zookeeper.ZooKeeper - Initiating client 
connection, connectString=127.0.0.1:2181 sessionTimeout=1 
watcher=org.apache.bookkeeper.zookeeper.ZooKeeperWatcherBase@5e8e02a3
   10:22:07.874 [main-SendThread(localhost:2181)] INFO  
org.apache.zookeeper.ClientCnxn - Opening socket connection to server 
localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown 
error)
   10:22:07.875 [main-SendThread(localhost:2181)] INFO  
org.apache.zookeeper.ClientCnxn - Socket connection established to 
localhost/127.0.0.1:2181, initiating session
   10:22:07.875 [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181] INFO  
org.apache.zookeeper.server.NIOServerCnxnFactory - Accepted socket connection 
from /127.0.0.1:56997
   10:22:07.876 [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181] INFO  
org.apache.zookeeper.server.ZooKeeperServer - Client attempting to establish 
new session at /127.0.0.1:56997
   10:22:07.876 [SyncThread:0] INFO  
org.apache.zookeeper.server.ZooKeeperServer - Established session 
0x100294ba83d0007 with negotiated timeout 1 for client /127.0.0.1:56997
   10:22:07.877 [main-SendThread(localhost:2181)] INFO  
org.apache.zookeeper.ClientCnxn - Session establishment complete on server 
localhost/127.0.0.1:2181, sessionid = 0x100294ba83d0007, negotiated timeout = 
1
   10:22:07.877 [main-EventThread] INFO  
org.apache.bookkeeper.zookeeper.ZooKeeperWatcherBase - ZooKeeper client is 
connected now.
   10:22:07.880 [main] INFO  
org.apache.bookkeeper.stream.server.service.RegistrationStateService - 
Successfully register myself under registration path 
/stream/servers/172.20.10.2:4181
   10:22:07.891 [main] INFO  
org.apache.bookkeeper.stream.server.service.ClusterControllerService - 
Successfully started the cluster controller.
   10:22:08.121 [client-scheduler-OrderedScheduler-0-0] INFO  
org.apache.bookkeeper.clients.impl.channel.StorageServerChannelManager - Added 
range server (hostname: "10.2.4.175"
   port: 4181
   ) into the channel manager.
   10:22:14.180 [SessionTracker] INFO  
org.apache.zookeeper.server.ZooKeeperServer - Expiring session 
0x1002940a6a5, timeout of 6000ms exceeded
   10:22:14.180 [ProcessThread(sid:0 cport:2181):] INFO  
org.apache.zookeeper.server.PrepRequestProcessor - Processed session 
termination for sessionid: 0x1002940a6a5
   10:22:17.182 [SessionTracker] INFO  
org.apache.zookeeper.server.ZooKeeperServer - Expiring session 
0x1002940a6a50001, timeout of 1ms exceeded
   10:22:17.182 [SessionTracker] INFO  
org.apache.zookeeper.server.ZooKeeperServer - Expiring session 
0x1002940a6a50007, timeout of 1ms exceeded
   10:22:17.183 [ProcessThread(sid:0 cport:2181):] INFO  
org.apache.zookeeper.server.PrepRequestProcessor - Processed session 
termination for sessionid: 0x1002940a6a50001

[GitHub] [pulsar] codelipenghui commented on issue #5254: Expose bookkeeperDiskWeightBasedPlacementEnabled in broker.conf

2019-09-23 Thread GitBox

codelipenghui commented on issue #5254: Expose 
bookkeeperDiskWeightBasedPlacementEnabled in broker.conf
URL: https://github.com/apache/pulsar/pull/5254#issuecomment-534360264
 
 
   When i test enable diskWeightBasedPlacement locally, standalone can't 
startup successful, the output log stoped at:


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [pulsar] Monica-zy commented on issue #5221: [doc] Improve Pulsar Administration Pulsar dashboard

2019-09-23 Thread GitBox

Monica-zy commented on issue #5221: [doc] Improve Pulsar Administration Pulsar 
dashboard
URL: https://github.com/apache/pulsar/pull/5221#issuecomment-534358056
 
 
   run cpp tests
   run integration tests
   run java8 tests


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [pulsar] aahmed-se commented on issue #5263: Fix Cmake to build _pulsar for osx

2019-09-23 Thread GitBox

aahmed-se commented on issue #5263: Fix Cmake to build _pulsar for osx
URL: https://github.com/apache/pulsar/pull/5263#issuecomment-534356959
 
 
   rerun cpp tests


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [pulsar] aahmed-se commented on issue #5263: Fix Cmake to build _pulsar for osx

2019-09-23 Thread GitBox

aahmed-se commented on issue #5263: Fix Cmake to build _pulsar for osx
URL: https://github.com/apache/pulsar/pull/5263#issuecomment-534352810
 
 
   rerun cpp tests


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [pulsar] aahmed-se commented on issue #5261: Fix client backoff

2019-09-23 Thread GitBox

aahmed-se commented on issue #5261: Fix client backoff
URL: https://github.com/apache/pulsar/pull/5261#issuecomment-534352572
 
 
   run integration tests


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [pulsar] aahmed-se commented on issue #5263: Fix Cmake to build _pulsar for osx

2019-09-23 Thread GitBox

aahmed-se commented on issue #5263: Fix Cmake to build _pulsar for osx
URL: https://github.com/apache/pulsar/pull/5263#issuecomment-534352505
 
 
   run cpp tests


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [pulsar] tuteng commented on issue #5258: [Doc] Add ElasticSearch sink connector guide

2019-09-23 Thread GitBox

tuteng commented on issue #5258: [Doc] Add *ElasticSearch sink connector guide*
URL: https://github.com/apache/pulsar/pull/5258#issuecomment-534352028
 
 
   run java8 tests


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [pulsar] tuteng commented on issue #5257: [Doc] Add Cassandra sink connector guide

2019-09-23 Thread GitBox

tuteng commented on issue #5257: [Doc] Add *Cassandra sink connector guide*
URL: https://github.com/apache/pulsar/pull/5257#issuecomment-534351606
 
 
   run java8 tests
   run integrations tests
   run cpp tests


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [pulsar] tuteng commented on issue #5257: [Doc] Add Cassandra sink connector guide

2019-09-23 Thread GitBox

tuteng commented on issue #5257: [Doc] Add *Cassandra sink connector guide*
URL: https://github.com/apache/pulsar/pull/5257#issuecomment-534351683
 
 
   run integration tests


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [pulsar] tuteng commented on a change in pull request #5256: [Doc] Add Aerospike sink connector guide

2019-09-23 Thread GitBox

tuteng commented on a change in pull request #5256: [Doc] Add *Aerospike sink 
connector guide*
URL: https://github.com/apache/pulsar/pull/5256#discussion_r327395620
 
 

 ##
 File path: site2/docs/io-aerospike-sink.md
 ##
 @@ -0,0 +1,25 @@
+---
+id: io-aerospike-sink
+title: Aerospike sink connector
+sidebar_label: Aerospike sink connector
+---
+
+The Aerospike sink connector pulls messages from Pulsar topics to Aerospike 
clusters.
+
+## Configuration
+
+The configuration of the Aerospike sink connector has the following properties.
+
+### Property
+
+| Name | Type|Required | Default | Description 
+|--|--|--|-|-|
+| `seedHosts` |String| true | 3000 | The comma-separated list of one or more 
Aerospike cluster hosts.Each host can be specified as a valid IP 
address or hostname followed by an optional port number. | 
 
 Review comment:
   The default value for this option seems incorrect.
   This is a required option. Its value is similar to 
`ip1:port1,ip2:port2,ip3:port3`, I think there is no need to add default values 
here.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [pulsar] tuteng commented on a change in pull request #5256: [Doc] Add Aerospike sink connector guide

2019-09-23 Thread GitBox

tuteng commented on a change in pull request #5256: [Doc] Add *Aerospike sink 
connector guide*
URL: https://github.com/apache/pulsar/pull/5256#discussion_r327395896
 
 

 ##
 File path: site2/docs/io-aerospike-sink.md
 ##
 @@ -0,0 +1,25 @@
+---
+id: io-aerospike-sink
+title: Aerospike sink connector
+sidebar_label: Aerospike sink connector
+---
+
+The Aerospike sink connector writes messages from Pulsar topics to Aerospike 
clusters.
+
+## Configuration
+
+The configuration of the Aerospike sink connector has the following properties.
+
+### Property
+
+| Name | Type|Required | Default | Description 
+|--|--|--|-|-|
+| `seedHosts` |String| true | 3000 | The comma-separated list of one or more 
Aerospike cluster hosts.Each host can be specified as a valid IP 
address or hostname followed by an optional port number. | 
+| `keyspace` | String| true | |The Aerospike namespace. |
+| `columnName` | String | true| |The Aerospike column name. |
+|`userName`|String|false||The Aerospike username.|
+|`password`|String|false||The Aerospike password.|
+| `keySet` | String|false | | The Aerospike set name. |
+| `maxConcurrentRequests` |int| false | 100 | The maximum number of concurrent 
Aerospike transactions that a sink can open. |
+| `timeoutMs` | int|false | 100 | This property controls `socketTimeout` and 
`totalTimeout` for Aerospike transactions.  |
+| `retries` | int|false | 1 |The maximum number of retries before aborting a 
write transaction to Aerospike. |
 
 Review comment:
   `userName` default value is `null`, `password` default value is `null`
   `keySet` default value is `null`


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [pulsar] aahmed-se commented on issue #5263: Fix Cmake to build _pulsar for osx

2019-09-23 Thread GitBox

aahmed-se commented on issue #5263: Fix Cmake to build _pulsar for osx
URL: https://github.com/apache/pulsar/pull/5263#issuecomment-534349352
 
 
   run cpp tests


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [pulsar] tuteng commented on issue #5255: [Doc] Add Rabbit source connector guide

2019-09-23 Thread GitBox

tuteng commented on issue #5255: [Doc] Add *Rabbit source connector guide*
URL: https://github.com/apache/pulsar/pull/5255#issuecomment-534349056
 
 
   run integration tests
   run java8 tests


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [pulsar] tuteng commented on a change in pull request #5253: [Doc] Add Netty source connector guide

2019-09-23 Thread GitBox

tuteng commented on a change in pull request #5253: [Doc] Add *Netty source 
connector guide*
URL: https://github.com/apache/pulsar/pull/5253#discussion_r327393960
 
 

 ##
 File path: site2/docs/io-netty-source.md
 ##
 @@ -0,0 +1,204 @@
+---
+id: io-netty
+title: Netty source connector
+sidebar_label: Netty source connector
+---
+
+The Netty source connector opens a port that accepts incoming data via the 
configured network protocol 
+and publish it to user-defined Pulsar topics.
+
+This connector can be used in a containerized (for example, k8s) deployment. 
Otherwise, if the connector is running in process or thread mode, the instance 
may be conflicting on listening to ports.
+
+## Configuration
+
+The configuration of the Netty source connector has the following properties.
+
+### Property
+
+| Name | Type|Required | Default | Description 
+|--|--|--|-|-|
+| `type` |String| true |tcp | The network protocol over which data is 
transmitted to netty. Below are the available 
options:tcphttpudp |
+| `host` | String|true | 127.0.0.1 | The host name or address on which the 
source instance listen. |
+| `port` | int|true | 10999 | The port on which the source instance listen. |
+| `numberOfThreads` |int| true |1 | The number of threads of Netty TCP server 
to accept incoming connections and handle the traffic of accepted connections. |
+
+
+### Example
+
+Before using the Netty source connector, you need to create a configuration 
file through one of the following methods.
+
+* JSON 
+
+```json
+{
+"type": "tcp",
+"host": "127.0.0.1",
+"port": "10911",
+"numberOfThreads": "1"
+}
+```
+
+* YAML
+
+```yaml
+configs:
+type: "tcp"
+host: "127.0.0.1"
+port: 10999
+numberOfThreads: 1
+```
+
+
+## Usage 
+
+The following examples show how to use the Netty source connector with TCP and 
HTTP.
+
+### TCP 
+
+1. Start Pulsar standalone.
+
+```bash
+$ docker pull apachepulsar/pulsar:{version}
+
+$ docker run -d -it -p 6650:6650 -p 8080:8080 -v $PWD/data:/pulsar/data 
--name pulsar-netty-standalone apachepulsar/pulsar:{version} bin/pulsar 
standalone
+```
+
+2. Create a configuration file _netty-source-config.yaml_.
+   
+```yaml
+configs:
+type: "tcp"
+host: "127.0.0.1"
+port: 10999
+numberOfThreads: 1
+```
+
+3. Copy the configuration file _netty-source-config.yaml_ to Pulsar server.
+
+```bash
+$ docker cp netty-source-config.yaml pulsar-netty-standalone:/pulsar/conf/
+```
+
+4. Download the Netty source connector.
+
+```bash
+$ docker exec -it pulsar-netty-standalone /bin/bash
+curl -O 
http://mirror-hk.koddos.net/apache/pulsar/pulsar-{version}/connectors/pulsar-io-netty-{version}.nar
+```
+
+5. Start the Netty source connector.
+   
+   ```bash
+   $ ./bin/pulsar-admin sources localrun \
+   --archive pulsar-io-{{pulsar:version}}.nar \
+   --tenant public \
+   --namespace default \
+   --name netty \
+   --destination-topic-name netty-topic \
+   --source-config-file netty-source-config.yaml \
+   --parallelism 1
+   ```
+
+6. Consume data.
+
+```bash
+$ docker exec -it pulsar-netty-standalone /bin/bash
+
+$ ./bin/pulsar-client consume -t Exclusive -s netty-sub netty-topic -n 0
+```
+
+7. Open another terminal window to send data to the Netty source.
+
+```bash
+$ docker exec -it pulsar-netty-standalone /bin/bash
+
+$ apt-get update
+
+$ apt-get -y install telnet
+
+$ root@1d19327b2c67:/pulsar# telnet 127.0.0.1 10999
+Trying 127.0.0.1...
+Connected to 127.0.0.1.
+Escape character is '^]'.
+hello
+world
+```
+
+8. The following information appears on the consumer terminal window.
+
+```bash
+- got message -
+hello
+
+- got message -
+world
+```
+
+### HTTP 
+
+1. Start Pulsar standalone.
+
+```bash
+$ docker pull apachepulsar/pulsar:{version}
+
+$ docker run -d -it -p 6650:6650 -p 8080:8080 -v $PWD/data:/pulsar/data 
--name pulsar-netty-standalone apachepulsar/pulsar:{version} bin/pulsar 
standalone
+```
+
+2. Create a configuration file _netty-source-config.yaml_.
+   
+```yaml
+configs:
+type: "http"
+host: "127.0.0.1"
+port: 10999
+numberOfThreads: 1
+```
+
+3. Copy the configuration file _netty-source-config.yaml_ to Pulsar server.
+   
+```bash
+$ docker cp netty-source-config.yaml pulsar-netty-standalone:/pulsar/conf/
+```
+
+4. Download the Netty source connector.
+
+```bash
+$ docker exec -it pulsar-netty-standalone /bin/bash
+curl -O 
http://mirror-hk.koddos.net/apache/pulsar/pulsar-{version}/connectors/pulsar-io-netty-{version}.nar
+```
+
+5. Start the Netty source connector.
+   
+   ```bash
+   $ ./bin/pulsar-admin sources localrun \
+   --archive

[GitHub] [pulsar] tuteng commented on issue #5253: [Doc] Add Netty source connector guide

2019-09-23 Thread GitBox

tuteng commented on issue #5253: [Doc] Add *Netty source connector guide*
URL: https://github.com/apache/pulsar/pull/5253#issuecomment-534348770
 
 
   run integration tests


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [pulsar] tuteng commented on a change in pull request #5240: [Doc] Add File source connector guide

2019-09-23 Thread GitBox

tuteng commented on a change in pull request #5240: [Doc] Add *File source 
connector guide*
URL: https://github.com/apache/pulsar/pull/5240#discussion_r327393385
 
 

 ##
 File path: site2/docs/io-file-source.md
 ##
 @@ -0,0 +1,69 @@
+---
+id: io-file
+title: File source connector
+sidebar_label: File source connector
+---
+
+The File source connector pulls messages from files in directories and 
persists the messages to Pulsar topics.
+
+## Configuration
+
+The configuration of the File source connector has the following properties.
+
+### Property
+
+| Name | Type|Required | Default | Description 
+|--|--|--|-|-|
+| `inputDirectory` | String|false|  | The input directory to pull files. |
+| `recurse` | Boolean|false |  | Whether to pull files from subdirectory or 
not.|
+| `keepFile` |Boolean|false | false | If set to true, the file is not deleted 
after it is processed, which means the file can be picked up continually. |
+| `fileFilter` | String|false| [^\\.].* | The file whose name matches the 
given regular expression is picked up. |
+| `pathFilter` | String |false |  | If `recurse` is set to true, the 
subdirectory whose path matches the given regular expression is scanned. |
+| `minimumFileAge` | Integer|false |  | The minimum age that a file can be 
processed. Any file younger than `minimumFileAge` (according to the 
last modification date) is ignored. |
+| `maximumFileAge` | Long|false | | The maximum age that a file can be 
processed. Any file older than `maximumFileAge` (according to last 
modification date) is ignored. |
+| `minimumSize` |Integer| false | | The minimum size (in bytes) that a file 
can be processed. |
+| `maximumSize` | Double|false || The maximum size (in bytes) that a file can 
be processed. |
+| `ignoreHiddenFiles` |Boolean| false | | Whether the hidden files should be 
ignored or not. |
+| `pollingInterval`|Long | false |  | Indicates how long to wait before 
performing a directory listing. |
+| `numWorkers` | Integer | false | 1 | The number of worker threads that 
process files. This allows you to process a larger number of files 
concurrently. However, setting this to a value greater than 1 makes the 
data from multiple files mixed in the target topic. |
+
 
 Review comment:
   The option `inputDirectory` is required.   
   
   The option `recurse` default value is `true`
   
   The option `pollingInterval` default value is `1L`  
   
   The option `minimumSize ` default value is `1`
   
   The options `maximumSize` default value is `Double.MAX_VALUE `  
 
   The option `minimumFileAge` default value is `0`   
   
   The option `maximumFileAge` default value is `Long.MAX_VALUE`
   
   The option `pathFilter` default value is `null` 
   
   reference 
https://github.com/apache/pulsar/blob/master/pulsar-io/file/src/main/java/org/apache/pulsar/io/file/FileListingThread.java


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [pulsar] aahmed-se commented on issue #5263: Fix Cmake to build _pulsar for osx

2019-09-23 Thread GitBox

aahmed-se commented on issue #5263: Fix Cmake to build _pulsar for osx
URL: https://github.com/apache/pulsar/pull/5263#issuecomment-534344791
 
 
   run cpp tests


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [pulsar] aahmed-se commented on issue #5263: Fix Cmake to build _pulsar for osx

2019-09-23 Thread GitBox

aahmed-se commented on issue #5263: Fix Cmake to build _pulsar for osx
URL: https://github.com/apache/pulsar/pull/5263#issuecomment-534342067
 
 
   @merlimat can we port this 2.4.2 ?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [pulsar] aahmed-se commented on issue #5263: Fix Cmake to build _pulsar for osx

2019-09-23 Thread GitBox

aahmed-se commented on issue #5263: Fix Cmake to build _pulsar for osx
URL: https://github.com/apache/pulsar/pull/5263#issuecomment-534342012
 
 
   run cpp tests


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [pulsar] aahmed-se opened a new pull request #5263: Fix Cmake to build _pulsar for osx

2019-09-23 Thread GitBox

aahmed-se opened a new pull request #5263: Fix Cmake to build _pulsar for osx
URL: https://github.com/apache/pulsar/pull/5263
 
 
   setting CMAKE_FIND_LIBRARY_SUFFIXES breaks osx builds this fixes it.
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [pulsar] JointHero opened a new issue #5262: pulsar SQL (Presto) can only query data less than 100000

2019-09-23 Thread GitBox

JointHero opened a new issue #5262: pulsar SQL (Presto) can only query data 
less than 10 
URL: https://github.com/apache/pulsar/issues/5262
 
 
   **Describe the bug**
   I create a persistent topic named "persistent://public/default/tusr1", use a 
nifi workflow to read user data from https://randomuser.me/ and push the 
userdata in to "persistent://public/default/tusr1", then I use pulsar sql to 
get the message count.
   the sql is simply: [ SELECT COUNT(ssn) FROM pulsar."public/default"."tusr1"; 
]
   I Expected the return value is the message count saved in the bookkeeper ;
   but when the messages count up to 10+, the sql will return 5+
   
   **To Reproduce**
   Steps to reproduce the behavior:
   1. push messages to a persistent topic
   2. keep querying "select count(*) from topic name " in presto
   3. wait the messages count up to 10+
   4. the "select count(*) from topic name " result will back to 5+
   
   **Expected behavior**
   I expected the query will return the right message count with the 
"pulsar-admin persistent stats-internal" command
   
   **Screenshots**
   I use "pulsar-admin persistent stats-internal 
persistent://public/default/tusr1" to get the internal stats , got below :
   {
   "entriesAddedCounter" : 149000,
   "numberOfEntries" : 11999,
   "totalSize" : 1670626,
   "currentLedgerEntries" : 2999,
   "currentLedgerSize" : 417685,
   "lastLedgerCreatedTimestamp" : "2019-09-23T00:16:38.965Z",
   "waitingCursorsCount" : 0,
   "pendingAddEntriesCount" : 0,
   "lastConfirmedEntry" : "8088:2998",
   "state" : "LedgerOpened",
   "ledgers" : [ {
   "ledgerId" : 5638,
   "entries" : 9000,
   "size" : 1252941,
   "offloaded" : false
   }, {
   "ledgerId" : 8088,
   "entries" : 0,
   "size" : 0,
   "offloaded" : false
   } ],
   "cursors" : { }
   }
   
   but "SELECT COUNT(ssn) FROM pulsar."public/default"."tusr1";" return 5+
   
   **Desktop (please complete the following information):**
- OS: [linux]  pulsar 2.4.1 presto pulsar edition with 2.4.1
   
   
   **Additional context**
   I guess the messages all stored in bookkeeper, but presto only can query 2 
segment? is that right ?
   
   anyone can help me to explain this ? and how chan I query all of the 
messages?
   
   thank you very much
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [pulsar] JointHero closed issue #5249: Where's the messages gone ? / pulsar SQL (Presto) can only query data less than 100000 ?

2019-09-23 Thread GitBox

JointHero closed issue #5249: Where's the messages gone ? / pulsar SQL (Presto) 
can only query data less than 10 ?
URL: https://github.com/apache/pulsar/issues/5249
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [pulsar] jerrypeng commented on issue #5261: Fix client backoff

2019-09-23 Thread GitBox

jerrypeng commented on issue #5261: Fix client backoff
URL: https://github.com/apache/pulsar/pull/5261#issuecomment-534337386
 
 
   rerun integration tests


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [pulsar] jerrypeng commented on issue #5243: Bug in Message Deduplication that may cause incorrect client/broker interaction

2019-09-23 Thread GitBox

jerrypeng commented on issue #5243: Bug in Message Deduplication that may cause 
incorrect client/broker interaction
URL: https://github.com/apache/pulsar/pull/5243#issuecomment-534337328
 
 
   rerun integration tests


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [pulsar] merlimat commented on issue #3613: Use individual netty-* artifacts instead of netty-all

2019-09-23 Thread GitBox

merlimat commented on issue #3613: Use individual netty-* artifacts instead of 
netty-all
URL: https://github.com/apache/pulsar/pull/3613#issuecomment-534334596
 
 
   run java8 tests


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [pulsar] aahmed-se commented on issue #5252: support cloud event

2019-09-23 Thread GitBox

aahmed-se commented on issue #5252: support cloud event
URL: https://github.com/apache/pulsar/issues/5252#issuecomment-534320094
 
 
   It's been discussed though not enough requests have been made to give 
consideration.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [pulsar] jerrypeng opened a new pull request #5261: Fix client backoff

2019-09-23 Thread GitBox

jerrypeng opened a new pull request #5261: Fix client backoff
URL: https://github.com/apache/pulsar/pull/5261
 
 
   
   ### Motivation
   
   The https://github.com/apache/pulsar/pull/3848 allowed users to be able to 
configure initial backoff delay and max backoff delay.  However, the PR doesn't 
actually properly propagate those user settings and defaults are used 
regardless of whether the user configured those settings in the pulsar client
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [pulsar] jerrypeng commented on issue #5243: Bug in Message Deduplication that may cause incorrect client/broker interaction

2019-09-23 Thread GitBox

jerrypeng commented on issue #5243: Bug in Message Deduplication that may cause 
incorrect client/broker interaction
URL: https://github.com/apache/pulsar/pull/5243#issuecomment-534292201
 
 
   rerun integration tests


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [pulsar] RAbraham closed issue #5248: Segmentation Fault: Mac OS Mojave

2019-09-23 Thread GitBox

RAbraham closed issue #5248: Segmentation Fault: Mac OS Mojave
URL: https://github.com/apache/pulsar/issues/5248
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [pulsar] RAbraham commented on issue #5248: Segmentation Fault: Mac OS Mojave

2019-09-23 Thread GitBox

RAbraham commented on issue #5248: Segmentation Fault: Mac OS Mojave
URL: https://github.com/apache/pulsar/issues/5248#issuecomment-534261384
 
 
   @pouledodue  It worked in 3.7.4 in a non pyenv environment. Thanks for that 
tip. However, I'm a bit confused as it worked for python 3.6.8 in a pyenv 
environment and  pyenv is such a blessing when it comes to managing multiple 
python environments for many projects. 


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [pulsar] mickdelaney commented on issue #271: Consider using gRPC as an externally exposed API

2019-09-23 Thread GitBox

mickdelaney commented on issue #271: Consider using gRPC as an externally 
exposed API
URL: https://github.com/apache/pulsar/issues/271#issuecomment-534252436
 
 
   Hi, 
   Sorry for the late reply. 
   
   So we use Kafka at the moment, confluent provide dotnet & python clients, 
based on librdkafka which 
   in theory gives a baseline for all clients that extend it. 
   
   The reality is that its very expensive to maintain all these language 
drivers, and so you get differences, you get things that are coming down the 
line, for example the schema/avro support in the various languages for kafka 
varies significantly, Java being very different than say C#. 
   
   So for teams using these drivers, you have to rely on different semantics, 
you have to create different approaches to dealing with things like schemas, 
and it increases costs. 
   
   Also you have to think about the teams providing the drivers, and the costs 
they have in maintaining them. Its not easy. 
   
   So if there's a possibility that GRPC will fit the semantis of pulsars 
protocol, it seems to me that its a win for everyone, the pulsar team in 
particular can focus they're attention on making the GRPC layer first class. 
   
   Thanks...
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [pulsar] jerrypeng commented on issue #5243: Bug in Message Deduplication that may cause incorrect client/broker interaction

2019-09-23 Thread GitBox

jerrypeng commented on issue #5243: Bug in Message Deduplication that may cause 
incorrect client/broker interaction
URL: https://github.com/apache/pulsar/pull/5243#issuecomment-534245207
 
 
   rerun integration tests
   
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [pulsar] merlimat commented on issue #5232: Upgrade dependencies for security fixes

2019-09-23 Thread GitBox

merlimat commented on issue #5232: Upgrade dependencies for security fixes
URL: https://github.com/apache/pulsar/pull/5232#issuecomment-534238850
 
 
   run java8 tests
   
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [pulsar] RAbraham commented on issue #5248: Segmentation Fault: Mac OS Mojave

2019-09-23 Thread GitBox

RAbraham commented on issue #5248: Segmentation Fault: Mac OS Mojave
URL: https://github.com/apache/pulsar/issues/5248#issuecomment-534224293
 
 
   @pouledodue I'l try but the python 3.6.8 version I tried it with and worked 
was using `pyenv`.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [pulsar] RAbraham closed issue #5260: How do I test Pulsar Functions Locally? ModuleNotFoundError: No module named '_pulsar'

2019-09-23 Thread GitBox

RAbraham closed issue #5260: How do I test Pulsar Functions Locally? 
ModuleNotFoundError: No module named '_pulsar'
URL: https://github.com/apache/pulsar/issues/5260
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [pulsar] RAbraham commented on issue #5260: How do I test Pulsar Functions Locally? ModuleNotFoundError: No module named '_pulsar'

2019-09-23 Thread GitBox

RAbraham commented on issue #5260: How do I test Pulsar Functions Locally? 
ModuleNotFoundError: No module named '_pulsar'
URL: https://github.com/apache/pulsar/issues/5260#issuecomment-534223831
 
 
   Yes, `pip install pulsar-client ` moved it forward but it failed looking for 
the latest python code I think. I'm running Pulsar 2.4.1 but on python 3.6.8, 
the python library is at 2.0 and my personal problem is that I can't get pulsar 
to work with 3.7. Thanks for that tip though.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [pulsar] merlimat commented on issue #5232: Upgrade dependencies for security fixes

2019-09-23 Thread GitBox

merlimat commented on issue #5232: Upgrade dependencies for security fixes
URL: https://github.com/apache/pulsar/pull/5232#issuecomment-534208898
 
 
   run java8 tests
   
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [pulsar] pouledodue commented on issue #5260: How do I test Pulsar Functions Locally? ModuleNotFoundError: No module named '_pulsar'

2019-09-23 Thread GitBox

pouledodue commented on issue #5260: How do I test Pulsar Functions Locally? 
ModuleNotFoundError: No module named '_pulsar'
URL: https://github.com/apache/pulsar/issues/5260#issuecomment-534199771
 
 
   you can run the docker standalone but it's python 3.5 #4944 


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [pulsar] pouledodue edited a comment on issue #5260: How do I test Pulsar Functions Locally? ModuleNotFoundError: No module named '_pulsar'

2019-09-23 Thread GitBox

pouledodue edited a comment on issue #5260: How do I test Pulsar Functions 
Locally? ModuleNotFoundError: No module named '_pulsar'
URL: https://github.com/apache/pulsar/issues/5260#issuecomment-534191127
 
 
   without using `pyenv` venv


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [pulsar] pouledodue edited a comment on issue #5260: How do I test Pulsar Functions Locally? ModuleNotFoundError: No module named '_pulsar'

2019-09-23 Thread GitBox

pouledodue edited a comment on issue #5260: How do I test Pulsar Functions 
Locally? ModuleNotFoundError: No module named '_pulsar'
URL: https://github.com/apache/pulsar/issues/5260#issuecomment-534191127
 
 
   without using `pyenv`


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [pulsar] pouledodue commented on issue #5260: How do I test Pulsar Functions Locally? ModuleNotFoundError: No module named '_pulsar'

2019-09-23 Thread GitBox

pouledodue commented on issue #5260: How do I test Pulsar Functions Locally? 
ModuleNotFoundError: No module named '_pulsar'
URL: https://github.com/apache/pulsar/issues/5260#issuecomment-534191127
 
 
   with or without using `pyenv`


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [pulsar] pouledodue edited a comment on issue #5248: Segmentation Fault: Mac OS Mojave

2019-09-23 Thread GitBox

pouledodue edited a comment on issue #5248: Segmentation Fault: Mac OS Mojave
URL: https://github.com/apache/pulsar/issues/5248#issuecomment-534188258
 
 
   did you try "not in a pyenv venv"


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [pulsar] pouledodue edited a comment on issue #5248: Segmentation Fault: Mac OS Mojave

2019-09-23 Thread GitBox

pouledodue edited a comment on issue #5248: Segmentation Fault: Mac OS Mojave
URL: https://github.com/apache/pulsar/issues/5248#issuecomment-534188258
 
 
   did you try outside pyenv venv


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [pulsar] pouledodue commented on issue #5248: Segmentation Fault: Mac OS Mojave

2019-09-23 Thread GitBox

pouledodue commented on issue #5248: Segmentation Fault: Mac OS Mojave
URL: https://github.com/apache/pulsar/issues/5248#issuecomment-534188258
 
 
   did you try outside pyenv


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [pulsar] pouledodue commented on issue #5248: Segmentation Fault: Mac OS Mojave

2019-09-23 Thread GitBox

pouledodue commented on issue #5248: Segmentation Fault: Mac OS Mojave
URL: https://github.com/apache/pulsar/issues/5248#issuecomment-534181390
 
 
   I used successfully 3.7.0 and 3.7.3 in the past with Mojave


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [pulsar] RAbraham opened a new issue #5260: How do I test Pulsar Functions Locally? ModuleNotFoundError: No module named '_pulsar'

2019-09-23 Thread GitBox

RAbraham opened a new issue #5260: How do I test Pulsar Functions Locally? 
ModuleNotFoundError: No module named '_pulsar'
URL: https://github.com/apache/pulsar/issues/5260
 
 
   **Describe the bug**
   How do I run functions on my local Mac OS(Mojave) machine? It fails with 
`ModuleNotFoundError: No module named '_pulsar'`
   
   **To Reproduce**
   Steps to reproduce the behavior:
   - In Terminal 1, I run `bin/pulsar functions-worker`
   - In Terminal 2, 
   ```bash
   ➜  apache-pulsar-2.4.1 bin/pulsar-admin functions localrun \
 --py 
/Users/rabraham/Documents/dev/fifteenrock/fifteenrock-connector/pulsar_lib/functions.py
 \
 --className functions.ArrowFunction \
 --inputs persistent://public/default/file-topic \
 --output persistent://public/default/result-topic
   12:19:14.644 [main] INFO  
org.apache.pulsar.functions.runtime.ProcessRuntimeFactory - Java instance jar 
location is not defined, using the location defined in system environment : 
/Users/rabraham/Documents/dev/trial/pulsar-trial/apache-pulsar-2.4.1/instances/java-instance.jar
   12:19:14.647 [main] INFO  
org.apache.pulsar.functions.runtime.ProcessRuntimeFactory - Python instance 
file location is not defined using the location defined in system environment : 
/Users/rabraham/Documents/dev/trial/pulsar-trial/apache-pulsar-2.4.1/instances/python-instance/python_instance_main.py
   12:19:14.654 [main] INFO  
org.apache.pulsar.functions.runtime.ProcessRuntimeFactory - No extra 
dependencies location is defined in either function worker config or system 
environment
   12:19:14.859 [main] INFO  org.apache.pulsar.functions.runtime.RuntimeSpawner 
- public/default/ArrowFunction-0 RuntimeSpawner starting function
   12:19:14.861 [main] INFO  org.apache.pulsar.functions.runtime.ProcessRuntime 
- Creating function log directory 
/Users/rabraham/Documents/dev/trial/pulsar-trial/apache-pulsar-2.4.1/logs/functions/public/default/ArrowFunction
   12:19:14.868 [main] INFO  org.apache.pulsar.functions.runtime.ProcessRuntime 
- Created or found function log directory 
/Users/rabraham/Documents/dev/trial/pulsar-trial/apache-pulsar-2.4.1/logs/functions/public/default/ArrowFunction
   12:19:14.869 [main] INFO  org.apache.pulsar.functions.runtime.ProcessRuntime 
- ProcessBuilder starting the process with args python 
/Users/rabraham/Documents/dev/trial/pulsar-trial/apache-pulsar-2.4.1/instances/python-instance/python_instance_main.py
 --py 
/Users/rabraham/Documents/dev/fifteenrock/fifteenrock-connector/pulsar_lib/functions.py
 --logging_directory 
/Users/rabraham/Documents/dev/trial/pulsar-trial/apache-pulsar-2.4.1/logs/functions
 --logging_file ArrowFunction --logging_config_file 
/Users/rabraham/Documents/dev/trial/pulsar-trial/apache-pulsar-2.4.1/conf/functions-logging/logging_config.ini
 --instance_id 0 --function_id 5373e0b5-9b1c-4656-8ab5-dba7de1c4bfb 
--function_version 74f6460f-8573-4ff4-a875-ae0acba78aef --function_details 
'{"tenant":"public","namespace":"default","name":"ArrowFunction","className":"functions.ArrowFunction","runtime":"PYTHON","autoAck":true,"parallelism":1,"source":{"inputSpecs":{"persistent://public/default/file-topic":{}},"cleanupSubscription":true},"sink":{"topic":"persistent://public/default/result-topic"},"resources":{"cpu":1.0,"ram":"1073741824","disk":"10737418240"},"componentType":"FUNCTION"}'
 --pulsar_serviceurl pulsar://localhost:6650 --use_tls false 
--tls_allow_insecure false --hostname_verification_enabled false 
--max_buffered_tuples 1024 --port 52463 --metrics_port 52464 
--expected_healthcheck_interval 30 --secrets_provider 
secretsprovider.ClearTextSecretsProvider --cluster_name local
   12:19:14.879 [main] INFO  org.apache.pulsar.functions.runtime.ProcessRuntime 
- Started process successfully
   Traceback (most recent call last):
 File 
"/Users/rabraham/Documents/dev/trial/pulsar-trial/apache-pulsar-2.4.1/instances/python-instance/python_instance_main.py",
 line 36, in 
   import pulsar
 File 
"/Users/rabraham/Documents/dev/trial/pulsar-trial/apache-pulsar-2.4.1/instances/python-instance/pulsar/__init__.py",
 line 102, in 
   import _pulsar
   ModuleNotFoundError: No module named '_pulsar'
   ```
   
   **Desktop (please complete the following information):**
- OS: Mac OS (Mojave)
   
   **Additional context**
   I don't have `pulsar-client` on my machine. I thought that local cluster 
would have it's own version of pulsar python code?
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [pulsar] RAbraham commented on issue #5248: Segmentation Fault: Mac OS Mojave

2019-09-23 Thread GitBox

RAbraham commented on issue #5248: Segmentation Fault: Mac OS Mojave
URL: https://github.com/apache/pulsar/issues/5248#issuecomment-534166313
 
 
   Fails on both 3.7.0 and 3.7.3. To check if my 3.7 binaries are ok, I 
installed `numpy`(because it may have c extensions like Pulsar)  and `numpy` 
worked.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [pulsar] pouledodue edited a comment on issue #5248: Segmentation Fault: Mac OS Mojave

2019-09-23 Thread GitBox

pouledodue edited a comment on issue #5248: Segmentation Fault: Mac OS Mojave
URL: https://github.com/apache/pulsar/issues/5248#issuecomment-534161355
 
 
   my issue #5127 is with Catalina, not Mojave. They need to build 
pulsar-client wheels for Catalina to fix it. In the meantime yes my fix is the 
above


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [pulsar] pouledodue commented on issue #5248: Segmentation Fault: Mac OS Mojave

2019-09-23 Thread GitBox

pouledodue commented on issue #5248: Segmentation Fault: Mac OS Mojave
URL: https://github.com/apache/pulsar/issues/5248#issuecomment-534161355
 
 
   my issue #5127 is with Catalina, not Mojave.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [pulsar] pouledodue commented on issue #5248: Segmentation Fault: Mac OS Mojave

2019-09-23 Thread GitBox

pouledodue commented on issue #5248: Segmentation Fault: Mac OS Mojave
URL: https://github.com/apache/pulsar/issues/5248#issuecomment-534157758
 
 
   does it work with 3.7.0 or 3.7.3


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [pulsar] ivankelly opened a new issue #5259: Calling statsInternal can cause topic creation to fail

2019-09-23 Thread GitBox

ivankelly opened a new issue #5259: Calling statsInternal can cause topic 
creation to fail
URL: https://github.com/apache/pulsar/issues/5259
 
 
   **Describe the bug**
   When creating a topic, if there is some other client querying stats (or 
probably many other things), then the creation can fail.
   
   This is because of how BrokerService ensures that only one client can create 
the topic at a time. A future is inserted into a map for the topic. However, 
the computation for this future differs depending on where the call to get the 
topic comes from. If it comes from the admin interface, then it doesn't create 
the topic if missing. If it comes from producer creation, then it does create 
the topic if missing.
   
   The problem occurs when a call comes in from the admin interface to get the 
topic, the future is inserted, then a call to get the topic comes in from 
producer creation. Producer creation will receive the previously saved future 
back which will not create the topic if missing, consequently failing the 
producer creation.
   
   **To Reproduce**
   Run this test case:
   
https://github.com/ivankelly/pulsar/commit/bda83c5321e47114f0e4b3742e2f6e39667a99cf
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [pulsar] codelipenghui removed a comment on issue #5254: Expose bookkeeperDiskWeightBasedPlacementEnabled in broker.conf

2019-09-23 Thread GitBox

codelipenghui removed a comment on issue #5254: Expose 
bookkeeperDiskWeightBasedPlacementEnabled in broker.conf
URL: https://github.com/apache/pulsar/pull/5254#issuecomment-534100429
 
 
   when i doing local testing and set the 
bookkeeperDiskWeightBasedPlacementEnabled=true found that standalone can't 
startup, log stop at:
   ```


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [pulsar] codelipenghui edited a comment on issue #5254: Expose bookkeeperDiskWeightBasedPlacementEnabled in broker.conf

2019-09-23 Thread GitBox

codelipenghui edited a comment on issue #5254: Expose 
bookkeeperDiskWeightBasedPlacementEnabled in broker.conf
URL: https://github.com/apache/pulsar/pull/5254#issuecomment-534100429
 
 
   when i doing local testing and set the 
bookkeeperDiskWeightBasedPlacementEnabled=true found that standalone can't 
startup, log stop at:
   ```


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [pulsar] codelipenghui commented on issue #5254: Expose bookkeeperDiskWeightBasedPlacementEnabled in broker.conf

2019-09-23 Thread GitBox

codelipenghui commented on issue #5254: Expose 
bookkeeperDiskWeightBasedPlacementEnabled in broker.conf
URL: https://github.com/apache/pulsar/pull/5254#issuecomment-534100429
 
 
   when i doing local testing and set the 
bookkeeperDiskWeightBasedPlacementEnabled=true found that standalone can't 
startup, log stop at:


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [pulsar] yjshen commented on issue #5258: [Doc] Add ElasticSearch sink connector guide

2019-09-23 Thread GitBox

yjshen commented on issue #5258: [Doc] Add *ElasticSearch sink connector guide*
URL: https://github.com/apache/pulsar/pull/5258#issuecomment-534093129
 
 
   run java8 tests
   run integration tests


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [pulsar] yjshen removed a comment on issue #5258: [Doc] Add ElasticSearch sink connector guide

2019-09-23 Thread GitBox

yjshen removed a comment on issue #5258: [Doc] Add *ElasticSearch sink 
connector guide*
URL: https://github.com/apache/pulsar/pull/5258#issuecomment-534092419
 
 
   run integration test
   run java8 test


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [pulsar] yjshen commented on issue #5258: [Doc] Add ElasticSearch sink connector guide

2019-09-23 Thread GitBox

yjshen commented on issue #5258: [Doc] Add *ElasticSearch sink connector guide*
URL: https://github.com/apache/pulsar/pull/5258#issuecomment-534092419
 
 
   run integration test
   run java8 test


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [pulsar] RAbraham commented on issue #5248: Segmentation Fault: Mac OS Mojave

2019-09-23 Thread GitBox

RAbraham commented on issue #5248: Segmentation Fault: Mac OS Mojave
URL: https://github.com/apache/pulsar/issues/5248#issuecomment-534091723
 
 
   @pouledodue. I see you had the same issue 
https://github.com/apache/pulsar/issues/5217. Was your resolution, the one 
above?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [pulsar] RAbraham commented on issue #5248: Segmentation Fault: Mac OS Mojave

2019-09-23 Thread GitBox

RAbraham commented on issue #5248: Segmentation Fault: Mac OS Mojave
URL: https://github.com/apache/pulsar/issues/5248#issuecomment-534090206
 
 
   @aahmed-se @pouledodue  could you try python3.7? I just tried on python2.7 
and python3.6 and it worked for me but not python3.7.4. @pouledodue I'll also 
try your solution but still wanted to raise the issue, just in case something 
went wrong with the python3.7 binaries. 


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [pulsar] codelipenghui removed a comment on issue #5254: Expose bookkeeperDiskWeightBasedPlacementEnabled in broker.conf

2019-09-23 Thread GitBox

codelipenghui removed a comment on issue #5254: Expose 
bookkeeperDiskWeightBasedPlacementEnabled in broker.conf
URL: https://github.com/apache/pulsar/pull/5254#issuecomment-534084571
 
 
   run integration tests


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [pulsar] codelipenghui removed a comment on issue #5254: Expose bookkeeperDiskWeightBasedPlacementEnabled in broker.conf

2019-09-23 Thread GitBox

codelipenghui removed a comment on issue #5254: Expose 
bookkeeperDiskWeightBasedPlacementEnabled in broker.conf
URL: https://github.com/apache/pulsar/pull/5254#issuecomment-534023828
 
 
   run integration tests


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [pulsar] codelipenghui commented on issue #5254: Expose bookkeeperDiskWeightBasedPlacementEnabled in broker.conf

2019-09-23 Thread GitBox

codelipenghui commented on issue #5254: Expose 
bookkeeperDiskWeightBasedPlacementEnabled in broker.conf
URL: https://github.com/apache/pulsar/pull/5254#issuecomment-534084571
 
 
   run integration tests


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [pulsar] tuteng commented on issue #5251: [Doc] Add Twitter Firehose source connector guide

2019-09-23 Thread GitBox

tuteng commented on issue #5251: [Doc] Add *Twitter Firehose source connector 
guide*
URL: https://github.com/apache/pulsar/pull/5251#issuecomment-534047108
 
 
   run integration tests
   run java8 tests


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [pulsar] tuteng commented on issue #5239: [Doc] Add HDFS3 sink connector guide

2019-09-23 Thread GitBox

tuteng commented on issue #5239: [Doc] Add *HDFS3 sink connector guide*
URL: https://github.com/apache/pulsar/pull/5239#issuecomment-534045467
 
 
   run integration tests


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [pulsar] tuteng commented on a change in pull request #5238: [Doc] Add Flume sink connector guide

2019-09-23 Thread GitBox

tuteng commented on a change in pull request #5238: [Doc] Add *Flume sink 
connector guide*
URL: https://github.com/apache/pulsar/pull/5238#discussion_r327046060
 
 

 ##
 File path: site2/docs/io-flume-sink.md
 ##
 @@ -0,0 +1,49 @@
+---
+id: io-flume-sink
+title: Flume sink connector
+sidebar_label: Flume sink connector
+---
+
+The Flume sink connector pulls messages from Pulsar topics to logs.
+
+## Configuration
+
+The configuration of the Flume sink connector has the following properties.
+
+### Property
+
+| Name | Type|Required | Default | Description 
+|--|--|--|-|-|
+`name`|String|true|"" (empty string)|The name of the agent.
+`confFile`|String|true|"" (empty string)|The configuration file.
+`noReloadConf`|Boolean|false|false|Whether to reload configuration file if 
changed.
+`zkConnString`|String|true|"" (empty string)|The ZooKeeper connection.
+`zkBasePath`|String|true|"" (empty string)|The base path in ZooKeeper for 
agent configuration.
+
+### Example
+
+Before using the Flume sink connector, you need to create a configuration file 
through one of the following methods.
+
 
 Review comment:
   A link to the 
[sink.conf](https://github.com/apache/pulsar/blob/master/pulsar-io/flume/src/main/resources/flume/sink.conf)
 file can be added here to make it easier for users to understand if and 
configure `sink.conf`.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [pulsar] tuteng commented on a change in pull request #5238: [Doc] Add Flume sink connector guide

2019-09-23 Thread GitBox

tuteng commented on a change in pull request #5238: [Doc] Add *Flume sink 
connector guide*
URL: https://github.com/apache/pulsar/pull/5238#discussion_r327045814
 
 

 ##
 File path: site2/docs/io-flume-source.md
 ##
 @@ -8,14 +8,42 @@ The Flume source connector pulls messages from logs to 
Pulsar topics.
 
 ## Configuration
 
-The configuration of Flume source connector has the following parameters.
+The configuration of the Flume source connector has the following properties.
 
-### Parameter
+### Property
 
 | Name | Type|Required | Default | Description 
 |--|--|--|-|-|
-`name`|String|true|"" (empty string)|Name of the agent
-`confFile`|String|true|"" (empty string)|Configuration file
-`noReloadConf`|Boolean|false|false|Whether to reload configuration file if 
changed
-`zkConnString`|String|true|"" (empty string)|ZooKeeper connection 
-`zkBasePath`|String|true|"" (empty string)|Base path in ZooKeeper for agent 
configuration
+`name`|String|true|"" (empty string)|The name of the agent.
+`confFile`|String|true|"" (empty string)|The configuration file.
+`noReloadConf`|Boolean|false|false|Whether to reload configuration file if 
changed.
+`zkConnString`|String|true|"" (empty string)|The ZooKeeper connection.
+`zkBasePath`|String|true|"" (empty string)|The base path in ZooKeeper for 
agent configuration.
+
+### Example
+
+Before using the Flume source connector, you need to create a configuration 
file through one of the following methods.
+
 
 Review comment:
   A link to the 
[source.conf](https://github.com/apache/pulsar/blob/master/pulsar-io/flume/src/main/resources/flume/source.conf)
 file can be added here to make it easier for users to understand if and 
configure `source.conf`.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [pulsar] Jennifer88huang commented on issue #5221: [doc] Improve Pulsar Administration Pulsar dashboard

2019-09-23 Thread GitBox

Jennifer88huang commented on issue #5221: [doc] Improve Pulsar Administration 
Pulsar dashboard
URL: https://github.com/apache/pulsar/pull/5221#issuecomment-534037194
 
 
   run cpp tests
   run integration tests
   run java8 tests


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [pulsar] Anonymitaet commented on issue #5250: [Doc] Add configs for broker

2019-09-23 Thread GitBox

Anonymitaet commented on issue #5250: [Doc] Add configs for broker
URL: https://github.com/apache/pulsar/pull/5250#issuecomment-534034196
 
 
   run integration tests


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [pulsar] Anonymitaet edited a comment on issue #5015: [Doc] Update Pulsar Connector Guide

2019-09-23 Thread GitBox

Anonymitaet edited a comment on issue #5015: [Doc] Update Pulsar Connector Guide
URL: https://github.com/apache/pulsar/issues/5015#issuecomment-524149120
 
 
   I've created the following PRs to resolve this issue:
   
   ## Overview
   * https://github.com/apache/pulsar/pull/5183
   * https://github.com/apache/pulsar/pull/5224
   
   ## Use 
   * https://github.com/apache/pulsar/pull/5071
   
   ## Built-in connector 
   * https://github.com/apache/pulsar/pull/5168
   * https://github.com/apache/pulsar/pull/5173
   * https://github.com/apache/pulsar/pull/5197
   * https://github.com/apache/pulsar/pull/5199
   * https://github.com/apache/pulsar/pull/5203
   * https://github.com/apache/pulsar/pull/5210
   * https://github.com/apache/pulsar/pull/5211
   * https://github.com/apache/pulsar/pull/5214
   * https://github.com/apache/pulsar/pull/5226
   * Flume sink: https://github.com/apache/pulsar/pull/5238
   * HDFS3 sink: https://github.com/apache/pulsar/pull/5239
   * File source: https://github.com/apache/pulsar/pull/5240
   * Twitter Firehose source: https://github.com/apache/pulsar/pull/5251
   * Netty source: https://github.com/apache/pulsar/pull/5253
   * Rabbit source: https://github.com/apache/pulsar/pull/5255
   * Aerospike sink: https://github.com/apache/pulsar/pull/5256
   * Cassandra sink: https://github.com/apache/pulsar/pull/5257
   * ElasticSearch sink: https://github.com/apache/pulsar/pull/5258
   
   ## CDC connector
   * https://github.com/apache/pulsar/pull/5198
   
   ## Debug 
   * https://github.com/apache/pulsar/pull/5041
   
   ## Connector Admin CLI
   * https://github.com/apache/pulsar/pull/5028
   * https://github.com/apache/pulsar/pull/5184


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [pulsar] Anonymitaet opened a new pull request #5258: [Doc] Add ElasticSearch sink connector guide

2019-09-23 Thread GitBox

Anonymitaet opened a new pull request #5258: [Doc] Add *ElasticSearch sink 
connector guide*
URL: https://github.com/apache/pulsar/pull/5258
 
 
   Fix https://github.com/apache/pulsar/issues/5015


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [pulsar] codelipenghui commented on issue #5254: Expose bookkeeperDiskWeightBasedPlacementEnabled in broker.conf

2019-09-23 Thread GitBox

codelipenghui commented on issue #5254: Expose 
bookkeeperDiskWeightBasedPlacementEnabled in broker.conf
URL: https://github.com/apache/pulsar/pull/5254#issuecomment-534023828
 
 
   run integration tests


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [pulsar] codelipenghui removed a comment on issue #5254: Expose bookkeeperDiskWeightBasedPlacementEnabled in broker.conf

2019-09-23 Thread GitBox

codelipenghui removed a comment on issue #5254: Expose 
bookkeeperDiskWeightBasedPlacementEnabled in broker.conf
URL: https://github.com/apache/pulsar/pull/5254#issuecomment-534023549
 
 
   run  Integration Tests


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [pulsar] codelipenghui commented on issue #5254: Expose bookkeeperDiskWeightBasedPlacementEnabled in broker.conf

2019-09-23 Thread GitBox

codelipenghui commented on issue #5254: Expose 
bookkeeperDiskWeightBasedPlacementEnabled in broker.conf
URL: https://github.com/apache/pulsar/pull/5254#issuecomment-534023549
 
 
   run  Integration Tests


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [pulsar] Anonymitaet commented on issue #5214: [Doc] Add Kinesis sink connector guide

2019-09-23 Thread GitBox

Anonymitaet commented on issue #5214: [Doc] Add *Kinesis sink connector guide* 
URL: https://github.com/apache/pulsar/pull/5214#issuecomment-534022863
 
 
   run java8 tests


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [pulsar] Anonymitaet commented on issue #5257: [Doc] Add Cassandra sink connector guide

2019-09-23 Thread GitBox

Anonymitaet commented on issue #5257: [Doc] Add *Cassandra sink connector guide*
URL: https://github.com/apache/pulsar/pull/5257#issuecomment-534021818
 
 
   @tuteng could you please help review? Thank you


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [pulsar] Anonymitaet edited a comment on issue #5015: [Doc] Update Pulsar Connector Guide

2019-09-23 Thread GitBox

Anonymitaet edited a comment on issue #5015: [Doc] Update Pulsar Connector Guide
URL: https://github.com/apache/pulsar/issues/5015#issuecomment-524149120
 
 
   I've created the following PRs to resolve this issue:
   
   ## Overview
   * https://github.com/apache/pulsar/pull/5183
   * https://github.com/apache/pulsar/pull/5224
   
   ## Use 
   * https://github.com/apache/pulsar/pull/5071
   
   ## Built-in connector 
   * https://github.com/apache/pulsar/pull/5168
   * https://github.com/apache/pulsar/pull/5173
   * https://github.com/apache/pulsar/pull/5197
   * https://github.com/apache/pulsar/pull/5199
   * https://github.com/apache/pulsar/pull/5203
   * https://github.com/apache/pulsar/pull/5210
   * https://github.com/apache/pulsar/pull/5211
   * https://github.com/apache/pulsar/pull/5214
   * https://github.com/apache/pulsar/pull/5226
   * Flume sink: https://github.com/apache/pulsar/pull/5238
   * HDFS3 sink: https://github.com/apache/pulsar/pull/5239
   * File source: https://github.com/apache/pulsar/pull/5240
   * Twitter Firehose source: https://github.com/apache/pulsar/pull/5251
   * Netty source: https://github.com/apache/pulsar/pull/5253
   * Rabbit source: https://github.com/apache/pulsar/pull/5255
   * Aerospike sink: https://github.com/apache/pulsar/pull/5256
   * Cassandra sink: https://github.com/apache/pulsar/pull/5257
   
   ## CDC connector
   * https://github.com/apache/pulsar/pull/5198
   
   ## Debug 
   * https://github.com/apache/pulsar/pull/5041
   
   ## Connector Admin CLI
   * https://github.com/apache/pulsar/pull/5028
   * https://github.com/apache/pulsar/pull/5184


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [pulsar] Anonymitaet opened a new pull request #5257: [Doc] Add Cassandra sink connector guide

2019-09-23 Thread GitBox

Anonymitaet opened a new pull request #5257: [Doc] Add *Cassandra sink 
connector guide*
URL: https://github.com/apache/pulsar/pull/5257
 
 
   Fix https://github.com/apache/pulsar/issues/5015


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [pulsar] Anonymitaet commented on a change in pull request #5256: [Doc] Add Aerospike sink connector guide

2019-09-23 Thread GitBox

Anonymitaet commented on a change in pull request #5256: [Doc] Add *Aerospike 
sink connector guide*
URL: https://github.com/apache/pulsar/pull/5256#discussion_r326997784
 
 

 ##
 File path: site2/docs/io-aerospike-sink.md
 ##
 @@ -0,0 +1,25 @@
+---
+id: io-aerospike-sink
+title: Aerospike sink connector
+sidebar_label: Aerospike sink connector
+---
+
+The Aerospike sink connector writes messages from Pulsar topics to Aerospike 
clusters.
+
+## Configuration
+
+The configuration of the Aerospike sink connector has the following properties.
+
+### Property
+
+| Name | Type|Required | Default | Description 
+|--|--|--|-|-|
+| `seedHosts` |String| true | 3000 | The comma-separated list of one or more 
Aerospike cluster hosts.Each host can be specified as a valid IP 
address or hostname followed by an optional port number. | 
+| `keyspace` | String| true | |The Aerospike namespace. |
+| `columnName` | String | true| |The Aerospike column name. |
+|`userName`|String|false||The Aerospike username.|
+|`password`|String|false||The Aerospike password.|
+| `keySet` | String|false | | The Aerospike set name. |
+| `maxConcurrentRequests` |int| false | 100 | The maximum number of concurrent 
Aerospike transactions that a sink can open. |
+| `timeoutMs` | int|false | 100 | This property controls `socketTimeout` and 
`totalTimeout` for Aerospike transactions.  |
+| `retries` | int|false | 1 |The maximum number of retries before aborting a 
write transaction to Aerospike. |
 
 Review comment:
   @tuteng what are the default values for line 18 to 22? 
   I don't find them in 
[AerospikeSinkConfig.java](https://github.com/apache/pulsar/blob/master/pulsar-io/aerospike/src/main/java/org/apache/pulsar/io/aerospike/AerospikeSinkConfig.java)


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [pulsar] Anonymitaet opened a new pull request #5256: [Doc] Add Aerospike sink connector guide

2019-09-23 Thread GitBox

Anonymitaet opened a new pull request #5256: [Doc] Add *Aerospike sink 
connector guide*
URL: https://github.com/apache/pulsar/pull/5256
 
 
   Fix https://github.com/apache/pulsar/issues/5015


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [pulsar] Anonymitaet edited a comment on issue #5015: [Doc] Update Pulsar Connector Guide

2019-09-23 Thread GitBox

Anonymitaet edited a comment on issue #5015: [Doc] Update Pulsar Connector Guide
URL: https://github.com/apache/pulsar/issues/5015#issuecomment-524149120
 
 
   I've created the following PRs to resolve this issue:
   
   ## Overview
   * https://github.com/apache/pulsar/pull/5183
   * https://github.com/apache/pulsar/pull/5224
   
   ## Use 
   * https://github.com/apache/pulsar/pull/5071
   
   ## Built-in connector 
   * https://github.com/apache/pulsar/pull/5168
   * https://github.com/apache/pulsar/pull/5173
   * https://github.com/apache/pulsar/pull/5197
   * https://github.com/apache/pulsar/pull/5199
   * https://github.com/apache/pulsar/pull/5203
   * https://github.com/apache/pulsar/pull/5210
   * https://github.com/apache/pulsar/pull/5211
   * https://github.com/apache/pulsar/pull/5214
   * https://github.com/apache/pulsar/pull/5226
   * Flume sink: https://github.com/apache/pulsar/pull/5238
   * HDFS3 sink: https://github.com/apache/pulsar/pull/5239
   * File source: https://github.com/apache/pulsar/pull/5240
   * Twitter Firehose source: https://github.com/apache/pulsar/pull/5251
   * Netty source: https://github.com/apache/pulsar/pull/5253
   * Rabbit source: https://github.com/apache/pulsar/pull/5255
   * Aerospike sink: https://github.com/apache/pulsar/pull/5256
   
   ## CDC connector
   * https://github.com/apache/pulsar/pull/5198
   
   ## Debug 
   * https://github.com/apache/pulsar/pull/5041
   
   ## Connector Admin CLI
   * https://github.com/apache/pulsar/pull/5028
   * https://github.com/apache/pulsar/pull/5184


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [pulsar] Anonymitaet edited a comment on issue #5015: [Doc] Update Pulsar Connector Guide

2019-09-23 Thread GitBox

Anonymitaet edited a comment on issue #5015: [Doc] Update Pulsar Connector Guide
URL: https://github.com/apache/pulsar/issues/5015#issuecomment-524149120
 
 
   I've created the following PRs to resolve this issue:
   
   ## Overview
   * https://github.com/apache/pulsar/pull/5183
   * https://github.com/apache/pulsar/pull/5224
   
   ## Use 
   * https://github.com/apache/pulsar/pull/5071
   
   ## Built-in connector 
   * https://github.com/apache/pulsar/pull/5168
   * https://github.com/apache/pulsar/pull/5173
   * https://github.com/apache/pulsar/pull/5197
   * https://github.com/apache/pulsar/pull/5199
   * https://github.com/apache/pulsar/pull/5203
   * https://github.com/apache/pulsar/pull/5210
   * https://github.com/apache/pulsar/pull/5211
   * https://github.com/apache/pulsar/pull/5214
   * https://github.com/apache/pulsar/pull/5226
   * Flume sink: https://github.com/apache/pulsar/pull/5238
   * HDFS3 sink: https://github.com/apache/pulsar/pull/5239
   * File source: https://github.com/apache/pulsar/pull/5240
   * Twitter Firehose source: https://github.com/apache/pulsar/pull/5251
   * Netty source: https://github.com/apache/pulsar/pull/5253
   * Rabbit source: https://github.com/apache/pulsar/pull/5255
   
   ## CDC connector
   * https://github.com/apache/pulsar/pull/5198
   
   ## Debug 
   * https://github.com/apache/pulsar/pull/5041
   
   ## Connector Admin CLI
   * https://github.com/apache/pulsar/pull/5028
   * https://github.com/apache/pulsar/pull/5184


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [pulsar] Anonymitaet commented on issue #5255: [Doc] Add Rabbit source connector guide

2019-09-23 Thread GitBox

Anonymitaet commented on issue #5255: [Doc] Add *Rabbit source connector guide*
URL: https://github.com/apache/pulsar/pull/5255#issuecomment-533998209
 
 
   @tuteng could you please help review? Thank you 


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [pulsar] Anonymitaet opened a new pull request #5255: [Doc] Add Rabbit source connector guide

2019-09-23 Thread GitBox

Anonymitaet opened a new pull request #5255: [Doc] Add *Rabbit source connector 
guide*
URL: https://github.com/apache/pulsar/pull/5255
 
 
   Fix https://github.com/apache/pulsar/issues/5015


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[pulsar.wiki] branch master updated: Updated PIP 38: Batch Receiving Messages (markdown)

2019-09-23 Thread penghui

This is an automated email from the ASF dual-hosted git repository.

penghui pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/pulsar.wiki.git


The following commit(s) were added to refs/heads/master by this push:
 new 9dbdc48  Updated PIP 38: Batch Receiving Messages (markdown)
9dbdc48 is described below

commit 9dbdc48f3abea64378ebb518b42288df8240389d
Author: lipenghui 
AuthorDate: Mon Sep 23 15:02:51 2019 +0800

Updated PIP 38: Batch Receiving Messages (markdown)
---
 PIP-38:-Batch-Receiving-Messages.md | 1 +
 1 file changed, 1 insertion(+)

diff --git a/PIP-38:-Batch-Receiving-Messages.md 
b/PIP-38:-Batch-Receiving-Messages.md
index 92a5001..263e073 100644
--- a/PIP-38:-Batch-Receiving-Messages.md
+++ b/PIP-38:-Batch-Receiving-Messages.md
@@ -2,6 +2,7 @@
 - Author: Penghui Li
 - Discussion Thread:
 - Issue:
+- Pull Request: https://github.com/apache/pulsar/pull/4621
 
 ## Motivation

[GitHub] [pulsar] codelipenghui opened a new pull request #5254: Expose bookkeeperDiskWeightBasedPlacementEnabled in broker.conf

2019-09-23 Thread GitBox

codelipenghui opened a new pull request #5254: Expose 
bookkeeperDiskWeightBasedPlacementEnabled in broker.conf
URL: https://github.com/apache/pulsar/pull/5254
 
 
   ### Motivation
   
   Allow users to enable bookkeeper DiskWeightBasedPlacement in broker.conf
   
   ### Does this pull request potentially affect one of the following parts:
   
   *If `yes` was chosen, please highlight the changes*
   
 - Dependencies (does it add or upgrade a dependency): (no)
 - The public API: (no)
 - The schema: (no)
 - The default values of configurations: (no)
 - The wire protocol: (no)
 - The rest endpoints: (no)
 - The admin cli options: (no)
 - Anything that affects deployment: (no)
   
   ### Documentation
   
 - Does this pull request introduce a new feature? (no)
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [pulsar] Anonymitaet commented on issue #5250: [Doc] Add configs for broker

2019-09-23 Thread GitBox

Anonymitaet commented on issue #5250: [Doc] Add configs for broker
URL: https://github.com/apache/pulsar/pull/5250#issuecomment-533978430
 
 
   run integration tests


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [pulsar] Anonymitaet commented on a change in pull request #5253: [Doc] Add Netty source connector guide

2019-09-23 Thread GitBox

Anonymitaet commented on a change in pull request #5253: [Doc] Add *Netty 
source connector guide*
URL: https://github.com/apache/pulsar/pull/5253#discussion_r326965009
 
 

 ##
 File path: site2/docs/io-netty-source.md
 ##
 @@ -0,0 +1,204 @@
+---
+id: io-netty
+title: Netty source connector
+sidebar_label: Netty source connector
+---
+
+The Netty source connector opens a port that accepts incoming data via the 
configured network protocol 
+and publish it to user-defined Pulsar topics.
+
+This connector can be used in a containerized (for example, k8s) deployment. 
Otherwise, if the connector is running in process or thread mode, the instance 
may be conflicting on listening to ports.
+
+## Configuration
+
+The configuration of the Netty source connector has the following properties.
+
+### Property
+
+| Name | Type|Required | Default | Description 
+|--|--|--|-|-|
+| `type` |String| true |tcp | The network protocol over which data is 
transmitted to netty. Below are the available 
options:tcphttpudp |
+| `host` | String|true | 127.0.0.1 | The host name or address on which the 
source instance listen. |
+| `port` | int|true | 10999 | The port on which the source instance listen. |
+| `numberOfThreads` |int| true |1 | The number of threads of Netty TCP server 
to accept incoming connections and handle the traffic of accepted connections. |
+
+
+### Example
+
+Before using the Netty source connector, you need to create a configuration 
file through one of the following methods.
+
+* JSON 
+
+```json
+{
+"type": "tcp",
+"host": "127.0.0.1",
+"port": "10911",
+"numberOfThreads": "1"
+}
+```
+
+* YAML
+
+```yaml
+configs:
+type: "tcp"
+host: "127.0.0.1"
+port: 10999
+numberOfThreads: 1
+```
+
+
+## Usage 
+
+The following examples show how to use the Netty source connector with TCP and 
HTTP.
+
+### TCP 
+
+1. Start Pulsar standalone.
+
+```bash
+$ docker pull apachepulsar/pulsar:{version}
+
+$ docker run -d -it -p 6650:6650 -p 8080:8080 -v $PWD/data:/pulsar/data 
--name pulsar-netty-standalone apachepulsar/pulsar:{version} bin/pulsar 
standalone
+```
+
+2. Create a configuration file _netty-source-config.yaml_.
+   
+```yaml
+configs:
+type: "tcp"
+host: "127.0.0.1"
+port: 10999
+numberOfThreads: 1
+```
+
+3. Copy the configuration file _netty-source-config.yaml_ to Pulsar server.
+
+```bash
+$ docker cp netty-source-config.yaml pulsar-netty-standalone:/pulsar/conf/
+```
+
+4. Download the Netty source connector.
+
+```bash
+$ docker exec -it pulsar-netty-standalone /bin/bash
+curl -O 
http://mirror-hk.koddos.net/apache/pulsar/pulsar-{version}/connectors/pulsar-io-netty-{version}.nar
+```
+
+5. Start the Netty source connector.
+   
+   ```bash
+   $ ./bin/pulsar-admin sources localrun \
+   --archive pulsar-io-{{pulsar:version}}.nar \
+   --tenant public \
+   --namespace default \
+   --name netty \
+   --destination-topic-name netty-topic \
+   --source-config-file netty-source-config.yaml \
+   --parallelism 1
+   ```
+
+6. Consume data.
+
+```bash
+$ docker exec -it pulsar-netty-standalone /bin/bash
+
+$ ./bin/pulsar-client consume -t Exclusive -s netty-sub netty-topic -n 0
+```
+
+7. Open another terminal window to send data to the Netty source.
+
+```bash
+$ docker exec -it pulsar-netty-standalone /bin/bash
+
+$ apt-get update
+
+$ apt-get -y install telnet
+
+$ root@1d19327b2c67:/pulsar# telnet 127.0.0.1 10999
+Trying 127.0.0.1...
+Connected to 127.0.0.1.
+Escape character is '^]'.
+hello
+world
+```
+
+8. The following information appears on the consumer terminal window.
+
+```bash
+- got message -
+hello
+
+- got message -
+world
+```
+
+### HTTP 
+
+1. Start Pulsar standalone.
+
+```bash
+$ docker pull apachepulsar/pulsar:{version}
+
+$ docker run -d -it -p 6650:6650 -p 8080:8080 -v $PWD/data:/pulsar/data 
--name pulsar-netty-standalone apachepulsar/pulsar:{version} bin/pulsar 
standalone
+```
+
+2. Create a configuration file _netty-source-config.yaml_.
+   
+```yaml
+configs:
+type: "http"
+host: "127.0.0.1"
+port: 10999
+numberOfThreads: 1
+```
+
+3. Copy the configuration file _netty-source-config.yaml_ to Pulsar server.
+   
+```bash
+$ docker cp netty-source-config.yaml pulsar-netty-standalone:/pulsar/conf/
+```
+
+4. Download the Netty source connector.
+
+```bash
+$ docker exec -it pulsar-netty-standalone /bin/bash
+curl -O 
http://mirror-hk.koddos.net/apache/pulsar/pulsar-{version}/connectors/pulsar-io-netty-{version}.nar
+```
+
+5. Start the Netty source connector.
+   
+   ```bash
+   $ ./bin/pulsar-admin sources localrun \
+   --archive

[GitHub] [pulsar] Anonymitaet commented on a change in pull request #5253: [Doc] Add Netty source connector guide

2019-09-23 Thread GitBox

Anonymitaet commented on a change in pull request #5253: [Doc] Add *Netty 
source connector guide*
URL: https://github.com/apache/pulsar/pull/5253#discussion_r326965009
 
 

 ##
 File path: site2/docs/io-netty-source.md
 ##
 @@ -0,0 +1,204 @@
+---
+id: io-netty
+title: Netty source connector
+sidebar_label: Netty source connector
+---
+
+The Netty source connector opens a port that accepts incoming data via the 
configured network protocol 
+and publish it to user-defined Pulsar topics.
+
+This connector can be used in a containerized (for example, k8s) deployment. 
Otherwise, if the connector is running in process or thread mode, the instance 
may be conflicting on listening to ports.
+
+## Configuration
+
+The configuration of the Netty source connector has the following properties.
+
+### Property
+
+| Name | Type|Required | Default | Description 
+|--|--|--|-|-|
+| `type` |String| true |tcp | The network protocol over which data is 
transmitted to netty. Below are the available 
options:tcphttpudp |
+| `host` | String|true | 127.0.0.1 | The host name or address on which the 
source instance listen. |
+| `port` | int|true | 10999 | The port on which the source instance listen. |
+| `numberOfThreads` |int| true |1 | The number of threads of Netty TCP server 
to accept incoming connections and handle the traffic of accepted connections. |
+
+
+### Example
+
+Before using the Netty source connector, you need to create a configuration 
file through one of the following methods.
+
+* JSON 
+
+```json
+{
+"type": "tcp",
+"host": "127.0.0.1",
+"port": "10911",
+"numberOfThreads": "1"
+}
+```
+
+* YAML
+
+```yaml
+configs:
+type: "tcp"
+host: "127.0.0.1"
+port: 10999
+numberOfThreads: 1
+```
+
+
+## Usage 
+
+The following examples show how to use the Netty source connector with TCP and 
HTTP.
+
+### TCP 
+
+1. Start Pulsar standalone.
+
+```bash
+$ docker pull apachepulsar/pulsar:{version}
+
+$ docker run -d -it -p 6650:6650 -p 8080:8080 -v $PWD/data:/pulsar/data 
--name pulsar-netty-standalone apachepulsar/pulsar:{version} bin/pulsar 
standalone
+```
+
+2. Create a configuration file _netty-source-config.yaml_.
+   
+```yaml
+configs:
+type: "tcp"
+host: "127.0.0.1"
+port: 10999
+numberOfThreads: 1
+```
+
+3. Copy the configuration file _netty-source-config.yaml_ to Pulsar server.
+
+```bash
+$ docker cp netty-source-config.yaml pulsar-netty-standalone:/pulsar/conf/
+```
+
+4. Download the Netty source connector.
+
+```bash
+$ docker exec -it pulsar-netty-standalone /bin/bash
+curl -O 
http://mirror-hk.koddos.net/apache/pulsar/pulsar-{version}/connectors/pulsar-io-netty-{version}.nar
+```
+
+5. Start the Netty source connector.
+   
+   ```bash
+   $ ./bin/pulsar-admin sources localrun \
+   --archive pulsar-io-{{pulsar:version}}.nar \
+   --tenant public \
+   --namespace default \
+   --name netty \
+   --destination-topic-name netty-topic \
+   --source-config-file netty-source-config.yaml \
+   --parallelism 1
+   ```
+
+6. Consume data.
+
+```bash
+$ docker exec -it pulsar-netty-standalone /bin/bash
+
+$ ./bin/pulsar-client consume -t Exclusive -s netty-sub netty-topic -n 0
+```
+
+7. Open another terminal window to send data to the Netty source.
+
+```bash
+$ docker exec -it pulsar-netty-standalone /bin/bash
+
+$ apt-get update
+
+$ apt-get -y install telnet
+
+$ root@1d19327b2c67:/pulsar# telnet 127.0.0.1 10999
+Trying 127.0.0.1...
+Connected to 127.0.0.1.
+Escape character is '^]'.
+hello
+world
+```
+
+8. The following information appears on the consumer terminal window.
+
+```bash
+- got message -
+hello
+
+- got message -
+world
+```
+
+### HTTP 
+
+1. Start Pulsar standalone.
+
+```bash
+$ docker pull apachepulsar/pulsar:{version}
+
+$ docker run -d -it -p 6650:6650 -p 8080:8080 -v $PWD/data:/pulsar/data 
--name pulsar-netty-standalone apachepulsar/pulsar:{version} bin/pulsar 
standalone
+```
+
+2. Create a configuration file _netty-source-config.yaml_.
+   
+```yaml
+configs:
+type: "http"
+host: "127.0.0.1"
+port: 10999
+numberOfThreads: 1
+```
+
+3. Copy the configuration file _netty-source-config.yaml_ to Pulsar server.
+   
+```bash
+$ docker cp netty-source-config.yaml pulsar-netty-standalone:/pulsar/conf/
+```
+
+4. Download the Netty source connector.
+
+```bash
+$ docker exec -it pulsar-netty-standalone /bin/bash
+curl -O 
http://mirror-hk.koddos.net/apache/pulsar/pulsar-{version}/connectors/pulsar-io-netty-{version}.nar
+```
+
+5. Start the Netty source connector.
+   
+   ```bash
+   $ ./bin/pulsar-admin sources localrun \
+   --archive

[GitHub] [pulsar] Anonymitaet commented on a change in pull request #5253: [Doc] Add Netty source connector guide

2019-09-23 Thread GitBox

Anonymitaet commented on a change in pull request #5253: [Doc] Add *Netty 
source connector guide*
URL: https://github.com/apache/pulsar/pull/5253#discussion_r326965009
 
 

 ##
 File path: site2/docs/io-netty-source.md
 ##
 @@ -0,0 +1,204 @@
+---
+id: io-netty
+title: Netty source connector
+sidebar_label: Netty source connector
+---
+
+The Netty source connector opens a port that accepts incoming data via the 
configured network protocol 
+and publish it to user-defined Pulsar topics.
+
+This connector can be used in a containerized (for example, k8s) deployment. 
Otherwise, if the connector is running in process or thread mode, the instance 
may be conflicting on listening to ports.
+
+## Configuration
+
+The configuration of the Netty source connector has the following properties.
+
+### Property
+
+| Name | Type|Required | Default | Description 
+|--|--|--|-|-|
+| `type` |String| true |tcp | The network protocol over which data is 
transmitted to netty. Below are the available 
options:tcphttpudp |
+| `host` | String|true | 127.0.0.1 | The host name or address on which the 
source instance listen. |
+| `port` | int|true | 10999 | The port on which the source instance listen. |
+| `numberOfThreads` |int| true |1 | The number of threads of Netty TCP server 
to accept incoming connections and handle the traffic of accepted connections. |
+
+
+### Example
+
+Before using the Netty source connector, you need to create a configuration 
file through one of the following methods.
+
+* JSON 
+
+```json
+{
+"type": "tcp",
+"host": "127.0.0.1",
+"port": "10911",
+"numberOfThreads": "1"
+}
+```
+
+* YAML
+
+```yaml
+configs:
+type: "tcp"
+host: "127.0.0.1"
+port: 10999
+numberOfThreads: 1
+```
+
+
+## Usage 
+
+The following examples show how to use the Netty source connector with TCP and 
HTTP.
+
+### TCP 
+
+1. Start Pulsar standalone.
+
+```bash
+$ docker pull apachepulsar/pulsar:{version}
+
+$ docker run -d -it -p 6650:6650 -p 8080:8080 -v $PWD/data:/pulsar/data 
--name pulsar-netty-standalone apachepulsar/pulsar:{version} bin/pulsar 
standalone
+```
+
+2. Create a configuration file _netty-source-config.yaml_.
+   
+```yaml
+configs:
+type: "tcp"
+host: "127.0.0.1"
+port: 10999
+numberOfThreads: 1
+```
+
+3. Copy the configuration file _netty-source-config.yaml_ to Pulsar server.
+
+```bash
+$ docker cp netty-source-config.yaml pulsar-netty-standalone:/pulsar/conf/
+```
+
+4. Download the Netty source connector.
+
+```bash
+$ docker exec -it pulsar-netty-standalone /bin/bash
+curl -O 
http://mirror-hk.koddos.net/apache/pulsar/pulsar-{version}/connectors/pulsar-io-netty-{version}.nar
+```
+
+5. Start the Netty source connector.
+   
+   ```bash
+   $ ./bin/pulsar-admin sources localrun \
+   --archive pulsar-io-{{pulsar:version}}.nar \
+   --tenant public \
+   --namespace default \
+   --name netty \
+   --destination-topic-name netty-topic \
+   --source-config-file netty-source-config.yaml \
+   --parallelism 1
+   ```
+
+6. Consume data.
+
+```bash
+$ docker exec -it pulsar-netty-standalone /bin/bash
+
+$ ./bin/pulsar-client consume -t Exclusive -s netty-sub netty-topic -n 0
+```
+
+7. Open another terminal window to send data to the Netty source.
+
+```bash
+$ docker exec -it pulsar-netty-standalone /bin/bash
+
+$ apt-get update
+
+$ apt-get -y install telnet
+
+$ root@1d19327b2c67:/pulsar# telnet 127.0.0.1 10999
+Trying 127.0.0.1...
+Connected to 127.0.0.1.
+Escape character is '^]'.
+hello
+world
+```
+
+8. The following information appears on the consumer terminal window.
+
+```bash
+- got message -
+hello
+
+- got message -
+world
+```
+
+### HTTP 
+
+1. Start Pulsar standalone.
+
+```bash
+$ docker pull apachepulsar/pulsar:{version}
+
+$ docker run -d -it -p 6650:6650 -p 8080:8080 -v $PWD/data:/pulsar/data 
--name pulsar-netty-standalone apachepulsar/pulsar:{version} bin/pulsar 
standalone
+```
+
+2. Create a configuration file _netty-source-config.yaml_.
+   
+```yaml
+configs:
+type: "http"
+host: "127.0.0.1"
+port: 10999
+numberOfThreads: 1
+```
+
+3. Copy the configuration file _netty-source-config.yaml_ to Pulsar server.
+   
+```bash
+$ docker cp netty-source-config.yaml pulsar-netty-standalone:/pulsar/conf/
+```
+
+4. Download the Netty source connector.
+
+```bash
+$ docker exec -it pulsar-netty-standalone /bin/bash
+curl -O 
http://mirror-hk.koddos.net/apache/pulsar/pulsar-{version}/connectors/pulsar-io-netty-{version}.nar
+```
+
+5. Start the Netty source connector.
+   
+   ```bash
+   $ ./bin/pulsar-admin sources localrun \
+   --archive

[GitHub] [pulsar] Anonymitaet opened a new pull request #5253: [Doc] Add Netty source connector guide

2019-09-23 Thread GitBox

Anonymitaet opened a new pull request #5253: [Doc] Add *Netty source connector 
guide*
URL: https://github.com/apache/pulsar/pull/5253
 
 
   Fix https://github.com/apache/pulsar/issues/5015


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

1 2 >

1 - 100 of 103 matches

Mail list logo