[GitHub] merlimat closed pull request #2095: Remove unused test annotation
merlimat closed pull request #2095: Remove unused test annotation URL: https://github.com/apache/incubator-pulsar/pull/2095 This is a PR merged from a forked repository. As GitHub hides the original diff on merge, it is displayed below for the sake of provenance: As this is a foreign pull request (from a fork), the diff is supplied below (as it won't show otherwise due to GitHub magic): diff --git a/tests/integration/semantics/src/test/java/org/apache/pulsar/tests/integration/functions/PulsarFunctionsTest.java b/tests/integration/semantics/src/test/java/org/apache/pulsar/tests/integration/functions/PulsarFunctionsTest.java index 625d800d54..d0dfafc18a 100644 --- a/tests/integration/semantics/src/test/java/org/apache/pulsar/tests/integration/functions/PulsarFunctionsTest.java +++ b/tests/integration/semantics/src/test/java/org/apache/pulsar/tests/integration/functions/PulsarFunctionsTest.java @@ -34,7 +34,6 @@ // Tests on uploading/downloading function packages. // -@Test public String checkUpload() throws Exception { String bkPkgPath = String.format("%s/%s/%s", "tenant-" + randomName(8), This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[incubator-pulsar] branch master updated: Remove unused test annotation (#2095)
This is an automated email from the ASF dual-hosted git repository. mmerli pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/incubator-pulsar.git The following commit(s) were added to refs/heads/master by this push: new 86f0a0d Remove unused test annotation (#2095) 86f0a0d is described below commit 86f0a0db90c1cfdd517fc25cd015df83835bbb4f Author: Ali Ahmed AuthorDate: Fri Jul 6 21:41:40 2018 -0700 Remove unused test annotation (#2095) --- .../apache/pulsar/tests/integration/functions/PulsarFunctionsTest.java | 1 - 1 file changed, 1 deletion(-) diff --git a/tests/integration/semantics/src/test/java/org/apache/pulsar/tests/integration/functions/PulsarFunctionsTest.java b/tests/integration/semantics/src/test/java/org/apache/pulsar/tests/integration/functions/PulsarFunctionsTest.java index 625d800..d0dfafc 100644 --- a/tests/integration/semantics/src/test/java/org/apache/pulsar/tests/integration/functions/PulsarFunctionsTest.java +++ b/tests/integration/semantics/src/test/java/org/apache/pulsar/tests/integration/functions/PulsarFunctionsTest.java @@ -34,7 +34,6 @@ public class PulsarFunctionsTest extends PulsarFunctionsTestBase { // Tests on uploading/downloading function packages. // -@Test public String checkUpload() throws Exception { String bkPkgPath = String.format("%s/%s/%s", "tenant-" + randomName(8),
[GitHub] merlimat commented on a change in pull request #2095: Remove unused test annotation
merlimat commented on a change in pull request #2095: Remove unused test annotation URL: https://github.com/apache/incubator-pulsar/pull/2095#discussion_r200802291 ## File path: tests/integration/semantics/src/test/java/org/apache/pulsar/tests/integration/functions/PulsarFunctionsTest.java ## @@ -34,7 +34,6 @@ // Tests on uploading/downloading function packages. // -@Test public String checkUpload() throws Exception { Review comment: Got it. Please leave these kind of comment in the PR description so that reviewer can understand the motivation and the mechanics of the change. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] jerrypeng commented on issue #2098: Add context to source and sink
jerrypeng commented on issue #2098: Add context to source and sink URL: https://github.com/apache/incubator-pulsar/pull/2098#issuecomment-403183103 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] jerrypeng opened a new pull request #2107: patching CmdSink and adding extensive unit tests
jerrypeng opened a new pull request #2107: patching CmdSink and adding extensive unit tests URL: https://github.com/apache/incubator-pulsar/pull/2107 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] aahmed-se commented on issue #2080: Extract out common code from standalone and build Embedded Pulsar
aahmed-se commented on issue #2080: Extract out common code from standalone and build Embedded Pulsar URL: https://github.com/apache/incubator-pulsar/pull/2080#issuecomment-403182287 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] aahmed-se commented on issue #2093: Disable empty methods in SequenceIdWithErrorTest
aahmed-se commented on issue #2093: Disable empty methods in SequenceIdWithErrorTest URL: https://github.com/apache/incubator-pulsar/pull/2093#issuecomment-403180091 Done This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] merlimat opened a new pull request #2106: Provide API and CLI to access list of connectors supported by system
merlimat opened a new pull request #2106: Provide API and CLI to access list of connectors supported by system URL: https://github.com/apache/incubator-pulsar/pull/2106 ### Motivation Added API and CLI command for clients to discover list of supported connectors ### Modifications * When `WorkerService` starts, it checks the "connectors" directory and compile the list of supported sources and sinks * Added API/CLI tools Notes: * Next PR will add the ability to submit connectors by referring to a pre-existing type * This is based on #2102, only last commit 319a1fb is relevant for this PR. I'll rebase once the 1st is merged This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] sijie commented on issue #2093: Fix SequenceIdWithErrorTest
sijie commented on issue #2093: Fix SequenceIdWithErrorTest URL: https://github.com/apache/incubator-pulsar/pull/2093#issuecomment-403179765 @aahmed-se can you update the caption and description? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] sijie commented on a change in pull request #2103: Issue 1433: Expose batch flushAsync() and flush() methods in Producer
sijie commented on a change in pull request #2103: Issue 1433: Expose batch flushAsync() and flush() methods in Producer URL: https://github.com/apache/incubator-pulsar/pull/2103#discussion_r200798912 ## File path: pulsar-client/src/main/java/org/apache/pulsar/client/impl/ProducerImpl.java ## @@ -1218,7 +1223,22 @@ public void run(Timeout timeout) throws Exception { }; @Override -protected void flush() { +public CompletableFuture flushAsync() { +if (isBatchMessagingEnabled()) { Review comment: oh i see - I set the lastSendFuture but I forgot to wait :) I will fix it This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] aahmed-se commented on issue #1995: [WIP] Migrate to test containers Part 5
aahmed-se commented on issue #1995: [WIP] Migrate to test containers Part 5 URL: https://github.com/apache/incubator-pulsar/pull/1995#issuecomment-403175899 this has been replaced by individual test migration pr's This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] aahmed-se commented on issue #2045: [WIP] Add Embedded Pulsar Class
aahmed-se commented on issue #2045: [WIP] Add Embedded Pulsar Class URL: https://github.com/apache/incubator-pulsar/pull/2045#issuecomment-403175794 this is replaced by https://github.com/apache/incubator-pulsar/pull/2080 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] aahmed-se commented on issue #2095: Remove unused test annotation
aahmed-se commented on issue #2095: Remove unused test annotation URL: https://github.com/apache/incubator-pulsar/pull/2095#issuecomment-403175839 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] aahmed-se closed pull request #2045: [WIP] Add Embedded Pulsar Class
aahmed-se closed pull request #2045: [WIP] Add Embedded Pulsar Class URL: https://github.com/apache/incubator-pulsar/pull/2045 This is a PR merged from a forked repository. As GitHub hides the original diff on merge, it is displayed below for the sake of provenance: As this is a foreign pull request (from a fork), the diff is supplied below (as it won't show otherwise due to GitHub magic): diff --git a/pulsar-broker/src/main/java/org/apache/pulsar/EmbeddedPulsar.java b/pulsar-broker/src/main/java/org/apache/pulsar/EmbeddedPulsar.java new file mode 100644 index 00..634c9fb9ed --- /dev/null +++ b/pulsar-broker/src/main/java/org/apache/pulsar/EmbeddedPulsar.java @@ -0,0 +1,197 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.pulsar; + +import com.google.common.collect.Sets; +import lombok.Builder; +import org.apache.pulsar.broker.PulsarService; +import org.apache.pulsar.broker.ServiceConfiguration; +import org.apache.pulsar.broker.ServiceConfigurationUtils; +import org.apache.pulsar.client.admin.PulsarAdmin; +import org.apache.pulsar.client.admin.PulsarAdminException; +import org.apache.pulsar.common.configuration.PulsarConfigurationLoader; +import org.apache.pulsar.common.naming.TopicName; +import org.apache.pulsar.common.policies.data.ClusterData; +import org.apache.pulsar.common.policies.data.TenantInfo; +import org.apache.pulsar.functions.worker.WorkerService; +import org.apache.pulsar.zookeeper.LocalBookkeeperEnsemble; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import java.net.URL; +import java.util.Optional; + +import static org.apache.commons.lang3.StringUtils.isBlank; + +@Builder(toBuilder = true) +public class EmbeddedPulsar { + +private static final Logger log = LoggerFactory.getLogger(EmbeddedPulsar.class); + +private PulsarService broker; +private PulsarAdmin admin; +private LocalBookkeeperEnsemble bkEnsemble; +private ServiceConfiguration config; +private WorkerService fnWorkerService; +private String configFile; + +@Builder.Default +private boolean wipeData = true; +@Builder.Default +private int numOfBk = 1; +@Builder.Default +private int zkPort = 2181; +@Builder.Default +private int bkPort = 3181; +@Builder.Default +private String zkDir = "data/standalone/zookeeper"; +@Builder.Default +private String bkDir = "data/standalone/bookkeeper"; +@Builder.Default +private boolean noBroker = false; +@Builder.Default +private boolean onlyBroker = false; +@Builder.Default +private String advertisedAddress = null; + +public void start() throws Exception { + +this.config = PulsarConfigurationLoader.create((ClassLoader.class.getResourceAsStream("/embedded.conf")), ServiceConfiguration.class); + +String zkServers = "127.0.0.1"; + +if (advertisedAddress != null) { +// Use advertised address from command line +config.setAdvertisedAddress(advertisedAddress); +zkServers = advertisedAddress; +} else if (isBlank(config.getAdvertisedAddress())) { +// Use advertised address as local hostname + config.setAdvertisedAddress(ServiceConfigurationUtils.unsafeLocalhostResolve()); +} else { +// Use advertised address from config file +} + +// Set ZK server's host to localhost +config.setZookeeperServers(zkServers + ":" + zkPort); +config.setConfigurationStoreServers(zkServers + ":" + zkPort); +config.setRunningStandalone(true); + +if (config == null) { +System.exit(1); +} + +log.debug("--- setup PulsarStandaloneStarter ---"); + +if (!onlyBroker) { +// Start LocalBookKeeper +bkEnsemble = new LocalBookkeeperEnsemble(numOfBk, zkPort, bkPort, zkDir, bkDir, wipeData, advertisedAddress); +bkEnsemble.startStandalone(); +} + +if (noBroker) { +return; +} + +// Start Broker +broker = new PulsarService(config, Optional.ofNullable(fnWorkerService)); +broker.start(); + +URL webServiceUrl = new URL( +
[GitHub] merlimat commented on a change in pull request #2103: Issue 1433: Expose batch flushAsync() and flush() methods in Producer
merlimat commented on a change in pull request #2103: Issue 1433: Expose batch flushAsync() and flush() methods in Producer URL: https://github.com/apache/incubator-pulsar/pull/2103#discussion_r200796299 ## File path: pulsar-client/src/main/java/org/apache/pulsar/client/impl/ProducerImpl.java ## @@ -1218,7 +1223,22 @@ public void run(Timeout timeout) throws Exception { }; @Override -protected void flush() { +public CompletableFuture flushAsync() { +if (isBatchMessagingEnabled()) { Review comment: @sijie I think this is the only place that needs to be fixed, since the `lastSendFuture` should already be set This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] jerrypeng commented on issue #2098: Add context to source and sink
jerrypeng commented on issue #2098: Add context to source and sink URL: https://github.com/apache/incubator-pulsar/pull/2098#issuecomment-403172092 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] sijie commented on issue #2102: Load connectors from NAR archives
sijie commented on issue #2102: Load connectors from NAR archives URL: https://github.com/apache/incubator-pulsar/pull/2102#issuecomment-403168125 @jerrypeng @srkukarni @rdhabalia please spend some time on reviewing this. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] sijie commented on a change in pull request #2103: Issue 1433: Expose batch flushAsync() and flush() methods in Producer
sijie commented on a change in pull request #2103: Issue 1433: Expose batch flushAsync() and flush() methods in Producer URL: https://github.com/apache/incubator-pulsar/pull/2103#discussion_r200787191 ## File path: pulsar-client/src/main/java/org/apache/pulsar/client/api/Producer.java ## @@ -74,6 +74,27 @@ */ CompletableFuture sendAsync(T message); +/** + * Flush all the messages buffered in the client when {@link ProducerBuilder#enableBatching(boolean)} is enabled. + * It is a no-op if batching is disabled. Review comment: sorry the behavior is consistent. it is waiting on the last future. I will improve the documentation. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] aahmed-se commented on issue #2045: [WIP] Add Embedded Pulsar Class
aahmed-se commented on issue #2045: [WIP] Add Embedded Pulsar Class URL: https://github.com/apache/incubator-pulsar/pull/2045#issuecomment-403162182 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] aahmed-se commented on issue #2095: Remove unused test annotation
aahmed-se commented on issue #2095: Remove unused test annotation URL: https://github.com/apache/incubator-pulsar/pull/2095#issuecomment-403161959 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] aahmed-se commented on issue #2080: Extract out common code from standalone and build Embedded Pulsar
aahmed-se commented on issue #2080: Extract out common code from standalone and build Embedded Pulsar URL: https://github.com/apache/incubator-pulsar/pull/2080#issuecomment-403158397 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[incubator-pulsar] branch master updated: Fixed path for Postgres in dashboard docker image (#2088)
This is an automated email from the ASF dual-hosted git repository. mmerli pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/incubator-pulsar.git The following commit(s) were added to refs/heads/master by this push: new 1ebe334 Fixed path for Postgres in dashboard docker image (#2088) 1ebe334 is described below commit 1ebe3349d3a185d441e22ad3d9767d0ea39871eb Author: Matteo Merli AuthorDate: Fri Jul 6 14:31:43 2018 -0700 Fixed path for Postgres in dashboard docker image (#2088) --- dashboard/Dockerfile | 2 +- dashboard/conf/postgresql.conf | 10 +- dashboard/conf/supervisor-app.conf | 2 +- 3 files changed, 7 insertions(+), 7 deletions(-) diff --git a/dashboard/Dockerfile b/dashboard/Dockerfile index faae137..a00d8b0 100644 --- a/dashboard/Dockerfile +++ b/dashboard/Dockerfile @@ -28,7 +28,7 @@ RUN apt-get -y install postgresql python sudo nginx supervisor RUN pip install uwsgi 'Django<2.0' psycopg2 pytz requests # Postgres configuration -COPY conf/postgresql.conf /etc/postgresql/9.4/main/ +COPY conf/postgresql.conf /etc/postgresql/9.6/main/ # Configure nginx and supervisor RUN echo "daemon off;" >> /etc/nginx/nginx.conf diff --git a/dashboard/conf/postgresql.conf b/dashboard/conf/postgresql.conf index d35dffa..201dbba 100644 --- a/dashboard/conf/postgresql.conf +++ b/dashboard/conf/postgresql.conf @@ -23,16 +23,16 @@ full_page_writes = off synchronous_commit = off # Default configs -data_directory = '/var/lib/postgresql/9.4/main' -hba_file = '/etc/postgresql/9.4/main/pg_hba.conf' -ident_file = '/etc/postgresql/9.4/main/pg_ident.conf' -external_pid_file = '/var/run/postgresql/9.4-main.pid' +data_directory = '/var/lib/postgresql/9.6/main' +hba_file = '/etc/postgresql/9.6/main/pg_hba.conf' +ident_file = '/etc/postgresql/9.6/main/pg_ident.conf' +external_pid_file = '/var/run/postgresql/9.6-main.pid' port = 5432 max_connections = 100 datestyle = 'iso, mdy' default_text_search_config = 'pg_catalog.english' -stats_temp_directory = '/var/run/postgresql/9.4-main.pg_stat_tmp' +stats_temp_directory = '/var/run/postgresql/9.6-main.pg_stat_tmp' timezone = 'UTC' log_timezone = 'UTC' diff --git a/dashboard/conf/supervisor-app.conf b/dashboard/conf/supervisor-app.conf index 30ff208..44a918c 100644 --- a/dashboard/conf/supervisor-app.conf +++ b/dashboard/conf/supervisor-app.conf @@ -18,7 +18,7 @@ # [program:postgres] -command = /usr/lib/postgresql/9.4/bin/postgres -D /etc/postgresql/9.4/main +command = /usr/lib/postgresql/9.6/bin/postgres -D /etc/postgresql/9.6/main user = postgres [program:uwsgi]
[GitHub] merlimat closed pull request #2088: Fixed path for Postgres in dashboard docker image
merlimat closed pull request #2088: Fixed path for Postgres in dashboard docker image URL: https://github.com/apache/incubator-pulsar/pull/2088 This is a PR merged from a forked repository. As GitHub hides the original diff on merge, it is displayed below for the sake of provenance: As this is a foreign pull request (from a fork), the diff is supplied below (as it won't show otherwise due to GitHub magic): diff --git a/dashboard/Dockerfile b/dashboard/Dockerfile index faae137d24..a00d8b03f9 100644 --- a/dashboard/Dockerfile +++ b/dashboard/Dockerfile @@ -28,7 +28,7 @@ RUN apt-get -y install postgresql python sudo nginx supervisor RUN pip install uwsgi 'Django<2.0' psycopg2 pytz requests # Postgres configuration -COPY conf/postgresql.conf /etc/postgresql/9.4/main/ +COPY conf/postgresql.conf /etc/postgresql/9.6/main/ # Configure nginx and supervisor RUN echo "daemon off;" >> /etc/nginx/nginx.conf diff --git a/dashboard/conf/postgresql.conf b/dashboard/conf/postgresql.conf index d35dffa4c2..201dbba9da 100644 --- a/dashboard/conf/postgresql.conf +++ b/dashboard/conf/postgresql.conf @@ -23,16 +23,16 @@ full_page_writes = off synchronous_commit = off # Default configs -data_directory = '/var/lib/postgresql/9.4/main' -hba_file = '/etc/postgresql/9.4/main/pg_hba.conf' -ident_file = '/etc/postgresql/9.4/main/pg_ident.conf' -external_pid_file = '/var/run/postgresql/9.4-main.pid' +data_directory = '/var/lib/postgresql/9.6/main' +hba_file = '/etc/postgresql/9.6/main/pg_hba.conf' +ident_file = '/etc/postgresql/9.6/main/pg_ident.conf' +external_pid_file = '/var/run/postgresql/9.6-main.pid' port = 5432 max_connections = 100 datestyle = 'iso, mdy' default_text_search_config = 'pg_catalog.english' -stats_temp_directory = '/var/run/postgresql/9.4-main.pg_stat_tmp' +stats_temp_directory = '/var/run/postgresql/9.6-main.pg_stat_tmp' timezone = 'UTC' log_timezone = 'UTC' diff --git a/dashboard/conf/supervisor-app.conf b/dashboard/conf/supervisor-app.conf index 30ff208717..44a918c7b2 100644 --- a/dashboard/conf/supervisor-app.conf +++ b/dashboard/conf/supervisor-app.conf @@ -18,7 +18,7 @@ # [program:postgres] -command = /usr/lib/postgresql/9.4/bin/postgres -D /etc/postgresql/9.4/main +command = /usr/lib/postgresql/9.6/bin/postgres -D /etc/postgresql/9.6/main user = postgres [program:uwsgi] This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[incubator-pulsar] branch master updated: Remove dependencies exclusions for org.apache.httpcomponents:httpclient (#2104)
This is an automated email from the ASF dual-hosted git repository. mmerli pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/incubator-pulsar.git The following commit(s) were added to refs/heads/master by this push: new 7bfc343 Remove dependencies exclusions for org.apache.httpcomponents:httpclient (#2104) 7bfc343 is described below commit 7bfc34340d627f9a6e01ada004550b4d7e09f1d6 Author: Matteo Merli AuthorDate: Fri Jul 6 14:15:22 2018 -0700 Remove dependencies exclusions for org.apache.httpcomponents:httpclient (#2104) --- pom.xml | 6 -- 1 file changed, 6 deletions(-) diff --git a/pom.xml b/pom.xml index 862a29d..ba705a7 100644 --- a/pom.xml +++ b/pom.xml @@ -188,12 +188,6 @@ flexible messaging model and an intuitive client API. org.apache.httpcomponents httpclient 4.5.5 - - -* -* - -
[GitHub] merlimat closed pull request #2104: Remove dependencies exclusions for org.apache.httpcomponents:httpclient
merlimat closed pull request #2104: Remove dependencies exclusions for org.apache.httpcomponents:httpclient URL: https://github.com/apache/incubator-pulsar/pull/2104 This is a PR merged from a forked repository. As GitHub hides the original diff on merge, it is displayed below for the sake of provenance: As this is a foreign pull request (from a fork), the diff is supplied below (as it won't show otherwise due to GitHub magic): diff --git a/pom.xml b/pom.xml index 862a29d020..ba705a7796 100644 --- a/pom.xml +++ b/pom.xml @@ -188,12 +188,6 @@ flexible messaging model and an intuitive client API. org.apache.httpcomponents httpclient 4.5.5 - - -* -* - - This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] rdhabalia commented on a change in pull request #2104: Remove dependencies exclusions for org.apache.httpcomponents:httpclient
rdhabalia commented on a change in pull request #2104: Remove dependencies exclusions for org.apache.httpcomponents:httpclient URL: https://github.com/apache/incubator-pulsar/pull/2104#discussion_r200752973 ## File path: pom.xml ## @@ -188,12 +188,6 @@ flexible messaging model and an intuitive client API. org.apache.httpcomponents httpclient 4.5.5 Review comment: I misread it, I thought this change was made in pulsar-client-pom.xml. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] merlimat commented on a change in pull request #2104: Remove dependencies exclusions for org.apache.httpcomponents:httpclient
merlimat commented on a change in pull request #2104: Remove dependencies exclusions for org.apache.httpcomponents:httpclient URL: https://github.com/apache/incubator-pulsar/pull/2104#discussion_r200750911 ## File path: pom.xml ## @@ -188,12 +188,6 @@ flexible messaging model and an intuitive client API. org.apache.httpcomponents httpclient 4.5.5 Review comment: I left the exclusion on the `pulsar-client/pom.xml` so that it get only applied there This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] merlimat commented on a change in pull request #2104: Remove dependencies exclusions for org.apache.httpcomponents:httpclient
merlimat commented on a change in pull request #2104: Remove dependencies exclusions for org.apache.httpcomponents:httpclient URL: https://github.com/apache/incubator-pulsar/pull/2104#discussion_r200750766 ## File path: pom.xml ## @@ -188,12 +188,6 @@ flexible messaging model and an intuitive client API. org.apache.httpcomponents httpclient 4.5.5 Review comment: This section is from the top-level dep-management This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] rdhabalia commented on a change in pull request #2104: Remove dependencies exclusions for org.apache.httpcomponents:httpclient
rdhabalia commented on a change in pull request #2104: Remove dependencies exclusions for org.apache.httpcomponents:httpclient URL: https://github.com/apache/incubator-pulsar/pull/2104#discussion_r200750533 ## File path: pom.xml ## @@ -188,12 +188,6 @@ flexible messaging model and an intuitive client API. org.apache.httpcomponents httpclient 4.5.5 Review comment: as this dependency already part of parent-dep-management so, should we remove version as well? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] merlimat opened a new pull request #2104: Remove dependencies exclusions for org.apache.httpcomponents:httpclient
merlimat opened a new pull request #2104: Remove dependencies exclusions for org.apache.httpcomponents:httpclient URL: https://github.com/apache/incubator-pulsar/pull/2104 ### Motivation In `pulsar-client` we have dependency on `org.apache.httpcomponents:httpclient` and we had exclusions since we don't need any of the dependencies. Since the exclusion is already set in `pulsar-client/pom.xml` we don't need it here. This blanket exclusion is causing problems for connectors that use this dependency, since Maven won't pull in any of the transitive dependencies. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] sijie commented on issue #2051: Move Security related documentation to a top-level category
sijie commented on issue #2051: Move Security related documentation to a top-level category URL: https://github.com/apache/incubator-pulsar/pull/2051#issuecomment-403122140 @merlimat I addressed your comment. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[incubator-pulsar] branch master updated: Kinesis sink publish full json message (#2079)
This is an automated email from the ASF dual-hosted git repository. rdhabalia pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/incubator-pulsar.git The following commit(s) were added to refs/heads/master by this push: new da8981c Kinesis sink publish full json message (#2079) da8981c is described below commit da8981c7becab94443cdac41a9495654943570ef Author: Rajan Dhabalia AuthorDate: Fri Jul 6 11:59:29 2018 -0700 Kinesis sink publish full json message (#2079) * Kinesis sink publish full json message * fix pulsar typo --- pulsar-io/core/pom.xml | 13 ++ .../org/apache/pulsar/io/kinesis/KinesisSink.java | 23 ++- .../pulsar/io/kinesis/KinesisSinkConfig.java | 26 +++- .../java/org/apache/pulsar/io/kinesis/Utils.java | 103 + .../org/apache/pulsar/io/kinesis/UtilsTest.java| 162 + 5 files changed, 324 insertions(+), 3 deletions(-) diff --git a/pulsar-io/core/pom.xml b/pulsar-io/core/pom.xml index 30da3d5..003e6d8 100644 --- a/pulsar-io/core/pom.xml +++ b/pulsar-io/core/pom.xml @@ -47,6 +47,19 @@ + + + ${project.groupId} + protobuf-shaded + ${project.version} + + + com.google.protobuf + protobuf-java + + + + diff --git a/pulsar-io/kinesis/src/main/java/org/apache/pulsar/io/kinesis/KinesisSink.java b/pulsar-io/kinesis/src/main/java/org/apache/pulsar/io/kinesis/KinesisSink.java index 5a43312..eaa2b91 100644 --- a/pulsar-io/kinesis/src/main/java/org/apache/pulsar/io/kinesis/KinesisSink.java +++ b/pulsar-io/kinesis/src/main/java/org/apache/pulsar/io/kinesis/KinesisSink.java @@ -35,6 +35,7 @@ import org.apache.commons.lang3.builder.ReflectionToStringBuilder; import org.apache.commons.lang3.builder.ToStringStyle; import org.apache.pulsar.io.core.RecordContext; import org.apache.pulsar.io.core.Sink; +import org.apache.pulsar.io.kinesis.KinesisSinkConfig.MessageFormat; import org.slf4j.Logger; import org.slf4j.LoggerFactory; @@ -66,9 +67,17 @@ import io.netty.util.Recycler.Handle; * which accepts json-map of credentials in awsCredentialPluginParam * eg: awsCredentialPluginParam = {"accessKey":"my-access-key","secretKey":"my-secret-key"} * 5. awsCredentialPluginParam: json-parameters to initialize {@link AwsCredentialProviderPlugin} - * + * 6. messageFormat: enum:["ONLY_RAW_PAYLOAD","FULL_MESSAGE_IN_JSON"] + * a. ONLY_RAW_PAYLOAD: publishes raw payload to stream + * b. FULL_MESSAGE_IN_JSON: publish full message (encryptionCtx + properties + payload) in json format + * json-schema: + * {"type":"object","properties":{"encryptionCtx":{"type":"object","properties":{"metadata":{"type":"object","additionalProperties":{"type":"string"}},"uncompressedMessageSize":{"type":"integer"},"keysMetadataMap":{"type":"object","additionalProperties":{"type":"object","additionalProperties":{"type":"string"}}},"keysMapBase64":{"type":"object","additionalProperties":{"type":"string"}},"encParamBase64":{"type":"string"},"compressionType":{"type":"string","enum":["NONE","LZ4","ZLIB"]}," [...] + * Example: + * {"payloadBase64":"cGF5bG9hZA==","properties":{"prop1":"value"},"encryptionCtx":{"keysMapBase64":{"key1":"dGVzdDE=","key2":"dGVzdDI="},"keysMetadataMap":{"key1":{"ckms":"cmks-1","version":"v1"},"key2":{"ckms":"cmks-2","version":"v2"}},"metadata":{"ckms":"cmks-1","version":"v1"},"encParamBase64":"cGFyYW0=","algorithm":"algo","compressionType":"LZ4","uncompressedMessageSize":10,"batchSize":10}} * * + * + * */ public class KinesisSink implements Sink { @@ -92,7 +101,8 @@ public class KinesisSink implements Sink { ? partitionedKey.substring(0, maxPartitionedKeyLength - 1) : partitionedKey; // partitionedKey Length must be at least one, and at most 256 ListenableFuture addRecordResult = kinesisProducer.addUserRecord(this.streamName, -partitionedKey, ByteBuffer.wrap(value)); +partitionedKey, + ByteBuffer.wrap(createKinesisMessage(kinesisSinkConfig.getMessageFormat(), inputRecordContext, value))); addCallback(addRecordResult, ProducerSendCallback.create(this.streamName, inputRecordContext, System.nanoTime()), directExecutor()); if (LOG.isDebugEnabled()) { @@ -263,4 +273,13 @@ public class KinesisSink implements Sink { }; } +public static byte[] createKinesisMessage(MessageFormat msgFormat, RecordContext recordCtx, byte[] data) { +if (MessageFormat.FULL_MESSAGE_IN_JSON.equals(msgFormat)) { +return Utils.serializeRecordToJson(recordCtx, data).getBytes(); +} else { +// send raw-message +return data; +} +} + } \ No newline at end of file diff --git
[GitHub] rdhabalia closed pull request #2079: Kinesis sink publish full json message
rdhabalia closed pull request #2079: Kinesis sink publish full json message URL: https://github.com/apache/incubator-pulsar/pull/2079 This is a PR merged from a forked repository. As GitHub hides the original diff on merge, it is displayed below for the sake of provenance: As this is a foreign pull request (from a fork), the diff is supplied below (as it won't show otherwise due to GitHub magic): diff --git a/pulsar-io/core/pom.xml b/pulsar-io/core/pom.xml index 30da3d527a..003e6d8611 100644 --- a/pulsar-io/core/pom.xml +++ b/pulsar-io/core/pom.xml @@ -47,6 +47,19 @@ + + + ${project.groupId} + protobuf-shaded + ${project.version} + + + com.google.protobuf + protobuf-java + + + + diff --git a/pulsar-io/kinesis/src/main/java/org/apache/pulsar/io/kinesis/KinesisSink.java b/pulsar-io/kinesis/src/main/java/org/apache/pulsar/io/kinesis/KinesisSink.java index 5a433122b3..eaa2b91845 100644 --- a/pulsar-io/kinesis/src/main/java/org/apache/pulsar/io/kinesis/KinesisSink.java +++ b/pulsar-io/kinesis/src/main/java/org/apache/pulsar/io/kinesis/KinesisSink.java @@ -35,6 +35,7 @@ import org.apache.commons.lang3.builder.ToStringStyle; import org.apache.pulsar.io.core.RecordContext; import org.apache.pulsar.io.core.Sink; +import org.apache.pulsar.io.kinesis.KinesisSinkConfig.MessageFormat; import org.slf4j.Logger; import org.slf4j.LoggerFactory; @@ -66,9 +67,17 @@ * which accepts json-map of credentials in awsCredentialPluginParam * eg: awsCredentialPluginParam = {"accessKey":"my-access-key","secretKey":"my-secret-key"} * 5. awsCredentialPluginParam: json-parameters to initialize {@link AwsCredentialProviderPlugin} - * + * 6. messageFormat: enum:["ONLY_RAW_PAYLOAD","FULL_MESSAGE_IN_JSON"] + * a. ONLY_RAW_PAYLOAD: publishes raw payload to stream + * b. FULL_MESSAGE_IN_JSON: publish full message (encryptionCtx + properties + payload) in json format + * json-schema: + * {"type":"object","properties":{"encryptionCtx":{"type":"object","properties":{"metadata":{"type":"object","additionalProperties":{"type":"string"}},"uncompressedMessageSize":{"type":"integer"},"keysMetadataMap":{"type":"object","additionalProperties":{"type":"object","additionalProperties":{"type":"string"}}},"keysMapBase64":{"type":"object","additionalProperties":{"type":"string"}},"encParamBase64":{"type":"string"},"compressionType":{"type":"string","enum":["NONE","LZ4","ZLIB"]},"batchSize":{"type":"integer"},"algorithm":{"type":"string"}}},"payloadBase64":{"type":"string"},"properties":{"type":"object","additionalProperties":{"type":"string" + * Example: + * {"payloadBase64":"cGF5bG9hZA==","properties":{"prop1":"value"},"encryptionCtx":{"keysMapBase64":{"key1":"dGVzdDE=","key2":"dGVzdDI="},"keysMetadataMap":{"key1":{"ckms":"cmks-1","version":"v1"},"key2":{"ckms":"cmks-2","version":"v2"}},"metadata":{"ckms":"cmks-1","version":"v1"},"encParamBase64":"cGFyYW0=","algorithm":"algo","compressionType":"LZ4","uncompressedMessageSize":10,"batchSize":10}} * * + * + * */ public class KinesisSink implements Sink { @@ -92,7 +101,8 @@ public void write(RecordContext inputRecordContext, byte[] value) throws Excepti ? partitionedKey.substring(0, maxPartitionedKeyLength - 1) : partitionedKey; // partitionedKey Length must be at least one, and at most 256 ListenableFuture addRecordResult = kinesisProducer.addUserRecord(this.streamName, -partitionedKey, ByteBuffer.wrap(value)); +partitionedKey, + ByteBuffer.wrap(createKinesisMessage(kinesisSinkConfig.getMessageFormat(), inputRecordContext, value))); addCallback(addRecordResult, ProducerSendCallback.create(this.streamName, inputRecordContext, System.nanoTime()), directExecutor()); if (LOG.isDebugEnabled()) { @@ -263,4 +273,13 @@ public void refresh() { }; } +public static byte[] createKinesisMessage(MessageFormat msgFormat, RecordContext recordCtx, byte[] data) { +if (MessageFormat.FULL_MESSAGE_IN_JSON.equals(msgFormat)) { +return Utils.serializeRecordToJson(recordCtx, data).getBytes(); +} else { +// send raw-message +return data; +} +} + } \ No newline at end of file diff --git a/pulsar-io/kinesis/src/main/java/org/apache/pulsar/io/kinesis/KinesisSinkConfig.java b/pulsar-io/kinesis/src/main/java/org/apache/pulsar/io/kinesis/KinesisSinkConfig.java index b7dbad441b..e3c8fde263 100644 --- a/pulsar-io/kinesis/src/main/java/org/apache/pulsar/io/kinesis/KinesisSinkConfig.java +++ b/pulsar-io/kinesis/src/main/java/org/apache/pulsar/io/kinesis/KinesisSinkConfig.java @@ -44,6 +44,7 @@ private String awsKinesisStreamName; private String awsCredentialPluginName; private String awsCredentialPluginParam; +private
[GitHub] merlimat commented on a change in pull request #2103: Issue 1433: Expose batch flushAsync() and flush() methods in Producer
merlimat commented on a change in pull request #2103: Issue 1433: Expose batch flushAsync() and flush() methods in Producer URL: https://github.com/apache/incubator-pulsar/pull/2103#discussion_r200742150 ## File path: pulsar-client/src/main/java/org/apache/pulsar/client/api/Producer.java ## @@ -74,6 +74,27 @@ */ CompletableFuture sendAsync(T message); +/** + * Flush all the messages buffered in the client when {@link ProducerBuilder#enableBatching(boolean)} is enabled. + * It is a no-op if batching is disabled. Review comment: The behavior could be consistent even when batching is not being used. For example, one could always do: ```java producer.sendAsync(msg1); producer.sendAsync(msg2); producer.sendAsync(msg3); producer.flush(); ``` This should be equivalent to just waiting on the last future. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] sijie commented on issue #2088: Fixed path for Postgres in dashboard docker image
sijie commented on issue #2088: Fixed path for Postgres in dashboard docker image URL: https://github.com/apache/incubator-pulsar/pull/2088#issuecomment-403118834 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] sijie opened a new pull request #2103: Issue 1433: Expose batch flushAsync() and flush() methods in Producer
sijie opened a new pull request #2103: Issue 1433: Expose batch flushAsync() and flush() methods in Producer URL: https://github.com/apache/incubator-pulsar/pull/2103 *Motivation* Currently when you're using batches server will wait until timeout to send messages. To archive the maximum theoretical performance I suggest to expose flushAsync method (and flush() for completeness) of the Producer class. That will force publisher to flush all batched messages for faster delivery (faster than waiting until batch timeout). *Changes* - Introduce `flush` and `flushAsync` in Producer interface - rename existing `flush` method to `triggerFlush` - flush is waiting for the completion of last write 7345ae1 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] aahmed-se commented on issue #2093: Fix SequenceIdWithErrorTest
aahmed-se commented on issue #2093: Fix SequenceIdWithErrorTest URL: https://github.com/apache/incubator-pulsar/pull/2093#issuecomment-403117550 @rdhabalia will need to take a deeper look at the handleProducer code , this change is mostly to disable empty test methods to remove them from reporting. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] merlimat opened a new pull request #2102: Load connectors from NAR archives
merlimat opened a new pull request #2102: Load connectors from NAR archives URL: https://github.com/apache/incubator-pulsar/pull/2102 ### Motivation Allow the connectors to be submitted as NAR bundles. Each NAR bundle contains the classes for the connector code plus all of its dependencies packaged in a Zip file. The bundle is unpacked before being used the first time and then the directory is cached unless the archive is changed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] sijie commented on issue #2092: Issue 2090: Links to connector source code are broken
sijie commented on issue #2092: Issue 2090: Links to connector source code are broken URL: https://github.com/apache/incubator-pulsar/pull/2092#issuecomment-403070345 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] sijie commented on issue #2080: Extract out common code from standalone and build Embedded Pulsar
sijie commented on issue #2080: Extract out common code from standalone and build Embedded Pulsar URL: https://github.com/apache/incubator-pulsar/pull/2080#issuecomment-403070170 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] sijie closed pull request #2096: Issue 1252: Enable bookie autorecovery daemon by default
sijie closed pull request #2096: Issue 1252: Enable bookie autorecovery daemon by default URL: https://github.com/apache/incubator-pulsar/pull/2096 This is a PR merged from a forked repository. As GitHub hides the original diff on merge, it is displayed below for the sake of provenance: As this is a foreign pull request (from a fork), the diff is supplied below (as it won't show otherwise due to GitHub magic): diff --git a/conf/bookkeeper.conf b/conf/bookkeeper.conf index 6742c365bf..dedb7d88d8 100644 --- a/conf/bookkeeper.conf +++ b/conf/bookkeeper.conf @@ -229,13 +229,13 @@ rereplicationEntryBatchSize=5000 # Auto-replication # The grace period, in seconds, that the replication worker waits before fencing and # replicating a ledger fragment that's still being written to upon bookie failure. -# openLedgerRereplicationGracePeriod=30 +openLedgerRereplicationGracePeriod=30 # Whether the bookie itself can start auto-recovery service also or not -# autoRecoveryDaemonEnabled=false +autoRecoveryDaemonEnabled=true # How long to wait, in seconds, before starting auto recovery of a lost bookie -# lostBookieRecoveryDelay=0 +lostBookieRecoveryDelay=0 # ## Netty server settings This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[incubator-pulsar] branch master updated: Issue 1252: Enable bookie autorecovery daemon by default (#2096)
This is an automated email from the ASF dual-hosted git repository. sijie pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/incubator-pulsar.git The following commit(s) were added to refs/heads/master by this push: new 3e9bac6 Issue 1252: Enable bookie autorecovery daemon by default (#2096) 3e9bac6 is described below commit 3e9bac60e4bc5521ed3864dee7f14efd6e5bc0d1 Author: Sijie Guo AuthorDate: Fri Jul 6 08:37:29 2018 -0700 Issue 1252: Enable bookie autorecovery daemon by default (#2096) *Motivation* Fixes #1252 AutoRecovery daemon should be enabled by default for new users. *Changes* Turn on `autoRecoveryDaemonEnabled`. uncommeted recovery related settings to allow them being configured in containers. --- conf/bookkeeper.conf | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/conf/bookkeeper.conf b/conf/bookkeeper.conf index 6742c36..dedb7d8 100644 --- a/conf/bookkeeper.conf +++ b/conf/bookkeeper.conf @@ -229,13 +229,13 @@ rereplicationEntryBatchSize=5000 # Auto-replication # The grace period, in seconds, that the replication worker waits before fencing and # replicating a ledger fragment that's still being written to upon bookie failure. -# openLedgerRereplicationGracePeriod=30 +openLedgerRereplicationGracePeriod=30 # Whether the bookie itself can start auto-recovery service also or not -# autoRecoveryDaemonEnabled=false +autoRecoveryDaemonEnabled=true # How long to wait, in seconds, before starting auto recovery of a lost bookie -# lostBookieRecoveryDelay=0 +lostBookieRecoveryDelay=0 # ## Netty server settings
[GitHub] sijie closed issue #1252: Enable auto-recovery by default in bookie configuration
sijie closed issue #1252: Enable auto-recovery by default in bookie configuration URL: https://github.com/apache/incubator-pulsar/issues/1252 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] sijie closed pull request #2099: Issue 1455: MessageID has always batch index 0 when sending messages in a batch
sijie closed pull request #2099: Issue 1455: MessageID has always batch index 0 when sending messages in a batch URL: https://github.com/apache/incubator-pulsar/pull/2099 This is a PR merged from a forked repository. As GitHub hides the original diff on merge, it is displayed below for the sake of provenance: As this is a foreign pull request (from a fork), the diff is supplied below (as it won't show otherwise due to GitHub magic): diff --git a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/nonpersistent/NonPersistentReplicator.java b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/nonpersistent/NonPersistentReplicator.java index 42b99adace..d505c4417f 100644 --- a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/nonpersistent/NonPersistentReplicator.java +++ b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/nonpersistent/NonPersistentReplicator.java @@ -212,7 +212,7 @@ protected ProducerSendCallback newObject(Handle handle) { }; @Override -public void addCallback(SendCallback scb) { +public void addCallback(MessageImpl msg, SendCallback scb) { // noop } @@ -221,6 +221,11 @@ public SendCallback getNextSendCallback() { return null; } +@Override +public MessageImpl getNextMessage() { +return null; +} + @Override public CompletableFuture getFuture() { return null; diff --git a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentReplicator.java b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentReplicator.java index e0df5ea75b..7d529f909d 100644 --- a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentReplicator.java +++ b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentReplicator.java @@ -381,7 +381,7 @@ protected ProducerSendCallback newObject(Handle handle) { }; @Override -public void addCallback(SendCallback scb) { +public void addCallback(MessageImpl msg, SendCallback scb) { // noop } @@ -390,6 +390,11 @@ public SendCallback getNextSendCallback() { return null; } +@Override +public MessageImpl getNextMessage() { +return null; +} + @Override public CompletableFuture getFuture() { return null; diff --git a/pulsar-client/src/main/java/org/apache/pulsar/client/impl/BatchMessageContainer.java b/pulsar-client/src/main/java/org/apache/pulsar/client/impl/BatchMessageContainer.java index a97e5249cc..4d2ca09907 100644 --- a/pulsar-client/src/main/java/org/apache/pulsar/client/impl/BatchMessageContainer.java +++ b/pulsar-client/src/main/java/org/apache/pulsar/client/impl/BatchMessageContainer.java @@ -96,7 +96,7 @@ void add(MessageImpl msg, SendCallback callback) { } if (previousCallback != null) { -previousCallback.addCallback(callback); +previousCallback.addCallback(msg, callback); } previousCallback = callback; diff --git a/pulsar-client/src/main/java/org/apache/pulsar/client/impl/ProducerImpl.java b/pulsar-client/src/main/java/org/apache/pulsar/client/impl/ProducerImpl.java index afb6177568..a621a0ec3e 100644 --- a/pulsar-client/src/main/java/org/apache/pulsar/client/impl/ProducerImpl.java +++ b/pulsar-client/src/main/java/org/apache/pulsar/client/impl/ProducerImpl.java @@ -207,6 +207,7 @@ public long getLastSequenceId() { sendAsync(message, new SendCallback() { SendCallback nextCallback = null; +MessageImpl nextMsg = null; long createdAt = System.nanoTime(); @Override @@ -219,6 +220,11 @@ public SendCallback getNextSendCallback() { return nextCallback; } +@Override +public MessageImpl getNextMessage() { +return nextMsg; +} + @Override public void sendComplete(Exception e) { if (e != null) { @@ -230,20 +236,22 @@ public void sendComplete(Exception e) { } while (nextCallback != null) { SendCallback sendCallback = nextCallback; +MessageImpl msg = nextMsg; if (e != null) { stats.incrementSendFailed(); sendCallback.getFuture().completeExceptionally(e); } else { - sendCallback.getFuture().complete(message.getMessageId()); +sendCallback.getFuture().complete(msg.getMessageId()); stats.incrementNumAcksReceived(System.nanoTime() - createdAt); } +nextMsg = nextCallback.getNextMessage();
[incubator-pulsar] branch master updated: Issue 1455: MessageID has always batch index 0 when sending messages in a batch (#2099)
This is an automated email from the ASF dual-hosted git repository. sijie pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/incubator-pulsar.git The following commit(s) were added to refs/heads/master by this push: new 83e3157 Issue 1455: MessageID has always batch index 0 when sending messages in a batch (#2099) 83e3157 is described below commit 83e31579f5df941ad49fb54ec269c1d917821504 Author: Sijie Guo AuthorDate: Fri Jul 6 08:37:07 2018 -0700 Issue 1455: MessageID has always batch index 0 when sending messages in a batch (#2099) *Motivation* Fixes #1455. Pulsar uses a callback chain for completing the list of callbacks for a batch. However the callback chain doesn't reference the message instance for completing the callback. so when callback chain is triggered, it always uses the first message id to complete the chain of callbacks. *Changes* Introduce a field to keep message instance in the callback chain. So when the chain is invoked, each callback can use the right message instance to complete the callback. Added an integration test to ensure it works correctly. --- .../nonpersistent/NonPersistentReplicator.java | 7 ++- .../service/persistent/PersistentReplicator.java | 7 ++- .../pulsar/client/impl/BatchMessageContainer.java | 2 +- .../apache/pulsar/client/impl/ProducerImpl.java| 14 - .../apache/pulsar/client/impl/SendCallback.java| 14 - .../tests/integration/semantics/SemanticsTest.java | 64 ++ 6 files changed, 99 insertions(+), 9 deletions(-) diff --git a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/nonpersistent/NonPersistentReplicator.java b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/nonpersistent/NonPersistentReplicator.java index 42b99ad..d505c44 100644 --- a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/nonpersistent/NonPersistentReplicator.java +++ b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/nonpersistent/NonPersistentReplicator.java @@ -212,7 +212,7 @@ public class NonPersistentReplicator extends AbstractReplicator implements Repli }; @Override -public void addCallback(SendCallback scb) { +public void addCallback(MessageImpl msg, SendCallback scb) { // noop } @@ -222,6 +222,11 @@ public class NonPersistentReplicator extends AbstractReplicator implements Repli } @Override +public MessageImpl getNextMessage() { +return null; +} + +@Override public CompletableFuture getFuture() { return null; } diff --git a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentReplicator.java b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentReplicator.java index e0df5ea..7d529f9 100644 --- a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentReplicator.java +++ b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentReplicator.java @@ -381,7 +381,7 @@ public class PersistentReplicator extends AbstractReplicator implements Replicat }; @Override -public void addCallback(SendCallback scb) { +public void addCallback(MessageImpl msg, SendCallback scb) { // noop } @@ -391,6 +391,11 @@ public class PersistentReplicator extends AbstractReplicator implements Replicat } @Override +public MessageImpl getNextMessage() { +return null; +} + +@Override public CompletableFuture getFuture() { return null; } diff --git a/pulsar-client/src/main/java/org/apache/pulsar/client/impl/BatchMessageContainer.java b/pulsar-client/src/main/java/org/apache/pulsar/client/impl/BatchMessageContainer.java index a97e524..4d2ca09 100644 --- a/pulsar-client/src/main/java/org/apache/pulsar/client/impl/BatchMessageContainer.java +++ b/pulsar-client/src/main/java/org/apache/pulsar/client/impl/BatchMessageContainer.java @@ -96,7 +96,7 @@ class BatchMessageContainer { } if (previousCallback != null) { -previousCallback.addCallback(callback); +previousCallback.addCallback(msg, callback); } previousCallback = callback; diff --git a/pulsar-client/src/main/java/org/apache/pulsar/client/impl/ProducerImpl.java b/pulsar-client/src/main/java/org/apache/pulsar/client/impl/ProducerImpl.java index afb6177..a621a0e 100644 --- a/pulsar-client/src/main/java/org/apache/pulsar/client/impl/ProducerImpl.java +++ b/pulsar-client/src/main/java/org/apache/pulsar/client/impl/ProducerImpl.java @@ -207,6 +207,7 @@ public class ProducerImpl extends ProducerBase implements TimerTask, Conne sendAsync(message, new SendCallback() {
[GitHub] sijie closed issue #1455: MessageID has always batch index 0 when sending messages in a batch
sijie closed issue #1455: MessageID has always batch index 0 when sending messages in a batch URL: https://github.com/apache/incubator-pulsar/issues/1455 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[incubator-pulsar] branch master updated: Fix wget argument (#2085)
This is an automated email from the ASF dual-hosted git repository. nkurihar pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/incubator-pulsar.git The following commit(s) were added to refs/heads/master by this push: new 0420e59 Fix wget argument (#2085) 0420e59 is described below commit 0420e594eeaf7268886f5dc307488151c9bfc27e Author: Yuto Furuta AuthorDate: Fri Jul 6 19:15:57 2018 +0900 Fix wget argument (#2085) * fix wget value * remove -incubating --- site/_includes/explanations/install-package.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/site/_includes/explanations/install-package.md b/site/_includes/explanations/install-package.md index 3350445..7047ec6 100644 --- a/site/_includes/explanations/install-package.md +++ b/site/_includes/explanations/install-package.md @@ -38,7 +38,7 @@ To get started running Pulsar, download a binary tarball release in one of the f * using [wget](https://www.gnu.org/software/wget): ```shell - $ wget {{ binary_release_url }} + $ wget 'https://www.apache.org/dyn/mirrors/mirrors.cgi?action=download=incubator/pulsar/pulsar-{{ site.current_version }}/apache-pulsar-{{ site.current_version }}-bin.tar.gz' -O apache-pulsar-{{ site.current_version }}-bin.tar.gz ``` Once the tarball is downloaded, untar it and `cd` into the resulting directory:
[GitHub] nkurihar closed pull request #2085: Fix wget argument
nkurihar closed pull request #2085: Fix wget argument URL: https://github.com/apache/incubator-pulsar/pull/2085 This is a PR merged from a forked repository. As GitHub hides the original diff on merge, it is displayed below for the sake of provenance: As this is a foreign pull request (from a fork), the diff is supplied below (as it won't show otherwise due to GitHub magic): diff --git a/site/_includes/explanations/install-package.md b/site/_includes/explanations/install-package.md index 3350445c82..7047ec6ec8 100644 --- a/site/_includes/explanations/install-package.md +++ b/site/_includes/explanations/install-package.md @@ -38,7 +38,7 @@ To get started running Pulsar, download a binary tarball release in one of the f * using [wget](https://www.gnu.org/software/wget): ```shell - $ wget {{ binary_release_url }} + $ wget 'https://www.apache.org/dyn/mirrors/mirrors.cgi?action=download=incubator/pulsar/pulsar-{{ site.current_version }}/apache-pulsar-{{ site.current_version }}-bin.tar.gz' -O apache-pulsar-{{ site.current_version }}-bin.tar.gz ``` Once the tarball is downloaded, untar it and `cd` into the resulting directory: This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[incubator-pulsar] branch asf-site updated: Updated site at revision 268161c
This is an automated email from the ASF dual-hosted git repository. mmerli pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/incubator-pulsar.git The following commit(s) were added to refs/heads/asf-site by this push: new 6db2ee8 Updated site at revision 268161c 6db2ee8 is described below commit 6db2ee829350ae92d8bbcaeee4e1b5a1fdad76ca Author: jenkins AuthorDate: Fri Jul 6 08:51:39 2018 + Updated site at revision 268161c --- content/docs/latest/adaptors/PulsarSpark/index.html| 2 +- content/docs/latest/adaptors/PulsarStorm/index.html| 2 +- content/docs/latest/admin-api/overview/index.html | 8 content/docs/latest/admin/Authz/index.html | 4 ++-- content/docs/latest/clients/Java/index.html| 2 +- content/docs/latest/cookbooks/PartitionedTopics/index.html | 2 +- content/docs/latest/cookbooks/RetentionExpiry/index.html | 2 +- content/docs/latest/cookbooks/compaction/index.html| 2 +- content/docs/latest/cookbooks/message-deduplication/index.html | 2 +- content/docs/latest/cookbooks/message-queue/index.html | 2 +- content/docs/latest/cookbooks/tiered-storage/index.html| 2 +- content/docs/latest/deployment/aws-cluster/index.html | 2 +- content/docs/latest/deployment/cluster/index.html | 2 +- content/docs/latest/deployment/instance/index.html | 2 +- content/docs/latest/reference/CliTools/index.html | 4 ++-- 15 files changed, 20 insertions(+), 20 deletions(-) diff --git a/content/docs/latest/adaptors/PulsarSpark/index.html b/content/docs/latest/adaptors/PulsarSpark/index.html index c71a007..a9397a3 100644 --- a/content/docs/latest/adaptors/PulsarSpark/index.html +++ b/content/docs/latest/adaptors/PulsarSpark/index.html @@ -1609,9 +1609,9 @@ - The Pulsar admin interface + The Pulsar admin interface diff --git a/content/docs/latest/adaptors/PulsarStorm/index.html b/content/docs/latest/adaptors/PulsarStorm/index.html index 0bfe25d..01b0b9d 100644 --- a/content/docs/latest/adaptors/PulsarStorm/index.html +++ b/content/docs/latest/adaptors/PulsarStorm/index.html @@ -1373,9 +1373,9 @@ - The Pulsar admin interface + The Pulsar admin interface diff --git a/content/docs/latest/admin-api/overview/index.html b/content/docs/latest/admin-api/overview/index.html index 468c2ad..5ab27cc 100644 --- a/content/docs/latest/admin-api/overview/index.html +++ b/content/docs/latest/admin-api/overview/index.html @@ -1145,9 +1145,9 @@ - The Pulsar admin interface + The Pulsar admin interface @@ -1385,9 +1385,9 @@ - The Pulsar admin interface + The Pulsar admin interface @@ -1621,9 +1621,9 @@ - The Pulsar admin interface + The Pulsar admin interface @@ -1867,9 +1867,9 @@ - The Pulsar admin interface + The Pulsar admin interface diff --git a/content/docs/latest/admin/Authz/index.html b/content/docs/latest/admin/Authz/index.html index 1cb4f79..ca57a56 100644 --- a/content/docs/latest/admin/Authz/index.html +++ b/content/docs/latest/admin/Authz/index.html @@ -1145,9 +1145,9 @@ - The Pulsar admin interface + The Pulsar admin interface @@ -2341,9 +2341,9 @@ - The Pulsar admin interface + The Pulsar admin interface diff --git a/content/docs/latest/clients/Java/index.html b/content/docs/latest/clients/Java/index.html index faabed8..53de505 100644 --- a/content/docs/latest/clients/Java/index.html +++ b/content/docs/latest/clients/Java/index.html @@ -1383,9 +1383,9 @@ - The Pulsar admin interface + The Pulsar admin interface diff --git a/content/docs/latest/cookbooks/PartitionedTopics/index.html b/content/docs/latest/cookbooks/PartitionedTopics/index.html index 74bd608..e1fe7a6 100644 --- a/content/docs/latest/cookbooks/PartitionedTopics/index.html +++ b/content/docs/latest/cookbooks/PartitionedTopics/index.html @@ -1617,9 +1617,9 @@ - The
[GitHub] aahmed-se commented on issue #2093: Fix SequenceIdWithErrorTest
aahmed-se commented on issue #2093: Fix SequenceIdWithErrorTest URL: https://github.com/apache/incubator-pulsar/pull/2093#issuecomment-402960417 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] aahmed-se commented on issue #2095: Remove unused test annotation
aahmed-se commented on issue #2095: Remove unused test annotation URL: https://github.com/apache/incubator-pulsar/pull/2095#issuecomment-402960382 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services