[GitHub] jai1 commented on a change in pull request #1451: Fix: NPE on http-lookup redirection
jai1 commented on a change in pull request #1451: Fix: NPE on http-lookup redirection URL: https://github.com/apache/incubator-pulsar/pull/1451#discussion_r177314778 ## File path: pulsar-broker/src/main/java/org/apache/pulsar/broker/web/PulsarWebResource.java ## @@ -264,7 +265,8 @@ protected void validateClusterOwnership(String cluster) throws WebApplicationExc private URI getRedirectionUrl(ClusterData differentClusterData) throws MalformedURLException { URL webUrl = null; -if (pulsar.getConfiguration().isTlsEnabled() && !differentClusterData.getServiceUrlTls().isEmpty()) { +if (isRequestHttps() && pulsar.getConfiguration().isTlsEnabled() Review comment: isTlsEnabled check is not required This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] merlimat commented on issue #1449: When running standalone broker, force the proxy redirection flag
merlimat commented on issue #1449: When running standalone broker, force the proxy redirection flag URL: https://github.com/apache/incubator-pulsar/pull/1449#issuecomment-376403603 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] merlimat commented on issue #1451: Fix: NPE on http-lookup redirection
merlimat commented on issue #1451: Fix: NPE on http-lookup redirection URL: https://github.com/apache/incubator-pulsar/pull/1451#issuecomment-376403137 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] merlimat commented on issue #1451: Fix: NPE on http-lookup redirection
merlimat commented on issue #1451: Fix: NPE on http-lookup redirection URL: https://github.com/apache/incubator-pulsar/pull/1451#issuecomment-376403137 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] merlimat commented on issue #1410: Convert admin client properties to use v2 endpoints.
merlimat commented on issue #1410: Convert admin client properties to use v2 endpoints. URL: https://github.com/apache/incubator-pulsar/pull/1410#issuecomment-376396644 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] merlimat commented on issue #1445: Add version warning for latest vs. 2.0
merlimat commented on issue #1445: Add version warning for latest vs. 2.0 URL: https://github.com/apache/incubator-pulsar/pull/1445#issuecomment-376396537 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] merlimat commented on issue #1445: Add version warning for latest vs. 2.0
merlimat commented on issue #1445: Add version warning for latest vs. 2.0 URL: https://github.com/apache/incubator-pulsar/pull/1445#issuecomment-376396537 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] zhaijack commented on issue #1428: Issue #1117: handle race in concurrent bundle split
zhaijack commented on issue #1428: Issue #1117: handle race in concurrent bundle split URL: https://github.com/apache/incubator-pulsar/pull/1428#issuecomment-376395978 @merlimat , test fail because of wrongly removed `getData` in LocalZooKeeperCacheService in commit [2f1e6e1](https://github.com/apache/incubator-pulsar/pull/1428/commits/2f1e6e151add53070249ee628951d2de5d2cfea2#diff-5579b63437652bdd1f5ebad02121f551L83), added it back. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] zhaijack commented on issue #1428: Issue #1117: handle race in concurrent bundle split
zhaijack commented on issue #1428: Issue #1117: handle race in concurrent bundle split URL: https://github.com/apache/incubator-pulsar/pull/1428#issuecomment-376395978 @merlimat , test fail because of wrongly removed `getData` in LocalZooKeeperCacheService in commit [2f1e6e1](https://github.com/apache/incubator-pulsar/pull/1428/commits/2f1e6e151add53070249ee628951d2de5d2cfea2), added it back. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] zhaijack commented on a change in pull request #1428: Issue #1117: handle race in concurrent bundle split
zhaijack commented on a change in pull request #1428: Issue #1117: handle race in concurrent bundle split URL: https://github.com/apache/incubator-pulsar/pull/1428#discussion_r177306959 ## File path: pulsar-broker/src/main/java/org/apache/pulsar/broker/cache/LocalZooKeeperCacheService.java ## @@ -105,6 +108,38 @@ public LocalPolicies deserialize(String path, byte[] content) throws Exception { return future; } + +@Override +public CompletableFuture>> getWithStatAsync(String path) { Review comment: Thanks, Seems a lot of other place call it indirectly, remove it will cause error of BrokerServiceTest/testCreateNamespacePolicy at least. Added it back in latest commit, which removed the duplicated logic. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] zubchenok opened a new issue #1452: reachedEndOfTopic is called twice if a topic has been terminated before subscription
zubchenok opened a new issue #1452: reachedEndOfTopic is called twice if a topic has been terminated before subscription URL: https://github.com/apache/incubator-pulsar/issues/1452 When you terminate a topic and then subscribe, MessageListener.reachedEndOfTopic is called two times. There is a unit test that reproduces the issue: ``` @Test public void testDoubledReachedEndOfTopic() throws Exception { String topicName = "persistent://test-prop/vv/ns1/test-" + UUID.randomUUID(); PulsarClient pulsarClient = PulsarClient.create("pulsar://vv:6650"); Producer producer = pulsarClient.createProducer(topicName); producer.close(); PulsarAdmin admin = new PulsarAdmin(new URL("http://vv:8080;), new ClientConfiguration()); admin.persistentTopics().terminateTopicAsync(topicName).get(); admin.close(); CountDownLatch latch = new CountDownLatch(2); Consumer consumer = pulsarClient.subscribe(topicName, "subscription", new ConsumerConfiguration() .setMessageListener(new MessageListener() { @Override public void reachedEndOfTopic(Consumer consumer) { System.out.println("reachedEndOfTopic"); latch.countDown(); } @Override public void received(Consumer consumer, Message message) { // do nothing } }) ); assertFalse(latch.await(3, TimeUnit.SECONDS)); assertTrue(latch.getCount() == 1); consumer.close(); pulsarClient.close(); } ``` This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] rdhabalia opened a new pull request #1451: Fix: NPE on http-lookup redirection
rdhabalia opened a new pull request #1451: Fix: NPE on http-lookup redirection URL: https://github.com/apache/incubator-pulsar/pull/1451 ### Motivation - If broker has TLS enabled and broker's TLS url is not setup then http-redirection throws NPE. - Also, broker should consider http-request's protocol (http/https) while redirecting to another cluster. Right now, if broker's TLS is enabled then it tries to redirect to HTTPS url even if request is made on HTTP. ### Modifications Handle NPE and http-request's protocol. ### Result - Broker will not throw NPE while http redirection - Broker will redirect to another cluster's service url based on request's protocol. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] merlimat opened a new pull request #1450: Added multiple position delete in ManagedLedger
merlimat opened a new pull request #1450: Added multiple position delete in ManagedLedger URL: https://github.com/apache/incubator-pulsar/pull/1450 ### Motivation To support the grouping of acknowledgements, we need to be able to pass multiple message ids to managed ledger to delete individually. This change will be used then in broker to do group acknowledgment This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] zhaijack commented on issue #1428: Issue #1117: handle race in concurrent bundle split
zhaijack commented on issue #1428: Issue #1117: handle race in concurrent bundle split URL: https://github.com/apache/incubator-pulsar/pull/1428#issuecomment-376367021 @merlimat thanks, will handle it now This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] merlimat commented on issue #1428: Issue #1117: handle race in concurrent bundle split
merlimat commented on issue #1428: Issue #1117: handle race in concurrent bundle split URL: https://github.com/apache/incubator-pulsar/pull/1428#issuecomment-376362244 @zhaijack Please merge with master that has fix for a very flaky test. Also it appears there's a test that might be genuiniley failing: https://builds.apache.org/job/pulsar-pull-request/2290/org.apache.pulsar$pulsar-broker/testReport/org.apache.pulsar.broker.service/BrokerServiceTest/testCreateNamespacePolicy/ I don't remember having seen that test failing before. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[incubator-pulsar] branch master updated: Extend Spark Streaming Receiver UT Coverage (#1448)
This is an automated email from the ASF dual-hosted git repository. mmerli pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/incubator-pulsar.git The following commit(s) were added to refs/heads/master by this push: new b1fbd3a Extend Spark Streaming Receiver UT Coverage (#1448) b1fbd3a is described below commit b1fbd3a7abb7f745af0c2dfc47931f9c8584cebb Author: Eren AvsarogullariAuthorDate: Tue Mar 27 01:32:01 2018 +0100 Extend Spark Streaming Receiver UT Coverage (#1448) --- .../spark/SparkStreamingPulsarReceiverTest.java| 46 +- 1 file changed, 36 insertions(+), 10 deletions(-) diff --git a/pulsar-spark/src/test/java/org/apache/pulsar/spark/SparkStreamingPulsarReceiverTest.java b/pulsar-spark/src/test/java/org/apache/pulsar/spark/SparkStreamingPulsarReceiverTest.java index 86e9604..b89cd58 100644 --- a/pulsar-spark/src/test/java/org/apache/pulsar/spark/SparkStreamingPulsarReceiverTest.java +++ b/pulsar-spark/src/test/java/org/apache/pulsar/spark/SparkStreamingPulsarReceiverTest.java @@ -22,11 +22,12 @@ import static org.mockito.Mockito.spy; import static org.mockito.Mockito.doNothing; import org.apache.pulsar.client.api.*; +import org.apache.spark.storage.StorageLevel; import org.mockito.ArgumentCaptor; import static org.testng.Assert.assertEquals; +import static org.testng.Assert.assertNotNull; -import org.apache.pulsar.spark.SparkStreamingPulsarReceiver; import org.testng.annotations.AfterClass; import org.testng.annotations.BeforeClass; import org.testng.annotations.Test; @@ -35,6 +36,11 @@ import org.apache.pulsar.broker.auth.MockedPulsarServiceBaseTest; public class SparkStreamingPulsarReceiverTest extends MockedPulsarServiceBaseTest { +private final String URL = "pulsar://127.0.0.1:" + BROKER_PORT + "/"; +private static final String TOPIC = "persistent://p1/c1/ns1/topic1"; +private static final String SUBS = "sub1"; +private static final String EXPECTED_MESSAGE = "pulsar-spark test message"; + @BeforeClass @Override protected void setup() throws Exception { @@ -51,12 +57,9 @@ public class SparkStreamingPulsarReceiverTest extends MockedPulsarServiceBaseTes public void testReceivedMessage() throws Exception { ClientConfiguration clientConf = new ClientConfiguration(); ConsumerConfiguration consConf = new ConsumerConfiguration(); -String url = "pulsar://127.0.0.1:" + BROKER_PORT + "/"; -String topic = "persistent://p1/c1/ns1/topic1"; -String subs = "sub1"; SparkStreamingPulsarReceiver receiver = spy( -new SparkStreamingPulsarReceiver(clientConf, consConf, url, topic, subs)); +new SparkStreamingPulsarReceiver(clientConf, consConf, URL, TOPIC, SUBS)); MessageListener msgListener = spy(new MessageListener() { @Override public void received(Consumer consumer, Message msg) { @@ -70,17 +73,40 @@ public class SparkStreamingPulsarReceiverTest extends MockedPulsarServiceBaseTes receiver.onStart(); waitForTransmission(); -PulsarClient pulsarClient = PulsarClient.create(url, clientConf); -Producer producer = pulsarClient.createProducer(topic, new ProducerConfiguration()); -producer.send("pulsar-spark test message".getBytes()); +PulsarClient pulsarClient = PulsarClient.create(URL, clientConf); +Producer producer = pulsarClient.createProducer(TOPIC, new ProducerConfiguration()); +producer.send(EXPECTED_MESSAGE.getBytes()); waitForTransmission(); receiver.onStop(); -assertEquals(new String(msgCaptor.getValue().getData()), "pulsar-spark test message"); +assertEquals(new String(msgCaptor.getValue().getData()), EXPECTED_MESSAGE); +} + +@Test +public void testDefaultSettingsOfReceiver() { +ClientConfiguration clientConf = new ClientConfiguration(); +ConsumerConfiguration consConf = new ConsumerConfiguration(); +SparkStreamingPulsarReceiver receiver = +new SparkStreamingPulsarReceiver(clientConf, consConf, URL, TOPIC, SUBS); +assertEquals(receiver.storageLevel(), StorageLevel.MEMORY_AND_DISK_2()); +assertEquals(consConf.getAckTimeoutMillis(), 60_000); +assertNotNull(consConf.getMessageListener()); +} + +@Test(expectedExceptions = NullPointerException.class, +expectedExceptionsMessageRegExp = "ClientConfiguration must not be null") +public void testReceiverWhenClientConfigurationIsNull() { +new SparkStreamingPulsarReceiver(null, new ConsumerConfiguration(), URL, TOPIC, SUBS); +} + +@Test(expectedExceptions = NullPointerException.class, +expectedExceptionsMessageRegExp = "ConsumerConfiguration must not be null") +public void testReceiverWhenConsumerConfigurationIsNull() { +new
[GitHub] merlimat closed pull request #1448: [Spark_Streaming] Extend Receiver UT Coverage
merlimat closed pull request #1448: [Spark_Streaming] Extend Receiver UT Coverage URL: https://github.com/apache/incubator-pulsar/pull/1448 This is a PR merged from a forked repository. As GitHub hides the original diff on merge, it is displayed below for the sake of provenance: As this is a foreign pull request (from a fork), the diff is supplied below (as it won't show otherwise due to GitHub magic): diff --git a/pulsar-spark/src/test/java/org/apache/pulsar/spark/SparkStreamingPulsarReceiverTest.java b/pulsar-spark/src/test/java/org/apache/pulsar/spark/SparkStreamingPulsarReceiverTest.java index 86e960463..b89cd58b0 100644 --- a/pulsar-spark/src/test/java/org/apache/pulsar/spark/SparkStreamingPulsarReceiverTest.java +++ b/pulsar-spark/src/test/java/org/apache/pulsar/spark/SparkStreamingPulsarReceiverTest.java @@ -22,11 +22,12 @@ import static org.mockito.Mockito.doNothing; import org.apache.pulsar.client.api.*; +import org.apache.spark.storage.StorageLevel; import org.mockito.ArgumentCaptor; import static org.testng.Assert.assertEquals; +import static org.testng.Assert.assertNotNull; -import org.apache.pulsar.spark.SparkStreamingPulsarReceiver; import org.testng.annotations.AfterClass; import org.testng.annotations.BeforeClass; import org.testng.annotations.Test; @@ -35,6 +36,11 @@ public class SparkStreamingPulsarReceiverTest extends MockedPulsarServiceBaseTest { +private final String URL = "pulsar://127.0.0.1:" + BROKER_PORT + "/"; +private static final String TOPIC = "persistent://p1/c1/ns1/topic1"; +private static final String SUBS = "sub1"; +private static final String EXPECTED_MESSAGE = "pulsar-spark test message"; + @BeforeClass @Override protected void setup() throws Exception { @@ -51,12 +57,9 @@ protected void cleanup() throws Exception { public void testReceivedMessage() throws Exception { ClientConfiguration clientConf = new ClientConfiguration(); ConsumerConfiguration consConf = new ConsumerConfiguration(); -String url = "pulsar://127.0.0.1:" + BROKER_PORT + "/"; -String topic = "persistent://p1/c1/ns1/topic1"; -String subs = "sub1"; SparkStreamingPulsarReceiver receiver = spy( -new SparkStreamingPulsarReceiver(clientConf, consConf, url, topic, subs)); +new SparkStreamingPulsarReceiver(clientConf, consConf, URL, TOPIC, SUBS)); MessageListener msgListener = spy(new MessageListener() { @Override public void received(Consumer consumer, Message msg) { @@ -70,17 +73,40 @@ public void received(Consumer consumer, Message msg) { receiver.onStart(); waitForTransmission(); -PulsarClient pulsarClient = PulsarClient.create(url, clientConf); -Producer producer = pulsarClient.createProducer(topic, new ProducerConfiguration()); -producer.send("pulsar-spark test message".getBytes()); +PulsarClient pulsarClient = PulsarClient.create(URL, clientConf); +Producer producer = pulsarClient.createProducer(TOPIC, new ProducerConfiguration()); +producer.send(EXPECTED_MESSAGE.getBytes()); waitForTransmission(); receiver.onStop(); -assertEquals(new String(msgCaptor.getValue().getData()), "pulsar-spark test message"); +assertEquals(new String(msgCaptor.getValue().getData()), EXPECTED_MESSAGE); +} + +@Test +public void testDefaultSettingsOfReceiver() { +ClientConfiguration clientConf = new ClientConfiguration(); +ConsumerConfiguration consConf = new ConsumerConfiguration(); +SparkStreamingPulsarReceiver receiver = +new SparkStreamingPulsarReceiver(clientConf, consConf, URL, TOPIC, SUBS); +assertEquals(receiver.storageLevel(), StorageLevel.MEMORY_AND_DISK_2()); +assertEquals(consConf.getAckTimeoutMillis(), 60_000); +assertNotNull(consConf.getMessageListener()); +} + +@Test(expectedExceptions = NullPointerException.class, +expectedExceptionsMessageRegExp = "ClientConfiguration must not be null") +public void testReceiverWhenClientConfigurationIsNull() { +new SparkStreamingPulsarReceiver(null, new ConsumerConfiguration(), URL, TOPIC, SUBS); +} + +@Test(expectedExceptions = NullPointerException.class, +expectedExceptionsMessageRegExp = "ConsumerConfiguration must not be null") +public void testReceiverWhenConsumerConfigurationIsNull() { +new SparkStreamingPulsarReceiver(new ClientConfiguration(), null, URL, TOPIC, SUBS); } private static void waitForTransmission() { try { -Thread.sleep(1000); +Thread.sleep(1_000); } catch (InterruptedException e) { } } This is an automated message from the Apache Git Service. To respond to the message, please log on
[GitHub] sijie commented on issue #1424: Improve batch message acking by removing batch message tracker
sijie commented on issue #1424: Improve batch message acking by removing batch message tracker URL: https://github.com/apache/incubator-pulsar/pull/1424#issuecomment-376352303 merged latest master This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] sijie commented on a change in pull request #1447: Send Log statements to log Topic for Java Functions
sijie commented on a change in pull request #1447: Send Log statements to log Topic for Java Functions URL: https://github.com/apache/incubator-pulsar/pull/1447#discussion_r177271917 ## File path: pulsar-functions/instance/src/main/java/org/apache/pulsar/functions/instance/JavaInstanceRunnable.java ## @@ -262,10 +268,12 @@ public void run() { } long processAt = System.currentTimeMillis(); stats.incrementProcessed(processAt); +addLogTopicHandler(); Review comment: this seems to be done at per message? Can we configure the log handler at the beginning of the instance and remove it at the end? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] sijie commented on a change in pull request #1441: Update the bookkeeper configuration file
sijie commented on a change in pull request #1441: Update the bookkeeper configuration file URL: https://github.com/apache/incubator-pulsar/pull/1441#discussion_r177271361 ## File path: conf/bookkeeper.conf ## @@ -39,101 +68,231 @@ bookiePort=3181 # set the listening interface. allowLoopback=false -# Configure a specific hostname or IP address that the bookie should use to advertise itself to -# clients. If not set, bookie will advertised its own IP address or hostname, depending on the -# listeningInterface and `seHostNameAsBookieID settings. -advertisedAddress= +# Interval to watch whether bookie is dead or not, in milliseconds +bookieDeathWatchInterval=1000 -# Directory Bookkeeper outputs its write ahead log -journalDirectory=data/bookkeeper/journal +# When entryLogPerLedgerEnabled is enabled, checkpoint doesn't happens +# when a new active entrylog is created / previous one is rolled over. +# Instead SyncThread checkpoints periodically with 'flushInterval' delay +# (in milliseconds) in between executions. Checkpoint flushes both ledger +# entryLogs and ledger index pages to disk. +# Flushing entrylog and index files will introduce much random disk I/O. +# If separating journal dir and ledger dirs each on different devices, +# flushing would not affect performance. But if putting journal dir +# and ledger dirs on same device, performance degrade significantly +# on too frequent flushing. You can consider increment flush interval +# to get better performance, but you need to pay more time on bookie +# server restart after failure. +# This config is used only when entryLogPerLedgerEnabled is enabled. +flushInterval=6 -# Directory Bookkeeper outputs ledger snapshots -# could define multi directories to store snapshots, separated by ',' -# For example: -# ledgerDirectories=/tmp/bk1-data,/tmp/bk2-data -# -# Ideally ledger dirs and journal dir are each in a differet device, -# which reduce the contention between random i/o and sequential write. -# It is possible to run with a single disk, but performance will be significantly lower. -ledgerDirectories=data/bookkeeper/ledgers -# Directories to store index files. If not specified, will use ledgerDirectories to store. -# indexDirectories=data/bookkeeper/ledgers +# Allow the expansion of bookie storage capacity. Newly added ledger +# and index dirs must be empty. +# allowStorageExpansion=false -# Ledger Manager Class -# What kind of ledger manager is used to manage how ledgers are stored, managed -# and garbage collected. Try to read 'BookKeeper Internals' for detail info. -ledgerManagerType=hierarchical +# Whether the bookie should use its hostname to register with the +# co-ordination service(eg: Zookeeper service). +# When false, bookie will use its ipaddress for the registration. +# Defaults to false. +useHostNameAsBookieID=false -# Root zookeeper path to store ledger metadata -# This parameter is used by zookeeper-based ledger manager as a root znode to -# store all ledgers. -zkLedgersRootPath=/ledgers +# Whether the bookie is allowed to use an ephemeral port (port 0) as its +# server port. By default, an ephemeral port is not allowed. +# Using an ephemeral port as the service port usually indicates a configuration +# error. However, in unit tests, using an ephemeral port will address port +# conflict problems and allow running tests in parallel. +# allowEphemeralPorts=false -# Ledger storage implementation class -ledgerStorageClass=org.apache.bookkeeper.bookie.storage.ldb.DbLedgerStorage +# Whether allow the bookie to listen for BookKeeper clients executed on the local JVM. +# enableLocalTransport=false -# Enable/Disable entry logger preallocation -entryLogFilePreallocationEnabled=true +# Whether allow the bookie to disable bind on network interfaces, +# this bookie will be available only to BookKeeper clients executed on the local JVM. +# disableServerSocketBind=false -# Max file size of entry logger, in bytes -# A new entry log file will be created when the old one reaches the file size limitation -logSizeLimit=1073741824 +# The number of bytes we should use as chunk allocation for +# org.apache.bookkeeper.bookie.SkipListArena +# skipListArenaChunkSize=4194304 -# Threshold of minor compaction -# For those entry log files whose remaining size percentage reaches below -# this threshold will be compacted in a minor compaction. -# If it is set to less than zero, the minor compaction is disabled. -minorCompactionThreshold=0.2 +# The max size we should allocate from the skiplist arena. Allocations +# larger than this should be allocated directly by the VM to avoid fragmentation. +# skipListArenaMaxAllocSize=131072 -# Interval to run minor compaction, in seconds -# If it is set to less than zero, the minor compaction is disabled. -minorCompactionInterval=3600 +# The bookie authentication provider factory class name. +# If this is null, no authentication will take place. +#
[GitHub] merlimat commented on issue #1424: Improve batch message acking by removing batch message tracker
merlimat commented on issue #1424: Improve batch message acking by removing batch message tracker URL: https://github.com/apache/incubator-pulsar/pull/1424#issuecomment-376349548 @sijie can you merge with master. It already has the flaky test fixed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] merlimat commented on issue #1424: Improve batch message acking by removing batch message tracker
merlimat commented on issue #1424: Improve batch message acking by removing batch message tracker URL: https://github.com/apache/incubator-pulsar/pull/1424#issuecomment-376349548 @sijie can you merge with master? It already has the flaky test fixed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] merlimat opened a new pull request #1449: When running standalone broker, force the proxy redirection flag
merlimat opened a new pull request #1449: When running standalone broker, force the proxy redirection flag URL: https://github.com/apache/incubator-pulsar/pull/1449 ### Motivation Since standalone is running a "real" Pulsar service (along with ZK and BookKeeper) it uses the same discovery mechanism used in a full blown cluster. One of the problematic aspect is that brokers need to advertise an IP/hostname that is routable by clients. Since in standalone mode we have 1 single broker, we can get away with that additional configuration by forcing client to treat the standalone as a proxy. ### Modifications * Added a configuration flag for standalone mode. When the flag is set it will force the clients to connect through the same service url they have been already connecting to. ### Result No need to specify `advertisedAddress` when running standalone service. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] erenavsarogullari commented on issue #1448: [Spark_Streaming] Extend Receiver UT Coverage
erenavsarogullari commented on issue #1448: [Spark_Streaming] Extend Receiver UT Coverage URL: https://github.com/apache/incubator-pulsar/pull/1448#issuecomment-376344347 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[incubator-pulsar] branch master updated: Enforce timeout on all TestNG tests, thread dump on timeout (#1347)
This is an automated email from the ASF dual-hosted git repository. mmerli pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/incubator-pulsar.git The following commit(s) were added to refs/heads/master by this push: new dafe123 Enforce timeout on all TestNG tests, thread dump on timeout (#1347) dafe123 is described below commit dafe123e2909ba138007d1dba3dc43c7852446b5 Author: Matteo MerliAuthorDate: Mon Mar 26 16:06:38 2018 -0700 Enforce timeout on all TestNG tests, thread dump on timeout (#1347) * Enforce timeout on all TestNG tests, thread dump on timeout * Increased default timeout to 120 seconds * Removed usage of log statement that were making the log4j appender test to fail --- .../apache/pulsar/tests/AnnotationListener.java| 12 +- .../apache/pulsar/tests/PulsarTestListener.java| 73 ++ .../org/apache/pulsar/tests/ThreadDumpUtil.java| 148 + managed-ledger/pom.xml | 12 -- pom.xml| 8 +- pulsar-broker/pom.xml | 12 -- pulsar-discovery-service/pom.xml | 17 --- 7 files changed, 239 insertions(+), 43 deletions(-) diff --git a/buildtools/src/main/java/org/apache/pulsar/tests/AnnotationListener.java b/buildtools/src/main/java/org/apache/pulsar/tests/AnnotationListener.java index 0682105..be03247 100644 --- a/buildtools/src/main/java/org/apache/pulsar/tests/AnnotationListener.java +++ b/buildtools/src/main/java/org/apache/pulsar/tests/AnnotationListener.java @@ -27,9 +27,19 @@ import org.testng.annotations.ITestAnnotation; @SuppressWarnings("rawtypes") public class AnnotationListener implements IAnnotationTransformer { +private static final int DEFAULT_TEST_TIMEOUT_MILLIS = 12; + +public AnnotationListener() { +System.out.println("Created annotation listener"); +} + @Override public void transform(ITestAnnotation annotation, Class testClass, Constructor testConstructor, Method testMethod) { annotation.setRetryAnalyzer(RetryAnalyzer.class); -} +// Enforce default test timeout +if (annotation.getTimeOut() == 0) { +annotation.setTimeOut(DEFAULT_TEST_TIMEOUT_MILLIS); +} +} } diff --git a/buildtools/src/main/java/org/apache/pulsar/tests/PulsarTestListener.java b/buildtools/src/main/java/org/apache/pulsar/tests/PulsarTestListener.java new file mode 100644 index 000..1b21a68 --- /dev/null +++ b/buildtools/src/main/java/org/apache/pulsar/tests/PulsarTestListener.java @@ -0,0 +1,73 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.pulsar.tests; + +import java.util.Arrays; + +import org.testng.ITestContext; +import org.testng.ITestListener; +import org.testng.ITestResult; +import org.testng.internal.thread.ThreadTimeoutException; + +public class PulsarTestListener implements ITestListener { + +@Override +public void onTestStart(ITestResult result) { +System.out.format("--- Starting test %s.%s(%s)---\n", result.getTestClass(), result.getTestName(), +Arrays.toString(result.getParameters())); +} + +@Override +public void onTestSuccess(ITestResult result) { +System.out.format("--- SUCCESS -- %s.%s(%s)---", result.getTestClass(), result.getTestName(), +Arrays.toString(result.getParameters())); +} + +@Override +public void onTestFailure(ITestResult result) { +System.out.format("! FAILURE-- %s.%s(%s)---\n", result.getTestClass(), result.getTestName(), +Arrays.toString(result.getParameters())); + +if (result.getThrowable() instanceof ThreadTimeoutException) { +System.out.println("== THREAD DUMPS =="); +System.out.println(ThreadDumpUtil.buildThreadDiagnosticString()); +} +} + +@Override +public void onTestSkipped(ITestResult result) { +System.out.format("~ SKIPPED -- %s.%s(%s)---", result.getTestClass(), result.getTestName(), +
[GitHub] merlimat closed pull request #1347: Enforce timeout on all TestNG tests, thread dump on timeout
merlimat closed pull request #1347: Enforce timeout on all TestNG tests, thread dump on timeout URL: https://github.com/apache/incubator-pulsar/pull/1347 This is a PR merged from a forked repository. As GitHub hides the original diff on merge, it is displayed below for the sake of provenance: As this is a foreign pull request (from a fork), the diff is supplied below (as it won't show otherwise due to GitHub magic): diff --git a/buildtools/src/main/java/org/apache/pulsar/tests/AnnotationListener.java b/buildtools/src/main/java/org/apache/pulsar/tests/AnnotationListener.java index 0682105cc..be03247e9 100644 --- a/buildtools/src/main/java/org/apache/pulsar/tests/AnnotationListener.java +++ b/buildtools/src/main/java/org/apache/pulsar/tests/AnnotationListener.java @@ -27,9 +27,19 @@ @SuppressWarnings("rawtypes") public class AnnotationListener implements IAnnotationTransformer { +private static final int DEFAULT_TEST_TIMEOUT_MILLIS = 12; + +public AnnotationListener() { +System.out.println("Created annotation listener"); +} + @Override public void transform(ITestAnnotation annotation, Class testClass, Constructor testConstructor, Method testMethod) { annotation.setRetryAnalyzer(RetryAnalyzer.class); -} +// Enforce default test timeout +if (annotation.getTimeOut() == 0) { +annotation.setTimeOut(DEFAULT_TEST_TIMEOUT_MILLIS); +} +} } diff --git a/buildtools/src/main/java/org/apache/pulsar/tests/PulsarTestListener.java b/buildtools/src/main/java/org/apache/pulsar/tests/PulsarTestListener.java new file mode 100644 index 0..1b21a681b --- /dev/null +++ b/buildtools/src/main/java/org/apache/pulsar/tests/PulsarTestListener.java @@ -0,0 +1,73 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.pulsar.tests; + +import java.util.Arrays; + +import org.testng.ITestContext; +import org.testng.ITestListener; +import org.testng.ITestResult; +import org.testng.internal.thread.ThreadTimeoutException; + +public class PulsarTestListener implements ITestListener { + +@Override +public void onTestStart(ITestResult result) { +System.out.format("--- Starting test %s.%s(%s)---\n", result.getTestClass(), result.getTestName(), +Arrays.toString(result.getParameters())); +} + +@Override +public void onTestSuccess(ITestResult result) { +System.out.format("--- SUCCESS -- %s.%s(%s)---", result.getTestClass(), result.getTestName(), +Arrays.toString(result.getParameters())); +} + +@Override +public void onTestFailure(ITestResult result) { +System.out.format("! FAILURE-- %s.%s(%s)---\n", result.getTestClass(), result.getTestName(), +Arrays.toString(result.getParameters())); + +if (result.getThrowable() instanceof ThreadTimeoutException) { +System.out.println("== THREAD DUMPS =="); +System.out.println(ThreadDumpUtil.buildThreadDiagnosticString()); +} +} + +@Override +public void onTestSkipped(ITestResult result) { +System.out.format("~ SKIPPED -- %s.%s(%s)---", result.getTestClass(), result.getTestName(), +Arrays.toString(result.getParameters())); +} + +@Override +public void onTestFailedButWithinSuccessPercentage(ITestResult result) { + +} + +@Override +public void onStart(ITestContext context) { + +} + +@Override +public void onFinish(ITestContext context) { + +} +} diff --git a/buildtools/src/main/java/org/apache/pulsar/tests/ThreadDumpUtil.java b/buildtools/src/main/java/org/apache/pulsar/tests/ThreadDumpUtil.java new file mode 100644 index 0..e5156d0ff --- /dev/null +++ b/buildtools/src/main/java/org/apache/pulsar/tests/ThreadDumpUtil.java @@ -0,0 +1,148 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + *
[incubator-pulsar] branch master updated: Avoid allocation of one PositionImpl object in write path (#1439)
This is an automated email from the ASF dual-hosted git repository. mmerli pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/incubator-pulsar.git The following commit(s) were added to refs/heads/master by this push: new 01a77aa Avoid allocation of one PositionImpl object in write path (#1439) 01a77aa is described below commit 01a77aa29d7c1a8376227211403bfec26b57abc7 Author: Matteo MerliAuthorDate: Mon Mar 26 15:24:12 2018 -0700 Avoid allocation of one PositionImpl object in write path (#1439) --- .../src/main/java/org/apache/bookkeeper/mledger/impl/OpAddEntry.java| 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/OpAddEntry.java b/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/OpAddEntry.java index b1f1d56..85c4da6 100644 --- a/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/OpAddEntry.java +++ b/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/OpAddEntry.java @@ -166,7 +166,7 @@ class OpAddEntry extends SafeRunnable implements AddCallback, CloseCallback { updateLatency(); AddEntryCallback cb = callbackUpdater.getAndSet(this, null); if (cb != null) { -cb.addComplete(PositionImpl.get(ledger.getId(), entryId), ctx); +cb.addComplete(lastEntry, ctx); ml.notifyCursors(); this.recycle(); } -- To stop receiving notification emails like this one, please contact mme...@apache.org.
[GitHub] merlimat commented on issue #1417: Replace cached data key from the map if present already
merlimat commented on issue #1417: Replace cached data key from the map if present already URL: https://github.com/apache/incubator-pulsar/pull/1417#issuecomment-376316438 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] merlimat closed pull request #1439: Avoid allocation of one PositionImpl object in write path
merlimat closed pull request #1439: Avoid allocation of one PositionImpl object in write path URL: https://github.com/apache/incubator-pulsar/pull/1439 This is a PR merged from a forked repository. As GitHub hides the original diff on merge, it is displayed below for the sake of provenance: As this is a foreign pull request (from a fork), the diff is supplied below (as it won't show otherwise due to GitHub magic): diff --git a/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/OpAddEntry.java b/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/OpAddEntry.java index b1f1d56a5..85c4da624 100644 --- a/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/OpAddEntry.java +++ b/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/OpAddEntry.java @@ -166,7 +166,7 @@ public void safeRun() { updateLatency(); AddEntryCallback cb = callbackUpdater.getAndSet(this, null); if (cb != null) { -cb.addComplete(PositionImpl.get(ledger.getId(), entryId), ctx); +cb.addComplete(lastEntry, ctx); ml.notifyCursors(); this.recycle(); } This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] srkukarni commented on issue #1447: Send Log statements to log Topic for Java Functions
srkukarni commented on issue #1447: Send Log statements to log Topic for Java Functions URL: https://github.com/apache/incubator-pulsar/pull/1447#issuecomment-376329446 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] erenavsarogullari opened a new pull request #1448: [Spark_Streaming] Extend Receiver UT Coverage
erenavsarogullari opened a new pull request #1448: [Spark_Streaming] Extend Receiver UT Coverage URL: https://github.com/apache/incubator-pulsar/pull/1448 ### Motivation Extending current `SparkStreamingPulsarReceiver` UT coverage - Default Settings - Negative cases (if `ClientConfiguration` or `ConsumerConfiguration` are not set) ### Modifications Adding 3 new UT cases to `SparkStreamingPulsarReceiverTest` ### Result Current UT coverage will have been extended. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] merlimat commented on issue #1417: Replace cached data key from the map if present already
merlimat commented on issue #1417: Replace cached data key from the map if present already URL: https://github.com/apache/incubator-pulsar/pull/1417#issuecomment-376316438 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] merlimat commented on issue #1362: More Pulsar Functions documentation
merlimat commented on issue #1362: More Pulsar Functions documentation URL: https://github.com/apache/incubator-pulsar/pull/1362#issuecomment-376315309 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[incubator-pulsar] branch master updated: Added Groovy DSL definition for C++/Java PR builds (#1444)
This is an automated email from the ASF dual-hosted git repository. mmerli pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/incubator-pulsar.git The following commit(s) were added to refs/heads/master by this push: new 02fede7 Added Groovy DSL definition for C++/Java PR builds (#1444) 02fede7 is described below commit 02fede7c2c1c6465fcdcd01105ac1966029e454c Author: Matteo MerliAuthorDate: Mon Mar 26 14:13:46 2018 -0700 Added Groovy DSL definition for C++/Java PR builds (#1444) * Added Groovy DSL definition for C++/Java PR builds * Added license check * Run PR builds in parallel * Use "Hadoop" label for Jenkins workers * Also run integration test concurrently --- .test-infra/jenkins/common_job_properties.groovy | 2 +- ...ests.groovy => job_pulsar_precommit_cpp.groovy} | 28 +++--- .../job_pulsar_precommit_integrationtests.groovy | 3 +++ ...ts.groovy => job_pulsar_precommit_java8.groovy} | 25 +++ .test-infra/jenkins/job_seed.groovy| 2 +- 5 files changed, 27 insertions(+), 33 deletions(-) diff --git a/.test-infra/jenkins/common_job_properties.groovy b/.test-infra/jenkins/common_job_properties.groovy index 08c883d..56e4df2 100644 --- a/.test-infra/jenkins/common_job_properties.groovy +++ b/.test-infra/jenkins/common_job_properties.groovy @@ -43,7 +43,7 @@ class common_job_properties { String branch = 'master', String jdkVersion = 'JDK 1.8 (latest)', int timeout = 200, - String jenkinsExecutorLabel = 'ubuntu') { + String jenkinsExecutorLabel = 'Hadoop') { // GitHub project. context.properties { githubProjectUrl('https://github.com/apache/incubator-pulsar/') diff --git a/.test-infra/jenkins/job_pulsar_precommit_integrationtests.groovy b/.test-infra/jenkins/job_pulsar_precommit_cpp.groovy similarity index 74% copy from .test-infra/jenkins/job_pulsar_precommit_integrationtests.groovy copy to .test-infra/jenkins/job_pulsar_precommit_cpp.groovy index 7a08418..8d3f496 100644 --- a/.test-infra/jenkins/job_pulsar_precommit_integrationtests.groovy +++ b/.test-infra/jenkins/job_pulsar_precommit_cpp.groovy @@ -19,35 +19,35 @@ import common_job_properties // This is the Java precommit which runs a maven install, and the current set of precommit tests. -freeStyleJob('pulsar_precommit_integrationtests') { -description('precommit integration test verification for pull requests of http://pulsar.apache.org;>Apache Pulsar.') +freeStyleJob('pulsar_precommit_cpp') { +description('precommit CPP/Python client tests verification for pull requests of http://pulsar.apache.org;>Apache Pulsar.') + +// Execute concurrent builds if necessary. +concurrentBuild() // Set common parameters. common_job_properties.setTopLevelMainJobProperties(delegate) // Sets that this is a PreCommit job. -common_job_properties.setPreCommit(delegate, 'Integration Tests') +common_job_properties.setPreCommit(delegate, 'C++ / Python Tests') steps { -shell('tests/scripts/pre-integ-tests.sh') - -// Build everything +// Build Java Code maven { // Set Maven parameters. common_job_properties.setMavenConfig(delegate) -goals('-B clean install -Pdocker') +goals('-B clean package') properties(skipTests: true, interactiveMode: false) } -maven { -// Set Maven parameters. -common_job_properties.setMavenConfig(delegate) -rootPOM('tests/pom.xml') -goals('-B test -DintegrationTests') -} +shell(''' +echo "Build C++ client library" +export CMAKE_ARGS="-DCMAKE_BUILD_TYPE=Debug -DBUILD_DYNAMIC_LIB=OFF" -shell('tests/scripts/post-integ-tests.sh') +pulsar-client-cpp/docker-build.sh +pulsar-client-cpp/docker-tests.sh +''') } publishers { diff --git a/.test-infra/jenkins/job_pulsar_precommit_integrationtests.groovy b/.test-infra/jenkins/job_pulsar_precommit_integrationtests.groovy index 7a08418..8a1bdf2 100644 --- a/.test-infra/jenkins/job_pulsar_precommit_integrationtests.groovy +++ b/.test-infra/jenkins/job_pulsar_precommit_integrationtests.groovy @@ -25,6 +25,9 @@ freeStyleJob('pulsar_precommit_integrationtests') { // Set common parameters. common_job_properties.setTopLevelMainJobProperties(delegate) +// Execute concurrent builds if necessary. +concurrentBuild() + // Sets that this is a PreCommit job. common_job_properties.setPreCommit(delegate, 'Integration Tests') diff --git a/.test-infra/jenkins/job_pulsar_precommit_integrationtests.groovy
[incubator-pulsar] branch pr-jobs deleted (was 46efedc)
This is an automated email from the ASF dual-hosted git repository. mmerli pushed a change to branch pr-jobs in repository https://gitbox.apache.org/repos/asf/incubator-pulsar.git. was 46efedc Also run integration test concurrently The revisions that were on this branch are still contained in other references; therefore, this change does not discard any commits from the repository. -- To stop receiving notification emails like this one, please contact mme...@apache.org.
[GitHub] merlimat closed pull request #1444: Added Groovy DSL definition for C++/Java PR builds
merlimat closed pull request #1444: Added Groovy DSL definition for C++/Java PR builds URL: https://github.com/apache/incubator-pulsar/pull/1444 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] merlimat closed pull request #1442: Revert "Recycle OpAddEntry when ledger-ops fails (#1415)"
merlimat closed pull request #1442: Revert "Recycle OpAddEntry when ledger-ops fails (#1415)" URL: https://github.com/apache/incubator-pulsar/pull/1442 This is a PR merged from a forked repository. As GitHub hides the original diff on merge, it is displayed below for the sake of provenance: As this is a foreign pull request (from a fork), the diff is supplied below (as it won't show otherwise due to GitHub magic): diff --git a/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/OpAddEntry.java b/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/OpAddEntry.java index 701757bca..b1f1d56a5 100644 --- a/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/OpAddEntry.java +++ b/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/OpAddEntry.java @@ -99,7 +99,6 @@ public void failed(ManagedLedgerException e) { if (cb != null) { cb.addFailed(e, ctx); ml.mbean.recordAddEntryError(); -this.recycle(); } } This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[incubator-pulsar] branch master updated: Revert "Recycle OpAddEntry when ledger-ops fails (#1415)" (#1442)
This is an automated email from the ASF dual-hosted git repository. mmerli pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/incubator-pulsar.git The following commit(s) were added to refs/heads/master by this push: new 358bbc2 Revert "Recycle OpAddEntry when ledger-ops fails (#1415)" (#1442) 358bbc2 is described below commit 358bbc20f151e2fb2f321ab42dce5f68b340bfd3 Author: Ivan KellyAuthorDate: Mon Mar 26 23:06:18 2018 +0200 Revert "Recycle OpAddEntry when ledger-ops fails (#1415)" (#1442) If the OpAddEntry is recycled before the callback is triggered, the callback is calling into a dead object. This reverts commit f5ffbbe1ed2b79be1caf4a59688eab006e932ced. --- .../src/main/java/org/apache/bookkeeper/mledger/impl/OpAddEntry.java | 1 - 1 file changed, 1 deletion(-) diff --git a/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/OpAddEntry.java b/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/OpAddEntry.java index 701757b..b1f1d56 100644 --- a/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/OpAddEntry.java +++ b/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/OpAddEntry.java @@ -99,7 +99,6 @@ class OpAddEntry extends SafeRunnable implements AddCallback, CloseCallback { if (cb != null) { cb.addFailed(e, ctx); ml.mbean.recordAddEntryError(); -this.recycle(); } } -- To stop receiving notification emails like this one, please contact mme...@apache.org.
[GitHub] srkukarni commented on issue #1447: Send Log statements to log Topic for Java Functions
srkukarni commented on issue #1447: Send Log statements to log Topic for Java Functions URL: https://github.com/apache/incubator-pulsar/pull/1447#issuecomment-376304711 @sijie @merlimat This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[incubator-pulsar] branch pr-jobs updated: Also run integration test concurrently
This is an automated email from the ASF dual-hosted git repository. mmerli pushed a commit to branch pr-jobs in repository https://gitbox.apache.org/repos/asf/incubator-pulsar.git The following commit(s) were added to refs/heads/pr-jobs by this push: new 46efedc Also run integration test concurrently 46efedc is described below commit 46efedca3c549fd0b7da35182a477e9bb466500e Author: Matteo MerliAuthorDate: Mon Mar 26 13:18:32 2018 -0700 Also run integration test concurrently --- .test-infra/jenkins/job_pulsar_precommit_integrationtests.groovy | 3 +++ 1 file changed, 3 insertions(+) diff --git a/.test-infra/jenkins/job_pulsar_precommit_integrationtests.groovy b/.test-infra/jenkins/job_pulsar_precommit_integrationtests.groovy index 7a08418..8a1bdf2 100644 --- a/.test-infra/jenkins/job_pulsar_precommit_integrationtests.groovy +++ b/.test-infra/jenkins/job_pulsar_precommit_integrationtests.groovy @@ -25,6 +25,9 @@ freeStyleJob('pulsar_precommit_integrationtests') { // Set common parameters. common_job_properties.setTopLevelMainJobProperties(delegate) +// Execute concurrent builds if necessary. +concurrentBuild() + // Sets that this is a PreCommit job. common_job_properties.setPreCommit(delegate, 'Integration Tests') -- To stop receiving notification emails like this one, please contact mme...@apache.org.
[GitHub] merlimat commented on issue #1442: Revert "Recycle OpAddEntry when ledger-ops fails (#1415)"
merlimat commented on issue #1442: Revert "Recycle OpAddEntry when ledger-ops fails (#1415)" URL: https://github.com/apache/incubator-pulsar/pull/1442#issuecomment-376296710 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[incubator-pulsar] branch pr-jobs updated (5e6bee7 -> fa2347f)
This is an automated email from the ASF dual-hosted git repository. mmerli pushed a change to branch pr-jobs in repository https://gitbox.apache.org/repos/asf/incubator-pulsar.git. from 5e6bee7 Added license check add fa2347f Run PR builds in parallel No new revisions were added by this update. Summary of changes: .test-infra/jenkins/job_pulsar_precommit_cpp.groovy | 3 +++ .test-infra/jenkins/job_pulsar_precommit_java8.groovy | 3 +++ 2 files changed, 6 insertions(+) -- To stop receiving notification emails like this one, please contact mme...@apache.org.
[GitHub] sijie commented on a change in pull request #1441: Update the bookkeeper configuration file
sijie commented on a change in pull request #1441: Update the bookkeeper configuration file URL: https://github.com/apache/incubator-pulsar/pull/1441#discussion_r177213493 ## File path: conf/bookkeeper.conf ## @@ -39,101 +68,231 @@ bookiePort=3181 # set the listening interface. allowLoopback=false -# Configure a specific hostname or IP address that the bookie should use to advertise itself to -# clients. If not set, bookie will advertised its own IP address or hostname, depending on the -# listeningInterface and `seHostNameAsBookieID settings. -advertisedAddress= +# Interval to watch whether bookie is dead or not, in milliseconds +bookieDeathWatchInterval=1000 -# Directory Bookkeeper outputs its write ahead log -journalDirectory=data/bookkeeper/journal +# When entryLogPerLedgerEnabled is enabled, checkpoint doesn't happens +# when a new active entrylog is created / previous one is rolled over. +# Instead SyncThread checkpoints periodically with 'flushInterval' delay +# (in milliseconds) in between executions. Checkpoint flushes both ledger +# entryLogs and ledger index pages to disk. +# Flushing entrylog and index files will introduce much random disk I/O. +# If separating journal dir and ledger dirs each on different devices, +# flushing would not affect performance. But if putting journal dir +# and ledger dirs on same device, performance degrade significantly +# on too frequent flushing. You can consider increment flush interval +# to get better performance, but you need to pay more time on bookie +# server restart after failure. +# This config is used only when entryLogPerLedgerEnabled is enabled. +flushInterval=6 -# Directory Bookkeeper outputs ledger snapshots -# could define multi directories to store snapshots, separated by ',' -# For example: -# ledgerDirectories=/tmp/bk1-data,/tmp/bk2-data -# -# Ideally ledger dirs and journal dir are each in a differet device, -# which reduce the contention between random i/o and sequential write. -# It is possible to run with a single disk, but performance will be significantly lower. -ledgerDirectories=data/bookkeeper/ledgers -# Directories to store index files. If not specified, will use ledgerDirectories to store. -# indexDirectories=data/bookkeeper/ledgers +# Allow the expansion of bookie storage capacity. Newly added ledger +# and index dirs must be empty. +# allowStorageExpansion=false -# Ledger Manager Class -# What kind of ledger manager is used to manage how ledgers are stored, managed -# and garbage collected. Try to read 'BookKeeper Internals' for detail info. -ledgerManagerType=hierarchical +# Whether the bookie should use its hostname to register with the +# co-ordination service(eg: Zookeeper service). +# When false, bookie will use its ipaddress for the registration. +# Defaults to false. +useHostNameAsBookieID=false -# Root zookeeper path to store ledger metadata -# This parameter is used by zookeeper-based ledger manager as a root znode to -# store all ledgers. -zkLedgersRootPath=/ledgers +# Whether the bookie is allowed to use an ephemeral port (port 0) as its +# server port. By default, an ephemeral port is not allowed. +# Using an ephemeral port as the service port usually indicates a configuration +# error. However, in unit tests, using an ephemeral port will address port +# conflict problems and allow running tests in parallel. +# allowEphemeralPorts=false -# Ledger storage implementation class -ledgerStorageClass=org.apache.bookkeeper.bookie.storage.ldb.DbLedgerStorage +# Whether allow the bookie to listen for BookKeeper clients executed on the local JVM. +# enableLocalTransport=false -# Enable/Disable entry logger preallocation -entryLogFilePreallocationEnabled=true +# Whether allow the bookie to disable bind on network interfaces, +# this bookie will be available only to BookKeeper clients executed on the local JVM. +# disableServerSocketBind=false -# Max file size of entry logger, in bytes -# A new entry log file will be created when the old one reaches the file size limitation -logSizeLimit=1073741824 +# The number of bytes we should use as chunk allocation for +# org.apache.bookkeeper.bookie.SkipListArena +# skipListArenaChunkSize=4194304 -# Threshold of minor compaction -# For those entry log files whose remaining size percentage reaches below -# this threshold will be compacted in a minor compaction. -# If it is set to less than zero, the minor compaction is disabled. -minorCompactionThreshold=0.2 +# The max size we should allocate from the skiplist arena. Allocations +# larger than this should be allocated directly by the VM to avoid fragmentation. +# skipListArenaMaxAllocSize=131072 -# Interval to run minor compaction, in seconds -# If it is set to less than zero, the minor compaction is disabled. -minorCompactionInterval=3600 +# The bookie authentication provider factory class name. +# If this is null, no authentication will take place. +#
[GitHub] lucperkins commented on issue #1446: Pulsar 2.0 Documentation
lucperkins commented on issue #1446: Pulsar 2.0 Documentation URL: https://github.com/apache/incubator-pulsar/issues/1446#issuecomment-376286514 - [ ] Message deduplication - [ ] Add "cookbooks" section with end-to-end instructions for specific tasks (like adding and removing nodes) - [ ] Cursors This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] sijie commented on issue #1446: Pulsar 2.0 Documentation
sijie commented on issue #1446: Pulsar 2.0 Documentation URL: https://github.com/apache/incubator-pulsar/issues/1446#issuecomment-376287138 @lucperkins added "message deduplication" This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] lucperkins commented on issue #1446: Pulsar 2.0 Documentation
lucperkins commented on issue #1446: Pulsar 2.0 Documentation URL: https://github.com/apache/incubator-pulsar/issues/1446#issuecomment-376286514 - [ ] Message deduplication This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] sijie opened a new issue #1446: Pulsar 2.0 Documentation
sijie opened a new issue #1446: Pulsar 2.0 Documentation URL: https://github.com/apache/incubator-pulsar/issues/1446 Created a master issue for tracking all the documentation tasks for pulsar 2.0. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] merlimat commented on issue #1444: Added Groovy DSL definition for C++/Java PR builds
merlimat commented on issue #1444: Added Groovy DSL definition for C++/Java PR builds URL: https://github.com/apache/incubator-pulsar/pull/1444#issuecomment-376284665 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] merlimat commented on issue #1444: Added Groovy DSL definition for C++/Java PR builds
merlimat commented on issue #1444: Added Groovy DSL definition for C++/Java PR builds URL: https://github.com/apache/incubator-pulsar/pull/1444#issuecomment-376284665 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] lucperkins opened a new pull request #1445: Add version warning for latest vs. 2.0
lucperkins opened a new pull request #1445: Add version warning for latest vs. 2.0 URL: https://github.com/apache/incubator-pulsar/pull/1445 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] merlimat commented on a change in pull request #1441: Update the bookkeeper configuration file
merlimat commented on a change in pull request #1441: Update the bookkeeper configuration file URL: https://github.com/apache/incubator-pulsar/pull/1441#discussion_r177199255 ## File path: conf/bookkeeper.conf ## @@ -39,101 +68,231 @@ bookiePort=3181 # set the listening interface. allowLoopback=false -# Configure a specific hostname or IP address that the bookie should use to advertise itself to -# clients. If not set, bookie will advertised its own IP address or hostname, depending on the -# listeningInterface and `seHostNameAsBookieID settings. -advertisedAddress= +# Interval to watch whether bookie is dead or not, in milliseconds +bookieDeathWatchInterval=1000 -# Directory Bookkeeper outputs its write ahead log -journalDirectory=data/bookkeeper/journal +# When entryLogPerLedgerEnabled is enabled, checkpoint doesn't happens +# when a new active entrylog is created / previous one is rolled over. +# Instead SyncThread checkpoints periodically with 'flushInterval' delay +# (in milliseconds) in between executions. Checkpoint flushes both ledger +# entryLogs and ledger index pages to disk. +# Flushing entrylog and index files will introduce much random disk I/O. +# If separating journal dir and ledger dirs each on different devices, +# flushing would not affect performance. But if putting journal dir +# and ledger dirs on same device, performance degrade significantly +# on too frequent flushing. You can consider increment flush interval +# to get better performance, but you need to pay more time on bookie +# server restart after failure. +# This config is used only when entryLogPerLedgerEnabled is enabled. +flushInterval=6 -# Directory Bookkeeper outputs ledger snapshots -# could define multi directories to store snapshots, separated by ',' -# For example: -# ledgerDirectories=/tmp/bk1-data,/tmp/bk2-data -# -# Ideally ledger dirs and journal dir are each in a differet device, -# which reduce the contention between random i/o and sequential write. -# It is possible to run with a single disk, but performance will be significantly lower. -ledgerDirectories=data/bookkeeper/ledgers -# Directories to store index files. If not specified, will use ledgerDirectories to store. -# indexDirectories=data/bookkeeper/ledgers +# Allow the expansion of bookie storage capacity. Newly added ledger +# and index dirs must be empty. +# allowStorageExpansion=false -# Ledger Manager Class -# What kind of ledger manager is used to manage how ledgers are stored, managed -# and garbage collected. Try to read 'BookKeeper Internals' for detail info. -ledgerManagerType=hierarchical +# Whether the bookie should use its hostname to register with the +# co-ordination service(eg: Zookeeper service). +# When false, bookie will use its ipaddress for the registration. +# Defaults to false. +useHostNameAsBookieID=false -# Root zookeeper path to store ledger metadata -# This parameter is used by zookeeper-based ledger manager as a root znode to -# store all ledgers. -zkLedgersRootPath=/ledgers +# Whether the bookie is allowed to use an ephemeral port (port 0) as its +# server port. By default, an ephemeral port is not allowed. +# Using an ephemeral port as the service port usually indicates a configuration +# error. However, in unit tests, using an ephemeral port will address port +# conflict problems and allow running tests in parallel. +# allowEphemeralPorts=false -# Ledger storage implementation class -ledgerStorageClass=org.apache.bookkeeper.bookie.storage.ldb.DbLedgerStorage +# Whether allow the bookie to listen for BookKeeper clients executed on the local JVM. +# enableLocalTransport=false -# Enable/Disable entry logger preallocation -entryLogFilePreallocationEnabled=true +# Whether allow the bookie to disable bind on network interfaces, +# this bookie will be available only to BookKeeper clients executed on the local JVM. +# disableServerSocketBind=false -# Max file size of entry logger, in bytes -# A new entry log file will be created when the old one reaches the file size limitation -logSizeLimit=1073741824 +# The number of bytes we should use as chunk allocation for +# org.apache.bookkeeper.bookie.SkipListArena +# skipListArenaChunkSize=4194304 -# Threshold of minor compaction -# For those entry log files whose remaining size percentage reaches below -# this threshold will be compacted in a minor compaction. -# If it is set to less than zero, the minor compaction is disabled. -minorCompactionThreshold=0.2 +# The max size we should allocate from the skiplist arena. Allocations +# larger than this should be allocated directly by the VM to avoid fragmentation. +# skipListArenaMaxAllocSize=131072 -# Interval to run minor compaction, in seconds -# If it is set to less than zero, the minor compaction is disabled. -minorCompactionInterval=3600 +# The bookie authentication provider factory class name. +# If this is null, no authentication will take place. +#
[GitHub] merlimat commented on issue #1442: Revert "Recycle OpAddEntry when ledger-ops fails (#1415)"
merlimat commented on issue #1442: Revert "Recycle OpAddEntry when ledger-ops fails (#1415)" URL: https://github.com/apache/incubator-pulsar/pull/1442#issuecomment-376273012 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] merlimat commented on issue #1442: Revert "Recycle OpAddEntry when ledger-ops fails (#1415)"
merlimat commented on issue #1442: Revert "Recycle OpAddEntry when ledger-ops fails (#1415)" URL: https://github.com/apache/incubator-pulsar/pull/1442#issuecomment-376273012 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] merlimat commented on issue #1442: Revert "Recycle OpAddEntry when ledger-ops fails (#1415)"
merlimat commented on issue #1442: Revert "Recycle OpAddEntry when ledger-ops fails (#1415)" URL: https://github.com/apache/incubator-pulsar/pull/1442#issuecomment-376271412 @rdhabalia I verified that test stops failing after removing the recycle. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] merlimat commented on issue #1442: Revert "Recycle OpAddEntry when ledger-ops fails (#1415)"
merlimat commented on issue #1442: Revert "Recycle OpAddEntry when ledger-ops fails (#1415)" URL: https://github.com/apache/incubator-pulsar/pull/1442#issuecomment-376266483 @rdhabalia I think this is probably what's happening: ``` 18:19:24.743 [BookKeeperClientWorker-OrderedSafeExecutor-0-0:org.apache.bookkeeper.common.util.SafeRunnable@38] ERROR org.apache.bookkeeper.common.util.SafeRunnable - Unexpected throwable caught java.lang.NullPointerException: null at org.apache.bookkeeper.mledger.impl.OpAddEntry.addComplete(OpAddEntry.java:108) ~[classes/:?] at org.apache.bookkeeper.client.AsyncCallback$AddCallback.addCompleteWithLatency(AsyncCallback.java:91) ~[bookkeeper-server-shaded-4.7.0-SNAPSHOT.jar:4.7.0-SNAPSHOT] at org.apache.bookkeeper.client.PendingAddOp.submitCallback(PendingAddOp.java:361) ~[bookkeeper-server-shaded-4.7.0-SNAPSHOT.jar:4.7.0-SNAPSHOT] at org.apache.bookkeeper.client.LedgerHandle.errorOutPendingAdds(LedgerHandle.java:1531) ~[bookkeeper-server-shaded-4.7.0-SNAPSHOT.jar:4.7.0-SNAPSHOT] at org.apache.bookkeeper.client.LedgerHandle$5.safeRun(LedgerHandle.java:508) ~[bookkeeper-server-shaded-4.7.0-SNAPSHOT.jar:4.7.0-SNAPSHOT] at org.apache.bookkeeper.common.util.SafeRunnable.run(SafeRunnable.java:36) [bookkeeper-server-shaded-4.7.0-SNAPSHOT.jar:4.7.0-SNAPSHOT] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_152] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_152] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_152] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [?:1.8.0_152] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_152] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_152] at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [netty-all-4.1.21.Final.jar:4.1.21.Final] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_152] ``` See https://builds.apache.org/job/pulsar_precommit_java8/2/testReport/org.apache.bookkeeper.mledger.impl/ManagedLedgerBkTest/managedLedgerClosed/ This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] sijie commented on issue #1424: Improve batch message acking by removing batch message tracker
sijie commented on issue #1424: Improve batch message acking by removing batch message tracker URL: https://github.com/apache/incubator-pulsar/pull/1424#issuecomment-376262456 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] sijie commented on issue #1440: Update default values for a few publisher settings
sijie commented on issue #1440: Update default values for a few publisher settings URL: https://github.com/apache/incubator-pulsar/pull/1440#issuecomment-376261949 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] sijie commented on issue #1441: Update the bookkeeper configuration file
sijie commented on issue #1441: Update the bookkeeper configuration file URL: https://github.com/apache/incubator-pulsar/pull/1441#issuecomment-376261402 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] merlimat opened a new pull request #1444: Added Groovy DSL definition for C++/Java PR builds
merlimat opened a new pull request #1444: Added Groovy DSL definition for C++/Java PR builds URL: https://github.com/apache/incubator-pulsar/pull/1444 ### Motivation Added Jenkins DSL definitions for new PR validation jobs. This will create one job for Java8 tests and one for C++/Python which will be running in parallel. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] saandrews commented on issue #1417: Replace cached data key from the map if present already
saandrews commented on issue #1417: Replace cached data key from the map if present already URL: https://github.com/apache/incubator-pulsar/pull/1417#issuecomment-376242499 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[incubator-pulsar] branch master updated: Fixed C++ formatting (#1443)
This is an automated email from the ASF dual-hosted git repository. mmerli pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/incubator-pulsar.git The following commit(s) were added to refs/heads/master by this push: new 3360b14 Fixed C++ formatting (#1443) 3360b14 is described below commit 3360b1485a950e473ca6bfb6c7d65b8d868174ca Author: Matteo MerliAuthorDate: Mon Mar 26 10:13:24 2018 -0700 Fixed C++ formatting (#1443) --- pulsar-client-cpp/lib/BatchAcknowledgementTracker.cc | 11 ++- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git a/pulsar-client-cpp/lib/BatchAcknowledgementTracker.cc b/pulsar-client-cpp/lib/BatchAcknowledgementTracker.cc index 0dcf3be..e780e27 100644 --- a/pulsar-client-cpp/lib/BatchAcknowledgementTracker.cc +++ b/pulsar-client-cpp/lib/BatchAcknowledgementTracker.cc @@ -108,11 +108,12 @@ bool BatchAcknowledgementTracker::isBatchReady(const MessageId& msgID, const proto::CommandAck_AckType ackType) { Lock lock(mutex_); // Remove batch index -MessageId batchMessageId = MessageId(msgID.partition(), msgID.ledgerId(), msgID.entryId(), - -1 /* Batch index */); +MessageId batchMessageId = +MessageId(msgID.partition(), msgID.ledgerId(), msgID.entryId(), -1 /* Batch index */); TrackerMap::iterator pos = trackerMap_.find(batchMessageId); -if (pos == trackerMap_.end() || std::find(sendList_.begin(), sendList_.end(), batchMessageId) != sendList_.end()) { +if (pos == trackerMap_.end() || +std::find(sendList_.begin(), sendList_.end(), batchMessageId) != sendList_.end()) { LOG_DEBUG( "Batch is ready since message present in sendList_ or not present in trackerMap_ [message ID = " << batchMessageId << "]"); @@ -146,8 +147,8 @@ const MessageId BatchAcknowledgementTracker::getGreatestCumulativeAckReady(const Lock lock(mutex_); // Remove batch index -MessageId batchMessageId = MessageId(messageId.partition(), messageId.ledgerId(), - messageId.entryId(), -1 /* Batch index */); +MessageId batchMessageId = +MessageId(messageId.partition(), messageId.ledgerId(), messageId.entryId(), -1 /* Batch index */); TrackerMap::iterator pos = trackerMap_.find(batchMessageId); // element not found -- To stop receiving notification emails like this one, please contact mme...@apache.org.
[GitHub] merlimat closed pull request #1443: Fixed C++ formatting
merlimat closed pull request #1443: Fixed C++ formatting URL: https://github.com/apache/incubator-pulsar/pull/1443 This is a PR merged from a forked repository. As GitHub hides the original diff on merge, it is displayed below for the sake of provenance: As this is a foreign pull request (from a fork), the diff is supplied below (as it won't show otherwise due to GitHub magic): diff --git a/pulsar-client-cpp/lib/BatchAcknowledgementTracker.cc b/pulsar-client-cpp/lib/BatchAcknowledgementTracker.cc index 0dcf3be95..e780e2742 100644 --- a/pulsar-client-cpp/lib/BatchAcknowledgementTracker.cc +++ b/pulsar-client-cpp/lib/BatchAcknowledgementTracker.cc @@ -108,11 +108,12 @@ bool BatchAcknowledgementTracker::isBatchReady(const MessageId& msgID, const proto::CommandAck_AckType ackType) { Lock lock(mutex_); // Remove batch index -MessageId batchMessageId = MessageId(msgID.partition(), msgID.ledgerId(), msgID.entryId(), - -1 /* Batch index */); +MessageId batchMessageId = +MessageId(msgID.partition(), msgID.ledgerId(), msgID.entryId(), -1 /* Batch index */); TrackerMap::iterator pos = trackerMap_.find(batchMessageId); -if (pos == trackerMap_.end() || std::find(sendList_.begin(), sendList_.end(), batchMessageId) != sendList_.end()) { +if (pos == trackerMap_.end() || +std::find(sendList_.begin(), sendList_.end(), batchMessageId) != sendList_.end()) { LOG_DEBUG( "Batch is ready since message present in sendList_ or not present in trackerMap_ [message ID = " << batchMessageId << "]"); @@ -146,8 +147,8 @@ const MessageId BatchAcknowledgementTracker::getGreatestCumulativeAckReady(const Lock lock(mutex_); // Remove batch index -MessageId batchMessageId = MessageId(messageId.partition(), messageId.ledgerId(), - messageId.entryId(), -1 /* Batch index */); +MessageId batchMessageId = +MessageId(messageId.partition(), messageId.ledgerId(), messageId.entryId(), -1 /* Batch index */); TrackerMap::iterator pos = trackerMap_.find(batchMessageId); // element not found This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] rdhabalia commented on issue #1442: Revert "Recycle OpAddEntry when ledger-ops fails (#1415)"
rdhabalia commented on issue #1442: Revert "Recycle OpAddEntry when ledger-ops fails (#1415)" URL: https://github.com/apache/incubator-pulsar/pull/1442#issuecomment-376236794 > the callback is calling into a dead object It seems, `cb.addFailed(e, ctx)` is being triggered before it recycles `OpAddEntry`. do we use `OpAddEntry` anywhere after recycling it? is it possible to provide stacktrace where you see this behavior? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] merlimat opened a new pull request #1443: Fixed C++ formatting
merlimat opened a new pull request #1443: Fixed C++ formatting URL: https://github.com/apache/incubator-pulsar/pull/1443 ### Motivation Recent commit merged broke the formatting. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] ivankelly opened a new pull request #1442: Revert "Recycle OpAddEntry when ledger-ops fails (#1415)"
ivankelly opened a new pull request #1442: Revert "Recycle OpAddEntry when ledger-ops fails (#1415)" URL: https://github.com/apache/incubator-pulsar/pull/1442 If the OpAddEntry is recycled before the callback is triggered, the callback is calling into a dead object. This reverts commit f5ffbbe1ed2b79be1caf4a59688eab006e932ced. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] ivankelly commented on issue #1415: Recycle OpAddEntry when ledger-ops fails
ivankelly commented on issue #1415: Recycle OpAddEntry when ledger-ops fails URL: https://github.com/apache/incubator-pulsar/pull/1415#issuecomment-376201587 @rdhabalia @merlimat This change is causing flake in ManagedLedgerBkTest#managedLedgerClosed If you close the ledger, the op is failed and the then recycled. The object may still be references in bookkeeper though, so when the callback triggers, it triggers on a dead object. It'd be much safer to not recycle on error until bookkeeper calls are cancellable. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] maskit commented on a change in pull request #1297: Add Configuration to set tlsClientAuth
maskit commented on a change in pull request #1297: Add Configuration to set tlsClientAuth URL: https://github.com/apache/incubator-pulsar/pull/1297#discussion_r177043153 ## File path: pulsar-broker/src/test/java/org/apache/pulsar/client/api/TlsProducerConsumerBase.java ## @@ -61,24 +67,42 @@ protected void internalSetUpForBroker() throws Exception { conf.setTlsEnabled(true); conf.setTlsCertificateFilePath(TLS_SERVER_CERT_FILE_PATH); conf.setTlsKeyFilePath(TLS_SERVER_KEY_FILE_PATH); +conf.setTlsTrustCertsFilePath(TLS_TRUST_CERT_FILE_PATH); conf.setClusterName(clusterName); +conf.setTlsRequireTrustedClientCertOnConnect(true); +Set tlsProtocols = Sets.newConcurrentHashSet(); +tlsProtocols.add("TLSv1.2"); +conf.setTlsProtocols(tlsProtocols); } -protected void internalSetUpForClient() throws Exception { -String lookupUrl = new URI("pulsar+ssl://localhost:" + BROKER_PORT_TLS).toString(); -pulsarClient = PulsarClient.builder().serviceUrl(lookupUrl).tlsTrustCertsFilePath(TLS_SERVER_CERT_FILE_PATH) -.enableTls(true).build(); +protected void internalSetUpForClient(boolean addCertificates, String lookupUrl) throws Exception { +ClientConfiguration clientConf = new ClientConfiguration(); +clientConf.setTlsTrustCertsFilePath(TLS_TRUST_CERT_FILE_PATH); +clientConf.setUseTls(true); +clientConf.setTlsAllowInsecureConnection(false); +if (addCertificates) { +MapauthParams = new HashMap<>(); +authParams.put("tlsCertFile", TLS_CLIENT_CERT_FILE_PATH); +authParams.put("tlsKeyFile", TLS_CLIENT_KEY_FILE_PATH); +clientConf.setAuthentication(AuthenticationTls.class.getName(), authParams); Review comment: > If TLS is enabled and we want to authenticate using the CNAME in the Client Certs (Default TLS authentication that pulsar provides) then we need to use AuthenticationTls irrespective of requireTrustedClientCertificate Correct. > If we want to use a different authentication method where the connection is TLS but we don't use client certs to authenticate then requireTrustedClientCertificate should be false. I’m OK with this, however, it depends on usecases. IMO, using both client certs for host authentication and a pair of usename and password for user authentication is a valid usecase. If we don’t allow to use requireTrustedClientCertificate without AuthenticationTls plugin, shouldn’t it be a configuration for the plugin but not for pulsar global? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] jai1 commented on a change in pull request #1297: Add Configuration to set tlsClientAuth
jai1 commented on a change in pull request #1297: Add Configuration to set tlsClientAuth URL: https://github.com/apache/incubator-pulsar/pull/1297#discussion_r177008602 ## File path: pulsar-broker/src/test/java/org/apache/pulsar/client/api/TlsProducerConsumerBase.java ## @@ -61,24 +67,42 @@ protected void internalSetUpForBroker() throws Exception { conf.setTlsEnabled(true); conf.setTlsCertificateFilePath(TLS_SERVER_CERT_FILE_PATH); conf.setTlsKeyFilePath(TLS_SERVER_KEY_FILE_PATH); +conf.setTlsTrustCertsFilePath(TLS_TRUST_CERT_FILE_PATH); conf.setClusterName(clusterName); +conf.setTlsRequireTrustedClientCertOnConnect(true); +Set tlsProtocols = Sets.newConcurrentHashSet(); +tlsProtocols.add("TLSv1.2"); +conf.setTlsProtocols(tlsProtocols); } -protected void internalSetUpForClient() throws Exception { -String lookupUrl = new URI("pulsar+ssl://localhost:" + BROKER_PORT_TLS).toString(); -pulsarClient = PulsarClient.builder().serviceUrl(lookupUrl).tlsTrustCertsFilePath(TLS_SERVER_CERT_FILE_PATH) -.enableTls(true).build(); +protected void internalSetUpForClient(boolean addCertificates, String lookupUrl) throws Exception { +ClientConfiguration clientConf = new ClientConfiguration(); +clientConf.setTlsTrustCertsFilePath(TLS_TRUST_CERT_FILE_PATH); +clientConf.setUseTls(true); +clientConf.setTlsAllowInsecureConnection(false); +if (addCertificates) { +MapauthParams = new HashMap<>(); +authParams.put("tlsCertFile", TLS_CLIENT_CERT_FILE_PATH); +authParams.put("tlsKeyFile", TLS_CLIENT_KEY_FILE_PATH); +clientConf.setAuthentication(AuthenticationTls.class.getName(), authParams); Review comment: > We need to use AuthenticationTls to SEND client certs whenever we enable requireTrustedClientCertificate If TLS is enabled and we want to authenticate using the CNAME in the Client Certs (Default TLS authentication that pulsar provides) then we need to use AuthenticationTls irrespective of requireTrustedClientCertificate If we want to use a different authentication method where the connection is TLS but we don't use client certs to authenticate then `requireTrustedClientCertificate` should be false. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] sijie opened a new pull request #1441: Update the bookkeeper configuration file
sijie opened a new pull request #1441: Update the bookkeeper configuration file URL: https://github.com/apache/incubator-pulsar/pull/1441 Update the bookkeeper configuration file to use the configuration file in bookkeeper (so that it includes all new settings available in 4.7). This change keeps all the settings same as before. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] jai1 commented on issue #1297: Add Configuration to set tlsClientAuth
jai1 commented on issue #1297: Add Configuration to set tlsClientAuth URL: https://github.com/apache/incubator-pulsar/pull/1297#issuecomment-376057341 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services