2019-09-04 11:47:29 UTC - vikash: How to update Pulsar standalone 2.4.0
to 2.4.1
----
2019-09-04 12:02:27 UTC - jia zhai: It would be easy. download new tar, unzip,
move data and config from 2.4.0 dir into 2.4.1 dir.
----
2019-09-04 12:38:14 UTC - vikash: thanks
----
2019-09-04 13:34:39 UTC - Ravi Shah: @Ravi Shah has joined the channel
----
2019-09-04 13:40:44 UTC - Ravi Shah: Hey Guys, I am new to Apache pulsar. Can
anyone tell me how can i set maxPendingMessages to infinite?
----
2019-09-04 13:41:14 UTC - Ravi Shah: and how can i make connection alive? Using
ping method or is there any alternative?
----
2019-09-04 13:43:00 UTC - Ravi Shah: Is there any recommendation for production
level pulsar configuration for high throughput?
----
2019-09-04 13:57:45 UTC - vikash: @jia zhai how to set max message size
----
2019-09-04 13:57:46 UTC - vikash: Text message size [1181966] exceeds maximum
size [1048576]
----
2019-09-04 13:58:17 UTC - vikash: i am getting this Error while sending
Max payload any limitation on max message we can send through pulsar
----
2019-09-04 14:02:56 UTC - vikash: i am using standalone
----
2019-09-04 14:16:16 UTC - balaji:
<https://github.com/apache/pulsar/issues/4721> Is there any plan to fix this
bug, or timeline, this is regarding pulsar producer as flink sink.
----
2019-09-04 14:56:47 UTC - David Kjerrumgaard: @vikash The `maxMessageSize`
property controls the maximum message size in Pulsar. Typically this is in the
`broker.conf` file, however since you are running in standalone mode you will
need to add that setting to the `standalone.conf` file.
----
2019-09-04 17:43:39 UTC - Bruno Bonnin: Hi, I'm making requests using the admin
API for creating a topic: `curl -XPUT
localhost:8080/admin/v2/persistent/public/default/test` and I always get a `405
Method Not Allowed` . What am I doing wrong?
----
2019-09-04 17:51:25 UTC - Devin G. Bost: I think the search box is broken
again. It's not working for me:
<http://pulsar.apache.org/docs/en/client-libraries-java/#configuring-messages>
----
2019-09-04 17:51:30 UTC - Devin G. Bost: @David Kjerrumgaard FYI.
----
2019-09-04 17:54:01 UTC - Devin G. Bost: Also, regarding that code example in
that link:
```
producer.newMessage()
.key("my-message-key")
.value("my-async-message".getBytes())
.property("my-key", "my-value")
.property("my-other-key", "my-other-value")
.send();
```
Is there any documentation on the difference between a "property" and a "key"
or "value"?
Someone we're working with is confused about the difference.
----
2019-09-04 17:55:10 UTC - Ali Ahmed: @Devin G. Bost key value are message
components, property is like a tag or metadata
----
2019-09-04 17:55:14 UTC - Devin G. Bost: My understanding is that properties
pass all the way through a pipeline, but I can understand how someone might not
understand the difference between a property/property-value and key/key-value.
----
2019-09-04 17:55:54 UTC - Devin G. Bost: In terms of the impact, when we
consume from a topic on the command line, only keys and values appear, right?
----
2019-09-04 17:56:42 UTC - Ali Ahmed: @Devin G. Bost Yes I don’t think we print
properties in the cli
----
2019-09-04 18:02:45 UTC - Poule: @Bruno Bonnin what happens if you set header
to {'Content-Type': 'application/json'}
----
2019-09-04 18:09:24 UTC - Devin G. Bost: @Ali Ahmed Let's say that you have
topic1 -> function1 -> topic2 -> function2 -> topic3
Let's say that a property is attached to a message going into topic1.
If that property is untouched, will it still exist on the message in topic3?
Or, does that property need to be re-attached in each function to ensure that
it remains on the message in topic3 (like a key/value would)?
----
2019-09-04 18:10:57 UTC - Ali Ahmed: I don’t think functions pass through
messages like that so the properties will have to be reinserted
----
2019-09-04 18:11:43 UTC - Devin G. Bost: So, then what's the functional
difference between properties and keys/values?
ok_hand : Poule
----
2019-09-04 18:16:50 UTC - Bruno Bonnin: Same error :disappointed:
----
2019-09-04 18:17:44 UTC - Bruno Bonnin: Is there any body to provide ?
----
2019-09-04 18:22:04 UTC - Poule: i don't think so
----
2019-09-04 18:23:05 UTC - Bruno Bonnin: I'm going to check the code of pulsar
:wink:
----
2019-09-04 18:23:40 UTC - Bruno Bonnin: maybe my url is not good, or any
missing parameters
----
2019-09-04 18:30:48 UTC - Poule: your command works on my side
----
2019-09-04 18:43:20 UTC - Jerry Peng: @Devin G. Bost properties of a message
was designed to serve as place to store metadata about the message. Key/Value
of the message is the actual payload/content. Think of properties as headers
in HTTP
----
2019-09-04 18:43:58 UTC - Zhenhao Li: hi team, I have a very general question.
Does Pulsar have any GDPR features?
----
2019-09-04 18:44:28 UTC - Zhenhao Li: e.g. allow personal data to be deleted at
a user’s request
----
2019-09-04 18:45:34 UTC - Zhenhao Li: to me, that translates to deletion based
on key
----
2019-09-04 18:49:33 UTC - Bruno Bonnin: I'm using pulsar standalone without
security config
----
2019-09-04 18:57:07 UTC - David Kjerrumgaard: @Zhenhao Li There is nothing
built into the platform specifically for GDPR, but in general data retention
policies could be used to ensure that data in topics that hold user data are
automatically purged periodically. If you are using Pulsar as just an
intermediary between the source and longer-term data repositories, such as a
database, then this should be sufficient. however, if you are planning on using
Pulsar's tiered-storage capability to retain data for longer periods of time,
then you will have to implement similar features to what you would do for say a
database, etc and handle that at the application layer.
----
2019-09-04 19:04:54 UTC - Zhenhao Li: thanks! but it seems to me that GDPR
really requires deletion on the data layer
----
2019-09-04 19:09:24 UTC - David Kjerrumgaard: @Zhenhao Li You are correct. So
in the first case, the data is automatically deleted due to data retention
inside Pulsar.
----
2019-09-04 19:12:39 UTC - David Kjerrumgaard: @Zhenhao Li In order to delete
the data from Pulsar tiered-storage will require knowledge of where the data is
kept in pulsar. So the deletion process would have to be driven by an external
application that has that knowledge
----
2019-09-04 19:16:34 UTC - Ali Ahmed: @Devin G. Bost there is only one key/value
per message there can be multiple properties per message, messages are
internally hashed by key not properties, on the function side I think the
current behavior is to pass through properties as is it makes sense to make
that a configurable flag for the function runtime.
----
2019-09-04 19:29:56 UTC - David Kjerrumgaard: Keys can also be used for the new
key-shared subscription type to group messages together by key.
+1 : Devin G. Bost
----
2019-09-04 19:33:40 UTC - Zhenhao Li: so it is possible? what is the API for
that?
----
2019-09-04 19:33:56 UTC - Zhenhao Li: and each deletion will requires whole
partition scan?
----
2019-09-04 20:13:16 UTC - Bruno Bonnin: I'm using Pulsar v2.3.2, this API is
available from v2.4.0
----
2019-09-04 20:13:20 UTC - Bruno Bonnin: :disappointed:
----
2019-09-04 20:15:32 UTC - Retardust: is there any option to prevent topics
deletion because of inactivity?
----
2019-09-04 20:17:17 UTC - David Kjerrumgaard: @Retardust Yes in `broker.conf`
there is a `brokerDeleteInactiveTopicsEnabled` property that takes a boolean
value.
----
2019-09-04 20:18:47 UTC - Retardust: thanks. The default is unsafe:(
----
2019-09-04 20:27:08 UTC - Poule: oh
----
2019-09-04 20:41:40 UTC - Sijie Guo: for the people who is seeing search box
is not working, I don’t think it is broken at pulsar website. it is just
algolia doc search doesn’t return the search results. it happens each time when
we publish the documentation for new release. please try it in a few hours
until algolia refreshes its index
----
2019-09-04 20:46:38 UTC - David Kjerrumgaard: It depends on the schema, just
like a DB table. So you would have a GDPR sign-up site, etc where users come in
an request to be "forgotten". You then have a process that takes those user IDs
and deletes the records from the topics which holds the user data, but Pulsar
won't know which topics those are. The GDPR purging application would have to
know that.
----
2019-09-04 21:07:06 UTC - Kirill Merkushev: @Zhenhao Li vote for
<https://github.com/apache/pulsar/issues/5059> :slightly_smiling_face:
+1 : David Kjerrumgaard, Zhenhao Li
----
2019-09-04 21:59:07 UTC - Horus Collins: @Horus Collins has joined the channel
----
2019-09-04 22:10:46 UTC - Sijie Guo: actually for people searching doc at
<https://pulsar.apache.org>, please switch to 2.4.0 or master :
<http://pulsar.apache.org/docs/en/2.4.0/standalone/>
The search for all old releases is working as expected. The index for 2.4.1 is
not yet built by algolia doc search.
+1 : Devin G. Bost
----
2019-09-04 23:29:58 UTC - Devin G. Bost: Does anyone have an example of a Go
function (using the Pulsar Function API)?
This example:
<https://pulsar.apache.org/docs/en/functions-quickstart/#write-and-run-a-go-function>
is pretty bare bones.
----
2019-09-04 23:35:48 UTC - Devin G. Bost: Also, I noticed that the context
object has a very different signature than the context object for Pulsar
functions in Java, so I'd like some guidance on how to grab userConfig values
from the context in Go.
----
2019-09-04 23:43:10 UTC - Devin G. Bost:
----
2019-09-04 23:52:08 UTC - Devin G. Bost: Looks like that context object is a Go
interface, not a Pulsar interface.
Based on the documentation here:
<https://github.com/apache/pulsar/wiki/PIP-32:-Go-Function-API,-Instance-and-LocalRun>
it looks like I want the `FunctionContext` object, not the `Context` object.
How do I access the `FunctionContext` object from my Pulsar function in Go?
----
2019-09-04 23:59:49 UTC - Devin G. Bost: I just found an example here:
<https://github.com/apache/pulsar/blob/master/pulsar-function-go/examples/contextFunc.go>
though I'm not sure why I don't get autocomplete.
----
2019-09-04 23:59:52 UTC - Devin G. Bost:
----
2019-09-05 03:46:30 UTC - Lewinma: @Lewinma has joined the channel
----
2019-09-05 05:20:23 UTC - Kim Christian Gaarder: I tried to use Pulsar SQL with
presto jdbc driver to search topic for a specific message, then to reset a
subscription to that message using seek. But when I call seek I get exceptions
like “Error when resetting subscription: 99:0
----
2019-09-05 05:23:19 UTC - Kim Christian Gaarder: Here is my code to extract the
message-id from jdbc, which is the message-id instance passed to seek. I
suspect maybe the way that I construct the message-id instance could affect
seek ?
----
2019-09-05 05:39:05 UTC - Sijie Guo: Go Function was designed in a way that is
closer to go developer’s habits. regarding `Context` and `FunctionContext`,
@xiaolong.ran can chime in with more details.
----
2019-09-05 05:41:49 UTC - xiaolong.ran: ```
func contextFunc(ctx context.Context) {
if fc, ok := pf.FromContext(ctx); ok {
fmt.Printf("function ID is:%s, ", fc.GetFuncID())
fmt.Printf("function version is:%s\n", fc.GetFuncVersion())
}
}
```
----
2019-09-05 05:45:15 UTC - xiaolong.ran: The `context` in the go language allows
the user to customize the `context` and add the required context object to the
`context.context`. We can get it from the `FromContext`.
----
2019-09-05 05:46:24 UTC - xiaolong.ran:
<https://golang.org/src/context/context.go#L61>
----
2019-09-05 05:54:07 UTC - Retardust: And now... search on
<http://pulsar.apache.org|pulsar.apache.org> is broken(
----
2019-09-05 05:57:26 UTC - tuteng:
<http://pulsar.apache.org/docs/en/2.4.0/standalone/>
----
2019-09-05 07:12:24 UTC - Ravi Shah: Got this error when more than 2 producers
try to connect
java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: Java heap
space
Any idea?
----
2019-09-05 07:12:41 UTC - Ravi Shah: I am using webSocket
----
2019-09-05 07:14:29 UTC - Zhenhao Li: great initiative
----
2019-09-05 07:17:46 UTC - Sijie Guo: @Retardust:
checkout my previous comment in the general slack channel:
---
actually for people searching doc at <https://pulsar.apache.org>, please switch
to 2.4.0 or master : <http://pulsar.apache.org/docs/en/2.4.0/standalone/>
The search for all old releases is working as expected. The index for 2.4.1 is
not yet built by algolia doc search.
---
<https://apache-pulsar.slack.com/archives/C5Z4T36F7/p1567635046238500>
---
Also if you are in dev@ or users@ pulsar mailing list, you should receive an
email about the updates. If not, try to subscribe to users@ to get updates from
Pulsar.
----
2019-09-05 08:19:20 UTC - Retardust: thanks!
----
