2019-04-04 10:20:17 UTC - Jayden Han: @Jayden Han has joined the channel ---- 2019-04-04 12:09:55 UTC - Roman Khlebnov: @Roman Khlebnov has joined the channel ---- 2019-04-04 12:35:38 UTC - Roman: @Roman has joined the channel ---- 2019-04-04 12:49:16 UTC - bhagesharora: @Jerry Peng yeah, its fine ---- 2019-04-04 15:14:12 UTC - Kendall Magesh-Davis: Is there a quick and dirty command to run with `pulsar-admin` to get the pulsar running version? ---- 2019-04-04 17:13:42 UTC - Jerry Peng: @bhagesharora are there any errors in the pulsar standalone cluster logs or the presto logs? ---- 2019-04-04 19:22:54 UTC - John Crawford: we’re seeing some issues with consumers not getting messages due to what looks like the broker not being able to read ledgers appropriately. here is an example of some logs we’re seeing in the broker: ``` 19:15:44.101 [bookkeeper-ml-scheduler-OrderedScheduler-7-0] WARN org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - [02efa741-a96f-4124-a463-ae13a704b8fc/pXeyflmLEUEnpjUa5radVPTTj44PSG1P/persistent/raw-lot_reference_codes]-6043 read entry timeout for 0-76 after 120 sec 19:15:44.102 [bookkeeper-ml-workers-OrderedExecutor-4-0] ERROR org.apache.bookkeeper.common.util.SafeRunnable - Unexpected throwable caught java.lang.ArrayIndexOutOfBoundsException: null ``` ---- 2019-04-04 19:23:26 UTC - John Crawford: any ideas? haven’t been able to find anything surfing through github issues except for maybe this one: <https://github.com/apache/pulsar/issues/3715> ---- 2019-04-04 21:13:07 UTC - Dustin: @Dustin has joined the channel ---- 2019-04-04 21:28:10 UTC - Dustin: Hi team. I heard about `Pulsar` from a recent Podcast. We are an archive for big(ish) data (15MB - 250GB files) and we distribute it with legacy software to different geolocations. Files to be PUT are sent to a single place and then replicated. We’re looking to update (replace) the legacy software.
Is using the messaging system in `Pulsar` a good fit for managing our archive files? It seems like it’s more meant for small messages rather than big data. ---- 2019-04-04 21:28:23 UTC - Dustin: The technology is extremely attractive, though. ---- 2019-04-04 21:29:44 UTC - Ali Ahmed: @Dustin pulsar is meant for messages current limit per message is 5MB, large files and be broken up and stored as bytes but that’s not a recommended way to use pulsar ---- 2019-04-04 21:30:08 UTC - Dustin: Much obliged, @Ali Ahmed. Thanks for the reply. ---- 2019-04-04 21:55:20 UTC - Jerry Peng: @John Crawford this is likely because there are failures in your bookkeeper cluster. More failures than tolerated. I would look in your bookie logs to see if there are any errors ---- 2019-04-05 00:03:02 UTC - Sree Vaddi: Welcome to Pulsar, @Dustin ---- 2019-04-05 00:03:46 UTC - Sree Vaddi: would network availability and speed a concern at those different geolocations ? ---- 2019-04-05 00:07:39 UTC - Matteo Merli: @Dustin While it’s not feasable to treat each big file as 1 Pulsar message, a common practice is to use messaging and replication to coordinate the copy across multiple regions. ---- 2019-04-05 00:08:31 UTC - Matteo Merli: Eg: the data copy happens through a different method (HTTP, ftp, etc..) but the metadata and eventual consistency is updated through Pulsar ---- 2019-04-05 00:11:23 UTC - Sree Vaddi: it's not recommended, but using messaging system for storage is not an uncommon practice. ---- 2019-04-05 00:11:35 UTC - Sree Vaddi: how long you need to store those files ? ---- 2019-04-05 05:11:59 UTC - Shivji Kumar Jha: I think you can NOT control quotas at topic level at the moment. You can manage it on the namespace level. <https://pulsar.apache.org/docs/en/cookbooks-retention-expiry/> <http://pulsar.apache.org/docs/v2.0.1-incubating/reference/CliTools/#pulsar-admin-namespaces-set-backlog-quota> ---- 2019-04-05 07:48:20 UTC - Matti-Pekka Laaksonen: I mentioned before that I had problems starting the Pulsar proxy. I downgraded to version 2.2.1, and now it runs just fine without any other changes (same config templates and Ansible playbooks) ---- 2019-04-05 07:52:21 UTC - Matti-Pekka Laaksonen: Unfortunately I didn't have really any logs to work with, so I'm unable to file an issue with more details ----
