[jira] [Created] (ZOOKEEPER-3966) Model ZooKeeper data tree using RocksDB primitives to enable on disk data tree storage
Michael Han created ZOOKEEPER-3966: -- Summary: Model ZooKeeper data tree using RocksDB primitives to enable on disk data tree storage Key: ZOOKEEPER-3966 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-3966 Project: ZooKeeper Issue Type: Sub-task Components: server Reporter: Michael Han -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (ZOOKEEPER-3965) Add documentation for RocksDB Snap feature
Michael Han created ZOOKEEPER-3965: -- Summary: Add documentation for RocksDB Snap feature Key: ZOOKEEPER-3965 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-3965 Project: ZooKeeper Issue Type: Sub-task Components: documentation Reporter: Michael Han -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (ZOOKEEPER-3964) Introduce RocksDB snap and implement change data capture to enable incremental snapshot
Michael Han created ZOOKEEPER-3964: -- Summary: Introduce RocksDB snap and implement change data capture to enable incremental snapshot Key: ZOOKEEPER-3964 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-3964 Project: ZooKeeper Issue Type: Sub-task Components: rocksdb, server Reporter: Michael Han Assignee: Michael Han This is the first step of enabling on disk storage engine for ZooKeeper by extending the existing Snap interface and implement a RocksDB backed snapshot. Comparing to file based snapshot, RocksDB based snapshot is superior for big in memory data tree as it supports incremental snapshot by only serializing the changed data between snapshots. High level overview: * Extend Snap interface so every thing that's need serialize has a presence on the interface. * Implement RocksDB based snapshot, and bidirectional conversations between File based snapshot and RocksDB snapshot, for back / forward compatibility. * Change data capture is implemented by buffering transactions applied to data tree, and applied to RocksDB when processing each transaction. An incremental snapshot thus only requires RocksDB flush. ZK will always do a full snapshot when first loading the data tree during the start process. * By default, this feature is disabled. Users need opt in by explicitly specify a Java system property to instantiate RocksDBSnap at runtime. This work is based on top of the patch attached to ZOOKEEPER-3783 (kudos to Fangmin and co at FB), with some bug / test fixes and adjustment so it can cleanly apply to master branch. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (ZOOKEEPER-3963) Revamp and Log Connection Disconnect Reason
David Mollitor created ZOOKEEPER-3963: - Summary: Revamp and Log Connection Disconnect Reason Key: ZOOKEEPER-3963 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-3963 Project: ZooKeeper Issue Type: Improvement Reporter: David Mollitor Assignee: David Mollitor -- This message was sent by Atlassian Jira (v8.3.4#803005)
Re: Custom SSLContext
Hi Enrico, Yes, the goal is to be strict about what protocols and ciphers to allow. We have an SSLContext factory we use consistently across NiFi to provide a better security guarantee. On Wed, Oct 7, 2020 at 5:13 PM Enrico Olivelli wrote: > Nathan, > > Il Mer 7 Ott 2020, 23:06 Nathan Gough ha scritto: > > > Hi, > > > > I develop for Apache NiFi and was working on adding TLS to one of our > > clients that use Zookeeper. I was wondering if it's possible to inject a > > custom SSLContext similar in concept to this ticket: > > > > https://issues.apache.org/jira/browse/ZOOKEEPER-2643 > > > > I can't see a way to provide this through the ZooKeeperAdmin interface. > > > > Why do you need to inject a custom SSLContext? Can you explain a little > more? Is it about limiting ciphers or protocols > > Enrico > > > > Thanks! > > Nathan > > >
Re: Custom SSLContext
Nathan, Il Mer 7 Ott 2020, 23:06 Nathan Gough ha scritto: > Hi, > > I develop for Apache NiFi and was working on adding TLS to one of our > clients that use Zookeeper. I was wondering if it's possible to inject a > custom SSLContext similar in concept to this ticket: > > https://issues.apache.org/jira/browse/ZOOKEEPER-2643 > > I can't see a way to provide this through the ZooKeeperAdmin interface. > Why do you need to inject a custom SSLContext? Can you explain a little more? Is it about limiting ciphers or protocols Enrico > Thanks! > Nathan >
Custom SSLContext
Hi, I develop for Apache NiFi and was working on adding TLS to one of our clients that use Zookeeper. I was wondering if it's possible to inject a custom SSLContext similar in concept to this ticket: https://issues.apache.org/jira/browse/ZOOKEEPER-2643 I can't see a way to provide this through the ZooKeeperAdmin interface. Thanks! Nathan
Re: hacktoberfest
On Wed, Oct 7, 2020 at 4:12 PM Enrico Olivelli wrote: > > Il Mer 7 Ott 2020, 21:02 Christopher ha scritto: > > > I created https://github.com/apache/zookeeper/pull/1489 > > (https://issues.apache.org/jira/browse/ZOOKEEPER-3962) > > > Thank you The sooner this gets updated, the sooner Hacktoberfest participants will get "credit" for the PRs they have created, and the easier it will be for new contributors to find ZK's repo. > > This should add the hacktoberfest label to the github repo. > > > > On a related note: I've noticed that a lot of people just do pull > > requests, but this project's committers are pretty strict about > > requiring a JIRA issue for everything, even trivial changes. That > > information is hard to find for new contributors. If you created a > > CONTRIBUTING.md file, GitHub would automatically provide a link to > > that for new contributors coming from GitHub. It would be easier to > > find than looking on the wiki/confluence page where that information > > is currently stored. (Alternatively, you could relax your constraints > > and not require a JIRA for everything... > > > We do not require a JIRA for very simple patches. But JIRA is super useful > in order to track release notes. So, I've seen some patches I thought were very simple, and a committer still commented on the PR asking them to create a JIRA, even though they could have just merged it as-is. I'm not really sure there's a clear demarcation point for users to know when something is considered "very simple" or not when the committers themselves aren't consistent on this. I think it's just confusing, and the requirement should go away entirely. If JIRA is useful to track some issues, sure, use it, but if somebody has already done the work, and it's self-contained in a PR, regardless of whether it's simple or not, I think just accepting it (if it's ready) is the most community-friendly thing to do, rather than ask a new contributor to create a JIRA at all (especially since they may not want to create a separate JIRA account, if they don't already have one). If a JIRA is really needed, the committer themselves can take on this responsibility instead of asking the contributor to do it... as a way of lowering the bar to contributing... especially if the user is a new contributor. If they are a regular contributor, then they should just be invited to be a committer. It's easier to hold committers to a higher standard of responsibility for these things. Personally, I don't find the generated JIRA "release notes" very useful. We used to use that in Accumulo, and we stopped doing it, because it was basically nothing more than a commit log. When the "release notes" are basically a replay of the git history, they lose their value. As a replacement, we started curating our release notes with the most important information users needed to know, and publishing that in a feed on our website alongside each release. We refer people (with a convenient link) to the full changes in the git history, if they are interested. Most of the git history does have references to issue numbers (because when we merge from the GitHub UI, it appends the PR number to the log message), if people want additional context from any conversation on the issue. I'm not suggesting ZK do this... it may not work for everybody. I'm merely describing how we did it in Accumulo and why, as an alternative perspective. > > Do you mind adding the CONTRIBUTING.md file into your PR? Just with a > simple link to confluence I'm not sure I want to do that with just a link to confluence. I don't think it's a good idea to make users click through multiple links to eventually get to the information. Personally, I think the information should be entirely contained in CONTRIBUTING.md and the confluence page should be decommissioned... and links to it should point to the CONTRIBUTING.md page in the repo instead. I'm not a fan of spreading documentation across multiple sites like that. I think everything should be on the project website or in the project repo. So, if I were to contribute a PR to update this, I would pursue it in that direction. Otherwise, I think somebody else should add CONTRIBUTING.md if they think it should be done differently. > > Enrico > > when Accumulo did this, it > > became much more friendly to new contributors.) > > > > On Wed, Oct 7, 2020 at 11:38 AM Enrico Olivelli > > wrote: > > > > > > Mate, > > > > > > Il giorno mer 7 ott 2020 alle ore 17:17 Szalay-Bekő Máté < > > > szalay.beko.m...@gmail.com> ha scritto: > > > > > > > Hello Devs! > > > > > > > > Are you participating in Hacktoberfest? What about tagging the > > > > ZooKeeper repo with the 'hacktoberfest' label to make it visible for > > > > Hacktoberfest automatically? > > > > > > > > > > This can be a very good idea. > > > IIUC if we open the project to that event we should take care of all of > > the > > > pull requests that will come, as we are doing with MuseDev/ApacheCon. > > > > > >
Re: hacktoberfest
Il Mer 7 Ott 2020, 21:02 Christopher ha scritto: > I created https://github.com/apache/zookeeper/pull/1489 > (https://issues.apache.org/jira/browse/ZOOKEEPER-3962) > > Christopher Thank you This should add the hacktoberfest label to the github repo. > > On a related note: I've noticed that a lot of people just do pull > requests, but this project's committers are pretty strict about > requiring a JIRA issue for everything, even trivial changes. That > information is hard to find for new contributors. If you created a > CONTRIBUTING.md file, GitHub would automatically provide a link to > that for new contributors coming from GitHub. It would be easier to > find than looking on the wiki/confluence page where that information > is currently stored. (Alternatively, you could relax your constraints > and not require a JIRA for everything... We do not require a JIRA for very simple patches. But JIRA is super useful in order to track release notes. Do you mind adding the CONTRIBUTING.md file into your PR? Just with a simple link to confluence Enrico when Accumulo did this, it > became much more friendly to new contributors.) > > On Wed, Oct 7, 2020 at 11:38 AM Enrico Olivelli > wrote: > > > > Mate, > > > > Il giorno mer 7 ott 2020 alle ore 17:17 Szalay-Bekő Máté < > > szalay.beko.m...@gmail.com> ha scritto: > > > > > Hello Devs! > > > > > > Are you participating in Hacktoberfest? What about tagging the > > > ZooKeeper repo with the 'hacktoberfest' label to make it visible for > > > Hacktoberfest automatically? > > > > > > > This can be a very good idea. > > IIUC if we open the project to that event we should take care of all of > the > > pull requests that will come, as we are doing with MuseDev/ApacheCon. > > > > So I would like to be sure that we have a least a couple of committers > that > > explicitly want to support the project and keep up with such pull > requests. > > > > Enrico > > > > > > > > > > Currently I see more than 32.000 open-source repositories in this > > > topic (https://github.com/topics/hacktoberfest), I think we should add > > > ZooKeeper too. > > > > > > see: > > > - https://hacktoberfest.digitalocean.com/hacktoberfest-update > > > - > > > > https://docs.github.com/en/free-pro-team@latest/github/administering-a-repository/classifying-your-repository-with-topics > > > > > > BR, > > > Mate > > > > > > p.s. I'm not sure who has the rights to assign our repo to this topic. > > > (I don't have.) > > > >
Re: hacktoberfest
I created https://github.com/apache/zookeeper/pull/1489 (https://issues.apache.org/jira/browse/ZOOKEEPER-3962) This should add the hacktoberfest label to the github repo. On a related note: I've noticed that a lot of people just do pull requests, but this project's committers are pretty strict about requiring a JIRA issue for everything, even trivial changes. That information is hard to find for new contributors. If you created a CONTRIBUTING.md file, GitHub would automatically provide a link to that for new contributors coming from GitHub. It would be easier to find than looking on the wiki/confluence page where that information is currently stored. (Alternatively, you could relax your constraints and not require a JIRA for everything... when Accumulo did this, it became much more friendly to new contributors.) On Wed, Oct 7, 2020 at 11:38 AM Enrico Olivelli wrote: > > Mate, > > Il giorno mer 7 ott 2020 alle ore 17:17 Szalay-Bekő Máté < > szalay.beko.m...@gmail.com> ha scritto: > > > Hello Devs! > > > > Are you participating in Hacktoberfest? What about tagging the > > ZooKeeper repo with the 'hacktoberfest' label to make it visible for > > Hacktoberfest automatically? > > > > This can be a very good idea. > IIUC if we open the project to that event we should take care of all of the > pull requests that will come, as we are doing with MuseDev/ApacheCon. > > So I would like to be sure that we have a least a couple of committers that > explicitly want to support the project and keep up with such pull requests. > > Enrico > > > > > > Currently I see more than 32.000 open-source repositories in this > > topic (https://github.com/topics/hacktoberfest), I think we should add > > ZooKeeper too. > > > > see: > > - https://hacktoberfest.digitalocean.com/hacktoberfest-update > > - > > https://docs.github.com/en/free-pro-team@latest/github/administering-a-repository/classifying-your-repository-with-topics > > > > BR, > > Mate > > > > p.s. I'm not sure who has the rights to assign our repo to this topic. > > (I don't have.) > >
[jira] [Created] (ZOOKEEPER-3962) Create .asf.yaml file for ZooKeeper repo
Christopher Tubbs created ZOOKEEPER-3962: Summary: Create .asf.yaml file for ZooKeeper repo Key: ZOOKEEPER-3962 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-3962 Project: ZooKeeper Issue Type: Task Reporter: Christopher Tubbs Assignee: Christopher Tubbs Add a .asf.yaml file to control GitHub project settings, such as URL, description, and topics -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (ZOOKEEPER-3961) Inaccurate Error Message
David Mollitor created ZOOKEEPER-3961:
-
Summary: Inaccurate Error Message
Key: ZOOKEEPER-3961
URL: https://issues.apache.org/jira/browse/ZOOKEEPER-3961
Project: ZooKeeper
Issue Type: Improvement
Reporter: David Mollitor
https://github.com/apache/zookeeper/blob/c47ef905e077184bc5b7f555a3e2dfeb6dc046e1/zookeeper-server/src/main/java/org/apache/zookeeper/server/ZooKeeperServer.java#L239-L244
{code:java|title=ZooKeeperServer.java}
if (intBufferStartingSizeBytes < 32) {
String msg = "Buffer starting size must be greater than 0."
+ "Configure with
\"-Dzookeeper.intBufferStartingSizeBytes=\" ";
LOG.error(msg);
throw new IllegalArgumentException(msg);
}
{code}
Actually the size must be _greater or equal to 32_
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
Re: hacktoberfest
Mate, Il giorno mer 7 ott 2020 alle ore 17:17 Szalay-Bekő Máté < szalay.beko.m...@gmail.com> ha scritto: > Hello Devs! > > Are you participating in Hacktoberfest? What about tagging the > ZooKeeper repo with the 'hacktoberfest' label to make it visible for > Hacktoberfest automatically? > This can be a very good idea. IIUC if we open the project to that event we should take care of all of the pull requests that will come, as we are doing with MuseDev/ApacheCon. So I would like to be sure that we have a least a couple of committers that explicitly want to support the project and keep up with such pull requests. Enrico > > Currently I see more than 32.000 open-source repositories in this > topic (https://github.com/topics/hacktoberfest), I think we should add > ZooKeeper too. > > see: > - https://hacktoberfest.digitalocean.com/hacktoberfest-update > - > https://docs.github.com/en/free-pro-team@latest/github/administering-a-repository/classifying-your-repository-with-topics > > BR, > Mate > > p.s. I'm not sure who has the rights to assign our repo to this topic. > (I don't have.) >
hacktoberfest
Hello Devs! Are you participating in Hacktoberfest? What about tagging the ZooKeeper repo with the 'hacktoberfest' label to make it visible for Hacktoberfest automatically? Currently I see more than 32.000 open-source repositories in this topic (https://github.com/topics/hacktoberfest), I think we should add ZooKeeper too. see: - https://hacktoberfest.digitalocean.com/hacktoberfest-update - https://docs.github.com/en/free-pro-team@latest/github/administering-a-repository/classifying-your-repository-with-topics BR, Mate p.s. I'm not sure who has the rights to assign our repo to this topic. (I don't have.)
[jira] [Created] (ZOOKEEPER-3960) Update ZooKeeper client documentation about key file format parameters
Mate Szalay-Beko created ZOOKEEPER-3960: --- Summary: Update ZooKeeper client documentation about key file format parameters Key: ZOOKEEPER-3960 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-3960 Project: ZooKeeper Issue Type: Improvement Affects Versions: 3.6.2, 3.5.8 Reporter: Mate Szalay-Beko Assignee: Mate Szalay-Beko Fix For: 3.7.0, 3.5.9, 3.6.3 The Programmers guide doesn't contain the parameters needed to be changed in order to specify keystore and truststore file formats for the Java Clients. see: https://zookeeper.apache.org/doc/r3.6.2/zookeeperProgrammers.html#sc_java_client_configuration -- This message was sent by Atlassian Jira (v8.3.4#803005)
Re: ZooKeeper 3.7 timeline? C SASL backport?
Sorry Damien, I thought I had answered to you but my message was never sent :-( I think that master branch is in very good shape and we had feedback from users about 3.6.x, in fact we already released 3.6.1 and 3.6.2. Cutting 3.7.0 will also be a sign that the current "stable" version is 3.6., and this is very good. Let's see if any committer volunteers to cut the release. I will have time by the end of the month if no one else is available Enrico Il giorno mer 7 ott 2020 alle ore 11:01 Damien Diederen < ddiede...@sinenomine.net> ha scritto: > > Hello, everybody. > > Would you have any feedback on this? > > Cheers, -D > > > > Damien Diederen writes: > > Greetings, all, > > > > I haven't initially tried to get the C SASL patches integrated in the > > 3.6 branch, as they were not very mature and as I had noticed the > > increased release cadence in the project. > > > > But I am now wondering: do we already have a time horizon for ZooKeeper > > 3.7? Or are we waiting for a sufficient number of features to > > "accumulate" in master? > > > > (I understand that each branch causes additional overhead, and that we > > may not want to have a long ladder of cherry-picks for each patch coming > > in.) > > > > In case there aren't any short-term plans, I would like to backport the > > C and Perl SASL patches to 3.6, as they've now matured a fair bit and as > > those patches shouldn't impact the rest of the system. > > > > What do you think? > > > > Cheers, -D >
Re: ZooKeeper 3.7 timeline? C SASL backport?
Hello, everybody. Would you have any feedback on this? Cheers, -D Damien Diederen writes: > Greetings, all, > > I haven't initially tried to get the C SASL patches integrated in the > 3.6 branch, as they were not very mature and as I had noticed the > increased release cadence in the project. > > But I am now wondering: do we already have a time horizon for ZooKeeper > 3.7? Or are we waiting for a sufficient number of features to > "accumulate" in master? > > (I understand that each branch causes additional overhead, and that we > may not want to have a long ladder of cherry-picks for each patch coming > in.) > > In case there aren't any short-term plans, I would like to backport the > C and Perl SASL patches to 3.6, as they've now matured a fair bit and as > those patches shouldn't impact the rest of the system. > > What do you think? > > Cheers, -D
[jira] [Created] (ZOOKEEPER-3959) Allow multiple superUsers with SASL
Damien Diederen created ZOOKEEPER-3959:
--
Summary: Allow multiple superUsers with SASL
Key: ZOOKEEPER-3959
URL: https://issues.apache.org/jira/browse/ZOOKEEPER-3959
Project: ZooKeeper
Issue Type: New Feature
Components: server
Affects Versions: 3.6.2
Reporter: Damien Diederen
Assignee: Damien Diederen
We would like to add support for multiple SASL-authenticated {{super}} users to
ZooKeeper.
There currently exists a {{zookeeper.superUser}} property which is documented
as:
{quote}
When this parameter is set to a [SASL] principal name, only an authenticated
client with that principal will be able to bypass ACL checking and have full
privileges to all znodes.
{quote}
Connections with an ID matching that property receive {{super}} powers [through
somewhat "hardcoded"
logic|https://github.com/apache/zookeeper/blob/c47ef905e077184bc5b7f555a3e2dfeb6dc046e1/zookeeper-server/src/main/java/org/apache/zookeeper/server/ZooKeeperServer.java#L1675-L1678]
which only admits a single principal name.
Our goals could simply be achieved by promoting the equality comparison to a
set membership test, and either:
# defining a new {{zookeeper.superUsers}} property holding a comma-separated
list of principal names, or
# overloading {{zookeeper.superUser}}, parsing it as a comma-separated list.
Another possibility, more complex but also more flexible, would be to implement
the solution suggested [in this {{SASLAuthenticationProvider}}
comment|https://github.com/apache/zookeeper/blob/fe940cdd8fb23ba09684cefb73233d570f4a20fa/zookeeper-server/src/main/java/org/apache/zookeeper/server/auth/SASLAuthenticationProvider.java#L30-L36]:
{quote}
TODO: consider substituting current implementation of direct ClientCnxn
manipulation with a call to this method
(SASLAuthenticationProvider:handleAuthentication()) at session initiation.
{quote}
This would allow plugging arbitrary subclasses of
{{SASLAuthenticationProvider}} (or, possibly, implementations of a new, ad-hoc
interface) carrying custom logic.
Of course, these solutions are not exclusive: it would be possible to first
implement the {{zookeeper.superUsers}} functionality in place, and to later
migrate it to an {{AuthenticationProvider}}. E.g., doing the former in the 3.6
branch and reserving the latter for 3.7/{{master}}.
What do you think?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
