Re: full Zookeeper authentication
Thank you. This is what I needed. -- Dmitrii Kishchukov. Leading software developer Submission Portal Team On 12/7/18, 2:52 PM, "Joseph Wu" wrote: There are currently three components of Mesos that use Zookeeper: *Master Detector:* This object is used by the Mesos Master, Agent, and Scheduler to find which Master is the leader. The existing detector code will parse a "zk://" URL if given here: https://github.com/apache/mesos/blob/1.7.x/src/master/detector/detector.cpp#L62 Not including tests, there are four call sites which pass in a ZK URL to the detector: - Master: https://github.com/apache/mesos/blob/1.7.x/src/master/main.cpp#L430-L433 - Agent: https://github.com/apache/mesos/blob/1.7.x/src/slave/main.cpp#L487-L490 - Scheduler: https://github.com/apache/mesos/blob/1.7.x/src/sched/sched.cpp#L152 - (Deprecated) CLI helper binary: https://github.com/apache/mesos/blob/1.7.x/src/cli/resolve.cpp#L95-L96 *Master Contender:* This object is used by the Mesos Master to contend for leadership of the cluster. The contender will parse a ZK URL just like the detector: https://github.com/apache/mesos/blob/1.7.x/src/master/contender/contender.cpp#L53 Unlike the detector, there is only a single call site for the contender: https://github.com/apache/mesos/blob/1.7.x/src/master/main.cpp#L418-L421 *Replicated Log Library:* This is a library which is used by the Mesos Master and some custom frameworks, to persist data via the Paxos algorithm. The Master's call site is straightforward: https://github.com/apache/mesos/blob/1.7.x/src/master/main.cpp#L383-L391 The library is built into a JAR for use by java frameworks, so there are two references in this JNI code: https://github.com/apache/mesos/blob/1.7.x/src/java/jni/org_apache_mesos_Log.cpp#L673 https://github.com/apache/mesos/blob/1.7.x/src/java/jni/org_apache_mesos_state_LogState.cpp#L75 Some other files that you will likely need to modify include: - The zookeeper::Authentication class: https://github.com/apache/mesos/blob/1.7.x/include/mesos/zookeeper/authentication.hpp This will need to be extended to allow non-digest schemes. It will currently exit if a non-digest scheme is passed in the URL. - The zookeeper::URL class: https://github.com/apache/mesos/blob/1.7.x/include/mesos/zookeeper/url.hpp Depending on how flexible the authentication schemes are, you may need to update the URL parsing logic, or scrap the URL altogether if there are authentication schemes that cannot be encoded in a URL. - The "--zk" flag for the Master: https://github.com/apache/mesos/blob/1.7.x/src/master/flags.cpp#L666-L673 You may need to update the documentation of this flag, or perhaps add new flags. - The "--master" flag for the Agent: https://github.com/apache/mesos/blob/1.7.x/src/slave/flags.cpp#L1421-L1427 This will look similar to the "--zk" Master flag, but it also supports non-ZK masters. Hopefully this list of code locations will give you some idea of where to start. Feel free to ping us in Slack too. On Fri, Dec 7, 2018 at 6:01 AM Kishchukov, Dmitrii (NIH/NLM/NCBI) [C] < dmitrii.kishchu...@nih.gov> wrote: > Yes. I want to do it. And it would be good if someone could give an advise > how to do it. For example is there one place where Authentication object > constructed for Zookeeper? > For me it looks like there many places which is strange. > > -- > > Dmitrii Kishchukov. > Leading software developer > Submission Portal Team > > > On 12/6/18, 12:56 PM, "Vinod Kone" wrote: > > Dmitrii. > > That approach sounds reasonable. Would you like to work on this? Are > you > looking for a reviewer/shepherd? > > On Thu, Dec 6, 2018 at 11:28 AM Kishchukov, Dmitrii (NIH/NLM/NCBI) [C] > < > dmitrii.kishchu...@nih.gov> wrote: > > > Mesos allow using only digest authentication scheme for Zookeeper. > Which > > is bad because Zookeeper has quite a flexible security model. > > It is easy to make you own authenticator with its own scheme name. > > > > To support fully Zookeeper authentication, Mesos has pass two items > into > > Zookeeper: > > scheme and credentials. > > credentials can have different format depending on authentication > scheme. > > For digest scheme it is ‘login:password’ > > > > All Mesos should do just pass scheme and credentials to Zookeeper. > > > > Another improvement might be be to configure credentials via file > instead > > of URI > > > > For example it
Re: full Zookeeper authentication
There are currently three components of Mesos that use Zookeeper: *Master Detector:* This object is used by the Mesos Master, Agent, and Scheduler to find which Master is the leader. The existing detector code will parse a "zk://" URL if given here: https://github.com/apache/mesos/blob/1.7.x/src/master/detector/detector.cpp#L62 Not including tests, there are four call sites which pass in a ZK URL to the detector: - Master: https://github.com/apache/mesos/blob/1.7.x/src/master/main.cpp#L430-L433 - Agent: https://github.com/apache/mesos/blob/1.7.x/src/slave/main.cpp#L487-L490 - Scheduler: https://github.com/apache/mesos/blob/1.7.x/src/sched/sched.cpp#L152 - (Deprecated) CLI helper binary: https://github.com/apache/mesos/blob/1.7.x/src/cli/resolve.cpp#L95-L96 *Master Contender:* This object is used by the Mesos Master to contend for leadership of the cluster. The contender will parse a ZK URL just like the detector: https://github.com/apache/mesos/blob/1.7.x/src/master/contender/contender.cpp#L53 Unlike the detector, there is only a single call site for the contender: https://github.com/apache/mesos/blob/1.7.x/src/master/main.cpp#L418-L421 *Replicated Log Library:* This is a library which is used by the Mesos Master and some custom frameworks, to persist data via the Paxos algorithm. The Master's call site is straightforward: https://github.com/apache/mesos/blob/1.7.x/src/master/main.cpp#L383-L391 The library is built into a JAR for use by java frameworks, so there are two references in this JNI code: https://github.com/apache/mesos/blob/1.7.x/src/java/jni/org_apache_mesos_Log.cpp#L673 https://github.com/apache/mesos/blob/1.7.x/src/java/jni/org_apache_mesos_state_LogState.cpp#L75 Some other files that you will likely need to modify include: - The zookeeper::Authentication class: https://github.com/apache/mesos/blob/1.7.x/include/mesos/zookeeper/authentication.hpp This will need to be extended to allow non-digest schemes. It will currently exit if a non-digest scheme is passed in the URL. - The zookeeper::URL class: https://github.com/apache/mesos/blob/1.7.x/include/mesos/zookeeper/url.hpp Depending on how flexible the authentication schemes are, you may need to update the URL parsing logic, or scrap the URL altogether if there are authentication schemes that cannot be encoded in a URL. - The "--zk" flag for the Master: https://github.com/apache/mesos/blob/1.7.x/src/master/flags.cpp#L666-L673 You may need to update the documentation of this flag, or perhaps add new flags. - The "--master" flag for the Agent: https://github.com/apache/mesos/blob/1.7.x/src/slave/flags.cpp#L1421-L1427 This will look similar to the "--zk" Master flag, but it also supports non-ZK masters. Hopefully this list of code locations will give you some idea of where to start. Feel free to ping us in Slack too. On Fri, Dec 7, 2018 at 6:01 AM Kishchukov, Dmitrii (NIH/NLM/NCBI) [C] < dmitrii.kishchu...@nih.gov> wrote: > Yes. I want to do it. And it would be good if someone could give an advise > how to do it. For example is there one place where Authentication object > constructed for Zookeeper? > For me it looks like there many places which is strange. > > -- > > Dmitrii Kishchukov. > Leading software developer > Submission Portal Team > > > On 12/6/18, 12:56 PM, "Vinod Kone" wrote: > > Dmitrii. > > That approach sounds reasonable. Would you like to work on this? Are > you > looking for a reviewer/shepherd? > > On Thu, Dec 6, 2018 at 11:28 AM Kishchukov, Dmitrii (NIH/NLM/NCBI) [C] > < > dmitrii.kishchu...@nih.gov> wrote: > > > Mesos allow using only digest authentication scheme for Zookeeper. > Which > > is bad because Zookeeper has quite a flexible security model. > > It is easy to make you own authenticator with its own scheme name. > > > > To support fully Zookeeper authentication, Mesos has pass two items > into > > Zookeeper: > > scheme and credentials. > > credentials can have different format depending on authentication > scheme. > > For digest scheme it is ‘login:password’ > > > > All Mesos should do just pass scheme and credentials to Zookeeper. > > > > Another improvement might be be to configure credentials via file > instead > > of URI > > > > For example it can be two command line options: > > --zk_auth_scheme and –zk_auth_credentials > > > > It can be used like this: > > --zk_auth_scheme=some_custome_scheme –zk_auth_credentials=filename > > > > --zk_auth_credentials can just get all contents of the file as > credentials > > string. > > > > Class Authentication in Mesos already contains all that we need. The > > problem is what Mesos pass to the constructor. > > > > > > -- > > > > Dmitrii Kishchukov. > > > > > > >
[NOTICE] Mandatory relocation of Apache git repositories on git-wip-us.apache.org
[IF YOUR PROJECT DOES NOT HAVE GIT REPOSITORIES ON GIT-WIP-US PLEASE DISREGARD THIS EMAIL; IT WAS MASS-MAILED TO ALL APACHE PROJECTS] Hello Apache projects, I am writing to you because you may have git repositories on the git-wip-us server, which is slated to be decommissioned in the coming months. All repositories will be moved to the new gitbox service which includes direct write access on github as well as the standard ASF commit access via gitbox.apache.org. ## Why this move? ## The move comes as a result of retiring the git-wip service, as the hardware it runs on is longing for retirement. In lieu of this, we have decided to consolidate the two services (git-wip and gitbox), to ease the management of our repository systems and future-proof the underlying hardware. The move is fully automated, and ideally, nothing will change in your workflow other than added features and access to GitHub. ## Timeframe for relocation ## Initially, we are asking that projects voluntarily request to move their repositories to gitbox, hence this email. The voluntary timeframe is between now and January 9th 2019, during which projects are free to either move over to gitbox or stay put on git-wip. After this phase, we will be requiring the remaining projects to move within one month, after which we will move the remaining projects over. To have your project moved in this initial phase, you will need: - Consensus in the project (documented via the mailing list) - File a JIRA ticket with INFRA to voluntarily move your project repos over to gitbox (as stated, this is highly automated and will take between a minute and an hour, depending on the size and number of your repositories) To sum up the preliminary timeline; - December 9th 2018 -> January 9th 2019: Voluntary (coordinated) relocation - January 9th -> February 6th: Mandated (coordinated) relocation - February 7th: All remaining repositories are mass migrated. This timeline may change to accommodate various scenarios. ## Using GitHub with ASF repositories ## When your project has moved, you are free to use either the ASF repository system (gitbox.apache.org) OR GitHub for your development and code pushes. To be able to use GitHub, please follow the primer at: https://reference.apache.org/committer/github We appreciate your understanding of this issue, and hope that your project can coordinate voluntarily moving your repositories in a timely manner. All settings, such as commit mail targets, issue linking, PR notification schemes etc will automatically be migrated to gitbox as well. With regards, Daniel on behalf of ASF Infra. PS:For inquiries, please reply to us...@infra.apache.org, not your project's dev list :-).
Re: full Zookeeper authentication
Yes. I want to do it. And it would be good if someone could give an advise how to do it. For example is there one place where Authentication object constructed for Zookeeper? For me it looks like there many places which is strange. -- Dmitrii Kishchukov. Leading software developer Submission Portal Team On 12/6/18, 12:56 PM, "Vinod Kone" wrote: Dmitrii. That approach sounds reasonable. Would you like to work on this? Are you looking for a reviewer/shepherd? On Thu, Dec 6, 2018 at 11:28 AM Kishchukov, Dmitrii (NIH/NLM/NCBI) [C] < dmitrii.kishchu...@nih.gov> wrote: > Mesos allow using only digest authentication scheme for Zookeeper. Which > is bad because Zookeeper has quite a flexible security model. > It is easy to make you own authenticator with its own scheme name. > > To support fully Zookeeper authentication, Mesos has pass two items into > Zookeeper: > scheme and credentials. > credentials can have different format depending on authentication scheme. > For digest scheme it is ‘login:password’ > > All Mesos should do just pass scheme and credentials to Zookeeper. > > Another improvement might be be to configure credentials via file instead > of URI > > For example it can be two command line options: > --zk_auth_scheme and –zk_auth_credentials > > It can be used like this: > --zk_auth_scheme=some_custome_scheme –zk_auth_credentials=filename > > --zk_auth_credentials can just get all contents of the file as credentials > string. > > Class Authentication in Mesos already contains all that we need. The > problem is what Mesos pass to the constructor. > > > -- > > Dmitrii Kishchukov. > >
