[jira] [Comment Edited] (SOLR-7176) allow zkcli to modify JSON
[
https://issues.apache.org/jira/browse/SOLR-7176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14503430#comment-14503430
]
Per Steffensen edited comment on SOLR-7176 at 4/20/15 7:02 PM:
---
Believe it is referred to as compare-and-swap, but anyway
{{Overseer.handleProp}} does not use that feature (it always uses version=-1
when calling zk.setData). But actually I just realized that it is not true that
{{CollectionsHandler.handleProp}} does the fetch-update and only leaves update
to Overseer (as I claimed above) - it actually does forward the entire
operation to Overseer so that the Overseer does fetch, update and store (just
at I wanted it to). So you are right, that the problem I sketched above is not
a problem in todays code, but it is not due to usage of compare-and-swap - it
is because two threads will never try to do updates at the same time - they
will ask the Overseer to do the updates (single-threaded = primitive
pessimistic lock).
So we cannot just eliminate the overseer from the picture completely, if we
still want to avoid the (more or less theoretical) problem I sketched above.
So the bullet-safe command-line solution should take this into consideration -
e.g.
* 1) Take the overseer-lock before performing the operation
* 2) Use compare-and-swap (and make sure Overseer also does)
I believe I would prefer 1) because it is the most generally useable solution
to the problem. Compare-and-swap (even combined with ZK multi-op feature) will
not always be sufficient for operations that want to update several znodes
atomically - and who knows, maybe some day we also want to that kind of stuff
using command-line. Taking a pessimistic lock (like the Overseer-lock) always
will be sufficient.
was (Author: steff1193):
Believe it is referred to as compare-and-swap, but anyway
{{Overseer.handleProp}} does not use that feature (it always uses version=-1
when calling zk.setData). But actually I just realized that it is not true that
{{CollectionsHandler.handleProp}} does the fetch-update and only leaves update
to Overseer (as I claimed above) - it actually does forward the entire
operation to Overseer so that the Overseer does fetch, update and store (just
at I wanted it to). So you are right, that the problem I sketched above is not
a problem in todays code, but it is not due to usage of compare-and-swap - it
is because two threads will never try to do updates at the same time - they
will ask the Overseer to do the updates (single-threaded = primitive
pessimistic lock).
So we cannot just eliminate the overseer from the picture completely, if we
still want to avoid the (more or less theoretical) problem I sketched above.
So the bullet-safe command-line solution should take this into consideration -
e.g.
* 1) Take the overseer-lock before performing the operation
* 2) Use compare-and-swap (and make sure Overseer also does)
I would prefer 1) because it is the most generally useable solution to the
problem. Compare-and-swap (even combined with ZK multi-op feature) will not
always be sufficient for operations that want to update several znodes
atomically - and who knows, maybe some day we also want to that kind of stuff
using command-line. Taking a pessimistic lock (like the Overseer-lock) always
will be sufficient.
allow zkcli to modify JSON
--
Key: SOLR-7176
URL: https://issues.apache.org/jira/browse/SOLR-7176
Project: Solr
Issue Type: New Feature
Reporter: Yonik Seeley
Assignee: Noble Paul
Priority: Minor
Attachments: SOLR-7176.patch, SOLR-7176.patch, SOLR-7176.patch
To enable SSL, we have instructions like the following:
{code}
server/scripts/cloud-scripts/zkcli.sh -zkhost localhost:2181 -cmd put
/clusterprops.json '{urlScheme:https}'
{code}
Overwriting the value won't work well when we have more properties to put in
clusterprops. We should be able to change individual values or perhaps merge
values.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Comment Edited] (SOLR-7176) allow zkcli to modify JSON
[
https://issues.apache.org/jira/browse/SOLR-7176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14500028#comment-14500028
]
Per Steffensen edited comment on SOLR-7176 at 4/17/15 3:27 PM:
---
I agree, but from time to time I want to add a (async) command to the overseer
while the cluster is not running, expecting the overseer to pick it up and
execute it when I start my cluster. If you would enable this tool to do this
kind of stuff, then suddenly most of the cluster-commands become relevant for
this tool - if it is able to both execute the command directly (if supported -
e.g. the {{CLUSTERPROP}} command) or to leave the command for execution by the
overseer.
And, if you have numerous machines that might or might not currently run a
Solr-node, maybe you actually want to be able to run the {{OVERSEERSTATUS}}
command as a command-line just to get a not running response.
was (Author: steff1193):
I agree, but from time to time I want to add a (async) command to the overseer
while the cluster is not running, expecting the overseer to pick it up and
execute it when I start my cluster. If you would enable this tool to do this
kind of stuff, then suddenly most of the cluster-commands become relevant for
this tool - if it is able to both execute the command directly (if supported -
e.g. by {{CLUSTERPROP}} command) or to leave the command for execution by the
overseer.
And, if you have numerous machines that might or might not currently run a
Solr-node, maybe you actually want to be able to run the {{OVERSEERSTATUS}}
command as a command-line just to get an not running response.
allow zkcli to modify JSON
--
Key: SOLR-7176
URL: https://issues.apache.org/jira/browse/SOLR-7176
Project: Solr
Issue Type: New Feature
Reporter: Yonik Seeley
Priority: Minor
To enable SSL, we have instructions like the following:
{code}
server/scripts/cloud-scripts/zkcli.sh -zkhost localhost:2181 -cmd put
/clusterprops.json '{urlScheme:https}'
{code}
Overwriting the value won't work well when we have more properties to put in
clusterprops. We should be able to change individual values or perhaps merge
values.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Comment Edited] (SOLR-7176) allow zkcli to modify JSON
[
https://issues.apache.org/jira/browse/SOLR-7176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14343271#comment-14343271
]
Per Steffensen edited comment on SOLR-7176 at 3/2/15 3:45 PM:
--
I think we should not make ZkCLI a Swiss Army Knife. ZkCLI should deal with ZK
operations, like uploading and downloading. Changing then content of a
json-file is not a ZK operation. Why not see it as different things. So what
you need to do if you want to make arbitrary changes to json-content of a znode
in ZK is
{code}
zkcli getfile thefile.json
use another tool for modifying thefile.json
zkcli putfile thefile.json
{code}
It would be nice to make different tools that can make operations on the
different types of jsons we have in a safe way - ensuring e.g. consistency,
integrity, validity etc. But I think those tools should use the classes already
handling those jsons, and not have anything to do with ZkCLI. E.g. a tool for
manipulating clusterstate.json should use classes from
org.apache.solr.common.cloud in solrj project, like ZkStateReader,
ClusterState, ClusterStateUpdater etc. Then it is always those classes that are
used to operate on a specific type of json, and we only have to build
consistency, integrity, validity etc into those classes (separation of
concern). The new thing should be that those classes can be used via an
external tool also when no Solr nodes are running.
At least make sure to use those existing classes for actually
reading/writing/verifying the jsons, and make separate tool-classes. Changing
ZkCLI to be able to trigger operations in those tool-classes is less important,
but personally I do not like - actually, if it has to be, I would rather see
e.g. an ClusterPropCLI-tool support a do-in-zk-flag making it use ZkCLI tool
to download first, do its manipulations and then use ZkCLI to upload again.
That is, new tools that use ZkCLI (if you are too lazy doing the
download/upload using ZkCLI yourself), instead of the other way around, where
ZkCLI uses other tools or even just does json-manipulation itself. Please do
not try to implement rules about what can and cannot be in a specific type of
json, what operations you can do on it etc two places in the code-base - use
what we already have.
Would like to see clean definitions of which classes own responsibilities. E.g.
* ClusterState.java own the responsibility of a clusterstate.json always being
consistent, valid etc. You never do changes to clusterstate.json with using
ClusterState.java
* ClusterStateUpdater own the set of allowed operations on clusterstate.json.
You never manipulate a clusterstate.json without going through
ClusterStateUpdater. The new off-line ClusterStateCLI-tool should use
ClusterStateUpdater to do its operations - just change ClusterStateUpdater, so
that it support receiving jobs not coming from overseer queue.
Yes, I know you primarily talk about ClusterProps, but the principle will be
the same. It is just that I know more about which classes handle cluster-state
than which handles cluster-props.
That said, maybe you can live with using e.g. http://trentm.com/json/ (or one
of the million other command-line json tools available) in use another tool
for modifying thefile.json above?
was (Author: steff1193):
I think we should not make ZkCLI a Swiss Army Knife. ZkCLI should deal with ZK
operations, like uploading and downloading. Changing then content of a
json-file is not a ZK operation. Why not see it as different things. So what
you need to do if you want to make arbitrary changes to json-content of a znode
in ZK is
{code}
zkcli getfile thefile.json
use another tool for modifying thefile.json
zkcli putfile thefile.json
{code}
It would be nice to make different tools that can make operations on the
different types of jsons we have in a safe way - ensuring e.g. consistency,
integrity, validity etc. But I think those tools should use the classes already
handling those jsons, and not have anything to do with ZkCLI. E.g. a tool for
manipulating clusterstate.json should use classes from
org.apache.solr.common.cloud in solrj project, like ZkStateReader, ClusterState
etc. Then it is always those classes that are used to operate on a specific
type of json, and we only have to build consistency, integrity, validity etc
into those classes (separation of concern). The new thing should be that those
classes can be used via an external tool also when no Solr nodes are running.
At least make sure to use those existing classes for actually
reading/writing/verifying the jsons, and make separate tool-classes. Changing
ZkCLI to be able to trigger operations in those tool-classes is less important,
but personally I do not like - actually, if it has to be, I would rather see
e.g. an ClusterPropCLI-tool support a do-in-zk-flag making it use ZkCLI tool
to download first, do its
[jira] [Comment Edited] (SOLR-7176) allow zkcli to modify JSON
[
https://issues.apache.org/jira/browse/SOLR-7176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14343271#comment-14343271
]
Per Steffensen edited comment on SOLR-7176 at 3/2/15 3:48 PM:
--
I think we should not make ZkCLI a Swiss Army Knife. ZkCLI should deal with ZK
operations, like uploading and downloading. Changing then content of a
json-file is not a ZK operation. Why not see it as different things. So what
you need to do if you want to make arbitrary changes to json-content of a znode
in ZK is
{code}
zkcli getfile thefile.json
use another tool for modifying thefile.json
zkcli putfile thefile.json
{code}
It would be nice to make different tools that can make operations on the
different types of jsons we have in a safe way - ensuring e.g. consistency,
integrity, validity etc. But I think those tools should use the classes already
handling those jsons, and not have anything to do with ZkCLI. E.g. a tool for
manipulating clusterstate.json should use classes from
org.apache.solr.common.cloud in solrj project, like ZkStateReader,
ClusterState, ClusterStateUpdater etc. Then it is always those classes that are
used to operate on a specific type of json, and we only have to build
consistency, integrity, validity etc into those classes (separation of
concern). The new thing should be that those classes can be used via an
external tool also when no Solr nodes are running.
At least make sure to use those existing classes for actually
reading/writing/verifying the jsons, and make separate tool-classes. Changing
ZkCLI to be able to trigger operations in those tool-classes is less important,
but personally I do not like - actually, if it has to be, I would rather see
e.g. an ClusterPropCLI-tool support a do-in-zk-flag making it use ZkCLI tool
to download first, do its manipulations and then use ZkCLI to upload again.
That is, new tools that use ZkCLI (if you are too lazy doing the
download/upload using ZkCLI yourself), instead of the other way around, where
ZkCLI uses other tools or even just does json-manipulation itself. Please do
not try to implement rules about what can and cannot be in a specific type of
json, what operations you can do on it etc two places in the code-base - use
what we already have.
Would like to see clean definitions of which classes own responsibilities. E.g.
* ClusterState.java own the responsibility of a clusterstate.json always being
consistent, valid etc. You never do changes to clusterstate.json with using
ClusterState.java
* ClusterStateUpdater own the set of allowed operations on clusterstate.json.
You never manipulate a clusterstate.json without going through
ClusterStateUpdater. The new off-line ClusterStateCLI-tool should use
ClusterStateUpdater to do its operations - just change ClusterStateUpdater, so
that it support receiving jobs not coming from other sources than overseer
queue.
Yes, I know you primarily talk about ClusterProps, but the principle will be
the same. It is just that I know more about which classes handle cluster-state
than which handle cluster-props.
That said, maybe you can live with using e.g. http://trentm.com/json/ (or one
of the million other command-line json tools available) in use another tool
for modifying thefile.json above?
was (Author: steff1193):
I think we should not make ZkCLI a Swiss Army Knife. ZkCLI should deal with ZK
operations, like uploading and downloading. Changing then content of a
json-file is not a ZK operation. Why not see it as different things. So what
you need to do if you want to make arbitrary changes to json-content of a znode
in ZK is
{code}
zkcli getfile thefile.json
use another tool for modifying thefile.json
zkcli putfile thefile.json
{code}
It would be nice to make different tools that can make operations on the
different types of jsons we have in a safe way - ensuring e.g. consistency,
integrity, validity etc. But I think those tools should use the classes already
handling those jsons, and not have anything to do with ZkCLI. E.g. a tool for
manipulating clusterstate.json should use classes from
org.apache.solr.common.cloud in solrj project, like ZkStateReader,
ClusterState, ClusterStateUpdater etc. Then it is always those classes that are
used to operate on a specific type of json, and we only have to build
consistency, integrity, validity etc into those classes (separation of
concern). The new thing should be that those classes can be used via an
external tool also when no Solr nodes are running.
At least make sure to use those existing classes for actually
reading/writing/verifying the jsons, and make separate tool-classes. Changing
ZkCLI to be able to trigger operations in those tool-classes is less important,
but personally I do not like - actually, if it has to be, I would rather see
e.g. an ClusterPropCLI-tool support a do-in-zk-flag making it
[jira] [Comment Edited] (SOLR-7176) allow zkcli to modify JSON
[
https://issues.apache.org/jira/browse/SOLR-7176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14343271#comment-14343271
]
Per Steffensen edited comment on SOLR-7176 at 3/2/15 3:45 PM:
--
I think we should not make ZkCLI a Swiss Army Knife. ZkCLI should deal with ZK
operations, like uploading and downloading. Changing then content of a
json-file is not a ZK operation. Why not see it as different things. So what
you need to do if you want to make arbitrary changes to json-content of a znode
in ZK is
{code}
zkcli getfile thefile.json
use another tool for modifying thefile.json
zkcli putfile thefile.json
{code}
It would be nice to make different tools that can make operations on the
different types of jsons we have in a safe way - ensuring e.g. consistency,
integrity, validity etc. But I think those tools should use the classes already
handling those jsons, and not have anything to do with ZkCLI. E.g. a tool for
manipulating clusterstate.json should use classes from
org.apache.solr.common.cloud in solrj project, like ZkStateReader,
ClusterState, ClusterStateUpdater etc. Then it is always those classes that are
used to operate on a specific type of json, and we only have to build
consistency, integrity, validity etc into those classes (separation of
concern). The new thing should be that those classes can be used via an
external tool also when no Solr nodes are running.
At least make sure to use those existing classes for actually
reading/writing/verifying the jsons, and make separate tool-classes. Changing
ZkCLI to be able to trigger operations in those tool-classes is less important,
but personally I do not like - actually, if it has to be, I would rather see
e.g. an ClusterPropCLI-tool support a do-in-zk-flag making it use ZkCLI tool
to download first, do its manipulations and then use ZkCLI to upload again.
That is, new tools that use ZkCLI (if you are too lazy doing the
download/upload using ZkCLI yourself), instead of the other way around, where
ZkCLI uses other tools or even just does json-manipulation itself. Please do
not try to implement rules about what can and cannot be in a specific type of
json, what operations you can do on it etc two places in the code-base - use
what we already have.
Would like to see clean definitions of which classes own responsibilities. E.g.
* ClusterState.java own the responsibility of a clusterstate.json always being
consistent, valid etc. You never do changes to clusterstate.json with using
ClusterState.java
* ClusterStateUpdater own the set of allowed operations on clusterstate.json.
You never manipulate a clusterstate.json without going through
ClusterStateUpdater. The new off-line ClusterStateCLI-tool should use
ClusterStateUpdater to do its operations - just change ClusterStateUpdater, so
that it support receiving jobs not coming from overseer queue.
Yes, I know you primarily talk about ClusterProps, but the principle will be
the same. It is just that I know more about which classes handle cluster-state
than which handles cluster-props.
That said, maybe you can live with using e.g. http://trentm.com/json/ (or one
of the million other command-line json tools available) in use another tool
for modifying thefile.json above?
was (Author: steff1193):
I think we should not make ZkCLI a Swiss Army Knife. ZkCLI should deal with ZK
operations, like uploading and downloading. Changing then content of a
json-file is not a ZK operation. Why not see it as different things. So what
you need to do if you want to make arbitrary changes to json-content of a znode
in ZK is
{code}
zkcli getfile thefile.json
use another tool for modifying thefile.json
zkcli putfile thefile.json
{code}
It would be nice to make different tools that can make operations on the
different types of jsons we have in a safe way - ensuring e.g. consistency,
integrity, validity etc. But I think those tools should use the classes already
handling those jsons, and not have anything to do with ZkCLI. E.g. a tool for
manipulating clusterstate.json should use classes from
org.apache.solr.common.cloud in solrj project, like ZkStateReader,
ClusterState, ClusterStateUpdater etc. Then it is always those classes that are
used to operate on a specific type of json, and we only have to build
consistency, integrity, validity etc into those classes (separation of
concern). The new thing should be that those classes can be used via an
external tool also when no Solr nodes are running.
At least make sure to use those existing classes for actually
reading/writing/verifying the jsons, and make separate tool-classes. Changing
ZkCLI to be able to trigger operations in those tool-classes is less important,
but personally I do not like - actually, if it has to be, I would rather see
e.g. an ClusterPropCLI-tool support a do-in-zk-flag making it use ZkCLI tool
to
[jira] [Comment Edited] (SOLR-7176) allow zkcli to modify JSON
[
https://issues.apache.org/jira/browse/SOLR-7176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14342833#comment-14342833
]
Ramkumar Aiyengar edited comment on SOLR-7176 at 3/2/15 6:39 AM:
-
Fair enough, if we do see and intend to use clusterprops.json as an interface,
I withdraw my objection.
bq. I guess I see those as orthogonal questions. Even if there is some sort of
clusterprop command for zkcli, the ability to change an arbitrary file seems to
fit right into the feature set of being able to upload arbitrary files (which
zkcli can do today).
+1, if we are adding a clusterprop command as a safety mechanism, it seems
better to add it to a solr config tool even if it just wraps zkcli. The
feature Yonik currently is proposing is a suitable feature to add to what zkcli
does today, being a sharp tool with the ability to modify/damage ZK arbitrarily
as it stands today. Having people moving away from it in favour of a nicer
wrapper tool which does specific things seems like a separate, good feature to
have..
was (Author: andyetitmoves):
Fair enough, if we do see and intend to use cluster props.json, I withdraw my
objection.
bq. I guess I see those as orthogonal questions. Even if there is some sort of
clusterprop command for zkcli, the ability to change an arbitrary file seems to
fit right into the feature set of being able to upload arbitrary files (which
zkcli can do today).
+1, if we are adding a clusterprop command as a safety mechanism, it seems
better to add it to a solr config tool even if it just wraps zkcli. The
feature Yonik currently is proposing is a suitable feature to add to what zkcli
does today, being a sharp tool with the ability to modify/damage ZK arbitrarily
as it stands today. Having people moving away from it in favour of a nicer
wrapper tool which does specific things seems like a separate, good feature to
have..
allow zkcli to modify JSON
--
Key: SOLR-7176
URL: https://issues.apache.org/jira/browse/SOLR-7176
Project: Solr
Issue Type: New Feature
Reporter: Yonik Seeley
Priority: Minor
To enable SSL, we have instructions like the following:
{code}
server/scripts/cloud-scripts/zkcli.sh -zkhost localhost:2181 -cmd put
/clusterprops.json '{urlScheme:https}'
{code}
Overwriting the value won't work well when we have more properties to put in
clusterprops. We should be able to change individual values or perhaps merge
values.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
