Re: Deprecating/removing PropertyFileSnitch?
If you guys are still seeing the problem, would be good to have a JIRA written up, as all the ones linked were fixed in 2017 and 2015. CASSANDRA-13700 was found during our testing, and we haven’t seen any other issues since fixing it. -Jeremiah > On Oct 22, 2018, at 10:12 PM, Sankalp Kohli wrote: > > No worries...I mentioned the issue not the JIRA number > >> On Oct 22, 2018, at 8:01 PM, Jeremiah D Jordan wrote: >> >> Sorry, maybe my spam filter got them or something, but I have never seen a >> JIRA number mentioned in the thread before this one. Just looked back >> through again to make sure, and this is the first email I have with one. >> >> -Jeremiah >> >>> On Oct 22, 2018, at 9:37 PM, sankalp kohli wrote: >>> >>> Here are some of the JIRAs which are fixed but actually did not fix the >>> issue. We have tried fixing this by several patches. May be it will be >>> fixed when Gossip is rewritten(CASSANDRA-12345). I should find or create a >>> new JIRA as this issue still exists. >>> https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_browse_CASSANDRA-2D10366=DwIFaQ=adz96Xi0w1RHqtPMowiL2g=CNZK3RiJDLqhsZDG6FQGnXn8WyPRCQhp4x_uBICNC0g=lI3KEen0YYUim6t3VWsvITHUZfFX8oYaczP_t3kk21o=W_HfejhgW1gmZ06L0CXOnp_EgBQ1oI5MLMoyz0OrvFw= >>> https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_browse_CASSANDRA-2D10089=DwIFaQ=adz96Xi0w1RHqtPMowiL2g=CNZK3RiJDLqhsZDG6FQGnXn8WyPRCQhp4x_uBICNC0g=lI3KEen0YYUim6t3VWsvITHUZfFX8oYaczP_t3kk21o=qXzh1nq2yE27J8SvwYoRf9HPQE83m07cKdKVHXyOyAE= >>> (related to it) >>> >>> Also the quote you are using was written as a follow on email. I have >>> already said what the bug I was referring to. >>> >>> "Say you restarted all instances in the cluster and status for some host >>> goes missing. Now when you start a host replacement, the new host won’t >>> learn about the host whose status is missing and the view of this host will >>> be wrong." >>> >>> - CASSANDRA-10366 >>> >>> >>> On Mon, Oct 22, 2018 at 7:22 PM Sankalp Kohli >>> wrote: >>> I will send the JIRAs of the bug which we thought we have fixed but it still exists. Have you done any correctness testing after doing all these tests...have you done the tests for 1000 instance clusters? It is great you have done these tests and I am hoping the gossiping snitch is good. Also was there any Gossip bug fixed post 3.0? May be I am seeing the bug which is fixed. > On Oct 22, 2018, at 7:09 PM, J. D. Jordan wrote: > > Do you have a specific gossip bug that you have seen recently which caused a problem that would make this happen? Do you have a specific JIRA in mind? “We can’t remove this because what if there is a bug” doesn’t seem like a good enough reason to me. If that was a reason we would never make any changes to anything. > I think many people have seen PFS actually cause real problems, where with GPFS the issue being talked about is predicated on some theoretical gossip bug happening. > In the past year at DataStax we have done a lot of testing on 3.0 and 3.11 around adding nodes, adding DC’s, replacing nodes, replacing racks, and replacing DC’s, all while using GPFS, and as far as I know we have not seen any “lost” rack/DC information during such testing. > > -Jeremiah > >> On Oct 22, 2018, at 5:46 PM, sankalp kohli wrote: >> >> We will have similar issues with Gossip but this will create more issues as >> more things will be relied on Gossip. >> >> I agree PFS should be removed but I dont see how it can be with issues like >> these or someone proves that it wont cause any issues. >> >> On Mon, Oct 22, 2018 at 2:21 PM Paulo Motta >> wrote: >> >>> I can understand keeping PFS for historical/compatibility reasons, but if >>> gossip is broken I think you will have similar ring view problems during >>> replace/bootstrap that would still occur with the use of PFS (such as >>> missing tokens, since those are propagated via gossip), so that doesn't >>> seem like a strong reason to keep it around. >>> >>> With PFS it's pretty easy to shoot yourself in the foot if you're not >>> careful enough to have identical files across nodes and updating it when >>> adding nodes/dcs, so it's seems to be less foolproof than other snitches. >>> While the rejection of verbs to invalid replicas on trunk could address >>> concerns raised by Jeremy, this would only happen after the new node joins >>> the ring, so you would need to re-bootstrap the node and lose all the work >>> done in the original bootstrap. >>> >>> Perhaps one good reason to use PFS is the ability to easily package it >>> across multiple nodes, as pointed out by Sean Durity on CASSANDRA-10745 >>> (which is also it's Achilles'
Re: Deprecating/removing PropertyFileSnitch?
No worries...I mentioned the issue not the JIRA number > On Oct 22, 2018, at 8:01 PM, Jeremiah D Jordan wrote: > > Sorry, maybe my spam filter got them or something, but I have never seen a > JIRA number mentioned in the thread before this one. Just looked back > through again to make sure, and this is the first email I have with one. > > -Jeremiah > >> On Oct 22, 2018, at 9:37 PM, sankalp kohli wrote: >> >> Here are some of the JIRAs which are fixed but actually did not fix the >> issue. We have tried fixing this by several patches. May be it will be >> fixed when Gossip is rewritten(CASSANDRA-12345). I should find or create a >> new JIRA as this issue still exists. >> https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_browse_CASSANDRA-2D10366=DwIFaQ=adz96Xi0w1RHqtPMowiL2g=CNZK3RiJDLqhsZDG6FQGnXn8WyPRCQhp4x_uBICNC0g=lI3KEen0YYUim6t3VWsvITHUZfFX8oYaczP_t3kk21o=W_HfejhgW1gmZ06L0CXOnp_EgBQ1oI5MLMoyz0OrvFw= >> https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_browse_CASSANDRA-2D10089=DwIFaQ=adz96Xi0w1RHqtPMowiL2g=CNZK3RiJDLqhsZDG6FQGnXn8WyPRCQhp4x_uBICNC0g=lI3KEen0YYUim6t3VWsvITHUZfFX8oYaczP_t3kk21o=qXzh1nq2yE27J8SvwYoRf9HPQE83m07cKdKVHXyOyAE= >> (related to it) >> >> Also the quote you are using was written as a follow on email. I have >> already said what the bug I was referring to. >> >> "Say you restarted all instances in the cluster and status for some host >> goes missing. Now when you start a host replacement, the new host won’t >> learn about the host whose status is missing and the view of this host will >> be wrong." >> >> - CASSANDRA-10366 >> >> >> On Mon, Oct 22, 2018 at 7:22 PM Sankalp Kohli >> wrote: >> >>> I will send the JIRAs of the bug which we thought we have fixed but it >>> still exists. >>> >>> Have you done any correctness testing after doing all these tests...have >>> you done the tests for 1000 instance clusters? >>> >>> It is great you have done these tests and I am hoping the gossiping snitch >>> is good. Also was there any Gossip bug fixed post 3.0? May be I am seeing >>> the bug which is fixed. >>> On Oct 22, 2018, at 7:09 PM, J. D. Jordan >>> wrote: Do you have a specific gossip bug that you have seen recently which >>> caused a problem that would make this happen? Do you have a specific JIRA >>> in mind? “We can’t remove this because what if there is a bug” doesn’t >>> seem like a good enough reason to me. If that was a reason we would never >>> make any changes to anything. I think many people have seen PFS actually cause real problems, where >>> with GPFS the issue being talked about is predicated on some theoretical >>> gossip bug happening. In the past year at DataStax we have done a lot of testing on 3.0 and >>> 3.11 around adding nodes, adding DC’s, replacing nodes, replacing racks, >>> and replacing DC’s, all while using GPFS, and as far as I know we have not >>> seen any “lost” rack/DC information during such testing. -Jeremiah > On Oct 22, 2018, at 5:46 PM, sankalp kohli >>> wrote: > > We will have similar issues with Gossip but this will create more >>> issues as > more things will be relied on Gossip. > > I agree PFS should be removed but I dont see how it can be with issues >>> like > these or someone proves that it wont cause any issues. > > On Mon, Oct 22, 2018 at 2:21 PM Paulo Motta > wrote: > >> I can understand keeping PFS for historical/compatibility reasons, but >>> if >> gossip is broken I think you will have similar ring view problems >>> during >> replace/bootstrap that would still occur with the use of PFS (such as >> missing tokens, since those are propagated via gossip), so that doesn't >> seem like a strong reason to keep it around. >> >> With PFS it's pretty easy to shoot yourself in the foot if you're not >> careful enough to have identical files across nodes and updating it >>> when >> adding nodes/dcs, so it's seems to be less foolproof than other >>> snitches. >> While the rejection of verbs to invalid replicas on trunk could address >> concerns raised by Jeremy, this would only happen after the new node >>> joins >> the ring, so you would need to re-bootstrap the node and lose all the >>> work >> done in the original bootstrap. >> >> Perhaps one good reason to use PFS is the ability to easily package it >> across multiple nodes, as pointed out by Sean Durity on CASSANDRA-10745 >> (which is also it's Achilles' heel). To keep this ability, we could >>> make >> GPFS compatible with the cassandra-topology.properties file, but >>> reading >> only the dc/rack info about the local node. >> >> Em seg, 22 de out de 2018 às 16:58, sankalp kohli < >>> kohlisank...@gmail.com> >> escreveu: >> >>> Yes it will happen. I am worried that same way DC or rack info can go >>> missing. >>>
Re: Deprecating/removing PropertyFileSnitch?
Sorry, maybe my spam filter got them or something, but I have never seen a JIRA number mentioned in the thread before this one. Just looked back through again to make sure, and this is the first email I have with one. -Jeremiah > On Oct 22, 2018, at 9:37 PM, sankalp kohli wrote: > > Here are some of the JIRAs which are fixed but actually did not fix the > issue. We have tried fixing this by several patches. May be it will be > fixed when Gossip is rewritten(CASSANDRA-12345). I should find or create a > new JIRA as this issue still exists. > https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_browse_CASSANDRA-2D10366=DwIFaQ=adz96Xi0w1RHqtPMowiL2g=CNZK3RiJDLqhsZDG6FQGnXn8WyPRCQhp4x_uBICNC0g=lI3KEen0YYUim6t3VWsvITHUZfFX8oYaczP_t3kk21o=W_HfejhgW1gmZ06L0CXOnp_EgBQ1oI5MLMoyz0OrvFw= > https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_browse_CASSANDRA-2D10089=DwIFaQ=adz96Xi0w1RHqtPMowiL2g=CNZK3RiJDLqhsZDG6FQGnXn8WyPRCQhp4x_uBICNC0g=lI3KEen0YYUim6t3VWsvITHUZfFX8oYaczP_t3kk21o=qXzh1nq2yE27J8SvwYoRf9HPQE83m07cKdKVHXyOyAE= > (related to it) > > Also the quote you are using was written as a follow on email. I have > already said what the bug I was referring to. > > "Say you restarted all instances in the cluster and status for some host > goes missing. Now when you start a host replacement, the new host won’t > learn about the host whose status is missing and the view of this host will > be wrong." > > - CASSANDRA-10366 > > > On Mon, Oct 22, 2018 at 7:22 PM Sankalp Kohli > wrote: > >> I will send the JIRAs of the bug which we thought we have fixed but it >> still exists. >> >> Have you done any correctness testing after doing all these tests...have >> you done the tests for 1000 instance clusters? >> >> It is great you have done these tests and I am hoping the gossiping snitch >> is good. Also was there any Gossip bug fixed post 3.0? May be I am seeing >> the bug which is fixed. >> >>> On Oct 22, 2018, at 7:09 PM, J. D. Jordan >> wrote: >>> >>> Do you have a specific gossip bug that you have seen recently which >> caused a problem that would make this happen? Do you have a specific JIRA >> in mind? “We can’t remove this because what if there is a bug” doesn’t >> seem like a good enough reason to me. If that was a reason we would never >> make any changes to anything. >>> I think many people have seen PFS actually cause real problems, where >> with GPFS the issue being talked about is predicated on some theoretical >> gossip bug happening. >>> In the past year at DataStax we have done a lot of testing on 3.0 and >> 3.11 around adding nodes, adding DC’s, replacing nodes, replacing racks, >> and replacing DC’s, all while using GPFS, and as far as I know we have not >> seen any “lost” rack/DC information during such testing. >>> >>> -Jeremiah >>> On Oct 22, 2018, at 5:46 PM, sankalp kohli >> wrote: We will have similar issues with Gossip but this will create more >> issues as more things will be relied on Gossip. I agree PFS should be removed but I dont see how it can be with issues >> like these or someone proves that it wont cause any issues. On Mon, Oct 22, 2018 at 2:21 PM Paulo Motta wrote: > I can understand keeping PFS for historical/compatibility reasons, but >> if > gossip is broken I think you will have similar ring view problems >> during > replace/bootstrap that would still occur with the use of PFS (such as > missing tokens, since those are propagated via gossip), so that doesn't > seem like a strong reason to keep it around. > > With PFS it's pretty easy to shoot yourself in the foot if you're not > careful enough to have identical files across nodes and updating it >> when > adding nodes/dcs, so it's seems to be less foolproof than other >> snitches. > While the rejection of verbs to invalid replicas on trunk could address > concerns raised by Jeremy, this would only happen after the new node >> joins > the ring, so you would need to re-bootstrap the node and lose all the >> work > done in the original bootstrap. > > Perhaps one good reason to use PFS is the ability to easily package it > across multiple nodes, as pointed out by Sean Durity on CASSANDRA-10745 > (which is also it's Achilles' heel). To keep this ability, we could >> make > GPFS compatible with the cassandra-topology.properties file, but >> reading > only the dc/rack info about the local node. > > Em seg, 22 de out de 2018 às 16:58, sankalp kohli < >> kohlisank...@gmail.com> > escreveu: > >> Yes it will happen. I am worried that same way DC or rack info can go >> missing. >> >> On Mon, Oct 22, 2018 at 12:52 PM Paulo Motta < >> pauloricard...@gmail.com> >> wrote: >> the new host won’t learn about the host whose status is missing and > the >>> view of this host will be wrong.
Re: Deprecating/removing PropertyFileSnitch?
+1 to fallback and like I said before removing PFS is a good idea as long it is safe > On Oct 22, 2018, at 7:41 PM, Jeff Jirsa wrote: > > On Mon, Oct 22, 2018 at 7:09 PM J. D. Jordan > wrote: > >> Do you have a specific gossip bug that you have seen recently which caused >> a problem that would make this happen? Do you have a specific JIRA in mind? > > > Sankalp linked a few others, but also > https://issues.apache.org/jira/browse/CASSANDRA-13700 > > >> “We can’t remove this because what if there is a bug” doesn’t seem like >> a good enough reason to me. If that was a reason we would never make any >> changes to anything. >> > > How about "we know that certain fields that are gossiped go missing even > after all of the known races are fixed, so removing an existing > low-maintenance feature and forcing users to rely on gossip for topology > may be worth some discussion". > > >> I think many people have seen PFS actually cause real problems, where with >> GPFS the issue being talked about is predicated on some theoretical gossip >> bug happening. >> > > How many of those were actually caused by incorrect fallback from GPFS to > PFS, rather than PFS itself? > > >> In the past year at DataStax we have done a lot of testing on 3.0 and 3.11 >> around adding nodes, adding DC’s, replacing nodes, replacing racks, and >> replacing DC’s, all while using GPFS, and as far as I know we have not seen >> any “lost” rack/DC information during such testing. >> > > I've also run very large GPFS clusters in the past without much gossip > pain, and I'm in the "we should deprecate PFS" camp, but it is also true > that PFS is low maintenance and mostly works. Perhaps the first step is > breaking the GPFS->PFS fallback that people don't know about, maybe that'll > help? - To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org For additional commands, e-mail: dev-h...@cassandra.apache.org
Re: Deprecating/removing PropertyFileSnitch?
On Mon, Oct 22, 2018 at 7:09 PM J. D. Jordan wrote: > Do you have a specific gossip bug that you have seen recently which caused > a problem that would make this happen? Do you have a specific JIRA in mind? Sankalp linked a few others, but also https://issues.apache.org/jira/browse/CASSANDRA-13700 > “We can’t remove this because what if there is a bug” doesn’t seem like > a good enough reason to me. If that was a reason we would never make any > changes to anything. > How about "we know that certain fields that are gossiped go missing even after all of the known races are fixed, so removing an existing low-maintenance feature and forcing users to rely on gossip for topology may be worth some discussion". > I think many people have seen PFS actually cause real problems, where with > GPFS the issue being talked about is predicated on some theoretical gossip > bug happening. > How many of those were actually caused by incorrect fallback from GPFS to PFS, rather than PFS itself? > In the past year at DataStax we have done a lot of testing on 3.0 and 3.11 > around adding nodes, adding DC’s, replacing nodes, replacing racks, and > replacing DC’s, all while using GPFS, and as far as I know we have not seen > any “lost” rack/DC information during such testing. > I've also run very large GPFS clusters in the past without much gossip pain, and I'm in the "we should deprecate PFS" camp, but it is also true that PFS is low maintenance and mostly works. Perhaps the first step is breaking the GPFS->PFS fallback that people don't know about, maybe that'll help?
Re: Deprecating/removing PropertyFileSnitch?
Here are some of the JIRAs which are fixed but actually did not fix the issue. We have tried fixing this by several patches. May be it will be fixed when Gossip is rewritten(CASSANDRA-12345). I should find or create a new JIRA as this issue still exists. https://issues.apache.org/jira/browse/CASSANDRA-10366 https://issues.apache.org/jira/browse/CASSANDRA-10089 (related to it) Also the quote you are using was written as a follow on email. I have already said what the bug I was referring to. "Say you restarted all instances in the cluster and status for some host goes missing. Now when you start a host replacement, the new host won’t learn about the host whose status is missing and the view of this host will be wrong." - CASSANDRA-10366 On Mon, Oct 22, 2018 at 7:22 PM Sankalp Kohli wrote: > I will send the JIRAs of the bug which we thought we have fixed but it > still exists. > > Have you done any correctness testing after doing all these tests...have > you done the tests for 1000 instance clusters? > > It is great you have done these tests and I am hoping the gossiping snitch > is good. Also was there any Gossip bug fixed post 3.0? May be I am seeing > the bug which is fixed. > > > On Oct 22, 2018, at 7:09 PM, J. D. Jordan > wrote: > > > > Do you have a specific gossip bug that you have seen recently which > caused a problem that would make this happen? Do you have a specific JIRA > in mind? “We can’t remove this because what if there is a bug” doesn’t > seem like a good enough reason to me. If that was a reason we would never > make any changes to anything. > > I think many people have seen PFS actually cause real problems, where > with GPFS the issue being talked about is predicated on some theoretical > gossip bug happening. > > In the past year at DataStax we have done a lot of testing on 3.0 and > 3.11 around adding nodes, adding DC’s, replacing nodes, replacing racks, > and replacing DC’s, all while using GPFS, and as far as I know we have not > seen any “lost” rack/DC information during such testing. > > > > -Jeremiah > > > >> On Oct 22, 2018, at 5:46 PM, sankalp kohli > wrote: > >> > >> We will have similar issues with Gossip but this will create more > issues as > >> more things will be relied on Gossip. > >> > >> I agree PFS should be removed but I dont see how it can be with issues > like > >> these or someone proves that it wont cause any issues. > >> > >> On Mon, Oct 22, 2018 at 2:21 PM Paulo Motta > >> wrote: > >> > >>> I can understand keeping PFS for historical/compatibility reasons, but > if > >>> gossip is broken I think you will have similar ring view problems > during > >>> replace/bootstrap that would still occur with the use of PFS (such as > >>> missing tokens, since those are propagated via gossip), so that doesn't > >>> seem like a strong reason to keep it around. > >>> > >>> With PFS it's pretty easy to shoot yourself in the foot if you're not > >>> careful enough to have identical files across nodes and updating it > when > >>> adding nodes/dcs, so it's seems to be less foolproof than other > snitches. > >>> While the rejection of verbs to invalid replicas on trunk could address > >>> concerns raised by Jeremy, this would only happen after the new node > joins > >>> the ring, so you would need to re-bootstrap the node and lose all the > work > >>> done in the original bootstrap. > >>> > >>> Perhaps one good reason to use PFS is the ability to easily package it > >>> across multiple nodes, as pointed out by Sean Durity on CASSANDRA-10745 > >>> (which is also it's Achilles' heel). To keep this ability, we could > make > >>> GPFS compatible with the cassandra-topology.properties file, but > reading > >>> only the dc/rack info about the local node. > >>> > >>> Em seg, 22 de out de 2018 às 16:58, sankalp kohli < > kohlisank...@gmail.com> > >>> escreveu: > >>> > Yes it will happen. I am worried that same way DC or rack info can go > missing. > > On Mon, Oct 22, 2018 at 12:52 PM Paulo Motta < > pauloricard...@gmail.com> > wrote: > > >> the new host won’t learn about the host whose status is missing and > >>> the > > view of this host will be wrong. > > > > Won't this happen even with PropertyFileSnitch as the token(s) for > this > > host will be missing from gossip/system.peers? > > > > Em sáb, 20 de out de 2018 às 00:34, Sankalp Kohli < > kohlisank...@gmail.com> > > escreveu: > > > >> Say you restarted all instances in the cluster and status for some > >>> host > >> goes missing. Now when you start a host replacement, the new host > >>> won’t > >> learn about the host whose status is missing and the view of this > >>> host > > will > >> be wrong. > >> > >> PS: I will be happy to be proved wrong as I can also start using > >>> Gossip > >> snitch :) > >> > >>> On Oct 19, 2018, at 2:41 PM, Jeremy Hanna < > jeremy.hanna1...@gmail.com> > >> wrote: > >>>
Re: Deprecating/removing PropertyFileSnitch?
I will send the JIRAs of the bug which we thought we have fixed but it still exists. Have you done any correctness testing after doing all these tests...have you done the tests for 1000 instance clusters? It is great you have done these tests and I am hoping the gossiping snitch is good. Also was there any Gossip bug fixed post 3.0? May be I am seeing the bug which is fixed. > On Oct 22, 2018, at 7:09 PM, J. D. Jordan wrote: > > Do you have a specific gossip bug that you have seen recently which caused a > problem that would make this happen? Do you have a specific JIRA in mind? > “We can’t remove this because what if there is a bug” doesn’t seem like a > good enough reason to me. If that was a reason we would never make any > changes to anything. > I think many people have seen PFS actually cause real problems, where with > GPFS the issue being talked about is predicated on some theoretical gossip > bug happening. > In the past year at DataStax we have done a lot of testing on 3.0 and 3.11 > around adding nodes, adding DC’s, replacing nodes, replacing racks, and > replacing DC’s, all while using GPFS, and as far as I know we have not seen > any “lost” rack/DC information during such testing. > > -Jeremiah > >> On Oct 22, 2018, at 5:46 PM, sankalp kohli wrote: >> >> We will have similar issues with Gossip but this will create more issues as >> more things will be relied on Gossip. >> >> I agree PFS should be removed but I dont see how it can be with issues like >> these or someone proves that it wont cause any issues. >> >> On Mon, Oct 22, 2018 at 2:21 PM Paulo Motta >> wrote: >> >>> I can understand keeping PFS for historical/compatibility reasons, but if >>> gossip is broken I think you will have similar ring view problems during >>> replace/bootstrap that would still occur with the use of PFS (such as >>> missing tokens, since those are propagated via gossip), so that doesn't >>> seem like a strong reason to keep it around. >>> >>> With PFS it's pretty easy to shoot yourself in the foot if you're not >>> careful enough to have identical files across nodes and updating it when >>> adding nodes/dcs, so it's seems to be less foolproof than other snitches. >>> While the rejection of verbs to invalid replicas on trunk could address >>> concerns raised by Jeremy, this would only happen after the new node joins >>> the ring, so you would need to re-bootstrap the node and lose all the work >>> done in the original bootstrap. >>> >>> Perhaps one good reason to use PFS is the ability to easily package it >>> across multiple nodes, as pointed out by Sean Durity on CASSANDRA-10745 >>> (which is also it's Achilles' heel). To keep this ability, we could make >>> GPFS compatible with the cassandra-topology.properties file, but reading >>> only the dc/rack info about the local node. >>> >>> Em seg, 22 de out de 2018 às 16:58, sankalp kohli >>> escreveu: >>> Yes it will happen. I am worried that same way DC or rack info can go missing. On Mon, Oct 22, 2018 at 12:52 PM Paulo Motta wrote: >> the new host won’t learn about the host whose status is missing and >>> the > view of this host will be wrong. > > Won't this happen even with PropertyFileSnitch as the token(s) for this > host will be missing from gossip/system.peers? > > Em sáb, 20 de out de 2018 às 00:34, Sankalp Kohli < kohlisank...@gmail.com> > escreveu: > >> Say you restarted all instances in the cluster and status for some >>> host >> goes missing. Now when you start a host replacement, the new host >>> won’t >> learn about the host whose status is missing and the view of this >>> host > will >> be wrong. >> >> PS: I will be happy to be proved wrong as I can also start using >>> Gossip >> snitch :) >> >>> On Oct 19, 2018, at 2:41 PM, Jeremy Hanna < jeremy.hanna1...@gmail.com> >> wrote: >>> >>> Do you mean to say that during host replacement there may be a time > when >> the old->new host isn’t fully propagated and therefore wouldn’t yet >>> be in >> all system tables? >>> On Oct 17, 2018, at 4:20 PM, sankalp kohli < >>> kohlisank...@gmail.com> >> wrote: This is not the case during host replacement correct? On Tue, Oct 16, 2018 at 10:04 AM Jeremiah D Jordan < jeremiah.jor...@gmail.com> wrote: > As long as we are correctly storing such things in the system tables >> and > reading them out of the system tables when we do not have the >> information > from gossip yet, it should not be a problem. (As far as I know >>> GPFS >> does > this, but I have not done extensive code diving or testing to >>> make >> sure all > edge cases are covered there) > > -Jeremiah > >> On Oct 16, 2018, at 11:56 AM, sankalp kohli <
Re: Deprecating/removing PropertyFileSnitch?
Do you have a specific gossip bug that you have seen recently which caused a problem that would make this happen? Do you have a specific JIRA in mind? “We can’t remove this because what if there is a bug” doesn’t seem like a good enough reason to me. If that was a reason we would never make any changes to anything. I think many people have seen PFS actually cause real problems, where with GPFS the issue being talked about is predicated on some theoretical gossip bug happening. In the past year at DataStax we have done a lot of testing on 3.0 and 3.11 around adding nodes, adding DC’s, replacing nodes, replacing racks, and replacing DC’s, all while using GPFS, and as far as I know we have not seen any “lost” rack/DC information during such testing. -Jeremiah > On Oct 22, 2018, at 5:46 PM, sankalp kohli wrote: > > We will have similar issues with Gossip but this will create more issues as > more things will be relied on Gossip. > > I agree PFS should be removed but I dont see how it can be with issues like > these or someone proves that it wont cause any issues. > > On Mon, Oct 22, 2018 at 2:21 PM Paulo Motta > wrote: > >> I can understand keeping PFS for historical/compatibility reasons, but if >> gossip is broken I think you will have similar ring view problems during >> replace/bootstrap that would still occur with the use of PFS (such as >> missing tokens, since those are propagated via gossip), so that doesn't >> seem like a strong reason to keep it around. >> >> With PFS it's pretty easy to shoot yourself in the foot if you're not >> careful enough to have identical files across nodes and updating it when >> adding nodes/dcs, so it's seems to be less foolproof than other snitches. >> While the rejection of verbs to invalid replicas on trunk could address >> concerns raised by Jeremy, this would only happen after the new node joins >> the ring, so you would need to re-bootstrap the node and lose all the work >> done in the original bootstrap. >> >> Perhaps one good reason to use PFS is the ability to easily package it >> across multiple nodes, as pointed out by Sean Durity on CASSANDRA-10745 >> (which is also it's Achilles' heel). To keep this ability, we could make >> GPFS compatible with the cassandra-topology.properties file, but reading >> only the dc/rack info about the local node. >> >> Em seg, 22 de out de 2018 às 16:58, sankalp kohli >> escreveu: >> >>> Yes it will happen. I am worried that same way DC or rack info can go >>> missing. >>> >>> On Mon, Oct 22, 2018 at 12:52 PM Paulo Motta >>> wrote: >>> > the new host won’t learn about the host whose status is missing and >> the view of this host will be wrong. Won't this happen even with PropertyFileSnitch as the token(s) for this host will be missing from gossip/system.peers? Em sáb, 20 de out de 2018 às 00:34, Sankalp Kohli < >>> kohlisank...@gmail.com> escreveu: > Say you restarted all instances in the cluster and status for some >> host > goes missing. Now when you start a host replacement, the new host >> won’t > learn about the host whose status is missing and the view of this >> host will > be wrong. > > PS: I will be happy to be proved wrong as I can also start using >> Gossip > snitch :) > >> On Oct 19, 2018, at 2:41 PM, Jeremy Hanna < >>> jeremy.hanna1...@gmail.com> > wrote: >> >> Do you mean to say that during host replacement there may be a time when > the old->new host isn’t fully propagated and therefore wouldn’t yet >> be >>> in > all system tables? >> >>> On Oct 17, 2018, at 4:20 PM, sankalp kohli < >> kohlisank...@gmail.com> > wrote: >>> >>> This is not the case during host replacement correct? >>> >>> On Tue, Oct 16, 2018 at 10:04 AM Jeremiah D Jordan < >>> jeremiah.jor...@gmail.com> wrote: >>> As long as we are correctly storing such things in the system >>> tables > and reading them out of the system tables when we do not have the > information from gossip yet, it should not be a problem. (As far as I know >> GPFS > does this, but I have not done extensive code diving or testing to >> make > sure all edge cases are covered there) -Jeremiah > On Oct 16, 2018, at 11:56 AM, sankalp kohli < >>> kohlisank...@gmail.com > wrote: > > Will GossipingPropertyFileSnitch not be vulnerable to Gossip >> bugs > where we > lose hostId or some other fields when we restart C* for large > clusters(~1000 instances)? > >> On Tue, Oct 16, 2018 at 7:59 AM Jeff Jirsa wrote: >> >> We should, but the 4.0 features that log/reject verbs to >> invalid replicas >> solves a lot of the concerns here >> >> -- >> Jeff Jirsa >> >>
Re: Deprecating/removing PropertyFileSnitch?
We will have similar issues with Gossip but this will create more issues as more things will be relied on Gossip. I agree PFS should be removed but I dont see how it can be with issues like these or someone proves that it wont cause any issues. On Mon, Oct 22, 2018 at 2:21 PM Paulo Motta wrote: > I can understand keeping PFS for historical/compatibility reasons, but if > gossip is broken I think you will have similar ring view problems during > replace/bootstrap that would still occur with the use of PFS (such as > missing tokens, since those are propagated via gossip), so that doesn't > seem like a strong reason to keep it around. > > With PFS it's pretty easy to shoot yourself in the foot if you're not > careful enough to have identical files across nodes and updating it when > adding nodes/dcs, so it's seems to be less foolproof than other snitches. > While the rejection of verbs to invalid replicas on trunk could address > concerns raised by Jeremy, this would only happen after the new node joins > the ring, so you would need to re-bootstrap the node and lose all the work > done in the original bootstrap. > > Perhaps one good reason to use PFS is the ability to easily package it > across multiple nodes, as pointed out by Sean Durity on CASSANDRA-10745 > (which is also it's Achilles' heel). To keep this ability, we could make > GPFS compatible with the cassandra-topology.properties file, but reading > only the dc/rack info about the local node. > > Em seg, 22 de out de 2018 às 16:58, sankalp kohli > escreveu: > > > Yes it will happen. I am worried that same way DC or rack info can go > > missing. > > > > On Mon, Oct 22, 2018 at 12:52 PM Paulo Motta > > wrote: > > > > > > the new host won’t learn about the host whose status is missing and > the > > > view of this host will be wrong. > > > > > > Won't this happen even with PropertyFileSnitch as the token(s) for this > > > host will be missing from gossip/system.peers? > > > > > > Em sáb, 20 de out de 2018 às 00:34, Sankalp Kohli < > > kohlisank...@gmail.com> > > > escreveu: > > > > > > > Say you restarted all instances in the cluster and status for some > host > > > > goes missing. Now when you start a host replacement, the new host > won’t > > > > learn about the host whose status is missing and the view of this > host > > > will > > > > be wrong. > > > > > > > > PS: I will be happy to be proved wrong as I can also start using > Gossip > > > > snitch :) > > > > > > > > > On Oct 19, 2018, at 2:41 PM, Jeremy Hanna < > > jeremy.hanna1...@gmail.com> > > > > wrote: > > > > > > > > > > Do you mean to say that during host replacement there may be a time > > > when > > > > the old->new host isn’t fully propagated and therefore wouldn’t yet > be > > in > > > > all system tables? > > > > > > > > > >> On Oct 17, 2018, at 4:20 PM, sankalp kohli < > kohlisank...@gmail.com> > > > > wrote: > > > > >> > > > > >> This is not the case during host replacement correct? > > > > >> > > > > >> On Tue, Oct 16, 2018 at 10:04 AM Jeremiah D Jordan < > > > > >> jeremiah.jor...@gmail.com> wrote: > > > > >> > > > > >>> As long as we are correctly storing such things in the system > > tables > > > > and > > > > >>> reading them out of the system tables when we do not have the > > > > information > > > > >>> from gossip yet, it should not be a problem. (As far as I know > GPFS > > > > does > > > > >>> this, but I have not done extensive code diving or testing to > make > > > > sure all > > > > >>> edge cases are covered there) > > > > >>> > > > > >>> -Jeremiah > > > > >>> > > > > On Oct 16, 2018, at 11:56 AM, sankalp kohli < > > kohlisank...@gmail.com > > > > > > > > >>> wrote: > > > > > > > > Will GossipingPropertyFileSnitch not be vulnerable to Gossip > bugs > > > > where > > > > >>> we > > > > lose hostId or some other fields when we restart C* for large > > > > clusters(~1000 instances)? > > > > > > > > > On Tue, Oct 16, 2018 at 7:59 AM Jeff Jirsa > > > wrote: > > > > > > > > > > We should, but the 4.0 features that log/reject verbs to > invalid > > > > >>> replicas > > > > > solves a lot of the concerns here > > > > > > > > > > -- > > > > > Jeff Jirsa > > > > > > > > > > > > > > >> On Oct 16, 2018, at 4:10 PM, Jeremy Hanna < > > > > jeremy.hanna1...@gmail.com> > > > > > wrote: > > > > >> > > > > >> We have had PropertyFileSnitch for a long time even though > > > > > GossipingPropertyFileSnitch is effectively a superset of what > it > > > > offers > > > > >>> and > > > > > is much less error prone. There are some unexpected behaviors > > when > > > > >>> things > > > > > aren’t configured correctly with PFS. For example, if you > > replace > > > > >>> nodes in > > > > > one DC and add those nodes to that DCs property files and not > the > > > > other > > > > >>> DCs > > > > > property files - the resulting problems aren’t very > > straightforward > > > > to > > > > >
Re: Deprecating/removing PropertyFileSnitch?
I can understand keeping PFS for historical/compatibility reasons, but if gossip is broken I think you will have similar ring view problems during replace/bootstrap that would still occur with the use of PFS (such as missing tokens, since those are propagated via gossip), so that doesn't seem like a strong reason to keep it around. With PFS it's pretty easy to shoot yourself in the foot if you're not careful enough to have identical files across nodes and updating it when adding nodes/dcs, so it's seems to be less foolproof than other snitches. While the rejection of verbs to invalid replicas on trunk could address concerns raised by Jeremy, this would only happen after the new node joins the ring, so you would need to re-bootstrap the node and lose all the work done in the original bootstrap. Perhaps one good reason to use PFS is the ability to easily package it across multiple nodes, as pointed out by Sean Durity on CASSANDRA-10745 (which is also it's Achilles' heel). To keep this ability, we could make GPFS compatible with the cassandra-topology.properties file, but reading only the dc/rack info about the local node. Em seg, 22 de out de 2018 às 16:58, sankalp kohli escreveu: > Yes it will happen. I am worried that same way DC or rack info can go > missing. > > On Mon, Oct 22, 2018 at 12:52 PM Paulo Motta > wrote: > > > > the new host won’t learn about the host whose status is missing and the > > view of this host will be wrong. > > > > Won't this happen even with PropertyFileSnitch as the token(s) for this > > host will be missing from gossip/system.peers? > > > > Em sáb, 20 de out de 2018 às 00:34, Sankalp Kohli < > kohlisank...@gmail.com> > > escreveu: > > > > > Say you restarted all instances in the cluster and status for some host > > > goes missing. Now when you start a host replacement, the new host won’t > > > learn about the host whose status is missing and the view of this host > > will > > > be wrong. > > > > > > PS: I will be happy to be proved wrong as I can also start using Gossip > > > snitch :) > > > > > > > On Oct 19, 2018, at 2:41 PM, Jeremy Hanna < > jeremy.hanna1...@gmail.com> > > > wrote: > > > > > > > > Do you mean to say that during host replacement there may be a time > > when > > > the old->new host isn’t fully propagated and therefore wouldn’t yet be > in > > > all system tables? > > > > > > > >> On Oct 17, 2018, at 4:20 PM, sankalp kohli > > > wrote: > > > >> > > > >> This is not the case during host replacement correct? > > > >> > > > >> On Tue, Oct 16, 2018 at 10:04 AM Jeremiah D Jordan < > > > >> jeremiah.jor...@gmail.com> wrote: > > > >> > > > >>> As long as we are correctly storing such things in the system > tables > > > and > > > >>> reading them out of the system tables when we do not have the > > > information > > > >>> from gossip yet, it should not be a problem. (As far as I know GPFS > > > does > > > >>> this, but I have not done extensive code diving or testing to make > > > sure all > > > >>> edge cases are covered there) > > > >>> > > > >>> -Jeremiah > > > >>> > > > On Oct 16, 2018, at 11:56 AM, sankalp kohli < > kohlisank...@gmail.com > > > > > > >>> wrote: > > > > > > Will GossipingPropertyFileSnitch not be vulnerable to Gossip bugs > > > where > > > >>> we > > > lose hostId or some other fields when we restart C* for large > > > clusters(~1000 instances)? > > > > > > > On Tue, Oct 16, 2018 at 7:59 AM Jeff Jirsa > > wrote: > > > > > > > > We should, but the 4.0 features that log/reject verbs to invalid > > > >>> replicas > > > > solves a lot of the concerns here > > > > > > > > -- > > > > Jeff Jirsa > > > > > > > > > > > >> On Oct 16, 2018, at 4:10 PM, Jeremy Hanna < > > > jeremy.hanna1...@gmail.com> > > > > wrote: > > > >> > > > >> We have had PropertyFileSnitch for a long time even though > > > > GossipingPropertyFileSnitch is effectively a superset of what it > > > offers > > > >>> and > > > > is much less error prone. There are some unexpected behaviors > when > > > >>> things > > > > aren’t configured correctly with PFS. For example, if you > replace > > > >>> nodes in > > > > one DC and add those nodes to that DCs property files and not the > > > other > > > >>> DCs > > > > property files - the resulting problems aren’t very > straightforward > > > to > > > > troubleshoot. > > > >> > > > >> We could try to improve the resilience and fail fast error > > checking > > > and > > > > error reporting of PFS, but honestly, why wouldn’t we deprecate > and > > > >>> remove > > > > PropertyFileSnitch? Are there reasons why GPFS wouldn’t be > > > sufficient > > > >>> to > > > > replace it? > > > >> > > > - > > > >> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org > > > >> For additional commands, e-mail: dev-h...@cassandra.apache.org > > >
Re: Deprecating/removing PropertyFileSnitch?
Yes it will happen. I am worried that same way DC or rack info can go missing. On Mon, Oct 22, 2018 at 12:52 PM Paulo Motta wrote: > > the new host won’t learn about the host whose status is missing and the > view of this host will be wrong. > > Won't this happen even with PropertyFileSnitch as the token(s) for this > host will be missing from gossip/system.peers? > > Em sáb, 20 de out de 2018 às 00:34, Sankalp Kohli > escreveu: > > > Say you restarted all instances in the cluster and status for some host > > goes missing. Now when you start a host replacement, the new host won’t > > learn about the host whose status is missing and the view of this host > will > > be wrong. > > > > PS: I will be happy to be proved wrong as I can also start using Gossip > > snitch :) > > > > > On Oct 19, 2018, at 2:41 PM, Jeremy Hanna > > wrote: > > > > > > Do you mean to say that during host replacement there may be a time > when > > the old->new host isn’t fully propagated and therefore wouldn’t yet be in > > all system tables? > > > > > >> On Oct 17, 2018, at 4:20 PM, sankalp kohli > > wrote: > > >> > > >> This is not the case during host replacement correct? > > >> > > >> On Tue, Oct 16, 2018 at 10:04 AM Jeremiah D Jordan < > > >> jeremiah.jor...@gmail.com> wrote: > > >> > > >>> As long as we are correctly storing such things in the system tables > > and > > >>> reading them out of the system tables when we do not have the > > information > > >>> from gossip yet, it should not be a problem. (As far as I know GPFS > > does > > >>> this, but I have not done extensive code diving or testing to make > > sure all > > >>> edge cases are covered there) > > >>> > > >>> -Jeremiah > > >>> > > On Oct 16, 2018, at 11:56 AM, sankalp kohli > > > >>> wrote: > > > > Will GossipingPropertyFileSnitch not be vulnerable to Gossip bugs > > where > > >>> we > > lose hostId or some other fields when we restart C* for large > > clusters(~1000 instances)? > > > > > On Tue, Oct 16, 2018 at 7:59 AM Jeff Jirsa > wrote: > > > > > > We should, but the 4.0 features that log/reject verbs to invalid > > >>> replicas > > > solves a lot of the concerns here > > > > > > -- > > > Jeff Jirsa > > > > > > > > >> On Oct 16, 2018, at 4:10 PM, Jeremy Hanna < > > jeremy.hanna1...@gmail.com> > > > wrote: > > >> > > >> We have had PropertyFileSnitch for a long time even though > > > GossipingPropertyFileSnitch is effectively a superset of what it > > offers > > >>> and > > > is much less error prone. There are some unexpected behaviors when > > >>> things > > > aren’t configured correctly with PFS. For example, if you replace > > >>> nodes in > > > one DC and add those nodes to that DCs property files and not the > > other > > >>> DCs > > > property files - the resulting problems aren’t very straightforward > > to > > > troubleshoot. > > >> > > >> We could try to improve the resilience and fail fast error > checking > > and > > > error reporting of PFS, but honestly, why wouldn’t we deprecate and > > >>> remove > > > PropertyFileSnitch? Are there reasons why GPFS wouldn’t be > > sufficient > > >>> to > > > replace it? > > >> > > - > > >> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org > > >> For additional commands, e-mail: dev-h...@cassandra.apache.org > > >> > > > > > > > - > > > To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org > > > For additional commands, e-mail: dev-h...@cassandra.apache.org > > > > > > > > >>> > > >>> > > >>> - > > >>> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org > > >>> For additional commands, e-mail: dev-h...@cassandra.apache.org > > >>> > > >>> > > > > > > > > > - > > > To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org > > > For additional commands, e-mail: dev-h...@cassandra.apache.org > > > > > > > - > > To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org > > For additional commands, e-mail: dev-h...@cassandra.apache.org > > > > >
Re: Deprecating/removing PropertyFileSnitch?
> the new host won’t learn about the host whose status is missing and the view of this host will be wrong. Won't this happen even with PropertyFileSnitch as the token(s) for this host will be missing from gossip/system.peers? Em sáb, 20 de out de 2018 às 00:34, Sankalp Kohli escreveu: > Say you restarted all instances in the cluster and status for some host > goes missing. Now when you start a host replacement, the new host won’t > learn about the host whose status is missing and the view of this host will > be wrong. > > PS: I will be happy to be proved wrong as I can also start using Gossip > snitch :) > > > On Oct 19, 2018, at 2:41 PM, Jeremy Hanna > wrote: > > > > Do you mean to say that during host replacement there may be a time when > the old->new host isn’t fully propagated and therefore wouldn’t yet be in > all system tables? > > > >> On Oct 17, 2018, at 4:20 PM, sankalp kohli > wrote: > >> > >> This is not the case during host replacement correct? > >> > >> On Tue, Oct 16, 2018 at 10:04 AM Jeremiah D Jordan < > >> jeremiah.jor...@gmail.com> wrote: > >> > >>> As long as we are correctly storing such things in the system tables > and > >>> reading them out of the system tables when we do not have the > information > >>> from gossip yet, it should not be a problem. (As far as I know GPFS > does > >>> this, but I have not done extensive code diving or testing to make > sure all > >>> edge cases are covered there) > >>> > >>> -Jeremiah > >>> > On Oct 16, 2018, at 11:56 AM, sankalp kohli > >>> wrote: > > Will GossipingPropertyFileSnitch not be vulnerable to Gossip bugs > where > >>> we > lose hostId or some other fields when we restart C* for large > clusters(~1000 instances)? > > > On Tue, Oct 16, 2018 at 7:59 AM Jeff Jirsa wrote: > > > > We should, but the 4.0 features that log/reject verbs to invalid > >>> replicas > > solves a lot of the concerns here > > > > -- > > Jeff Jirsa > > > > > >> On Oct 16, 2018, at 4:10 PM, Jeremy Hanna < > jeremy.hanna1...@gmail.com> > > wrote: > >> > >> We have had PropertyFileSnitch for a long time even though > > GossipingPropertyFileSnitch is effectively a superset of what it > offers > >>> and > > is much less error prone. There are some unexpected behaviors when > >>> things > > aren’t configured correctly with PFS. For example, if you replace > >>> nodes in > > one DC and add those nodes to that DCs property files and not the > other > >>> DCs > > property files - the resulting problems aren’t very straightforward > to > > troubleshoot. > >> > >> We could try to improve the resilience and fail fast error checking > and > > error reporting of PFS, but honestly, why wouldn’t we deprecate and > >>> remove > > PropertyFileSnitch? Are there reasons why GPFS wouldn’t be > sufficient > >>> to > > replace it? > >> > - > >> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org > >> For additional commands, e-mail: dev-h...@cassandra.apache.org > >> > > > > - > > To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org > > For additional commands, e-mail: dev-h...@cassandra.apache.org > > > > > >>> > >>> > >>> - > >>> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org > >>> For additional commands, e-mail: dev-h...@cassandra.apache.org > >>> > >>> > > > > > > - > > To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org > > For additional commands, e-mail: dev-h...@cassandra.apache.org > > > > - > To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org > For additional commands, e-mail: dev-h...@cassandra.apache.org > >
