Re: create or setData in transaction?
As Michael correctly said, isolation only makes sense when you allow concurrent queries. Of the four ACID properties, the multi op satisfies A, C and D while I is essentially irrelevant (or could be said to be trivially satisfied since there are no concurrent queries. On Wed, Aug 14, 2019 at 6:45 PM Zili Chen wrote: > Thanks for your reply Ted. > > I cannot understand the statement "That leaves isolated which is kind of > hard to talk about with ZK since all operations are fast and sequential." > well. Could you explain a bit? What is "that" means and where is the "hard" > comes from? > > Best, > tison. > > > Ted Dunning 于2019年8月15日周四 上午9:40写道： > > > The multi op is atomic (all other operations will be before or after teh > > multi), consistent (all viewers will see all the effects or none, and > > durable (because ZK is linearized anyway). > > > > That leaves isolated which is kind of hard to talk about with ZK since > all > > operations are fast and sequential. > > > > On Wed, Aug 14, 2019 at 3:12 PM Michael Han wrote: > > > > > ... > > > Ted can correct me if I am wrong, since he added the multi op feature, > > but > > > my understanding is "multi op" is branded from day one as the > transaction > > > support for zookeeper (we even provide an API with exact name: > > > Transaction). If we use the traditional semantic for transaction in > > > database context, the ACID properties multi-op satisfies at least > > atomicity > > > and durability. So saying zookeeper does not support transaction seems > a > > > strong argument that against the properties of multi-op and existing > > > literatures related to zookeeper. On the other side, typically bulk > > > operations does not support atomicity, which will not take care of > > rolling > > > back failed operations. > > > > > > > > > > > > >
Re: create or setData in transaction?
>> where is the "hard" comes from? Isolation levels is only meaningful when concurrent transactions are allowed. For zookeeper, there is no concurrent transactions, as every transaction (and write operations in general) is serialized. On Wed, Aug 14, 2019 at 6:45 PM Zili Chen wrote: > Thanks for your reply Ted. > > I cannot understand the statement "That leaves isolated which is kind of > hard to talk about with ZK since all operations are fast and sequential." > well. Could you explain a bit? What is "that" means and where is the "hard" > comes from? > > Best, > tison. > > > Ted Dunning 于2019年8月15日周四 上午9:40写道： > > > The multi op is atomic (all other operations will be before or after teh > > multi), consistent (all viewers will see all the effects or none, and > > durable (because ZK is linearized anyway). > > > > That leaves isolated which is kind of hard to talk about with ZK since > all > > operations are fast and sequential. > > > > On Wed, Aug 14, 2019 at 3:12 PM Michael Han wrote: > > > > > ... > > > Ted can correct me if I am wrong, since he added the multi op feature, > > but > > > my understanding is "multi op" is branded from day one as the > transaction > > > support for zookeeper (we even provide an API with exact name: > > > Transaction). If we use the traditional semantic for transaction in > > > database context, the ACID properties multi-op satisfies at least > > atomicity > > > and durability. So saying zookeeper does not support transaction seems > a > > > strong argument that against the properties of multi-op and existing > > > literatures related to zookeeper. On the other side, typically bulk > > > operations does not support atomicity, which will not take care of > > rolling > > > back failed operations. > > > > > > > > > > > > >
Re: create or setData in transaction?
Thanks for your reply Ted. I cannot understand the statement "That leaves isolated which is kind of hard to talk about with ZK since all operations are fast and sequential." well. Could you explain a bit? What is "that" means and where is the "hard" comes from? Best, tison. Ted Dunning 于2019年8月15日周四 上午9:40写道： > The multi op is atomic (all other operations will be before or after teh > multi), consistent (all viewers will see all the effects or none, and > durable (because ZK is linearized anyway). > > That leaves isolated which is kind of hard to talk about with ZK since all > operations are fast and sequential. > > On Wed, Aug 14, 2019 at 3:12 PM Michael Han wrote: > > > ... > > Ted can correct me if I am wrong, since he added the multi op feature, > but > > my understanding is "multi op" is branded from day one as the transaction > > support for zookeeper (we even provide an API with exact name: > > Transaction). If we use the traditional semantic for transaction in > > database context, the ACID properties multi-op satisfies at least > atomicity > > and durability. So saying zookeeper does not support transaction seems a > > strong argument that against the properties of multi-op and existing > > literatures related to zookeeper. On the other side, typically bulk > > operations does not support atomicity, which will not take care of > rolling > > back failed operations. > > > > > > >
Re: create or setData in transaction?
OuO
Let me take a closer look...
Best,
tison.
Ted Dunning 于2019年8月15日周四 上午9:37写道：
> The multi operation *is* like a transaction. All the operations will
> succeed or none.
>
>
>
> On Wed, Aug 14, 2019 at 9:33 AM Andor Molnar wrote:
>
> > "it's said that ZooKeeper has a transaction mechanism”
> >
> > I’m still confused with this. ZooKeeper doesn’t support transactions to
> my
> > best knowledge. It has a `multi` operation feature, but that’s more like
> a
> > bulk operation, not transaction.
> >
> > "I want to tell ZooKeeper that the check and setData should be
> successful”
> >
> > I don’t think you can do that. ZK has no check-and-set support either.
> >
> > Maybe we should step back first and see what’s your use case exactly that
> > you’re trying to solve with ZooKeeper. I suspect that you’re trying to
> > follow the wrong approach or misusing ZooKeeper.
> >
> > Have you checked our tutorial and recipes page?
> > You can find some recommended usage patterns:
> > https://zookeeper.apache.org/doc/r3.5.5/recipes.html
> > https://zookeeper.apache.org/doc/r3.5.5/zookeeperTutorial.html
> >
> > If that’s not enough, you could also try Curator which has even more
> > built-in high level functionalities on top of basic ZK commands.
> >
> > Andor
> >
> >
> >
> > > On 2019. Aug 14., at 17:52, Zili Chen wrote:
> > >
> > > Hi Andor,
> > >
> > > Thanks for your attention.
> > >
> > > The problem is that in concurrent scenario zk.setData() could still
> > failed
> > > if there is another thread delete the node. I know with proper lock
> > strategy
> > > and ownership separation this can be avoid but it's said that ZooKeeper
> > has
> > > a transaction mechanism so I'd like to see whether I can make use of
> it.
> > >
> > > There is where I turn to
> > >
> > > zk.multi(Op.check(path1), Op.setData(path2, data)); // path1 == or !=
> > path2
> > > is irrelevant
> > >
> > > when the existence of a mark node(path1) guarded a condition and I want
> > to
> > > make
> > > sure that setData successes only if the mark node exist. If I check the
> > > existence
> > > first and commit setData, a remove to the node could break the guard.
> In
> > > other
> > > words, I want to tell ZooKeeper that the check and setData should be
> > > successful
> > > committed or fail to be committed atomically.
> > >
> > > Best,
> > > tison.
> > >
> > >
> > > Andor Molnar 于2019年8月14日周三 下午11:12写道：
> > >
> > >> Hi Zili,
> > >>
> > >> There’s no such functionality in ZooKeeper as far as I’m concerned. I
> > >> think your multi example (zk.multi(Op.check(path), Op.setData(path,
> > data)))
> > >> is already a usage pattern which multi is not designed to support.
> > >>
> > >> Why do you need to do this in “transactions” (there’s no transaction
> in
> > >> ZK)?
> > >>
> > >> In Java you can do:
> > >>
> > >> try {
> > >> zk.create();
> > >> } catch (NodeExistsException e) {
> > >> // swallow exception
> > >> }
> > >> zk.setData();
> > >> …
> > >>
> > >> Regards,
> > >> Andor
> > >>
> > >>
> > >>
> > >>
> > >>> On 2019. Aug 6., at 14:44, Zili Chen wrote:
> > >>>
> > >>> Hi Enrico,
> > >>>
> > >>> Thanks for your reply.
> > >>>
> > In this case usually you use conditional setData, using the
> 'version'
> > of
> > thr znode
> > >>>
> > >>>
> > >>> what if the caller has no idea on whether the node exist?(see also
> > >>> my if-else pseudo-code above.)
> > >>>
> > >>> IIRC if we call `setData` on a non-exist path a NoNodeException
> > >>> will be thrown.
> > >>> Best,
> > >>> tison.
> > >>>
> > >>>
> > >>> Enrico Olivelli 于2019年8月6日周二 下午8:27写道：
> > >>>
> > Il mar 6 ago 2019, 13:47 Zili Chen ha
> scritto:
> >
> > > Any ideas?
> > >
> > >
> > > Zili Chen 于2019年7月29日周一 上午11:12写道：
> > >
> > >> Hi ZooKeepers,
> > >>
> > >> Currently our transaction mechanism supports doing
> > >> create/setData/checkExist/delete in transaction. However, taking
> > this
> > >> scenario into consideration, we want to put data in path "/path"
> but
> > >> don't know whether the znode exists or not. Let's say we program
> as
> > >> below
> > >>
> > >> if (zk.exist(path)) {
> > >> zk.setData(path, data);
> > >> } else {
> > >> zk.create(path, data);
> > >> }
> > >
> >
> > Do you need to perform other ops in the same transaction?
> > In this case usually you use conditional setData, using the
> 'version'
> > of
> > thr znode
> >
> >
> > Enrico
> >
> > >
> > >> if we want to do the check and "put" in transaction, it would be
> > like
> > >>
> > >> zk.multi(Op.check(path), Op.setData(path, data));
> > >>
> > >> but we cannot add a "else" branch. ZooKeeper's transaction would
> all
> > >> success or fail.
> > >>
> > >> Is there any way to do an "if-else" transaction?
> > >>
> > >> Best,
> > >> tison.
> > >>
> > >
> >
> > >>
> > >>
> >
> >
>
Re: create or setData in transaction?
The multi op is atomic (all other operations will be before or after teh multi), consistent (all viewers will see all the effects or none, and durable (because ZK is linearized anyway). That leaves isolated which is kind of hard to talk about with ZK since all operations are fast and sequential. On Wed, Aug 14, 2019 at 3:12 PM Michael Han wrote: > ... > Ted can correct me if I am wrong, since he added the multi op feature, but > my understanding is "multi op" is branded from day one as the transaction > support for zookeeper (we even provide an API with exact name: > Transaction). If we use the traditional semantic for transaction in > database context, the ACID properties multi-op satisfies at least atomicity > and durability. So saying zookeeper does not support transaction seems a > strong argument that against the properties of multi-op and existing > literatures related to zookeeper. On the other side, typically bulk > operations does not support atomicity, which will not take care of rolling > back failed operations. > >
Re: create or setData in transaction?
The multi operation *is* like a transaction. All the operations will
succeed or none.
On Wed, Aug 14, 2019 at 9:33 AM Andor Molnar wrote:
> "it's said that ZooKeeper has a transaction mechanism”
>
> I’m still confused with this. ZooKeeper doesn’t support transactions to my
> best knowledge. It has a `multi` operation feature, but that’s more like a
> bulk operation, not transaction.
>
> "I want to tell ZooKeeper that the check and setData should be successful”
>
> I don’t think you can do that. ZK has no check-and-set support either.
>
> Maybe we should step back first and see what’s your use case exactly that
> you’re trying to solve with ZooKeeper. I suspect that you’re trying to
> follow the wrong approach or misusing ZooKeeper.
>
> Have you checked our tutorial and recipes page?
> You can find some recommended usage patterns:
> https://zookeeper.apache.org/doc/r3.5.5/recipes.html
> https://zookeeper.apache.org/doc/r3.5.5/zookeeperTutorial.html
>
> If that’s not enough, you could also try Curator which has even more
> built-in high level functionalities on top of basic ZK commands.
>
> Andor
>
>
>
> > On 2019. Aug 14., at 17:52, Zili Chen wrote:
> >
> > Hi Andor,
> >
> > Thanks for your attention.
> >
> > The problem is that in concurrent scenario zk.setData() could still
> failed
> > if there is another thread delete the node. I know with proper lock
> strategy
> > and ownership separation this can be avoid but it's said that ZooKeeper
> has
> > a transaction mechanism so I'd like to see whether I can make use of it.
> >
> > There is where I turn to
> >
> > zk.multi(Op.check(path1), Op.setData(path2, data)); // path1 == or !=
> path2
> > is irrelevant
> >
> > when the existence of a mark node(path1) guarded a condition and I want
> to
> > make
> > sure that setData successes only if the mark node exist. If I check the
> > existence
> > first and commit setData, a remove to the node could break the guard. In
> > other
> > words, I want to tell ZooKeeper that the check and setData should be
> > successful
> > committed or fail to be committed atomically.
> >
> > Best,
> > tison.
> >
> >
> > Andor Molnar 于2019年8月14日周三 下午11:12写道：
> >
> >> Hi Zili,
> >>
> >> There’s no such functionality in ZooKeeper as far as I’m concerned. I
> >> think your multi example (zk.multi(Op.check(path), Op.setData(path,
> data)))
> >> is already a usage pattern which multi is not designed to support.
> >>
> >> Why do you need to do this in “transactions” (there’s no transaction in
> >> ZK)?
> >>
> >> In Java you can do:
> >>
> >> try {
> >> zk.create();
> >> } catch (NodeExistsException e) {
> >> // swallow exception
> >> }
> >> zk.setData();
> >> …
> >>
> >> Regards,
> >> Andor
> >>
> >>
> >>
> >>
> >>> On 2019. Aug 6., at 14:44, Zili Chen wrote:
> >>>
> >>> Hi Enrico,
> >>>
> >>> Thanks for your reply.
> >>>
> In this case usually you use conditional setData, using the 'version'
> of
> thr znode
> >>>
> >>>
> >>> what if the caller has no idea on whether the node exist?(see also
> >>> my if-else pseudo-code above.)
> >>>
> >>> IIRC if we call `setData` on a non-exist path a NoNodeException
> >>> will be thrown.
> >>> Best,
> >>> tison.
> >>>
> >>>
> >>> Enrico Olivelli 于2019年8月6日周二 下午8:27写道：
> >>>
> Il mar 6 ago 2019, 13:47 Zili Chen ha scritto:
>
> > Any ideas?
> >
> >
> > Zili Chen 于2019年7月29日周一 上午11:12写道：
> >
> >> Hi ZooKeepers,
> >>
> >> Currently our transaction mechanism supports doing
> >> create/setData/checkExist/delete in transaction. However, taking
> this
> >> scenario into consideration, we want to put data in path "/path" but
> >> don't know whether the znode exists or not. Let's say we program as
> >> below
> >>
> >> if (zk.exist(path)) {
> >> zk.setData(path, data);
> >> } else {
> >> zk.create(path, data);
> >> }
> >
>
> Do you need to perform other ops in the same transaction?
> In this case usually you use conditional setData, using the 'version'
> of
> thr znode
>
>
> Enrico
>
> >
> >> if we want to do the check and "put" in transaction, it would be
> like
> >>
> >> zk.multi(Op.check(path), Op.setData(path, data));
> >>
> >> but we cannot add a "else" branch. ZooKeeper's transaction would all
> >> success or fail.
> >>
> >> Is there any way to do an "if-else" transaction?
> >>
> >> Best,
> >> tison.
> >>
> >
>
> >>
> >>
>
>
Re: create or setData in transaction?
Hi Jordan, Thanks for your reply. The problem is that we cannot confirm an action is committed atomically with checking the leadership. The thread shows that we are unable to access the latch znode. There is another thread[1] in Curator list and we could continue the discussion there. Best, tison. [1] https://www.mail-archive.com/user@curator.apache.org/msg01238.html Jordan Zimmerman 于2019年8月15日周四 上午8:23写道： > FYI - that "wart" had a workaround. You can't use the same workaround? > > -Jordan > > > On Aug 14, 2019, at 5:19 PM, Zili Chen wrote: > > > > Actually I want something like LeaderLatch of Curator but the > > implementation in Curator has some wart[1] > >
Re: create or setData in transaction?
FYI - that "wart" had a workaround. You can't use the same workaround? -Jordan > On Aug 14, 2019, at 5:19 PM, Zili Chen wrote: > > Actually I want something like LeaderLatch of Curator but the > implementation in Curator has some wart[1]
Re: create or setData in transaction?
> Actually I want something like LeaderLatch of Curator but the > implementation in Curator has some wart[1] so that I need to > reimplement it by myself. Or we can enhance Curator :D Why not add a new recipe for what you want in Curator. I (or one of the other committers) will help ypu. -Jordan > On Aug 14, 2019, at 5:19 PM, Zili Chen wrote: > > @Enrico > > The problem is that an operation should be rejected not only if > the version mismatched, but also if the mark node did not exist > even if the version matched. > > @Andor > > Thanks for your clarification. Let's then refer it as multi op. > > Actually I want something like LeaderLatch of Curator but the > implementation in Curator has some wart[1] so that I need to > reimplement it by myself. The semantic "bulk operations" can also > fit my case if ZooKeeper ensures the check is immediately followed > by setData. Since check doesn't commit a modification the problem > that it wouldn't roll back doesn't matter. > > @Michael > > Thanks for your clarification, too. Now I know bulk operation won't > take care of rolling back failed operations and thus it is different > from traditional transactions. It is a misunderstanding from my side. > > Best, > tison. > > [1] https://www.mail-archive.com/user@curator.apache.org/msg00912.html > > > Michael Han 于2019年8月15日周四 上午6:12写道： > Is there any way to do an "if-else" transaction? >> >> No for your use case. The only remotely related conditional operation you >> can express with multi-op is by using check operator (Op.check), where you >> can check a zNode's version and only execute subsequent operation in multi >> op when version matches. Though, even this would only allow you to express >> "if" rather than "if / else". >> ZooKeeper doesn’t support transactions to my best knowledge. It has a >> `multi` operation feature, but that’s more like a bulk operation, not >> transaction. >> >> Ted can correct me if I am wrong, since he added the multi op feature, but >> my understanding is "multi op" is branded from day one as the transaction >> support for zookeeper (we even provide an API with exact name: >> Transaction). If we use the traditional semantic for transaction in >> database context, the ACID properties multi-op satisfies at least atomicity >> and durability. So saying zookeeper does not support transaction seems a >> strong argument that against the properties of multi-op and existing >> literatures related to zookeeper. On the other side, typically bulk >> operations does not support atomicity, which will not take care of rolling >> back failed operations. >> >> On Wed, Aug 14, 2019 at 9:33 AM Andor Molnar wrote: >> >>> "it's said that ZooKeeper has a transaction mechanism” >>> >>> I’m still confused with this. ZooKeeper doesn’t support transactions to >> my >>> best knowledge. It has a `multi` operation feature, but that’s more like >> a >>> bulk operation, not transaction. >>> >>> "I want to tell ZooKeeper that the check and setData should be >> successful” >>> >>> I don’t think you can do that. ZK has no check-and-set support either. >>> >>> Maybe we should step back first and see what’s your use case exactly that >>> you’re trying to solve with ZooKeeper. I suspect that you’re trying to >>> follow the wrong approach or misusing ZooKeeper. >>> >>> Have you checked our tutorial and recipes page? >>> You can find some recommended usage patterns: >>> https://zookeeper.apache.org/doc/r3.5.5/recipes.html >>> https://zookeeper.apache.org/doc/r3.5.5/zookeeperTutorial.html >>> >>> If that’s not enough, you could also try Curator which has even more >>> built-in high level functionalities on top of basic ZK commands. >>> >>> Andor >>> >>> >>> On 2019. Aug 14., at 17:52, Zili Chen wrote: Hi Andor, Thanks for your attention. The problem is that in concurrent scenario zk.setData() could still >>> failed if there is another thread delete the node. I know with proper lock >>> strategy and ownership separation this can be avoid but it's said that ZooKeeper >>> has a transaction mechanism so I'd like to see whether I can make use of >> it. There is where I turn to zk.multi(Op.check(path1), Op.setData(path2, data)); // path1 == or != >>> path2 is irrelevant when the existence of a mark node(path1) guarded a condition and I want >>> to make sure that setData successes only if the mark node exist. If I check the existence first and commit setData, a remove to the node could break the guard. >> In other words, I want to tell ZooKeeper that the check and setData should be successful committed or fail to be committed atomically. Best, tison. Andor Molnar 于2019年8月14日周三 下午11:12写道： > Hi Zili, > > There’s no such functionality in ZooKeeper as far as I’m concerned. I > think your multi example
Re: create or setData in transaction?
@Enrico
The problem is that an operation should be rejected not only if
the version mismatched, but also if the mark node did not exist
even if the version matched.
@Andor
Thanks for your clarification. Let's then refer it as multi op.
Actually I want something like LeaderLatch of Curator but the
implementation in Curator has some wart[1] so that I need to
reimplement it by myself. The semantic "bulk operations" can also
fit my case if ZooKeeper ensures the check is immediately followed
by setData. Since check doesn't commit a modification the problem
that it wouldn't roll back doesn't matter.
@Michael
Thanks for your clarification, too. Now I know bulk operation won't
take care of rolling back failed operations and thus it is different
from traditional transactions. It is a misunderstanding from my side.
Best,
tison.
[1] https://www.mail-archive.com/user@curator.apache.org/msg00912.html
Michael Han 于2019年8月15日周四 上午6:12写道：
> >> Is there any way to do an "if-else" transaction?
>
> No for your use case. The only remotely related conditional operation you
> can express with multi-op is by using check operator (Op.check), where you
> can check a zNode's version and only execute subsequent operation in multi
> op when version matches. Though, even this would only allow you to express
> "if" rather than "if / else".
>
> >> ZooKeeper doesn’t support transactions to my best knowledge. It has a
> `multi` operation feature, but that’s more like a bulk operation, not
> transaction.
>
> Ted can correct me if I am wrong, since he added the multi op feature, but
> my understanding is "multi op" is branded from day one as the transaction
> support for zookeeper (we even provide an API with exact name:
> Transaction). If we use the traditional semantic for transaction in
> database context, the ACID properties multi-op satisfies at least atomicity
> and durability. So saying zookeeper does not support transaction seems a
> strong argument that against the properties of multi-op and existing
> literatures related to zookeeper. On the other side, typically bulk
> operations does not support atomicity, which will not take care of rolling
> back failed operations.
>
> On Wed, Aug 14, 2019 at 9:33 AM Andor Molnar wrote:
>
> > "it's said that ZooKeeper has a transaction mechanism”
> >
> > I’m still confused with this. ZooKeeper doesn’t support transactions to
> my
> > best knowledge. It has a `multi` operation feature, but that’s more like
> a
> > bulk operation, not transaction.
> >
> > "I want to tell ZooKeeper that the check and setData should be
> successful”
> >
> > I don’t think you can do that. ZK has no check-and-set support either.
> >
> > Maybe we should step back first and see what’s your use case exactly that
> > you’re trying to solve with ZooKeeper. I suspect that you’re trying to
> > follow the wrong approach or misusing ZooKeeper.
> >
> > Have you checked our tutorial and recipes page?
> > You can find some recommended usage patterns:
> > https://zookeeper.apache.org/doc/r3.5.5/recipes.html
> > https://zookeeper.apache.org/doc/r3.5.5/zookeeperTutorial.html
> >
> > If that’s not enough, you could also try Curator which has even more
> > built-in high level functionalities on top of basic ZK commands.
> >
> > Andor
> >
> >
> >
> > > On 2019. Aug 14., at 17:52, Zili Chen wrote:
> > >
> > > Hi Andor,
> > >
> > > Thanks for your attention.
> > >
> > > The problem is that in concurrent scenario zk.setData() could still
> > failed
> > > if there is another thread delete the node. I know with proper lock
> > strategy
> > > and ownership separation this can be avoid but it's said that ZooKeeper
> > has
> > > a transaction mechanism so I'd like to see whether I can make use of
> it.
> > >
> > > There is where I turn to
> > >
> > > zk.multi(Op.check(path1), Op.setData(path2, data)); // path1 == or !=
> > path2
> > > is irrelevant
> > >
> > > when the existence of a mark node(path1) guarded a condition and I want
> > to
> > > make
> > > sure that setData successes only if the mark node exist. If I check the
> > > existence
> > > first and commit setData, a remove to the node could break the guard.
> In
> > > other
> > > words, I want to tell ZooKeeper that the check and setData should be
> > > successful
> > > committed or fail to be committed atomically.
> > >
> > > Best,
> > > tison.
> > >
> > >
> > > Andor Molnar 于2019年8月14日周三 下午11:12写道：
> > >
> > >> Hi Zili,
> > >>
> > >> There’s no such functionality in ZooKeeper as far as I’m concerned. I
> > >> think your multi example (zk.multi(Op.check(path), Op.setData(path,
> > data)))
> > >> is already a usage pattern which multi is not designed to support.
> > >>
> > >> Why do you need to do this in “transactions” (there’s no transaction
> in
> > >> ZK)?
> > >>
> > >> In Java you can do:
> > >>
> > >> try {
> > >> zk.create();
> > >> } catch (NodeExistsException e) {
> > >> // swallow exception
> > >> }
> > >> zk.setData();
> > >> …
> > >>
> > >> Regards,
> >
Re: create or setData in transaction?
>> Is there any way to do an "if-else" transaction?
No for your use case. The only remotely related conditional operation you
can express with multi-op is by using check operator (Op.check), where you
can check a zNode's version and only execute subsequent operation in multi
op when version matches. Though, even this would only allow you to express
"if" rather than "if / else".
>> ZooKeeper doesn’t support transactions to my best knowledge. It has a
`multi` operation feature, but that’s more like a bulk operation, not
transaction.
Ted can correct me if I am wrong, since he added the multi op feature, but
my understanding is "multi op" is branded from day one as the transaction
support for zookeeper (we even provide an API with exact name:
Transaction). If we use the traditional semantic for transaction in
database context, the ACID properties multi-op satisfies at least atomicity
and durability. So saying zookeeper does not support transaction seems a
strong argument that against the properties of multi-op and existing
literatures related to zookeeper. On the other side, typically bulk
operations does not support atomicity, which will not take care of rolling
back failed operations.
On Wed, Aug 14, 2019 at 9:33 AM Andor Molnar wrote:
> "it's said that ZooKeeper has a transaction mechanism”
>
> I’m still confused with this. ZooKeeper doesn’t support transactions to my
> best knowledge. It has a `multi` operation feature, but that’s more like a
> bulk operation, not transaction.
>
> "I want to tell ZooKeeper that the check and setData should be successful”
>
> I don’t think you can do that. ZK has no check-and-set support either.
>
> Maybe we should step back first and see what’s your use case exactly that
> you’re trying to solve with ZooKeeper. I suspect that you’re trying to
> follow the wrong approach or misusing ZooKeeper.
>
> Have you checked our tutorial and recipes page?
> You can find some recommended usage patterns:
> https://zookeeper.apache.org/doc/r3.5.5/recipes.html
> https://zookeeper.apache.org/doc/r3.5.5/zookeeperTutorial.html
>
> If that’s not enough, you could also try Curator which has even more
> built-in high level functionalities on top of basic ZK commands.
>
> Andor
>
>
>
> > On 2019. Aug 14., at 17:52, Zili Chen wrote:
> >
> > Hi Andor,
> >
> > Thanks for your attention.
> >
> > The problem is that in concurrent scenario zk.setData() could still
> failed
> > if there is another thread delete the node. I know with proper lock
> strategy
> > and ownership separation this can be avoid but it's said that ZooKeeper
> has
> > a transaction mechanism so I'd like to see whether I can make use of it.
> >
> > There is where I turn to
> >
> > zk.multi(Op.check(path1), Op.setData(path2, data)); // path1 == or !=
> path2
> > is irrelevant
> >
> > when the existence of a mark node(path1) guarded a condition and I want
> to
> > make
> > sure that setData successes only if the mark node exist. If I check the
> > existence
> > first and commit setData, a remove to the node could break the guard. In
> > other
> > words, I want to tell ZooKeeper that the check and setData should be
> > successful
> > committed or fail to be committed atomically.
> >
> > Best,
> > tison.
> >
> >
> > Andor Molnar 于2019年8月14日周三 下午11:12写道：
> >
> >> Hi Zili,
> >>
> >> There’s no such functionality in ZooKeeper as far as I’m concerned. I
> >> think your multi example (zk.multi(Op.check(path), Op.setData(path,
> data)))
> >> is already a usage pattern which multi is not designed to support.
> >>
> >> Why do you need to do this in “transactions” (there’s no transaction in
> >> ZK)?
> >>
> >> In Java you can do:
> >>
> >> try {
> >> zk.create();
> >> } catch (NodeExistsException e) {
> >> // swallow exception
> >> }
> >> zk.setData();
> >> …
> >>
> >> Regards,
> >> Andor
> >>
> >>
> >>
> >>
> >>> On 2019. Aug 6., at 14:44, Zili Chen wrote:
> >>>
> >>> Hi Enrico,
> >>>
> >>> Thanks for your reply.
> >>>
> In this case usually you use conditional setData, using the 'version'
> of
> thr znode
> >>>
> >>>
> >>> what if the caller has no idea on whether the node exist?(see also
> >>> my if-else pseudo-code above.)
> >>>
> >>> IIRC if we call `setData` on a non-exist path a NoNodeException
> >>> will be thrown.
> >>> Best,
> >>> tison.
> >>>
> >>>
> >>> Enrico Olivelli 于2019年8月6日周二 下午8:27写道：
> >>>
> Il mar 6 ago 2019, 13:47 Zili Chen ha scritto:
>
> > Any ideas?
> >
> >
> > Zili Chen 于2019年7月29日周一 上午11:12写道：
> >
> >> Hi ZooKeepers,
> >>
> >> Currently our transaction mechanism supports doing
> >> create/setData/checkExist/delete in transaction. However, taking
> this
> >> scenario into consideration, we want to put data in path "/path" but
> >> don't know whether the znode exists or not. Let's say we program as
> >> below
> >>
> >> if (zk.exist(path)) {
> >> zk.setData(path, data);
> >> } else {
> >> zk.create(path, data);
> >>
Re: create or setData in transaction?
"it's said that ZooKeeper has a transaction mechanism”
I’m still confused with this. ZooKeeper doesn’t support transactions to my best
knowledge. It has a `multi` operation feature, but that’s more like a bulk
operation, not transaction.
"I want to tell ZooKeeper that the check and setData should be successful”
I don’t think you can do that. ZK has no check-and-set support either.
Maybe we should step back first and see what’s your use case exactly that
you’re trying to solve with ZooKeeper. I suspect that you’re trying to follow
the wrong approach or misusing ZooKeeper.
Have you checked our tutorial and recipes page?
You can find some recommended usage patterns:
https://zookeeper.apache.org/doc/r3.5.5/recipes.html
https://zookeeper.apache.org/doc/r3.5.5/zookeeperTutorial.html
If that’s not enough, you could also try Curator which has even more built-in
high level functionalities on top of basic ZK commands.
Andor
> On 2019. Aug 14., at 17:52, Zili Chen wrote:
>
> Hi Andor,
>
> Thanks for your attention.
>
> The problem is that in concurrent scenario zk.setData() could still failed
> if there is another thread delete the node. I know with proper lock strategy
> and ownership separation this can be avoid but it's said that ZooKeeper has
> a transaction mechanism so I'd like to see whether I can make use of it.
>
> There is where I turn to
>
> zk.multi(Op.check(path1), Op.setData(path2, data)); // path1 == or != path2
> is irrelevant
>
> when the existence of a mark node(path1) guarded a condition and I want to
> make
> sure that setData successes only if the mark node exist. If I check the
> existence
> first and commit setData, a remove to the node could break the guard. In
> other
> words, I want to tell ZooKeeper that the check and setData should be
> successful
> committed or fail to be committed atomically.
>
> Best,
> tison.
>
>
> Andor Molnar 于2019年8月14日周三 下午11:12写道：
>
>> Hi Zili,
>>
>> There’s no such functionality in ZooKeeper as far as I’m concerned. I
>> think your multi example (zk.multi(Op.check(path), Op.setData(path, data)))
>> is already a usage pattern which multi is not designed to support.
>>
>> Why do you need to do this in “transactions” (there’s no transaction in
>> ZK)?
>>
>> In Java you can do:
>>
>> try {
>> zk.create();
>> } catch (NodeExistsException e) {
>> // swallow exception
>> }
>> zk.setData();
>> …
>>
>> Regards,
>> Andor
>>
>>
>>
>>
>>> On 2019. Aug 6., at 14:44, Zili Chen wrote:
>>>
>>> Hi Enrico,
>>>
>>> Thanks for your reply.
>>>
In this case usually you use conditional setData, using the 'version' of
thr znode
>>>
>>>
>>> what if the caller has no idea on whether the node exist?(see also
>>> my if-else pseudo-code above.)
>>>
>>> IIRC if we call `setData` on a non-exist path a NoNodeException
>>> will be thrown.
>>> Best,
>>> tison.
>>>
>>>
>>> Enrico Olivelli 于2019年8月6日周二 下午8:27写道：
>>>
Il mar 6 ago 2019, 13:47 Zili Chen ha scritto:
> Any ideas?
>
>
> Zili Chen 于2019年7月29日周一 上午11:12写道：
>
>> Hi ZooKeepers,
>>
>> Currently our transaction mechanism supports doing
>> create/setData/checkExist/delete in transaction. However, taking this
>> scenario into consideration, we want to put data in path "/path" but
>> don't know whether the znode exists or not. Let's say we program as
>> below
>>
>> if (zk.exist(path)) {
>> zk.setData(path, data);
>> } else {
>> zk.create(path, data);
>> }
>
Do you need to perform other ops in the same transaction?
In this case usually you use conditional setData, using the 'version' of
thr znode
Enrico
>
>> if we want to do the check and "put" in transaction, it would be like
>>
>> zk.multi(Op.check(path), Op.setData(path, data));
>>
>> but we cannot add a "else" branch. ZooKeeper's transaction would all
>> success or fail.
>>
>> Is there any way to do an "if-else" transaction?
>>
>> Best,
>> tison.
>>
>
>>
>>
Re: create or setData in transaction?
Il mer 14 ago 2019, 17:55 Zili Chen ha scritto:
> FYI, etcd provides a transaction mechanism which supports if-else like
> commits.[1]
>
As we have setData with version check usually you don't need a
'transaction' with some kind of rowlevel lock.
Maybe I am missing something in your case.
Enrico
> Best,
> tison.
>
> [1] https://godoc.org/github.com/coreos/etcd/clientv3#OpTxn
>
>
> Zili Chen 于2019年8月14日周三 下午11:52写道：
>
> > Hi Andor,
> >
> > Thanks for your attention.
> >
> > The problem is that in concurrent scenario zk.setData() could still
> failed
> > if there is another thread delete the node. I know with proper lock
> > strategy
> > and ownership separation this can be avoid but it's said that ZooKeeper
> has
> > a transaction mechanism so I'd like to see whether I can make use of it.
> >
> > There is where I turn to
> >
> > zk.multi(Op.check(path1), Op.setData(path2, data)); // path1 == or !=
> > path2 is irrelevant
> >
> > when the existence of a mark node(path1) guarded a condition and I want
> to
> > make
> > sure that setData successes only if the mark node exist. If I check the
> > existence
> > first and commit setData, a remove to the node could break the guard. In
> > other
> > words, I want to tell ZooKeeper that the check and setData should be
> > successful
> > committed or fail to be committed atomically.
> >
> > Best,
> > tison.
> >
> >
> > Andor Molnar 于2019年8月14日周三 下午11:12写道：
> >
> >> Hi Zili,
> >>
> >> There’s no such functionality in ZooKeeper as far as I’m concerned. I
> >> think your multi example (zk.multi(Op.check(path), Op.setData(path,
> data)))
> >> is already a usage pattern which multi is not designed to support.
> >>
> >> Why do you need to do this in “transactions” (there’s no transaction in
> >> ZK)?
> >>
> >> In Java you can do:
> >>
> >> try {
> >> zk.create();
> >> } catch (NodeExistsException e) {
> >> // swallow exception
> >> }
> >> zk.setData();
> >> …
> >>
> >> Regards,
> >> Andor
> >>
> >>
> >>
> >>
> >> > On 2019. Aug 6., at 14:44, Zili Chen wrote:
> >> >
> >> > Hi Enrico,
> >> >
> >> > Thanks for your reply.
> >> >
> >> >> In this case usually you use conditional setData, using the 'version'
> >> of
> >> >> thr znode
> >> >
> >> >
> >> > what if the caller has no idea on whether the node exist?(see also
> >> > my if-else pseudo-code above.)
> >> >
> >> > IIRC if we call `setData` on a non-exist path a NoNodeException
> >> > will be thrown.
> >> > Best,
> >> > tison.
> >> >
> >> >
> >> > Enrico Olivelli 于2019年8月6日周二 下午8:27写道：
> >> >
> >> >> Il mar 6 ago 2019, 13:47 Zili Chen ha
> scritto:
> >> >>
> >> >>> Any ideas?
> >> >>>
> >> >>>
> >> >>> Zili Chen 于2019年7月29日周一 上午11:12写道：
> >> >>>
> >> Hi ZooKeepers,
> >>
> >> Currently our transaction mechanism supports doing
> >> create/setData/checkExist/delete in transaction. However, taking
> this
> >> scenario into consideration, we want to put data in path "/path"
> but
> >> don't know whether the znode exists or not. Let's say we program as
> >> below
> >>
> >> if (zk.exist(path)) {
> >> zk.setData(path, data);
> >> } else {
> >> zk.create(path, data);
> >> }
> >> >>>
> >> >>
> >> >> Do you need to perform other ops in the same transaction?
> >> >> In this case usually you use conditional setData, using the 'version'
> >> of
> >> >> thr znode
> >> >>
> >> >>
> >> >> Enrico
> >> >>
> >> >>>
> >> if we want to do the check and "put" in transaction, it would be
> like
> >>
> >> zk.multi(Op.check(path), Op.setData(path, data));
> >>
> >> but we cannot add a "else" branch. ZooKeeper's transaction would
> all
> >> success or fail.
> >>
> >> Is there any way to do an "if-else" transaction?
> >>
> >> Best,
> >> tison.
> >>
> >> >>>
> >> >>
> >>
> >>
>
Re: create or setData in transaction?
FYI, etcd provides a transaction mechanism which supports if-else like
commits.[1]
Best,
tison.
[1] https://godoc.org/github.com/coreos/etcd/clientv3#OpTxn
Zili Chen 于2019年8月14日周三 下午11:52写道：
> Hi Andor,
>
> Thanks for your attention.
>
> The problem is that in concurrent scenario zk.setData() could still failed
> if there is another thread delete the node. I know with proper lock
> strategy
> and ownership separation this can be avoid but it's said that ZooKeeper has
> a transaction mechanism so I'd like to see whether I can make use of it.
>
> There is where I turn to
>
> zk.multi(Op.check(path1), Op.setData(path2, data)); // path1 == or !=
> path2 is irrelevant
>
> when the existence of a mark node(path1) guarded a condition and I want to
> make
> sure that setData successes only if the mark node exist. If I check the
> existence
> first and commit setData, a remove to the node could break the guard. In
> other
> words, I want to tell ZooKeeper that the check and setData should be
> successful
> committed or fail to be committed atomically.
>
> Best,
> tison.
>
>
> Andor Molnar 于2019年8月14日周三 下午11:12写道：
>
>> Hi Zili,
>>
>> There’s no such functionality in ZooKeeper as far as I’m concerned. I
>> think your multi example (zk.multi(Op.check(path), Op.setData(path, data)))
>> is already a usage pattern which multi is not designed to support.
>>
>> Why do you need to do this in “transactions” (there’s no transaction in
>> ZK)?
>>
>> In Java you can do:
>>
>> try {
>> zk.create();
>> } catch (NodeExistsException e) {
>> // swallow exception
>> }
>> zk.setData();
>> …
>>
>> Regards,
>> Andor
>>
>>
>>
>>
>> > On 2019. Aug 6., at 14:44, Zili Chen wrote:
>> >
>> > Hi Enrico,
>> >
>> > Thanks for your reply.
>> >
>> >> In this case usually you use conditional setData, using the 'version'
>> of
>> >> thr znode
>> >
>> >
>> > what if the caller has no idea on whether the node exist?(see also
>> > my if-else pseudo-code above.)
>> >
>> > IIRC if we call `setData` on a non-exist path a NoNodeException
>> > will be thrown.
>> > Best,
>> > tison.
>> >
>> >
>> > Enrico Olivelli 于2019年8月6日周二 下午8:27写道：
>> >
>> >> Il mar 6 ago 2019, 13:47 Zili Chen ha scritto:
>> >>
>> >>> Any ideas?
>> >>>
>> >>>
>> >>> Zili Chen 于2019年7月29日周一 上午11:12写道：
>> >>>
>> Hi ZooKeepers,
>>
>> Currently our transaction mechanism supports doing
>> create/setData/checkExist/delete in transaction. However, taking this
>> scenario into consideration, we want to put data in path "/path" but
>> don't know whether the znode exists or not. Let's say we program as
>> below
>>
>> if (zk.exist(path)) {
>> zk.setData(path, data);
>> } else {
>> zk.create(path, data);
>> }
>> >>>
>> >>
>> >> Do you need to perform other ops in the same transaction?
>> >> In this case usually you use conditional setData, using the 'version'
>> of
>> >> thr znode
>> >>
>> >>
>> >> Enrico
>> >>
>> >>>
>> if we want to do the check and "put" in transaction, it would be like
>>
>> zk.multi(Op.check(path), Op.setData(path, data));
>>
>> but we cannot add a "else" branch. ZooKeeper's transaction would all
>> success or fail.
>>
>> Is there any way to do an "if-else" transaction?
>>
>> Best,
>> tison.
>>
>> >>>
>> >>
>>
>>
Re: create or setData in transaction?
Hi Andor,
Thanks for your attention.
The problem is that in concurrent scenario zk.setData() could still failed
if there is another thread delete the node. I know with proper lock strategy
and ownership separation this can be avoid but it's said that ZooKeeper has
a transaction mechanism so I'd like to see whether I can make use of it.
There is where I turn to
zk.multi(Op.check(path1), Op.setData(path2, data)); // path1 == or != path2
is irrelevant
when the existence of a mark node(path1) guarded a condition and I want to
make
sure that setData successes only if the mark node exist. If I check the
existence
first and commit setData, a remove to the node could break the guard. In
other
words, I want to tell ZooKeeper that the check and setData should be
successful
committed or fail to be committed atomically.
Best,
tison.
Andor Molnar 于2019年8月14日周三 下午11:12写道：
> Hi Zili,
>
> There’s no such functionality in ZooKeeper as far as I’m concerned. I
> think your multi example (zk.multi(Op.check(path), Op.setData(path, data)))
> is already a usage pattern which multi is not designed to support.
>
> Why do you need to do this in “transactions” (there’s no transaction in
> ZK)?
>
> In Java you can do:
>
> try {
> zk.create();
> } catch (NodeExistsException e) {
> // swallow exception
> }
> zk.setData();
> …
>
> Regards,
> Andor
>
>
>
>
> > On 2019. Aug 6., at 14:44, Zili Chen wrote:
> >
> > Hi Enrico,
> >
> > Thanks for your reply.
> >
> >> In this case usually you use conditional setData, using the 'version' of
> >> thr znode
> >
> >
> > what if the caller has no idea on whether the node exist?(see also
> > my if-else pseudo-code above.)
> >
> > IIRC if we call `setData` on a non-exist path a NoNodeException
> > will be thrown.
> > Best,
> > tison.
> >
> >
> > Enrico Olivelli 于2019年8月6日周二 下午8:27写道：
> >
> >> Il mar 6 ago 2019, 13:47 Zili Chen ha scritto:
> >>
> >>> Any ideas?
> >>>
> >>>
> >>> Zili Chen 于2019年7月29日周一 上午11:12写道：
> >>>
> Hi ZooKeepers,
>
> Currently our transaction mechanism supports doing
> create/setData/checkExist/delete in transaction. However, taking this
> scenario into consideration, we want to put data in path "/path" but
> don't know whether the znode exists or not. Let's say we program as
> below
>
> if (zk.exist(path)) {
> zk.setData(path, data);
> } else {
> zk.create(path, data);
> }
> >>>
> >>
> >> Do you need to perform other ops in the same transaction?
> >> In this case usually you use conditional setData, using the 'version' of
> >> thr znode
> >>
> >>
> >> Enrico
> >>
> >>>
> if we want to do the check and "put" in transaction, it would be like
>
> zk.multi(Op.check(path), Op.setData(path, data));
>
> but we cannot add a "else" branch. ZooKeeper's transaction would all
> success or fail.
>
> Is there any way to do an "if-else" transaction?
>
> Best,
> tison.
>
> >>>
> >>
>
>
Re: Clarification: SSL Client: Need of keystore?
Il mar 30 lug 2019, 20:49 Jörn Franke ha scritto: > Hi, > > I have a kerberized Zookeeper cluster and would like to add SSL on the > client side and to the quorum. > > So far the server configuration is clear. However, according to > > https://cwiki.apache.org/confluence/display/ZOOKEEPER/ZooKeeper+SSL+User+Guide > > I need to specify on the client side > zookeeper.ssl.keyStore.location="/path/to/your/keystore" > zookeeper.ssl.keyStore.password="keystore_password" > zookeeper.ssl.trustStore.location="/path/to/your/truststore" > zookeeper.ssl.trustStore.password="truststore_password" > > I do understand the need to provide a truststore, but why does the client > need a keystore. As far as I understood the keystore is only needed for > X509 authentication, but I use the Kerberos authentication. > Your question is fair. Did you try not to configure a keystore for the client? Enrico > Does it mean the SSL client connection requires X509 authentication and > Kerberos is not possible? > Can you please clarify? > > thank you. > > best regards >
Re: create or setData in transaction?
Hi Zili,
There’s no such functionality in ZooKeeper as far as I’m concerned. I think
your multi example (zk.multi(Op.check(path), Op.setData(path, data))) is
already a usage pattern which multi is not designed to support.
Why do you need to do this in “transactions” (there’s no transaction in ZK)?
In Java you can do:
try {
zk.create();
} catch (NodeExistsException e) {
// swallow exception
}
zk.setData();
…
Regards,
Andor
> On 2019. Aug 6., at 14:44, Zili Chen wrote:
>
> Hi Enrico,
>
> Thanks for your reply.
>
>> In this case usually you use conditional setData, using the 'version' of
>> thr znode
>
>
> what if the caller has no idea on whether the node exist?(see also
> my if-else pseudo-code above.)
>
> IIRC if we call `setData` on a non-exist path a NoNodeException
> will be thrown.
> Best,
> tison.
>
>
> Enrico Olivelli 于2019年8月6日周二 下午8:27写道：
>
>> Il mar 6 ago 2019, 13:47 Zili Chen ha scritto:
>>
>>> Any ideas?
>>>
>>>
>>> Zili Chen 于2019年7月29日周一 上午11:12写道：
>>>
Hi ZooKeepers,
Currently our transaction mechanism supports doing
create/setData/checkExist/delete in transaction. However, taking this
scenario into consideration, we want to put data in path "/path" but
don't know whether the znode exists or not. Let's say we program as
below
if (zk.exist(path)) {
zk.setData(path, data);
} else {
zk.create(path, data);
}
>>>
>>
>> Do you need to perform other ops in the same transaction?
>> In this case usually you use conditional setData, using the 'version' of
>> thr znode
>>
>>
>> Enrico
>>
>>>
if we want to do the check and "put" in transaction, it would be like
zk.multi(Op.check(path), Op.setData(path, data));
but we cannot add a "else" branch. ZooKeeper's transaction would all
success or fail.
Is there any way to do an "if-else" transaction?
Best,
tison.
>>>
>>
Re: Clarification: SSL Client: Need of keystore?
Hi Jorn, I cannot test this unfortunately, because I don’t have a working Kerberos environment at the moment. If you comment out keystore.location, ZooKeeper won’t start, because it’s unable to build the TrustManager. Would you please try to create a fake (possibly empty) truststore and see how it goes? Andor > On 2019. Jul 30., at 20:49, Jörn Franke wrote: > > Hi, > > I have a kerberized Zookeeper cluster and would like to add SSL on the > client side and to the quorum. > > So far the server configuration is clear. However, according to > https://cwiki.apache.org/confluence/display/ZOOKEEPER/ZooKeeper+SSL+User+Guide > > I need to specify on the client side > zookeeper.ssl.keyStore.location="/path/to/your/keystore" > zookeeper.ssl.keyStore.password="keystore_password" > zookeeper.ssl.trustStore.location="/path/to/your/truststore" > zookeeper.ssl.trustStore.password="truststore_password" > > I do understand the need to provide a truststore, but why does the client > need a keystore. As far as I understood the keystore is only needed for > X509 authentication, but I use the Kerberos authentication. > > Does it mean the SSL client connection requires X509 authentication and > Kerberos is not possible? > Can you please clarify? > > thank you. > > best regards
Re: Issue migrating from Zookeeper 3.4.14 to 3.5.5
After some digging it turned out that this is an outstanding issue in 3.4->3.5 upgrade. I’ve found the following e-mail thread about it: https://markmail.org/thread/rbhzbro6nszypwwp …and an open Jira: https://issues.apache.org/jira/browse/ZOOKEEPER-3056 Unfortunately, patch is still not available, but essentially the solution is to force ZooKeeper to create snapshot file somehow. Sorry, Admin interface is not available in 3.4, it was my bad to recommend it. In the last Jira comment there’s a workaround: To perform an upgrade (3.4 -> 3.5): • download the "snapshot.0" file attached • copy it to the versioned directory (e.g. "version-2") within your data directory (parameter "dataDir" in your config - this is the directory containing the "myid" file for a peer) • restart the peer • upgrade the peer (this can be combined with the above step if you like) Would you please give it a try? Andor > On 2019. Aug 14., at 10:44, Andor Molnar wrote: > > Hi Jorn, > > Thanks for reaching out to us, this is a very important exercise to make sure > the upgrade path works as expected. > > - Please do an `ls -al` in your data dir to make sure you have valid snapshot > files. > - It would be also useful to expose the Admin port (8080/tcp by default) and > check the output of `lastSnapshotCommand`. > > Regards, > Andor > > > > > >> On 2019. Aug 14., at 7:13, Jörn Franke wrote: >> >> For me the issue occurred only in standalone mode. With the ensemble I >> simply cleared the data directory and it received the zookeeper data from >> the quorum. >> >>> Am 13.08.2019 um 15:42 schrieb Koen De Groote : >>> >>> I would also like to know if this is possible. >>> >>> From going over the github page, it seems there is a JMX method to force >>> the creation of a snapshot. Yet the docker image is configured as such that >>> a port will never be assigned to the JMX process. >>> >>> Is there any way to bypass this? >>> On Tue, Jul 30, 2019 at 8:51 AM Jörn Franke wrote: Thanks. It is possible to force Zookeeper to create a snapshot? I will check I think the snapshot count is set to 1 in the cfg > Am 30.07.2019 um 08:06 schrieb Enrico Olivelli : > > Il giorno lun 29 lug 2019 alle ore 23:59 Jörn Franke < jornfra...@gmail.com> > ha scritto: > >> ok, then let me verify tomorrow if a snapshot file is indeed there. If it >> is missing then I wonder why it was missing. There was no crash or whatever >> and 3.4.14 works without issue, but of course it could have loaded them >> from the log files. However, then I wonder why it does not create one. >> > > > > I remember now that some other user, I think Sijie, reported a similar > problem some month ago, that it is not possible to upgrade from 3.4 to 3.5 > if no snapshot is present. > IIRC The fix was to force the creation of at least one snapshot file and > then upgrade > > Enrico > > >> >> On Mon, Jul 29, 2019 at 11:45 PM Michael Han wrote: >> > I just wonder why it does not find a valid snapshot. >>> >>> If there are local snapshot files and the files are valid, then it's a >> bug >>> that server fails to load them. >>> > Is it because the format changed in 3.5.5 compared to 3.4.14? >>> >>> Not I am aware of. There are some format changes (added compression >>> support) in master branch, but that's not shipped with 3.5.5. >>> >>> >>> >>> On Mon, Jul 29, 2019 at 2:31 PM Jörn Franke >> wrote: >>> ok, then it affects basically all standalone nodes? This is fine, >> despite that it means some extra work (for uncritical lab environments). I am not sure it is ZOOKEEPER-2325, but I don't know the full history behind it).The logs are fine (it works in 3.4.14 without issues, even >>> after downgrading back). There is no issue with disk space and there are no 0 byte files. I just wonder why it does not find a valid snapshot. Is it because the format changed in 3.5.5 compared to 3.4.14? On Mon, Jul 29, 2019 at 11:25 PM Michael Han wrote: >>> java.io.IOException: No snapshot found, but there are log entries. > Something is broken! > > This is expected behavior introduced in ZOOKEEPER-2325. We don't want >>> to > end up with potential inconsistent state across the ensemble when > recovering from empty snapshot. > > To continue upgrade, just delete all txn log files and let the node >>> sync > the snapshot from the quorum. > > > On Mon, Jul 29, 2019 at 1:38 PM Enrico Olivelli >> > wrote: > >> Il lun 29 lug 2019, 22:32 Jörn Franke ha scritto: >>
Re: Issue migrating from Zookeeper 3.4.14 to 3.5.5
Hi Jorn, Thanks for reaching out to us, this is a very important exercise to make sure the upgrade path works as expected. - Please do an `ls -al` in your data dir to make sure you have valid snapshot files. - It would be also useful to expose the Admin port (8080/tcp by default) and check the output of `lastSnapshotCommand`. Regards, Andor > On 2019. Aug 14., at 7:13, Jörn Franke wrote: > > For me the issue occurred only in standalone mode. With the ensemble I simply > cleared the data directory and it received the zookeeper data from the > quorum. > >> Am 13.08.2019 um 15:42 schrieb Koen De Groote : >> >> I would also like to know if this is possible. >> >> From going over the github page, it seems there is a JMX method to force >> the creation of a snapshot. Yet the docker image is configured as such that >> a port will never be assigned to the JMX process. >> >> Is there any way to bypass this? >> >>> On Tue, Jul 30, 2019 at 8:51 AM Jörn Franke wrote: >>> >>> Thanks. It is possible to force Zookeeper to create a snapshot? I will >>> check I think the snapshot count is set to 1 in the cfg >>> Am 30.07.2019 um 08:06 schrieb Enrico Olivelli : Il giorno lun 29 lug 2019 alle ore 23:59 Jörn Franke < >>> jornfra...@gmail.com> ha scritto: > ok, then let me verify tomorrow if a snapshot file is indeed there. If >>> it > is missing then I wonder why it was missing. There was no crash or >>> whatever > and 3.4.14 works without issue, but of course it could have loaded them > from the log files. However, then I wonder why it does not create one. > I remember now that some other user, I think Sijie, reported a similar problem some month ago, that it is not possible to upgrade from 3.4 to >>> 3.5 if no snapshot is present. IIRC The fix was to force the creation of at least one snapshot file and then upgrade Enrico > > On Mon, Jul 29, 2019 at 11:45 PM Michael Han wrote: > I just wonder why it does not find a valid snapshot. >> >> If there are local snapshot files and the files are valid, then it's a > bug >> that server fails to load them. >> Is it because the format changed in 3.5.5 compared to 3.4.14? >> >> Not I am aware of. There are some format changes (added compression >> support) in master branch, but that's not shipped with 3.5.5. >> >> >> >> On Mon, Jul 29, 2019 at 2:31 PM Jörn Franke > wrote: >> >>> ok, then it affects basically all standalone nodes? This is fine, > despite >>> that it means some extra work (for uncritical lab environments). >>> I am not sure it is ZOOKEEPER-2325, but I don't know the full history >>> behind it).The logs are fine (it works in 3.4.14 without issues, even >> after >>> downgrading back). There is no issue with disk space and there are no >>> 0 >>> byte files. I just wonder why it does not find a valid snapshot. Is >>> it >>> because the format changed in 3.5.5 compared to 3.4.14? >>> >>> On Mon, Jul 29, 2019 at 11:25 PM Michael Han wrote: >>> >> java.io.IOException: No snapshot found, but there are log entries. Something is broken! This is expected behavior introduced in ZOOKEEPER-2325. We don't want >> to end up with potential inconsistent state across the ensemble when recovering from empty snapshot. To continue upgrade, just delete all txn log files and let the node >> sync the snapshot from the quorum. On Mon, Jul 29, 2019 at 1:38 PM Enrico Olivelli > wrote: > Il lun 29 lug 2019, 22:32 Jörn Franke ha >>> scritto: > >> It also seems that 3.5.5 does not attempt to read all of the >> logfiles (I >> have to still confirm), but the two it reads exist, it has access >> and > they >> are much more than 0 byte >> > > We should have the stackstace of the EOFException. > > Anyone on this list has a better idea? > > Enrico > > >> On Mon, Jul 29, 2019 at 10:13 PM Jörn Franke < > jornfra...@gmail.com >>> > wrote: >> >>> (of course i do not run them at the same time) >>> >>> On Mon, Jul 29, 2019 at 10:10 PM Jörn Franke < >> jornfra...@gmail.com >> wrote: >>> thank you for the quick reply. They read from the same disk >> paths and have the same access rights (in fact the RHEL service executes >>> them as >> the same specific user). On Mon, Jul 29, 2019 at 10:09 PM Enrico Olivelli < eolive...@gmail.com >>
