Re: [saga-actuator]consider about adding StreamBasedSaga engine to integrate with shardingsphere

[zhaojun]

I have just created a issue[1] in JIRA

[1]: https://issues.apache.org/jira/browse/SCB-1276 


--
Zhao Jun
Apache Sharding-Sphere & ServiceComb

> On May 5, 2019, at 8:46 AM, zhaojun  wrote:
> 
> OK, I’ll try to write a JIRA today.
> 
> --
> Zhao Jun
> Apache Sharding-Sphere & ServiceComb
> 
> 
>> On May 2, 2019, at 3:18 PM, Willem Jiang  wrote:
>> 
>> It sounds good to me.
>> 
>> Willem Jiang
>> 
>> Twitter: willemjiang
>> Weibo: 姜宁willem
>> 
>> On Thu, May 2, 2019 at 11:14 AM 赵俊  wrote:
>>> 
>>> Let SQL executor running command one by one is good to me.
>>> 
>>> One logic-sql will be routed to multi-actual SQLs, then executed in 
>>> sharding-sphere execute-engine.
>>> A actual SQL will be wrapped as a SQL execute-callback in sharding-sphere.
>>> We could provide a SPI for this SQL execute-callback.
>>> For integrated with saga, we could implement a saga execute-callback, it 
>>> will wrap event sourcing logic like saga-actuator task have done.
>>> 
>>> 
 On May 1, 2019, at 5:06 PM, Willem Jiang  wrote:
 
 We could do some enhancement on it.
 From my understanding, We could create a calling graph which has three
 sub graph of it, or we could let the SQL executor running the command
 one by one.
 Any thoughts?
 
 Willem Jiang
 
 Twitter: willemjiang
 Weibo: 姜宁willem
 
 On Wed, May 1, 2019 at 4:29 PM zhaojun  wrote:
> 
> But saga-actuator don’t support 3 logic SQL is a global transaction while 
> you submit 3 times separately.
> 
> --
> Zhao Jun
> Apache Sharding-Sphere & ServiceComb
> 
>> On May 1, 2019, at 8:38 AM, Willem Jiang  wrote:
>> 
>> Willem Jiang
>> 
>> Twitter: willemjiang
>> Weibo: 姜宁willem
>> 
>> On Mon, Apr 29, 2019 at 7:58 PM zhaojun  wrote:
>>> 
>>> Hi, Willem
>>> 
>>> Saga actuator could not compatible with ShardingSphere currently,  
>>> there are 2 main problem exist.
>>> 1. If sql execute in saga transport instead of ShardingSphere, we can 
>>> not see the result of logic-sql even in one transaction, it is like 
>>> this:
>>> update t_order set status=’start’ where usr_id=’tom’;
>>> select status from t_order where usr_id=’tom’;  
>>> —> we can’t query ’start’ record as actuator haven’t executed.
>>> Insert into t_order values(?,?,?) ;
>> 
>> I think we can break the whole SQL interactions into smaller steps,
>> and let saga-actuator do the job one by one.
>> In this case, you can break the whole calling graph into several sub
>> calling graphs.
>> Any thought?
>> 
>>> 2. If logic-sql execute in ShardingSphere, we cannot handle recovery 
>>> before submit graph result, as event log only wrote at saga engine.
>>> 
>>> so we should integrate saga-transaction like omega send event log step 
>>> by step.
>>> It is better to make every instance do recovery Independently, instead 
>>> of providing another coordinator center service.
>>> I feel that embed saga should have following  capability.
>>> 1).It can provide service in jar package independent
>>> 2).each embed saga only recovery their own transaction-data of this 
>>> instance.
>>>If one instance crashed, we can introduce external service to do 
>>> failover.
>>> 
>>> so we consider about exending saga-acutator, if it can support submit 
>>> task step by step in one transaction, it is a good choice.
>>> of course, there have many things we should do.
>>> 
>>> --
>>> Zhao Jun
>>> Apache Sharding-Sphere & ServiceComb
>>> 
 On Apr 29, 2019, at 5:17 PM, Willem Jiang  
 wrote:
 
 First Saga actuator need to build up the calling grapha before sending
 out the request.  I don't think you can do the step by step SQL
 invocation with Saga actuator.
 If you want to call the SQL execution step by step , you may need to
 switch to ServiceComb Pack project which has a coordinator to take
 care of the distributed transaction. But that introduce another
 endpoint(Alpha) to shardingsphere.
 
 From my understanding, Saga actuator is most efficient way to execute
 the SQL across different data nodes.
 
 
 Willem Jiang
 
 Twitter: willemjiang
 Weibo: 姜宁willem
 
 On Fri, Apr 26, 2019 at 6:41 PM zhaojun  wrote:
> 
> Hi, all
> 
> currently, we have integrated with saga using graph based engine in 
> shardingsphere[1]
> it need us to collect all participated actual SQL, then submit to 
> saga actuator in commit/rollback phase.
> if application crashed before invoking saga actuator, undo log of 
> 

Re: [saga-actuator]consider about adding StreamBasedSaga engine to integrate with shardingsphere

[zhaojun]

OK, I’ll try to write a JIRA today.

--
Zhao Jun
Apache Sharding-Sphere & ServiceComb


> On May 2, 2019, at 3:18 PM, Willem Jiang  wrote:
> 
> It sounds good to me.
> 
> Willem Jiang
> 
> Twitter: willemjiang
> Weibo: 姜宁willem
> 
> On Thu, May 2, 2019 at 11:14 AM 赵俊  wrote:
>> 
>> Let SQL executor running command one by one is good to me.
>> 
>> One logic-sql will be routed to multi-actual SQLs, then executed in 
>> sharding-sphere execute-engine.
>> A actual SQL will be wrapped as a SQL execute-callback in sharding-sphere.
>> We could provide a SPI for this SQL execute-callback.
>> For integrated with saga, we could implement a saga execute-callback, it 
>> will wrap event sourcing logic like saga-actuator task have done.
>> 
>> 
>>> On May 1, 2019, at 5:06 PM, Willem Jiang  wrote:
>>> 
>>> We could do some enhancement on it.
>>> From my understanding, We could create a calling graph which has three
>>> sub graph of it, or we could let the SQL executor running the command
>>> one by one.
>>> Any thoughts?
>>> 
>>> Willem Jiang
>>> 
>>> Twitter: willemjiang
>>> Weibo: 姜宁willem
>>> 
>>> On Wed, May 1, 2019 at 4:29 PM zhaojun  wrote:
 
 But saga-actuator don’t support 3 logic SQL is a global transaction while 
 you submit 3 times separately.
 
 --
 Zhao Jun
 Apache Sharding-Sphere & ServiceComb
 
> On May 1, 2019, at 8:38 AM, Willem Jiang  wrote:
> 
> Willem Jiang
> 
> Twitter: willemjiang
> Weibo: 姜宁willem
> 
> On Mon, Apr 29, 2019 at 7:58 PM zhaojun  wrote:
>> 
>> Hi, Willem
>> 
>> Saga actuator could not compatible with ShardingSphere currently,  there 
>> are 2 main problem exist.
>> 1. If sql execute in saga transport instead of ShardingSphere, we can 
>> not see the result of logic-sql even in one transaction, it is like this:
>>  update t_order set status=’start’ where usr_id=’tom’;
>>  select status from t_order where usr_id=’tom’;  
>> —> we can’t query ’start’ record as actuator haven’t executed.
>>  Insert into t_order values(?,?,?) ;
> 
> I think we can break the whole SQL interactions into smaller steps,
> and let saga-actuator do the job one by one.
> In this case, you can break the whole calling graph into several sub
> calling graphs.
> Any thought?
> 
>> 2. If logic-sql execute in ShardingSphere, we cannot handle recovery 
>> before submit graph result, as event log only wrote at saga engine.
>> 
>> so we should integrate saga-transaction like omega send event log step 
>> by step.
>> It is better to make every instance do recovery Independently, instead 
>> of providing another coordinator center service.
>> I feel that embed saga should have following  capability.
>> 1).It can provide service in jar package independent
>> 2).each embed saga only recovery their own transaction-data of this 
>> instance.
>> If one instance crashed, we can introduce external service to do 
>> failover.
>> 
>> so we consider about exending saga-acutator, if it can support submit 
>> task step by step in one transaction, it is a good choice.
>> of course, there have many things we should do.
>> 
>> --
>> Zhao Jun
>> Apache Sharding-Sphere & ServiceComb
>> 
>>> On Apr 29, 2019, at 5:17 PM, Willem Jiang  
>>> wrote:
>>> 
>>> First Saga actuator need to build up the calling grapha before sending
>>> out the request.  I don't think you can do the step by step SQL
>>> invocation with Saga actuator.
>>> If you want to call the SQL execution step by step , you may need to
>>> switch to ServiceComb Pack project which has a coordinator to take
>>> care of the distributed transaction. But that introduce another
>>> endpoint(Alpha) to shardingsphere.
>>> 
>>> From my understanding, Saga actuator is most efficient way to execute
>>> the SQL across different data nodes.
>>> 
>>> 
>>> Willem Jiang
>>> 
>>> Twitter: willemjiang
>>> Weibo: 姜宁willem
>>> 
>>> On Fri, Apr 26, 2019 at 6:41 PM zhaojun  wrote:
 
 Hi, all
 
 currently, we have integrated with saga using graph based engine in 
 shardingsphere[1]
 it need us to collect all participated actual SQL, then submit to saga 
 actuator in commit/rollback phase.
 if application crashed before invoking saga actuator, undo log of 
 branch transaction SQL will not be saved,
 so recovery thread will not be executed correctly.
 
 it's better that encapsulating every actual SQL as a saga task in 
 shardingsphere side,
 then submit to saga actuator realtime instead of batch processing all 
 the SQLs at commit/rollback phase.
 this architecture will make the boundary more 

Re: [saga-actuator]consider about adding StreamBasedSaga engine to integrate with shardingsphere

[Willem Jiang]

It sounds good to me.

Willem Jiang

Twitter: willemjiang
Weibo: 姜宁willem

On Thu, May 2, 2019 at 11:14 AM 赵俊  wrote:
>
> Let SQL executor running command one by one is good to me.
>
> One logic-sql will be routed to multi-actual SQLs, then executed in 
> sharding-sphere execute-engine.
> A actual SQL will be wrapped as a SQL execute-callback in sharding-sphere.
> We could provide a SPI for this SQL execute-callback.
> For integrated with saga, we could implement a saga execute-callback, it will 
> wrap event sourcing logic like saga-actuator task have done.
>
>
> > On May 1, 2019, at 5:06 PM, Willem Jiang  wrote:
> >
> > We could do some enhancement on it.
> > From my understanding, We could create a calling graph which has three
> > sub graph of it, or we could let the SQL executor running the command
> > one by one.
> > Any thoughts?
> >
> > Willem Jiang
> >
> > Twitter: willemjiang
> > Weibo: 姜宁willem
> >
> > On Wed, May 1, 2019 at 4:29 PM zhaojun  wrote:
> >>
> >> But saga-actuator don’t support 3 logic SQL is a global transaction while 
> >> you submit 3 times separately.
> >>
> >> --
> >> Zhao Jun
> >> Apache Sharding-Sphere & ServiceComb
> >>
> >>> On May 1, 2019, at 8:38 AM, Willem Jiang  wrote:
> >>>
> >>> Willem Jiang
> >>>
> >>> Twitter: willemjiang
> >>> Weibo: 姜宁willem
> >>>
> >>> On Mon, Apr 29, 2019 at 7:58 PM zhaojun  wrote:
> 
>  Hi, Willem
> 
>  Saga actuator could not compatible with ShardingSphere currently,  there 
>  are 2 main problem exist.
>  1. If sql execute in saga transport instead of ShardingSphere, we can 
>  not see the result of logic-sql even in one transaction, it is like this:
>    update t_order set status=’start’ where usr_id=’tom’;
>    select status from t_order where usr_id=’tom’; 
>   —> we can’t query ’start’ record as actuator haven’t executed.
>    Insert into t_order values(?,?,?) ;
> >>>
> >>> I think we can break the whole SQL interactions into smaller steps,
> >>> and let saga-actuator do the job one by one.
> >>> In this case, you can break the whole calling graph into several sub
> >>> calling graphs.
> >>> Any thought?
> >>>
>  2. If logic-sql execute in ShardingSphere, we cannot handle recovery 
>  before submit graph result, as event log only wrote at saga engine.
> 
>  so we should integrate saga-transaction like omega send event log step 
>  by step.
>  It is better to make every instance do recovery Independently, instead 
>  of providing another coordinator center service.
>  I feel that embed saga should have following  capability.
>  1).It can provide service in jar package independent
>  2).each embed saga only recovery their own transaction-data of this 
>  instance.
>   If one instance crashed, we can introduce external service to do 
>  failover.
> 
>  so we consider about exending saga-acutator, if it can support submit 
>  task step by step in one transaction, it is a good choice.
>  of course, there have many things we should do.
> 
>  --
>  Zhao Jun
>  Apache Sharding-Sphere & ServiceComb
> 
> > On Apr 29, 2019, at 5:17 PM, Willem Jiang  
> > wrote:
> >
> > First Saga actuator need to build up the calling grapha before sending
> > out the request.  I don't think you can do the step by step SQL
> > invocation with Saga actuator.
> > If you want to call the SQL execution step by step , you may need to
> > switch to ServiceComb Pack project which has a coordinator to take
> > care of the distributed transaction. But that introduce another
> > endpoint(Alpha) to shardingsphere.
> >
> > From my understanding, Saga actuator is most efficient way to execute
> > the SQL across different data nodes.
> >
> >
> > Willem Jiang
> >
> > Twitter: willemjiang
> > Weibo: 姜宁willem
> >
> > On Fri, Apr 26, 2019 at 6:41 PM zhaojun  wrote:
> >>
> >> Hi, all
> >>
> >> currently, we have integrated with saga using graph based engine in 
> >> shardingsphere[1]
> >> it need us to collect all participated actual SQL, then submit to saga 
> >> actuator in commit/rollback phase.
> >> if application crashed before invoking saga actuator, undo log of 
> >> branch transaction SQL will not be saved,
> >> so recovery thread will not be executed correctly.
> >>
> >> it's better that encapsulating every actual SQL as a saga task in 
> >> shardingsphere side,
> >> then submit to saga actuator realtime instead of batch processing all 
> >> the SQLs at commit/rollback phase.
> >> this architecture will make the boundary more clear between 
> >> shardingsphre and saga, currently we have done some additional works 
> >> for integrating saga.
> >>
> >> any thought?
> >>
> >> 

Re: [saga-actuator]consider about adding StreamBasedSaga engine to integrate with shardingsphere

[Willem Jiang]

We could do some enhancement on it.
>From my understanding, We could create a calling graph which has three
sub graph of it, or we could let the SQL executor running the command
one by one.
Any thoughts?

Willem Jiang

Twitter: willemjiang
Weibo: 姜宁willem

On Wed, May 1, 2019 at 4:29 PM zhaojun  wrote:
>
> But saga-actuator don’t support 3 logic SQL is a global transaction while you 
> submit 3 times separately.
>
> --
> Zhao Jun
> Apache Sharding-Sphere & ServiceComb
>
> > On May 1, 2019, at 8:38 AM, Willem Jiang  wrote:
> >
> > Willem Jiang
> >
> > Twitter: willemjiang
> > Weibo: 姜宁willem
> >
> > On Mon, Apr 29, 2019 at 7:58 PM zhaojun  wrote:
> >>
> >> Hi, Willem
> >>
> >> Saga actuator could not compatible with ShardingSphere currently,  there 
> >> are 2 main problem exist.
> >> 1. If sql execute in saga transport instead of ShardingSphere, we can not 
> >> see the result of logic-sql even in one transaction, it is like this:
> >>update t_order set status=’start’ where usr_id=’tom’;
> >>select status from t_order where usr_id=’tom’;  
> >> —> we can’t query ’start’ record as actuator haven’t executed.
> >>Insert into t_order values(?,?,?) ;
> >
> > I think we can break the whole SQL interactions into smaller steps,
> > and let saga-actuator do the job one by one.
> > In this case, you can break the whole calling graph into several sub
> > calling graphs.
> > Any thought?
> >
> >> 2. If logic-sql execute in ShardingSphere, we cannot handle recovery 
> >> before submit graph result, as event log only wrote at saga engine.
> >>
> >> so we should integrate saga-transaction like omega send event log step by 
> >> step.
> >> It is better to make every instance do recovery Independently, instead of 
> >> providing another coordinator center service.
> >> I feel that embed saga should have following  capability.
> >>  1).It can provide service in jar package independent
> >>  2).each embed saga only recovery their own transaction-data of this 
> >> instance.
> >>   If one instance crashed, we can introduce external service to do 
> >> failover.
> >>
> >> so we consider about exending saga-acutator, if it can support submit task 
> >> step by step in one transaction, it is a good choice.
> >> of course, there have many things we should do.
> >>
> >> --
> >> Zhao Jun
> >> Apache Sharding-Sphere & ServiceComb
> >>
> >>> On Apr 29, 2019, at 5:17 PM, Willem Jiang  wrote:
> >>>
> >>> First Saga actuator need to build up the calling grapha before sending
> >>> out the request.  I don't think you can do the step by step SQL
> >>> invocation with Saga actuator.
> >>> If you want to call the SQL execution step by step , you may need to
> >>> switch to ServiceComb Pack project which has a coordinator to take
> >>> care of the distributed transaction. But that introduce another
> >>> endpoint(Alpha) to shardingsphere.
> >>>
> >>> From my understanding, Saga actuator is most efficient way to execute
> >>> the SQL across different data nodes.
> >>>
> >>>
> >>> Willem Jiang
> >>>
> >>> Twitter: willemjiang
> >>> Weibo: 姜宁willem
> >>>
> >>> On Fri, Apr 26, 2019 at 6:41 PM zhaojun  wrote:
> 
>  Hi, all
> 
>  currently, we have integrated with saga using graph based engine in 
>  shardingsphere[1]
>  it need us to collect all participated actual SQL, then submit to saga 
>  actuator in commit/rollback phase.
>  if application crashed before invoking saga actuator, undo log of branch 
>  transaction SQL will not be saved,
>  so recovery thread will not be executed correctly.
> 
>  it's better that encapsulating every actual SQL as a saga task in 
>  shardingsphere side,
>  then submit to saga actuator realtime instead of batch processing all 
>  the SQLs at commit/rollback phase.
>  this architecture will make the boundary more clear between 
>  shardingsphre and saga, currently we have done some additional works for 
>  integrating saga.
> 
>  any thought?
> 
>  [1]:https://github.com/sharding-sphere/shardingsphere-spi-impl/tree/master/sharding-transaction-spi-impl/sharding-transaction-base-spi-impl/sharding-transaction-base-saga
>   
>  
> 
> 
> 
>  --
>  Zhao Jun
>  Apache Sharding-Sphere & ServiceComb
> 
> >>
>

Re: [saga-actuator]consider about adding StreamBasedSaga engine to integrate with shardingsphere

[zhaojun]

But saga-actuator don’t support 3 logic SQL is a global transaction while you 
submit 3 times separately.

--
Zhao Jun
Apache Sharding-Sphere & ServiceComb

> On May 1, 2019, at 8:38 AM, Willem Jiang  wrote:
> 
> Willem Jiang
> 
> Twitter: willemjiang
> Weibo: 姜宁willem
> 
> On Mon, Apr 29, 2019 at 7:58 PM zhaojun  wrote:
>> 
>> Hi, Willem
>> 
>> Saga actuator could not compatible with ShardingSphere currently,  there are 
>> 2 main problem exist.
>> 1. If sql execute in saga transport instead of ShardingSphere, we can not 
>> see the result of logic-sql even in one transaction, it is like this:
>>update t_order set status=’start’ where usr_id=’tom’;
>>select status from t_order where usr_id=’tom’;  
>> —> we can’t query ’start’ record as actuator haven’t executed.
>>Insert into t_order values(?,?,?) ;
> 
> I think we can break the whole SQL interactions into smaller steps,
> and let saga-actuator do the job one by one.
> In this case, you can break the whole calling graph into several sub
> calling graphs.
> Any thought?
> 
>> 2. If logic-sql execute in ShardingSphere, we cannot handle recovery before 
>> submit graph result, as event log only wrote at saga engine.
>> 
>> so we should integrate saga-transaction like omega send event log step by 
>> step.
>> It is better to make every instance do recovery Independently, instead of 
>> providing another coordinator center service.
>> I feel that embed saga should have following  capability.
>>  1).It can provide service in jar package independent
>>  2).each embed saga only recovery their own transaction-data of this 
>> instance.
>>   If one instance crashed, we can introduce external service to do 
>> failover.
>> 
>> so we consider about exending saga-acutator, if it can support submit task 
>> step by step in one transaction, it is a good choice.
>> of course, there have many things we should do.
>> 
>> --
>> Zhao Jun
>> Apache Sharding-Sphere & ServiceComb
>> 
>>> On Apr 29, 2019, at 5:17 PM, Willem Jiang  wrote:
>>> 
>>> First Saga actuator need to build up the calling grapha before sending
>>> out the request.  I don't think you can do the step by step SQL
>>> invocation with Saga actuator.
>>> If you want to call the SQL execution step by step , you may need to
>>> switch to ServiceComb Pack project which has a coordinator to take
>>> care of the distributed transaction. But that introduce another
>>> endpoint(Alpha) to shardingsphere.
>>> 
>>> From my understanding, Saga actuator is most efficient way to execute
>>> the SQL across different data nodes.
>>> 
>>> 
>>> Willem Jiang
>>> 
>>> Twitter: willemjiang
>>> Weibo: 姜宁willem
>>> 
>>> On Fri, Apr 26, 2019 at 6:41 PM zhaojun  wrote:
 
 Hi, all
 
 currently, we have integrated with saga using graph based engine in 
 shardingsphere[1]
 it need us to collect all participated actual SQL, then submit to saga 
 actuator in commit/rollback phase.
 if application crashed before invoking saga actuator, undo log of branch 
 transaction SQL will not be saved,
 so recovery thread will not be executed correctly.
 
 it's better that encapsulating every actual SQL as a saga task in 
 shardingsphere side,
 then submit to saga actuator realtime instead of batch processing all the 
 SQLs at commit/rollback phase.
 this architecture will make the boundary more clear between shardingsphre 
 and saga, currently we have done some additional works for integrating 
 saga.
 
 any thought?
 
 [1]:https://github.com/sharding-sphere/shardingsphere-spi-impl/tree/master/sharding-transaction-spi-impl/sharding-transaction-base-spi-impl/sharding-transaction-base-saga
  
 
 
 
 
 --
 Zhao Jun
 Apache Sharding-Sphere & ServiceComb
 
>> 

Re: [saga-actuator]consider about adding StreamBasedSaga engine to integrate with shardingsphere

[Sheng Wu]

Feedback from our Seata and SkyWalking integration.

We will provide plugins for there TC and TM.
Also, Seata team has the plan to distribute their binary including
SkyWalking binary tar in TC, for user convenience.
Then users could use a single command in TC to open SkyWalking agent
working with the globally distributed tracing system, and benefits from
SkyWalking other monitoring/analysis capabilities.

Sheng Wu 吴晟

Apache SkyWalking, ShardingSphere, Zipkin
Twitter, wusheng1108


Willem Jiang  于2019年5月1日周三 上午8:44写道：

> Willem Jiang
>
> Twitter: willemjiang
> Weibo: 姜宁willem
>
> On Mon, Apr 29, 2019 at 7:58 PM zhaojun  wrote:
> >
> > Hi, Willem
> >
> > Saga actuator could not compatible with ShardingSphere currently,  there
> are 2 main problem exist.
> >  1. If sql execute in saga transport instead of ShardingSphere, we can
> not see the result of logic-sql even in one transaction, it is like this:
> > update t_order set status=’start’ where usr_id=’tom’;
> > select status from t_order where usr_id=’tom’;
> —> we can’t query ’start’ record as actuator haven’t executed.
> > Insert into t_order values(?,?,?) ;
>
> I think we can break the whole SQL interactions into smaller steps,
> and let saga-actuator do the job one by one.
> In this case, you can break the whole calling graph into several sub
> calling graphs.
> Any thought?
>
> >  2. If logic-sql execute in ShardingSphere, we cannot handle recovery
> before submit graph result, as event log only wrote at saga engine.
> >
> > so we should integrate saga-transaction like omega send event log step
> by step.
> > It is better to make every instance do recovery Independently, instead
> of providing another coordinator center service.
> > I feel that embed saga should have following  capability.
> >   1).It can provide service in jar package independent
> >   2).each embed saga only recovery their own transaction-data of this
> instance.
> >If one instance crashed, we can introduce external service to do
> failover.
> >
> > so we consider about exending saga-acutator, if it can support submit
> task step by step in one transaction, it is a good choice.
> > of course, there have many things we should do.
> >
> > --
> > Zhao Jun
> > Apache Sharding-Sphere & ServiceComb
> >
> > > On Apr 29, 2019, at 5:17 PM, Willem Jiang 
> wrote:
> > >
> > > First Saga actuator need to build up the calling grapha before sending
> > > out the request.  I don't think you can do the step by step SQL
> > > invocation with Saga actuator.
> > > If you want to call the SQL execution step by step , you may need to
> > > switch to ServiceComb Pack project which has a coordinator to take
> > > care of the distributed transaction. But that introduce another
> > > endpoint(Alpha) to shardingsphere.
> > >
> > > From my understanding, Saga actuator is most efficient way to execute
> > > the SQL across different data nodes.
> > >
> > >
> > > Willem Jiang
> > >
> > > Twitter: willemjiang
> > > Weibo: 姜宁willem
> > >
> > > On Fri, Apr 26, 2019 at 6:41 PM zhaojun  wrote:
> > >>
> > >> Hi, all
> > >>
> > >> currently, we have integrated with saga using graph based engine in
> shardingsphere[1]
> > >> it need us to collect all participated actual SQL, then submit to
> saga actuator in commit/rollback phase.
> > >> if application crashed before invoking saga actuator, undo log of
> branch transaction SQL will not be saved,
> > >> so recovery thread will not be executed correctly.
> > >>
> > >> it's better that encapsulating every actual SQL as a saga task in
> shardingsphere side,
> > >> then submit to saga actuator realtime instead of batch processing all
> the SQLs at commit/rollback phase.
> > >> this architecture will make the boundary more clear between
> shardingsphre and saga, currently we have done some additional works for
> integrating saga.
> > >>
> > >> any thought?
> > >>
> > >> [1]:
> https://github.com/sharding-sphere/shardingsphere-spi-impl/tree/master/sharding-transaction-spi-impl/sharding-transaction-base-spi-impl/sharding-transaction-base-saga
> <
> https://github.com/sharding-sphere/shardingsphere-spi-impl/tree/master/sharding-transaction-spi-impl/sharding-transaction-base-spi-impl/sharding-transaction-base-saga
> >
> > >>
> > >>
> > >>
> > >> --
> > >> Zhao Jun
> > >> Apache Sharding-Sphere & ServiceComb
> > >>
> >
>

Re: [saga-actuator]consider about adding StreamBasedSaga engine to integrate with shardingsphere

[Willem Jiang]

Willem Jiang

Twitter: willemjiang
Weibo: 姜宁willem

On Mon, Apr 29, 2019 at 7:58 PM zhaojun  wrote:
>
> Hi, Willem
>
> Saga actuator could not compatible with ShardingSphere currently,  there are 
> 2 main problem exist.
>  1. If sql execute in saga transport instead of ShardingSphere, we can not 
> see the result of logic-sql even in one transaction, it is like this:
> update t_order set status=’start’ where usr_id=’tom’;
> select status from t_order where usr_id=’tom’;  
> —> we can’t query ’start’ record as actuator haven’t executed.
> Insert into t_order values(?,?,?) ;

I think we can break the whole SQL interactions into smaller steps,
and let saga-actuator do the job one by one.
In this case, you can break the whole calling graph into several sub
calling graphs.
Any thought?

>  2. If logic-sql execute in ShardingSphere, we cannot handle recovery before 
> submit graph result, as event log only wrote at saga engine.
>
> so we should integrate saga-transaction like omega send event log step by 
> step.
> It is better to make every instance do recovery Independently, instead of 
> providing another coordinator center service.
> I feel that embed saga should have following  capability.
>   1).It can provide service in jar package independent
>   2).each embed saga only recovery their own transaction-data of this 
> instance.
>If one instance crashed, we can introduce external service to do 
> failover.
>
> so we consider about exending saga-acutator, if it can support submit task 
> step by step in one transaction, it is a good choice.
> of course, there have many things we should do.
>
> --
> Zhao Jun
> Apache Sharding-Sphere & ServiceComb
>
> > On Apr 29, 2019, at 5:17 PM, Willem Jiang  wrote:
> >
> > First Saga actuator need to build up the calling grapha before sending
> > out the request.  I don't think you can do the step by step SQL
> > invocation with Saga actuator.
> > If you want to call the SQL execution step by step , you may need to
> > switch to ServiceComb Pack project which has a coordinator to take
> > care of the distributed transaction. But that introduce another
> > endpoint(Alpha) to shardingsphere.
> >
> > From my understanding, Saga actuator is most efficient way to execute
> > the SQL across different data nodes.
> >
> >
> > Willem Jiang
> >
> > Twitter: willemjiang
> > Weibo: 姜宁willem
> >
> > On Fri, Apr 26, 2019 at 6:41 PM zhaojun  wrote:
> >>
> >> Hi, all
> >>
> >> currently, we have integrated with saga using graph based engine in 
> >> shardingsphere[1]
> >> it need us to collect all participated actual SQL, then submit to saga 
> >> actuator in commit/rollback phase.
> >> if application crashed before invoking saga actuator, undo log of branch 
> >> transaction SQL will not be saved,
> >> so recovery thread will not be executed correctly.
> >>
> >> it's better that encapsulating every actual SQL as a saga task in 
> >> shardingsphere side,
> >> then submit to saga actuator realtime instead of batch processing all the 
> >> SQLs at commit/rollback phase.
> >> this architecture will make the boundary more clear between shardingsphre 
> >> and saga, currently we have done some additional works for integrating 
> >> saga.
> >>
> >> any thought?
> >>
> >> [1]:https://github.com/sharding-sphere/shardingsphere-spi-impl/tree/master/sharding-transaction-spi-impl/sharding-transaction-base-spi-impl/sharding-transaction-base-saga
> >>  
> >> 
> >>
> >>
> >>
> >> --
> >> Zhao Jun
> >> Apache Sharding-Sphere & ServiceComb
> >>
>

Re: [saga-actuator]consider about adding StreamBasedSaga engine to integrate with shardingsphere

[zhaojun]

Hi, Willem

Saga actuator could not compatible with ShardingSphere currently,  there are 2 
main problem exist.
 1. If sql execute in saga transport instead of ShardingSphere, we can not see 
the result of logic-sql even in one transaction, it is like this:
update t_order set status=’start’ where usr_id=’tom’;  
select status from t_order where usr_id=’tom’;  —> 
we can’t query ’start’ record as actuator haven’t executed.
Insert into t_order values(?,?,?) ;
 2. If logic-sql execute in ShardingSphere, we cannot handle recovery before 
submit graph result, as event log only wrote at saga engine.

so we should integrate saga-transaction like omega send event log step by step. 
It is better to make every instance do recovery Independently, instead of 
providing another coordinator center service.
I feel that embed saga should have following  capability.
  1).It can provide service in jar package independent
  2).each embed saga only recovery their own transaction-data of this instance.
   If one instance crashed, we can introduce external service to do 
failover. 

so we consider about exending saga-acutator, if it can support submit task step 
by step in one transaction, it is a good choice.
of course, there have many things we should do.

--
Zhao Jun
Apache Sharding-Sphere & ServiceComb

> On Apr 29, 2019, at 5:17 PM, Willem Jiang  wrote:
> 
> First Saga actuator need to build up the calling grapha before sending
> out the request.  I don't think you can do the step by step SQL
> invocation with Saga actuator.
> If you want to call the SQL execution step by step , you may need to
> switch to ServiceComb Pack project which has a coordinator to take
> care of the distributed transaction. But that introduce another
> endpoint(Alpha) to shardingsphere.
> 
> From my understanding, Saga actuator is most efficient way to execute
> the SQL across different data nodes.
> 
> 
> Willem Jiang
> 
> Twitter: willemjiang
> Weibo: 姜宁willem
> 
> On Fri, Apr 26, 2019 at 6:41 PM zhaojun  wrote:
>> 
>> Hi, all
>> 
>> currently, we have integrated with saga using graph based engine in 
>> shardingsphere[1]
>> it need us to collect all participated actual SQL, then submit to saga 
>> actuator in commit/rollback phase.
>> if application crashed before invoking saga actuator, undo log of branch 
>> transaction SQL will not be saved,
>> so recovery thread will not be executed correctly.
>> 
>> it's better that encapsulating every actual SQL as a saga task in 
>> shardingsphere side,
>> then submit to saga actuator realtime instead of batch processing all the 
>> SQLs at commit/rollback phase.
>> this architecture will make the boundary more clear between shardingsphre 
>> and saga, currently we have done some additional works for integrating saga.
>> 
>> any thought?
>> 
>> [1]:https://github.com/sharding-sphere/shardingsphere-spi-impl/tree/master/sharding-transaction-spi-impl/sharding-transaction-base-spi-impl/sharding-transaction-base-saga
>>  
>> 
>> 
>> 
>> 
>> --
>> Zhao Jun
>> Apache Sharding-Sphere & ServiceComb
>> 

Re: [saga-actuator]consider about adding StreamBasedSaga engine to integrate with shardingsphere

[Willem Jiang]

First Saga actuator need to build up the calling grapha before sending
out the request.  I don't think you can do the step by step SQL
invocation with Saga actuator.
If you want to call the SQL execution step by step , you may need to
switch to ServiceComb Pack project which has a coordinator to take
care of the distributed transaction. But that introduce another
endpoint(Alpha) to shardingsphere.

>From my understanding, Saga actuator is most efficient way to execute
the SQL across different data nodes.


Willem Jiang

Twitter: willemjiang
Weibo: 姜宁willem

On Fri, Apr 26, 2019 at 6:41 PM zhaojun  wrote:
>
> Hi, all
>
> currently, we have integrated with saga using graph based engine in 
> shardingsphere[1]
> it need us to collect all participated actual SQL, then submit to saga 
> actuator in commit/rollback phase.
> if application crashed before invoking saga actuator, undo log of branch 
> transaction SQL will not be saved,
> so recovery thread will not be executed correctly.
>
> it's better that encapsulating every actual SQL as a saga task in 
> shardingsphere side,
> then submit to saga actuator realtime instead of batch processing all the 
> SQLs at commit/rollback phase.
> this architecture will make the boundary more clear between shardingsphre and 
> saga, currently we have done some additional works for integrating saga.
>
> any thought?
>
> [1]:https://github.com/sharding-sphere/shardingsphere-spi-impl/tree/master/sharding-transaction-spi-impl/sharding-transaction-base-spi-impl/sharding-transaction-base-saga
>  
> 
>
>
>
> --
> Zhao Jun
> Apache Sharding-Sphere & ServiceComb
>

Re: [saga-actuator]consider about adding StreamBasedSaga engine to integrate with shardingsphere

[zhaojun]

Hi, zheng, Sheng

Revert SQL have been in compensate method method automatically. We have 
completed this feature.
You can see more detail from our example[1].
Also we have some issue [2] to do. shardingsphere-transaction-base-saga could 
be using in cloud native environment. I think this model is more effective.

[1] 
https://github.com/apache/incubator-shardingsphere-example/tree/dev/sharding-jdbc-example/transaction-example/transaction-base-saga-example
 

[2]https://github.com/sharding-sphere/shardingsphere-spi-impl/issues?q=is%3Aopen+is%3Aissue+label%3Asaga
 





--
Zhao Jun
Apache Sharding-Sphere & ServiceComb

> On Apr 26, 2019, at 7:16 PM, Sheng Wu  wrote:
> 
> Hi
> 
> SkyWalking will do agent based auto instrumentation by runtime code
> manipulate tech. So we would depend on SPI or particular interface.
> SkyWalking plans to intercept some core APIs.
> 
> ServiceComb could be supported in same way.
> 
> I cc'd dev@skywalking
> 
> Zhenxu Ke and Yuguang Zhao, you could share your design to Shardingsphere
> community too.
> 
> Zheng Feng 于2019年4月26日 周五下午7:02写道：
> 
>> Thanks Sheng for pointing out this and I just wonder the SkyWalking will
>> trace the transaction event according to the issue [1]. And this
>> transaction event could have the common SPI interface with the different
>> vendor implementation, e.g.  the ServiceComb and the Seata ?
>> 
>> Thanks,
>> Zheng Feng
>> 
>> [1] https://github.com/seata/seata/issues/714
>> 
>> Sheng Wu  于2019年4月26日周五 下午6:47写道：
>> 
>>> Please be advised, SkyWalking and seata integration are ongoing too.
>>> 
>>> https://github.com/seata/seata/issues/714
>>> 
>>> SkyWalking agent even will be binding in Seata TC binary release.
>>> 
>>> As shardingsphere pmc are developing SkyWalking plugin too. Please be
>>> careful, don't make the conflict.
>>> 
>>> zhaojun 于2019年4月26日 周五下午6:41写道：
>>> 
 Hi, all
 
 currently, we have integrated with saga using graph based engine in
 shardingsphere[1]
 it need us to collect all participated actual SQL, then submit to saga
 actuator in commit/rollback phase.
 if application crashed before invoking saga actuator, undo log of
>> branch
 transaction SQL will not be saved,
 so recovery thread will not be executed correctly.
 
 it's better that encapsulating every actual SQL as a saga task in
 shardingsphere side,
 then submit to saga actuator realtime instead of batch processing all
>> the
 SQLs at commit/rollback phase.
 this architecture will make the boundary more clear between
>> shardingsphre
 and saga, currently we have done some additional works for integrating
>>> saga.
 
 any thought?
 
 [1]:
 
>>> 
>> https://github.com/sharding-sphere/shardingsphere-spi-impl/tree/master/sharding-transaction-spi-impl/sharding-transaction-base-spi-impl/sharding-transaction-base-saga
 <
 
>>> 
>> https://github.com/sharding-sphere/shardingsphere-spi-impl/tree/master/sharding-transaction-spi-impl/sharding-transaction-base-spi-impl/sharding-transaction-base-saga
> 
 
 
 
 --
 Zhao Jun
 Apache Sharding-Sphere & ServiceComb
 
 --
>>> Sheng Wu
>>> SkyWalking, Shardingsphere and Zipkin
>>> 
>> 
> -- 
> Sheng Wu
> SkyWalking, Shardingsphere and Zipkin

Re: [saga-actuator]consider about adding StreamBasedSaga engine to integrate with shardingsphere

[Zheng Feng]

Hi Zhao Jun,

It looks good to me and it could have the revert SQL in the compensate
method automatically ?

Regards,
Zheng Feng

zhaojun  于2019年4月26日周五 下午6:41写道：

> Hi, all
>
> currently, we have integrated with saga using graph based engine in
> shardingsphere[1]
> it need us to collect all participated actual SQL, then submit to saga
> actuator in commit/rollback phase.
> if application crashed before invoking saga actuator, undo log of branch
> transaction SQL will not be saved,
> so recovery thread will not be executed correctly.
>
> it's better that encapsulating every actual SQL as a saga task in
> shardingsphere side,
> then submit to saga actuator realtime instead of batch processing all the
> SQLs at commit/rollback phase.
> this architecture will make the boundary more clear between shardingsphre
> and saga, currently we have done some additional works for integrating saga.
>
> any thought?
>
> [1]:
> https://github.com/sharding-sphere/shardingsphere-spi-impl/tree/master/sharding-transaction-spi-impl/sharding-transaction-base-spi-impl/sharding-transaction-base-saga
> <
> https://github.com/sharding-sphere/shardingsphere-spi-impl/tree/master/sharding-transaction-spi-impl/sharding-transaction-base-spi-impl/sharding-transaction-base-saga
> >
>
>
>
> --
> Zhao Jun
> Apache Sharding-Sphere & ServiceComb
>
>

[saga-actuator]consider about adding StreamBasedSaga engine to integrate with shardingsphere

[zhaojun]

Hi, all

currently, we have integrated with saga using graph based engine in 
shardingsphere[1]
it need us to collect all participated actual SQL, then submit to saga actuator 
in commit/rollback phase.
if application crashed before invoking saga actuator, undo log of branch 
transaction SQL will not be saved, 
so recovery thread will not be executed correctly.

it's better that encapsulating every actual SQL as a saga task in 
shardingsphere side, 
then submit to saga actuator realtime instead of batch processing all the SQLs 
at commit/rollback phase. 
this architecture will make the boundary more clear between shardingsphre and 
saga, currently we have done some additional works for integrating saga.

any thought?

[1]:https://github.com/sharding-sphere/shardingsphere-spi-impl/tree/master/sharding-transaction-spi-impl/sharding-transaction-base-spi-impl/sharding-transaction-base-saga
 




--
Zhao Jun
Apache Sharding-Sphere & ServiceComb