OK. I think I got the point. In my mind, one new storage provider is a better idea.
I recommend you to read IRegisterLockDAO and IRegisterDAO implementations in the storage provider(ES's, I think). Then, you should try to write your own register implementation. Like 1. Use one place to generate ID, such as using a central Redis to generate ID. Across cloud accessible is the key point. 2. Replicated write to both clouds for metadata. 3. The query should keep your current change. You should be able to run all the things much more stable and easier than what you do today. Sheng Wu 吴晟 Twitter, wusheng1108 成龙 <[email protected]> 于2019年12月11日周三 下午9:40写道： > > > Thanks for your reply! > > > Oh, yes, to avoid ID conflicts, we made some slight changes to OAP server, > and write a extra sync service. Details is as follows: > > > 1. First of all, we add a config item to es storage, called "startIndex: > ${SW_STORAGE_ES_START_INDEX:100000}", means all register ID is start from > this value(At the moment, we've written down a big enough number, it's not > a good idea, we are optimizing,figure out how to make the id unique). > Therefore, Skywalking in different cloud has a different sectional id. > 2. The extra sync service in Public Cloudwill subscribe thePublic Cloud es > data changes(only metadata), and send the data to Public Cloud > RocketMQ(Internet HTTP Endpoint), meanwhile the extra sync service in > Finance Cloud will receive the message, and execute the normal register > process. > 3. The message is carrying the ID which from cross-cloud, we use a map to > storage the relationship of the normal register id and the cross-cloud id. > 4. In the segment parse process, if the trace is across the cloud, we use > the id-relationship-map to exchange the cross-cloud id to normal id, make > Skywalking just like deal with the local trace. > 5. Of course, also need to make a slight change to > server-query-plugin#TraceQuery, make sure the UI shows the cross-cloud > trace properly. > > > I don't know if it's explained clearly. Thanks again for your reply! > > > > Lilong@iBOXCHAIN > Middleware Developer > Shenzhen iBOXCHAIN technology co. LTD > [email protected] > On 12/11/2019 20:31，Sheng Wu<[email protected]> wrote： > Hi Lilong > > Thanks for sharing the details of your scenario. > I may need a little more about the detail. > > I don't know how do you sync the metadata across the cloud? The register > could happen concurrently, how could you guarantee the ID is unique in the > global? > > > Sheng Wu 吴晟 > Twitter, wusheng1108 > > > 成龙 <[email protected]> 于2019年12月11日周三 下午6:24写道： > > > > Hello everyone, > > > We know that trace, topology and metrics across cloud was not expect, but > for a variety of reasons, we are now faced with the cross-cloud headache! > > > Our core payment service is in Alibaba Finance Cloud because of the > compliance audit, and most of the business service is in Alibaba Public > Cloud because of the cost considerations. > Here are some of our problems: > 1. Each cloud has thier own Skywalking which responsible for the > service tracing in their respective cloud regions; > 2. The network between the cloud is separated, we can't send the trace > segment to another directly; > 3. Most of payment requests in Finance are came from Public Cloud, so > we can't just ignore these traces; > 4. Besides if the trace is across cloud, it will throws > NullPointerException becuase of referenceId exchange > (ReferenceIdExchanger.java:60, baseed on 6.5.0), OAP server will print too > much exception stack. > > > To solve these problems, we simply sync the register metadata across the > cloud to make sure Skywalking in Finance Cloud can properly process the > cross-cloud segment. In this case, we can collect all segment whether the > trace is across cloud. Also we have the same traceId, we can find the same > trace in both side, even though it's just part of the whole trace. > > > I know maybe this is not a good idea, but it can solve our current > probolems. So I really look forward to a better solution, or if someone has > any good ideas or suggestions on this, please let me know! > > > Thanks. > > > | | > Lilong@iBOXCHAIN > Middleware Developer > | > | > Shenzhen iBOXCHAIN technology co. LTD > [email protected] > | >
