OK, I will try to write a configurable register implementation.
Here are some of my thoughts:
1. Choose a central middleware to generate global ID and provide a channel
to pub/sub the register message, Redis or Zookeeper should be fine, but
consider the need for an external network address, Redis would be better.
2. All the register global ID is from Redis, and the register request will
get to both clouds through pub/sub channel of Redis.
3. Of course, this is a configurable feature, we can open the config item
when we truely faced with the cross-cloud problem.
I wonder if those thoughts will be accepted by the communitry as a feature, we
wanted to give all the changes back to the community, and follow the version
iteration of the community.
By the way, if the trace is from different Skywalking but in different company,
like we provide a payment service as a trading channel to another company, and
we both use Skywalking. Is there a problem with this scenario?
_______________________________
Lilong@iBOXCHAIN
Middleware Developer
On 12/11/2019 21:48，Sheng Wu<[email protected]> wrote：
OK. I think I got the point.
In my mind, one new storage provider is a better idea.
I recommend you to read IRegisterLockDAO and IRegisterDAO implementations
in the storage provider(ES's, I think).
Then, you should try to write your own register implementation. Like
1. Use one place to generate ID, such as using a central Redis to generate
ID. Across cloud accessible is the key point.
2. Replicated write to both clouds for metadata.
3. The query should keep your current change.
You should be able to run all the things much more stable and easier than
what you do today.
Sheng Wu 吴晟
Twitter, wusheng1108
成龙 <[email protected]> 于2019年12月11日周三 下午9:40写道：
Thanks for your reply!
Oh, yes, to avoid ID conflicts, we made some slight changes to OAP server,
and write a extra sync service. Details is as follows:
1. First of all, we add a config item to es storage, called "startIndex:
${SW_STORAGE_ES_START_INDEX:100000}", means all register ID is start from
this value(At the moment, we've written down a big enough number, it's not
a good idea, we are optimizing,figure out how to make the id unique).
Therefore, Skywalking in different cloud has a different sectional id.
2. The extra sync service in Public Cloudwill subscribe thePublic Cloud es
data changes(only metadata), and send the data to Public Cloud
RocketMQ(Internet HTTP Endpoint), meanwhile the extra sync service in
Finance Cloud will receive the message, and execute the normal register
process.
3. The message is carrying the ID which from cross-cloud, we use a map to
storage the relationship of the normal register id and the cross-cloud id.
4. In the segment parse process, if the trace is across the cloud, we use
the id-relationship-map to exchange the cross-cloud id to normal id, make
Skywalking just like deal with the local trace.
5. Of course, also need to make a slight change to
server-query-plugin#TraceQuery, make sure the UI shows the cross-cloud
trace properly.
I don't know if it's explained clearly. Thanks again for your reply!
Lilong@iBOXCHAIN
Middleware Developer
Shenzhen iBOXCHAIN technology co. LTD
[email protected]
On 12/11/2019 20:31，Sheng Wu<[email protected]> wrote：
Hi Lilong
Thanks for sharing the details of your scenario.
I may need a little more about the detail.
I don't know how do you sync the metadata across the cloud? The register
could happen concurrently, how could you guarantee the ID is unique in the
global?
Sheng Wu 吴晟
Twitter, wusheng1108
成龙 <[email protected]> 于2019年12月11日周三 下午6:24写道：
Hello everyone,
We know that trace, topology and metrics across cloud was not expect, but
for a variety of reasons, we are now faced with the cross-cloud headache!
Our core payment service is in Alibaba Finance Cloud because of the
compliance audit, and most of the business service is in Alibaba Public
Cloud because of the cost considerations.
Here are some of our problems:
1. Each cloud has thier own Skywalking which responsible for the
service tracing in their respective cloud regions;
2. The network between the cloud is separated, we can't send the trace
segment to another directly;
3. Most of payment requests in Finance are came from Public Cloud, so
we can't just ignore these traces;
4. Besides if the trace is across cloud, it will throws
NullPointerException becuase of referenceId exchange
(ReferenceIdExchanger.java:60, baseed on 6.5.0), OAP server will print too
much exception stack.
To solve these problems, we simply sync the register metadata across the
cloud to make sure Skywalking in Finance Cloud can properly process the
cross-cloud segment. In this case, we can collect all segment whether the
trace is across cloud. Also we have the same traceId, we can find the same
trace in both side, even though it's just part of the whole trace.
I know maybe this is not a good idea, but it can solve our current
probolems. So I really look forward to a better solution, or if someone has
any good ideas or suggestions on this, please let me know!
Thanks.
| |
Lilong@iBOXCHAIN
Middleware Developer
|
|
Shenzhen iBOXCHAIN technology co. LTD
[email protected]
|
