hello, community,
After some discussion, I made a summary and listed some function points
that need to be collaborated. The content is as follows:
- Protocol
Firstly, use http protocol to send access log to skywalking, and in the
future use GRPC protocol.
- Structure
AccessLogProto (ALP) [1]
- Community Cooperation
Apache APISIX
1. Support the collection of fields required by access log
2. Extend plugin, use http protocol to send to the target address
Apache Skywalking
1. Implement http receiver, support AccessLogProto (ALP) format analysis,
and write to OAP
Welcome to add and discuss, thanks.
[1] Data structure definition for reference
```go
message AccessLogEntry {
AccessLogCommon common = 1;
HTTPRequestProperties request = 2;
HTTPResponseProperties response = 3;
}
```
```go
message AccessLogCommon {
double sample_rate = 1;
string downstream_remote_address = 2;
string downstream_local_address = 3;
string upstream_remote_address = 4;
string upstream_local_address = 5;
google.protobuf.Timestamp start_time = 6;
// Interval between the first downstream byte received and the first
upstream byte sent
google.protobuf.Duration time_to_first_upstream_tx_byte = 7;
string route_name = 8;
HTTPRequestProperties request = 9;
HTTPResponseProperties response = 10;
}
```
```go
message HTTPRequestProperties {
string method = 1;
string scheme = 2;
google.protobuf.UInt32Value port = 3;
string path = 4;
string user_agent = 5;
string forwarded_for = 6;
tring request_id = 7;
uint64 request_headers_bytes = 8;
uint64 request_body_bytes = 9;
map<string, string> request_headers = 10;
}
```
```go
message HTTPResponseProperties {
uint64 response_code = 1;
uint64 response_headers_bytes = 2;
uint64 response_body_bytes = 3;
map<string, string> response_headers = 4;
}
```
Sheng Wu <wu.sheng.841...@gmail.com> 于2021年4月6日周二 下午4:45写道：
> This seems a very abstract question.
> What kind of access log do you have? Is there a design or format document
> to describe that?
>
> Sheng Wu 吴晟
> Twitter, wusheng1108
>
>
> Ming Wen <wenm...@apache.org> 于2021年4月6日周二 下午4:17写道：
>
> > Hi, sheng wu,
> > How can Apache APISIX send ALS to skywalking?
> > Are we implementing it in the existing skywalking plugin? Or need to add
> a
> > new plug-in?
> >
> > Thanks,
> > Ming Wen, Apache APISIX PMC Chair
> > Twitter: _WenMing
> >
> >
> > Sheng Wu <wu.sheng.841...@gmail.com> 于2021年3月23日周二 下午12:07写道：
> >
> > > Ming Wen <wenm...@apache.org> 于2021年3月23日周二 上午11:25写道：
> > >
> > > > > The biggest advantage of ALS, is replacing the physical local
> access
> > > log
> > > > > file. So you have metrics, topology(dependency), and logs in one
> > > package
> > > > of
> > > > > data.
> > > >
> > > > If we also record the metrics and tracing data with the physical
> local
> > > > access log, can we achieve the same effect?
> > > >
> > >
> > > Technically, it is practicable. Once we add file scanning capability in
> > the
> > > satellite, or use 3rd party tool, and forward to the OAP.
> > > The question is, do we really need the physical log file, as basically,
> > it
> > > costs unnecessary resources.
> > >
> > >
> > > Sheng Wu 吴晟
> > > Twitter, wusheng1108
> > >
> > >
> > > >
> > > > > If you want to discuss most ALS in APISIX, I am glad to help with
> the
> > > > > design and integration solution.
> > > >
> > > > yes, I want to discuss more details.
> > > >
> > > > Thanks,
> > > > Ming Wen, Apache APISIX PMC Chair
> > > > Twitter: _WenMing
> > > >
> > > >
> > > > wei jin <k...@apache.org> 于2021年3月22日周一 下午3:58写道：
> > > >
> > > > > Zexuan Luo <spacewan...@apache.org> 于2021年3月21日周日 下午2:18写道：
> > > > >
> > > > > > I am not familiar with
> > > https://github.com/apache/skywalking-satellite.
> > > > > > Can we use this project to convert plain HTTP to GRPC?
> > > > >
> > > > >
> > > > > Skywalking satellite is usually deployed as a sidecar. It can
> convert
> > > > > APISIX gateway metrics to other data formats, do some
> preprocessing,
> > > and
> > > > > even change the data collection method from pull to push.
> > > > >
> > > > > However, the original prometheus metrics are still generated by
> > APISIX,
> > > > > which is not suitable for URI-level statistics.
> > > > >
> > > > > wei jin <k...@apache.org> 于2021年3月22日周一 下午3:57写道：
> > > > >
> > > > > > Zexuan Luo <spacewan...@apache.org> 于2021年3月21日周日 下午2:18写道：
> > > > > >
> > > > > > > I am not familiar with
> > > > https://github.com/apache/skywalking-satellite.
> > > > > > > Can we use this project to convert plain HTTP to GRPC?
> > > > > >
> > > > > >
> > > > > > Skywalking satellite is usually deployed as a sidecar. It can
> > convert
> > > > > > APISIX gateway metrics to other data formats, do some
> > preprocessing,
> > > > and
> > > > > > even change the data collection method from pull to push.
> > > > > >
> > > > > > However, the original prometheus metrics are still generated by
> > > APISIX,
> > > > > > which is not suitable for URI-level statistics.
> > > > > >
> > > > > >
> > > > > > YuanSheng Wang <membp...@apache.org> 于2021年3月21日周日 下午10:08写道：
> > > > > >
> > > > > >> For APISIX, gRPC calls need to be supported.
> > > > > >>
> > > > > >> Either subrequest or pure Lua mode is ok, this feature is also
> > very
> > > > > useful
> > > > > >> in other cases.
> > > > > >>
> > > > > >> We can use `subrequest` to implement the first version, this way
> > > seems
> > > > > >> simpler.
> > > > > >>
> > > > > >>
> > > > > >> On Sun, Mar 21, 2021 at 2:34 PM Zexuan Luo <
> > spacewan...@apache.org>
> > > > > >> wrote:
> > > > > >>
> > > > > >> > Another solution is writing a Lua GRPC library with
> > > > > >> > https://github.com/tokers/lua-resty-http2
> > > > > >> > https://github.com/starwing/lua-protobuf
> > > > > >> >
> > > > > >> > Zexuan Luo <spacewan...@apache.org> 于2021年3月21日周日 下午2:18写道：
> > > > > >> > >
> > > > > >> > > I am not familiar with
> > > > > https://github.com/apache/skywalking-satellite
> > > > > >> .
> > > > > >> > > Can we use this project to convert plain HTTP to GRPC?
> > > > > >> > >
> > > > > >> > > Sheng Wu <wu.sheng.841...@gmail.com> 于2021年3月21日周日
> 下午1:59写道：
> > > > > >> > > >
> > > > > >> > > > Oh, you mean that.
> > > > > >> > > > SkyWalking natively provides HTTP APIs for nearly all
> > > protocols,
> > > > > >> even
> > > > > >> > if
> > > > > >> > > > not, we are happy to add. There is no block between APISIX
> > and
> > > > > >> > SkyWalking.
> > > > > >> > > > The point to discuss the gRPC service here is about the
> > > > > performance,
> > > > > >> > > > including proto's no field name in transmission, enum as
> > int,
> > > > > small
> > > > > >> > int as
> > > > > >> > > > small, and especially streaming.
> > > > > >> > > > If you want to implement ALS, then this is super important
> > to
> > > > > >> provide
> > > > > >> > > > better performance.
> > > > > >> > > >
> > > > > >> > > > Sheng Wu 吴晟
> > > > > >> > > > Twitter, wusheng1108
> > > > > >> > > >
> > > > > >> > > >
> > > > > >> > > > Chao Zhang <zchao1...@gmail.com> 于2021年3月21日周日 下午1:55写道：
> > > > > >> > > >
> > > > > >> > > > > It’s not so easy to use gRPC in APISIX as the lack of
> > > > ecosystem,
> > > > > >> but
> > > > > >> > with
> > > > > >> > > > > the help of gRPC gateway (
> > > > > >> > https://github.com/grpc-ecosystem/grpc-gateway).
> > > > > >> > > > > We can use the restful APIs to talk with gRPC endpoints.
> > > Just
> > > > > like
> > > > > >> > the way
> > > > > >> > > > > APISIX communicates with ETCD V3 APIs.
> > > > > >> > > > >
> > > > > >> > > > > Chao Zhang
> > > > > >> > > > > https://github.com/tokers
> > > > > >> > > > >
> > > > > >> > > > > On March 21, 2021 at 1:53:41 PM, Sheng Wu (
> > > > > >> wu.sheng.841...@gmail.com
> > > > > >> > )
> > > > > >> > > > > wrote:
> > > > > >> > > > >
> > > > > >> > > > > Chao Zhang <zchao1...@gmail.com> 于2021年3月21日周日
> 下午1:46写道：
> > > > > >> > > > >
> > > > > >> > > > > > Does the gRPC service enables grpc gateway? If so,
> it’s
> > > easy
> > > > > for
> > > > > >> > APISIX
> > > > > >> > > > > to
> > > > > >> > > > > > integrate it?
> > > > > >> > > > > >
> > > > > >> > > > >
> > > > > >> > > > > What is the relationship with grpc gateway?
> > > > > >> > > > >
> > > > > >> > > > >
> > > > > >> > > > >
> > > > > >> > > > > >
> > > > > >> > > > > > Zexuan Luo <spacewan...@apache.org>于2021年3月21日
> > 周日13:35写道：
> > > > > >> > > > > >
> > > > > >> > > > > > > Another solution is wrapping the C++ implementation
> of
> > > the
> > > > > >> GPRC
> > > > > >> > > > > > > client. But it requires you to prebuild the proto
> file
> > > > into
> > > > > >> *.c
> > > > > >> > source
> > > > > >> > > > > > > and can only be distributed as part of
> > apisix-openresty.
> > > > > >> > > > > > >
> > > > > >> > > > > > > Zexuan Luo <spacewan...@apache.org> 于2021年3月21日周日
> > > > 下午1:25写道：
> > > > > >> > > > > > > >
> > > > > >> > > > > > > > AFAIK, there is not library support making GPRC
> > > > directly.
> > > > > >> > > > > > > > One solution is subrequest + http2grpc, but this
> way
> > > > > doesn't
> > > > > >> > support
> > > > > >> > > > > > > > stream mode. So it is not a good solution.
> > > > > >> > > > > > > >
> > > > > >> > > > > > > > YuanSheng Wang <membp...@apache.org>
> 于2021年3月20日周六
> > > > > >> 下午10:53写道：
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > > LGTM
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > > > The biggest advantage of ALS, is replacing the
> > > > > physical
> > > > > >> > local
> > > > > >> > > > > > access
> > > > > >> > > > > > > log
> > > > > >> > > > > > > > > file.
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > > The APISIX has a plugin `http-logger`, I think
> > > APISIX
> > > > is
> > > > > >> > easy to
> > > > > >> > > > > > report
> > > > > >> > > > > > > > > those data to SkyWalking.
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > > And I think we can try to use `gRPC` to report
> > data.
> > > > > What
> > > > > >> do
> > > > > >> > > > > > > > > you think? @ming @spacewander
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > > On Fri, Mar 19, 2021 at 11:43 PM Ming Wen <
> > > > > >> > wenm...@apache.org>
> > > > > >> > > > > > wrote:
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > > > Cool, I think we can try ALS
> > > > > >> > > > > > > > > >
> > > > > >> > > > > > > > > > Sheng Wu <wu.sheng.841...@gmail.com
> >于2021年3月19日
> > > > > >> > 周五下午11:21写道：
> > > > > >> > > > > > > > > >
> > > > > >> > > > > > > > > > > We have run the Envoy ALS performance test a
> > > long
> > > > > time
> > > > > >> > ago(2
> > > > > >> > > > > > > years, I
> > > > > >> > > > > > > > > > > think). Sadly, no public blog.
> > > > > >> > > > > > > > > > > It could easily support 10k-20k/s traffic
> > with a
> > > > > very
> > > > > >> > low load
> > > > > >> > > > > to
> > > > > >> > > > > > > the
> > > > > >> > > > > > > > > > > proxy. But, it is based on gRPC, so
> basically,
> > > it
> > > > > >> > doesn't send
> > > > > >> > > > > > many
> > > > > >> > > > > > > > > > things
> > > > > >> > > > > > > > > > > out, mostly only flags and latency value.
> > > > > >> > > > > > > > > > > APISIX is using HTTP1.1 mostly, so, this
> could
> > > be
> > > > > >> > different.
> > > > > >> > > > > > > > > > > Also, at the same time, besides the ALS, we
> > > have a
> > > > > >> > chance to
> > > > > >> > > > > > > provide a
> > > > > >> > > > > > > > > > push
> > > > > >> > > > > > > > > > > mode LUA SDK in skywalking meter format. It
> > > could
> > > > > >> provide
> > > > > >> > > > > > URI-level
> > > > > >> > > > > > > > > > metrics
> > > > > >> > > > > > > > > > > at reasonable resource cost, especially it
> > works
> > > > > with
> > > > > >> > > > > SkyWalking
> > > > > >> > > > > > > > > > satellite,
> > > > > >> > > > > > > > > > > which could leverage local disk to cache the
> > > data.
> > > > > >> > > > > > > > > > > The biggest advantage of ALS, is replacing
> the
> > > > > >> physical
> > > > > >> > local
> > > > > >> > > > > > > access log
> > > > > >> > > > > > > > > > > file. So you have metrics,
> > topology(dependency),
> > > > and
> > > > > >> > logs in
> > > > > >> > > > > one
> > > > > >> > > > > > > package
> > > > > >> > > > > > > > > > of
> > > > > >> > > > > > > > > > > data.
> > > > > >> > > > > > > > > > >
> > > > > >> > > > > > > > > > > If you want to discuss most ALS in APISIX, I
> > am
> > > > glad
> > > > > >> to
> > > > > >> > help
> > > > > >> > > > > with
> > > > > >> > > > > > > the
> > > > > >> > > > > > > > > > > design and integration solution.
> > > > > >> > > > > > > > > > >
> > > > > >> > > > > > > > > > > Sheng Wu 吴晟
> > > > > >> > > > > > > > > > > Twitter, wusheng1108
> > > > > >> > > > > > > > > > >
> > > > > >> > > > > > > > > > >
> > > > > >> > > > > > > > > > > Ming Wen <wenm...@apache.org> 于2021年3月19日周五
> > > > > >> 下午11:08写道：
> > > > > >> > > > > > > > > > >
> > > > > >> > > > > > > > > > > > yes, Prometheus is not designed for
> > URI-level
> > > > > >> metrics.
> > > > > >> > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > Does envoy ALS will send too much data and
> > > cause
> > > > > >> > performance
> > > > > >> > > > > > > problems?
> > > > > >> > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > And I know skywalking will supoort envoy
> > WASM,
> > > > is
> > > > > >> there
> > > > > >> > > > > > anything
> > > > > >> > > > > > > apisix
> > > > > >> > > > > > > > > > > can
> > > > > >> > > > > > > > > > > > learn from this？
> > > > > >> > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > Sheng Wu <wu.sheng.841...@gmail.com
> > > >于2021年3月19日
> > > > > >> > 周五下午8:11写道：
> > > > > >> > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > Hi APISIX & SkyWalking community
> > > > > >> > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > I was discussing with Wei Jin about
> using
> > > > > >> Prometheus
> > > > > >> > SDK
> > > > > >> > > > > to
> > > > > >> > > > > > > monitor
> > > > > >> > > > > > > > > > > > > URI-level metrics of APISIX/OpenResty.
> > > > > >> > > > > > > > > > > > > Wei received feedback that way of using
> > > > > Prometheus
> > > > > >> > SDK
> > > > > >> > > > > > causing
> > > > > >> > > > > > > > > > > > > a significant performance impact. From
> the
> > > > > >> > observability
> > > > > >> > > > > > > perspective,
> > > > > >> > > > > > > > > > > > Prom
> > > > > >> > > > > > > > > > > > > is not good at gathering a large scale
> of
> > > > > >> entities'
> > > > > >> > > > > metrics
> > > > > >> > > > > > > due to
> > > > > >> > > > > > > > > > its
> > > > > >> > > > > > > > > > > > > local cache and pull mode.
> > > > > >> > > > > > > > > > > > > Instead, we should consider providing a
> > push
> > > > way
> > > > > >> like
> > > > > >> > > > > Envoy
> > > > > >> > > > > > > ALS[1]
> > > > > >> > > > > > > > > > and
> > > > > >> > > > > > > > > > > > > Metrics Service[2] did(it does through
> > gRPC,
> > > > and
> > > > > >> > > > > SkyWalking
> > > > > >> > > > > > > > > > recommended
> > > > > >> > > > > > > > > > > > in
> > > > > >> > > > > > > > > > > > > the prod environment.
> > > > > >> > > > > > > > > > > > > If this is accepted and agreed by the
> > APISIX
> > > > > >> > community, we
> > > > > >> > > > > > > could look
> > > > > >> > > > > > > > > > > for
> > > > > >> > > > > > > > > > > > > implementation in the
> > > skywalking-nginx-lua[3]
> > > > > >> repo to
> > > > > >> > > > > collect
> > > > > >> > > > > > > > > > > > metrics(meter
> > > > > >> > > > > > > > > > > > > in SkyWalking) and use HTTP(in
> SkyWalking
> > > > native
> > > > > >> > meter
> > > > > >> > > > > > format)
> > > > > >> > > > > > > to
> > > > > >> > > > > > > > > > > report
> > > > > >> > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > This mail has been sent to
> > > > > d...@apisix.apache.org
> > > > > >> and
> > > > > >> > > > > > > > > > > > > dev@skywalking.apache.org. Please reply
> > all
> > > > > when
> > > > > >> you
> > > > > >> > > > > discuss
> > > > > >> > > > > > > in the
> > > > > >> > > > > > > > > > > > > thread.
> > > > > >> > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > [1]
> > > > > >> > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > >
> > > > > >> > > > > > > > > > >
> > > > > >> > > > > > > > > >
> > > > > >> > > > > > >
> > > > > >> > > > > >
> > > > > >> > > > >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
> https://www.envoyproxy.io/docs/envoy/latest/api-v2/config/accesslog/v2/als.proto
> > > > > >> > > > > > > > > > > > > [2]
> > > > > >> > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > >
> > > > > >> > > > > > > > > > >
> > > > > >> > > > > > > > > >
> > > > > >> > > > > > >
> > > > > >> > > > > >
> > > > > >> > > > >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
> https://www.envoyproxy.io/docs/envoy/latest/api-v2/config/metrics/v2/metrics_service.proto
> > > > > >> > > > > > > > > > > > > [3]
> > > > > >> https://github.com/apache/skywalking-nginx-lua
> > > > > >> > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > Sheng Wu 吴晟
> > > > > >> > > > > > > > > > > > > Twitter, wusheng1108
> > > > > >> > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > --
> > > > > >> > > > > > > > > > > > Thanks,
> > > > > >> > > > > > > > > > > > Ming Wen, Apache APISIX PMC Chair
> > > > > >> > > > > > > > > > > > Twitter: _WenMing
> > > > > >> > > > > > > > > > > >
> > > > > >> > > > > > > > > > >
> > > > > >> > > > > > > > > > --
> > > > > >> > > > > > > > > > Thanks,
> > > > > >> > > > > > > > > > Ming Wen, Apache APISIX PMC Chair
> > > > > >> > > > > > > > > > Twitter: _WenMing
> > > > > >> > > > > > > > > >
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > > --
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > > *MembPhis*
> > > > > >> > > > > > > > > My GitHub: https://github.com/membphis
> > > > > >> > > > > > > > > Apache APISIX: https://github.com/apache/apisix
> > > > > >> > > > > > >
> > > > > >> > > > > > --
> > > > > >> > > > > > Best regards
> > > > > >> > > > > > Alex Zhang
> > > > > >> > > > > >
> > > > > >> > > > > > https://github.com/tokers
> > > > > >> > > > > > https://github.com/upyun/upyun-resty
> > > > > >> > > > > >
> > > > > >> > > > >
> > > > > >> > > > >
> > > > > >> >
> > > > > >>
> > > > > >>
> > > > > >> --
> > > > > >>
> > > > > >> *MembPhis*
> > > > > >> My GitHub: https://github.com/membphis
> > > > > >> Apache APISIX: https://github.com/apache/apisix
> > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
>
