(skywalking) branch swip-1 updated (50d8de4b9b -> dce1c0458b)
This is an automated email from the ASF dual-hosted git repository. wusheng pushed a change to branch swip-1 in repository https://gitbox.apache.org/repos/asf/skywalking.git from 50d8de4b9b Fix add dce1c0458b Fix readme. No new revisions were added by this update. Summary of changes: docs/README.md | 38 ++ 1 file changed, 38 insertions(+) create mode 100644 docs/README.md
(skywalking) branch swip-1 updated (2edbdb6b08 -> 50d8de4b9b)
This is an automated email from the ASF dual-hosted git repository. wusheng pushed a change to branch swip-1 in repository https://gitbox.apache.org/repos/asf/skywalking.git from 2edbdb6b08 SWIP-1 Create and detect Service Hierarchy Relationship add 50d8de4b9b Fix No new revisions were added by this update. Summary of changes: docs/README.md | 62 -- docs/en/swip/readme.md | 11 ++--- 2 files changed, 8 insertions(+), 65 deletions(-) delete mode 100644 docs/README.md
[PR] SWIP-1 Create and detect Service Hierarchy Relationship [skywalking]
wu-sheng opened a new pull request, #11670: URL: https://github.com/apache/skywalking/pull/11670 - [ ] If this pull request closes/resolves/fixes an existing issue, replace the issue number. Closes #. - [x] Update the [`CHANGES` log](https://github.com/apache/skywalking/blob/master/docs/en/changes/changes.md). @wankai123 and I are working on this. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: notifications-unsubscr...@skywalking.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
(skywalking) 01/01: SWIP-1 Create and detect Service Hierarchy Relationship
This is an automated email from the ASF dual-hosted git repository. wusheng pushed a commit to branch swip-1 in repository https://gitbox.apache.org/repos/asf/skywalking.git commit 2edbdb6b089c8590ee5e932a74700d2438e42dc3 Author: Wu Sheng AuthorDate: Mon Dec 18 11:44:32 2023 +0800 SWIP-1 Create and detect Service Hierarchy Relationship --- docs/README.md | 74 ++ docs/en/changes/changes.md | 1 + docs/en/swip/SWIP-1.md | 73 + 3 files changed, 123 insertions(+), 25 deletions(-) diff --git a/docs/README.md b/docs/README.md index 20b6c88213..2acae7d5f9 100644 --- a/docs/README.md +++ b/docs/README.md @@ -1,38 +1,62 @@ -# Welcome +# SWIP - SkyWalking Improvement Proposal -**This is the official documentation of SkyWalking 9. Welcome to the SkyWalking community!** +SWIP - SkyWalking Improvement Proposal, is an official document to propose a new feature and/or feature improvement, +which +are relative to end users and developers. -Here you can learn all you need to know about **SkyWalking**’s architecture, understand how to deploy and use -SkyWalking, and contribute to the project based on SkyWalking's contributing guidelines. +SkyWalking has been very stable since v9.x. We are getting over the rapid changing stage. The core concepts, protocols for +reporting telemetry and query, 3rd party integration, and the streaming process kernel are very stable. From now(2024) on, +SkyWalking community would focus more on improvement and controllable improvement. All major changes should be evaluated +more seriously, and try as good as possible to avoid incompatible breaking changes. -- **Concepts and Designs**. You'll find the core logic behind SkyWalking. You may start from here if you want to - understand what is going on under our cool features and visualization. +## What is considered a major change? -- **Setup**. A guide to installing SkyWalking for different use cases. It is an observability platform that supports - multiple observability modes. +The catalogs of a major change are listed as follows -- **Contributing Guides**. If you are a PMC member, a committer, or a new contributor, learn how to start contributing - with these guides! +- New Feature. A feature doesn't exist for the latest version. +- Any change of the network Interfaces, especially + for [Query Protocol](https://github.com/apache/skywalking-query-protocol), + [Data Collect Protocols](https://github.com/apache/skywalking-data-collect-protocol), + Dynamic Configuration APIs, Exporting APIs, AI pipeline APIs. +- Any change of storage structure. -- **Protocols**. The protocols show how agents/probes and the backend communicate with one another. Anyone interested in - uplink telemetry data should definitely read this. +**Q: Is Agent side feature or change considered a SWIP?** -- **FAQs**. A manifest of known issues with setup and secondary developments processes. Should you encounter any - problems, check here first. +A: Right now, SWIP targets on OAP and UI side changes. All agent side changes are pending on the reviews from the +committers of those agents. -You might also find these links interesting: +## SWIP Template -- The latest and old releases are all available - at [Apache SkyWalking release page](https://skywalking.apache.org/downloads/). The change logs can be - found [here](https://github.com/apache/skywalking/tree/master/changes). +The purpose of this template should not be considered a hard requirement. The major purpose of SWIP is helping the PMC +and community member to understand the proposal better. -- [SkyWalking WIKI](https://cwiki.apache.org/confluence/display/SKYWALKING/Home) hosts the context of some changes and - events. +* Title: SWIP-1234 -- You can find the conference schedules, video recordings, and articles about SkyWalking in - the [community resource catalog](https://github.com/OpenSkywalking/Community). +- Motivation - the description of new feature or improvement. +- Architecture Graph - describing the relationship between your new proposal part and existing components. +- Proposed Changes - state your proposal in detail. +- Imported Dependencies libs and their licenses. +- Compatibility - Whether breaking configuration, storage structure, protocols. +- General usage docs - This doesn't have to be final version, but help the reviews to understand how to use this new + feature. -We're always looking for help to improve our documentation and codes, so please don’t hesitate -to [file an issue](https://github.com/apache/skywalking/issues/new) if you see any problems. Or better yet, directly -contribute by submitting a pull request to help us get better! +## SWIP Process +Here is the process for starting a SWIP. + +1. Start a SWIP discussion at [GitHub Discussion Page](https://github.com/apache/skywalking/discussions/categories/swip) +
(skywalking) branch swip-1 created (now 2edbdb6b08)
This is an automated email from the ASF dual-hosted git repository. wusheng pushed a change to branch swip-1 in repository https://gitbox.apache.org/repos/asf/skywalking.git at 2edbdb6b08 SWIP-1 Create and detect Service Hierarchy Relationship This branch includes the following new commits: new 2edbdb6b08 SWIP-1 Create and detect Service Hierarchy Relationship The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference.
Re: [PR] Add Pulsar configuration to determine the collection of message contents [skywalking-java]
CommissarXia commented on PR #657: URL: https://github.com/apache/skywalking-java/pull/657#issuecomment-1859423670 I was sick last week. I have time to deal with it this week. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: notifications-unsubscr...@skywalking.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
(skywalking) branch swip deleted (was 1465223420)
This is an automated email from the ASF dual-hosted git repository. wusheng pushed a change to branch swip in repository https://gitbox.apache.org/repos/asf/skywalking.git was 1465223420 Fix a word. The revisions that were on this branch are still contained in other references; therefore, this change does not discard any commits from the repository.
(skywalking) branch master updated: Begin the process of `SWIP - SkyWalking Improvement Proposal`. (#11668)
This is an automated email from the ASF dual-hosted git repository. wusheng pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/skywalking.git The following commit(s) were added to refs/heads/master by this push: new 6304cd6223 Begin the process of `SWIP - SkyWalking Improvement Proposal`. (#11668) 6304cd6223 is described below commit 6304cd6223745bcbb3f28c039d0c324ba194a091 Author: 吴晟 Wu Sheng AuthorDate: Mon Dec 18 09:37:52 2023 +0800 Begin the process of `SWIP - SkyWalking Improvement Proposal`. (#11668) --- docs/en/changes/changes.md | 1 + docs/en/swip/readme.md | 57 ++ docs/menu.yml | 2 ++ 3 files changed, 60 insertions(+) diff --git a/docs/en/changes/changes.md b/docs/en/changes/changes.md index 0a2b100a5f..d8410c1ee5 100644 --- a/docs/en/changes/changes.md +++ b/docs/en/changes/changes.md @@ -21,5 +21,6 @@ * Update the release notification mail a little. * Polish docs structure. Move customization docs separately from the introduction docs. * Add webhook/gRPC hooks settings example for `backend-alarm.md`. +* Begin the process of `SWIP - SkyWalking Improvement Proposal`. All issues and pull requests are [here](https://github.com/apache/skywalking/milestone/202?closed=1) diff --git a/docs/en/swip/readme.md b/docs/en/swip/readme.md new file mode 100644 index 00..dd18a75a5e --- /dev/null +++ b/docs/en/swip/readme.md @@ -0,0 +1,57 @@ +# SWIP - SkyWalking Improvement Proposal + +SWIP - SkyWalking Improvement Proposal, is an official document to propose a new feature and/or feature improvement, +which +are relative to end users and developers. + +SkyWalking has been very stable since v9.x. We are getting over the rapid changing stage. The core concepts, protocols for +reporting telemetry and query, 3rd party integration, and the streaming process kernel are very stable. From now(2024) on, +SkyWalking community would focus more on improvement and controllable improvement. All major changes should be evaluated +more seriously, and try as good as possible to avoid incompatible breaking changes. + +## What is considered a major change? + +The catalogs of a major change are listed as follows + +- New Feature. A feature doesn't exist for the latest version. +- Any change of the network Interfaces, especially + for [Query Protocol](https://github.com/apache/skywalking-query-protocol), + [Data Collect Protocols](https://github.com/apache/skywalking-data-collect-protocol), + Dynamic Configuration APIs, Exporting APIs, AI pipeline APIs. +- Any change of storage structure. + +**Q: Is Agent side feature or change considered a SWIP?** + +A: Right now, SWIP targets on OAP and UI side changes. All agent side changes are pending on the reviews from the +committers of those agents. + +## SWIP Template + +The purpose of this template should not be considered a hard requirement. The major purpose of SWIP is helping the PMC +and community member to understand the proposal better. + +* Title: SWIP-1234 + +- Motivation - the description of new feature or improvement. +- Architecture Graph - describing the relationship between your new proposal part and existing components. +- Imported Dependencies libs and their licenses. +- Compatibility - Whether breaking configuration, storage structure, protocols. +- General usage docs - This doesn't have to be final version, but help the reviews to understand how to use this new + feature. + +## SWIP Process + +Here is the process for starting a SWIP. + +1. Start a discussion at [GitHub Discussion Page](https://github.com/apache/skywalking/discussions) with + title `[DISCUSS] `. +2. Fill in the sections as described above in `SWIP Template`. +3. At least one SkyWalking committer commented on the discussion to show interests to adopt it. +4. This committer could update this page to grant a **SWIP ID**, and update the title to `[SWIP-ID NO.] [DISCUSS] `. +5. All further discussion could happen on the discussion page. +6. Once the consensus is made by enough committer supporters, and/or through a mail list vote, this SWIP should be + added [here](./) as `SWIP-ID NO.md` and listed in the below as `Known SWIPs`. + +## Known SWIPs + +Next SWIP Number: 2 diff --git a/docs/menu.yml b/docs/menu.yml index 6a6432919a..d88ad1cdbf 100644 --- a/docs/menu.yml +++ b/docs/menu.yml @@ -370,6 +370,8 @@ catalog: path: "/en/guides/how-to-bump-up-zipkin" - name: "I18n" path: "/en/guides/i18n" + - name: "SWIP" +path: "/en/swip/readme" - name: "Changelog" catalog: - name: "Current Version"
Re: [PR] Begin the process of `SWIP - SkyWalking Improvement Proposal`. [skywalking]
wu-sheng merged PR #11668: URL: https://github.com/apache/skywalking/pull/11668 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: notifications-unsubscr...@skywalking.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
(skywalking-banyandb) 01/01: Pass integration test
This is an automated email from the ASF dual-hosted git repository.
hanahmily pushed a commit to branch storage-column
in repository https://gitbox.apache.org/repos/asf/skywalking-banyandb.git
commit 00d456e9595ce41d9edc36385126933cf05d66de
Author: Gao Hongtao
AuthorDate: Mon Dec 18 05:45:33 2023 +0800
Pass integration test
Signed-off-by: Gao Hongtao
---
banyand/measure/block.go | 19 +++---
banyand/measure/query.go | 4 +++
banyand/measure/tstable_test.go| 30 +++---
pkg/index/inverted/inverted_series.go | 11 ++--
pkg/index/inverted/inverted_series_test.go | 1 +
pkg/query/logical/index_filter.go | 2 +-
.../measure/measure_plan_indexscan_local.go| 20 ---
test/cases/measure/measure.go | 4 +--
8 files changed, 59 insertions(+), 32 deletions(-)
diff --git a/banyand/measure/block.go b/banyand/measure/block.go
index fd0661e1..1730d022 100644
--- a/banyand/measure/block.go
+++ b/banyand/measure/block.go
@@ -213,9 +213,10 @@ func (b *block) unmarshalTagFamily(decoder
*encoding.BytesBlockDecoder, tfIndex
cc := b.tagFamilies[tfIndex].resizeColumns(len(tagProjection))
for j := range tagProjection {
- for i := range cc {
+ for i := range cfm.columnMetadata {
if tagProjection[j] == cfm.columnMetadata[i].name {
- cc[i].mustReadValues(decoder, valueReader,
cfm.columnMetadata[i], uint64(b.Len()))
+ cc[j].mustReadValues(decoder, valueReader,
cfm.columnMetadata[i], uint64(b.Len()))
+ break
}
}
}
@@ -372,7 +373,7 @@ func (bc *blockCursor) init(p *part, bm blockMetadata,
queryOpts QueryOptions) {
func (bc *blockCursor) copyAllTo(r *pbv1.Result) {
r.SID = bc.bm.seriesID
- r.Timestamps = bc.timestamps
+ r.Timestamps = append(r.Timestamps, bc.timestamps...)
for _, cf := range bc.tagFamilies {
tf := pbv1.TagFamily{
Name: cf.Name,
@@ -479,6 +480,12 @@ func (bc *blockCursor) loadData(tmpBlock *block) bool {
Name: cf.Columns[i].Name,
ValueType: cf.Columns[i].ValueType,
}
+ if len(cf.Columns[i].Values) == 0 {
+ continue
+ }
+ if len(cf.Columns[i].Values) !=
len(tmpBlock.timestamps) {
+ logger.Panicf("unexpected number of values for
tags %q: got %d; want %d", cf.Columns[i].Name, len(cf.Columns[i].Values),
len(tmpBlock.timestamps))
+ }
column.Values = append(column.Values,
cf.Columns[i].Values[start:end]...)
tf.Columns = append(tf.Columns, column)
}
@@ -486,6 +493,12 @@ func (bc *blockCursor) loadData(tmpBlock *block) bool {
}
bc.fields.Name = tmpBlock.field.Name
for i := range tmpBlock.field.Columns {
+ if len(tmpBlock.field.Columns[i].Values) == 0 {
+ continue
+ }
+ if len(tmpBlock.field.Columns[i].Values) !=
len(tmpBlock.timestamps) {
+ logger.Panicf("unexpected number of values for fields
%q: got %d; want %d", tmpBlock.field.Columns[i].Name,
len(tmpBlock.field.Columns[i].Values), len(tmpBlock.timestamps))
+ }
column := Column{
Name: tmpBlock.field.Columns[i].Name,
ValueType: tmpBlock.field.Columns[i].ValueType,
diff --git a/banyand/measure/query.go b/banyand/measure/query.go
index 15a9193f..6a41ca45 100644
--- a/banyand/measure/query.go
+++ b/banyand/measure/query.go
@@ -70,6 +70,9 @@ func (s *measure) Query(ctx context.Context, mqo
pbv1.MeasureQueryOptions) (pbv1
if mqo.TimeRange == nil || mqo.Entity == nil {
return nil, errors.New("invalid query options: timeRange and
series are required")
}
+ if len(mqo.TagProjection) == 0 && len(mqo.FieldProjection) == 0 {
+ return nil, errors.New("invalid query options: tagProjection or
fieldProjection is required")
+ }
tsdb := s.databaseSupplier.SupplyTSDB().(storage.TSDB[*tsTable])
tabWrappers := tsdb.SelectTSTables(*mqo.TimeRange)
sl, err := tsdb.IndexDB().Search(ctx, {Subject: mqo.Name,
EntityValues: mqo.Entity}, mqo.Filter, mqo.Order)
@@ -123,6 +126,7 @@ func (s *measure) Query(ctx context.Context, mqo
pbv1.MeasureQueryOptions) (pbv1
if mqo.Order.Sort == modelv1.Sort_SORT_ASC || mqo.Order.Sort ==
modelv1.Sort_SORT_UNSPECIFIED {
result.ascTS = true
}
+
(skywalking-banyandb) branch storage-column updated (7f157e80 -> 00d456e9)
This is an automated email from the ASF dual-hosted git repository. hanahmily pushed a change to branch storage-column in repository https://gitbox.apache.org/repos/asf/skywalking-banyandb.git discard 7f157e80 Pass integration test new 00d456e9 Pass integration test This update added new revisions after undoing existing revisions. That is to say, some revisions that were in the old version of the branch are not in the new version. This situation occurs when a user --force pushes a change and generates a repository containing something like this: * -- * -- B -- O -- O -- O (7f157e80) \ N -- N -- N refs/heads/storage-column (00d456e9) You should already have received notification emails for all of the O revisions, and so the following emails describe only the N revisions from the common base, B. Any revisions marked "omit" are not gone; other references still refer to them. Any revisions marked "discard" are gone forever. The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: banyand/measure/tstable_test.go | 30 +++--- 1 file changed, 15 insertions(+), 15 deletions(-)
(skywalking-banyandb) 02/07: Test part
This is an automated email from the ASF dual-hosted git repository.
hanahmily pushed a commit to branch storage-column
in repository https://gitbox.apache.org/repos/asf/skywalking-banyandb.git
commit 3fe134f3170a679ae3d8c408f599ea5a047aafd4
Author: Gao Hongtao
AuthorDate: Thu Dec 14 05:53:25 2023 +0800
Test part
Signed-off-by: Gao Hongtao
---
banyand/measure/block.go | 9 +-
banyand/measure/block_metadata.go | 2 +-
banyand/measure/measure.go| 2 +
banyand/measure/measure_query.go | 2 +-
banyand/measure/part.go | 9 +-
banyand/measure/part_iter.go | 166 +++
banyand/measure/part_iter_test.go | 142 +++
banyand/measure/part_test.go | 182 ++
banyand/measure/tstable.go| 201 ++
pkg/bytes/buffer.go | 8 +-
10 files changed, 434 insertions(+), 289 deletions(-)
diff --git a/banyand/measure/block.go b/banyand/measure/block.go
index 2c0407ce..8fd64781 100644
--- a/banyand/measure/block.go
+++ b/banyand/measure/block.go
@@ -322,14 +322,14 @@ type blockCursor struct {
columnValuesDecoder encoding.BytesBlockDecoder
p *part
- bm *blockMetadata
+ bm blockMetadata
opts*QueryOptions
}
func (bc *blockCursor) reset() {
bc.idx = 0
bc.p = nil
- bc.bm = nil
+ bc.bm = blockMetadata{}
bc.opts = nil
bc.timestamps = bc.timestamps[:0]
@@ -342,7 +342,7 @@ func (bc *blockCursor) reset() {
bc.fields.reset()
}
-func (bc *blockCursor) init(p *part, bm *blockMetadata, opts *QueryOptions) {
+func (bc *blockCursor) init(p *part, bm blockMetadata, opts *QueryOptions) {
bc.reset()
bc.p = p
bc.bm = bm
@@ -350,9 +350,8 @@ func (bc *blockCursor) init(p *part, bm *blockMetadata,
opts *QueryOptions) {
}
func (bc *blockCursor) loadData(tmpBlock *block) bool {
- bc.reset()
tmpBlock.reset()
- tmpBlock.mustReadFrom(, bc.p, bc.bm)
+ tmpBlock.mustReadFrom(, bc.p, )
start, end, ok := findRange(tmpBlock.timestamps, bc.opts.minTimestamp,
bc.opts.maxTimestamp)
if !ok {
diff --git a/banyand/measure/block_metadata.go
b/banyand/measure/block_metadata.go
index 857f9125..b77f0b03 100644
--- a/banyand/measure/block_metadata.go
+++ b/banyand/measure/block_metadata.go
@@ -175,7 +175,7 @@ func (bh *blockMetadata) unmarshal(src []byte) ([]byte,
error) {
return src, nil
}
-func (bh *blockMetadata) less(other *blockMetadata) bool {
+func (bh blockMetadata) less(other blockMetadata) bool {
if bh.seriesID == other.seriesID {
return bh.timestamps.min < other.timestamps.min
}
diff --git a/banyand/measure/measure.go b/banyand/measure/measure.go
index 739698b8..bfe7d7b2 100644
--- a/banyand/measure/measure.go
+++ b/banyand/measure/measure.go
@@ -44,6 +44,8 @@ const (
maxTagFamiliesMetadataSize = 8 * 1024 * 1024
maxUncompressedBlockSize= 2 * 1024 * 1024
maxUncompressedPrimaryBlockSize = 128 * 1024
+
+ maxBlockLength = 8 * 1024
)
type measure struct {
diff --git a/banyand/measure/measure_query.go b/banyand/measure/measure_query.go
index 2175459c..4b2960c9 100644
--- a/banyand/measure/measure_query.go
+++ b/banyand/measure/measure_query.go
@@ -243,7 +243,7 @@ func (s *measure) Query(ctx context.Context, mqo
pbv1.MeasureQueryOptions) (pbv1
for tstIter.NextBlock() {
bc := generateBlockCursor()
p := tstIter.piHeap[0]
- bc.init(p.p, p.bm, qo)
+ bc.init(p.p, p.curBlock, qo)
result.data = append(result.data, bc)
}
if mqo.Order != nil {
diff --git a/banyand/measure/part.go b/banyand/measure/part.go
index dd211449..d24b70d4 100644
--- a/banyand/measure/part.go
+++ b/banyand/measure/part.go
@@ -129,7 +129,8 @@ func (mp *memPart) mustInitFromDataPoints(dps *dataPoints) {
sidPrev = sid
}
- if uncompressedBlockSizeBytes >= maxUncompressedBlockSize ||
sid != sidPrev {
+ if uncompressedBlockSizeBytes >= maxUncompressedBlockSize ||
+ (i-indexPrev) > maxBlockLength || sid != sidPrev {
bsw.MustWriteDataPoints(sidPrev,
dps.timestamps[indexPrev:i], dps.tagFamilies[indexPrev:i],
dps.fields[indexPrev:i])
sidPrev = sid
indexPrev = i
@@ -157,7 +158,7 @@ func uncompressedDataPointSizeBytes(index int, dps
*dataPoints) uint64 {
return n
}
-func getMemPart() *memPart {
+func generateMemPart() *memPart {
v := memPartPool.Get()
if v == nil {
return {}
@@ -165,7 +166,7 @@ func getMemPart() *memPart {
return v.(*memPart)
}
-func putMemPart(mp *memPart)
(skywalking-banyandb) branch storage-column updated (0862fadd -> 7f157e80)
This is an automated email from the ASF dual-hosted git repository.
hanahmily pushed a change to branch storage-column
in repository https://gitbox.apache.org/repos/asf/skywalking-banyandb.git
from 0862fadd Add column test
new 835b36fe Finish test block
new 3fe134f3 Test part
new d91f5f26 Test tsTable
new 7cb0650c Test query result
new ad3f5c8e Pass the smoke test of the integration test.
new c651c164 Add projection
new 7f157e80 Pass integration test
The 7 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
banyand/internal/storage/index.go | 2 +-
banyand/internal/storage/tsdb.go | 39 +-
banyand/liaison/grpc/measure.go| 9 +-
banyand/measure/block.go | 260 --
banyand/measure/block_metadata.go | 6 +-
banyand/measure/block_test.go | 265 --
banyand/measure/block_writer.go| 35 +-
banyand/measure/column_metadata.go | 2 +-
banyand/measure/datapoints.go | 2 +-
banyand/measure/measure.go | 2 +
banyand/measure/measure_query.go | 572 -
banyand/measure/part.go| 9 +-
banyand/measure/part_iter.go | 166 +++---
banyand/measure/part_iter_test.go | 142 +
banyand/measure/part_metadata.go | 2 +
banyand/measure/part_test.go | 180 +++
banyand/measure/query.go | 430
banyand/measure/query_test.go | 443
banyand/measure/{measure_topn.go => topn.go} | 13 +-
banyand/measure/tstable.go | 223 +---
banyand/measure/tstable_test.go| 318
banyand/measure/{measure_write.go => write.go} | 26 +-
banyand/queue/local.go | 21 +-
pkg/bytes/buffer.go| 8 +-
pkg/index/inverted/inverted_series.go | 11 +-
pkg/index/inverted/inverted_series_test.go | 1 +
pkg/pb/v1/metadata.go | 4 +-
pkg/pb/v1/value.go | 5 +
pkg/query/logical/index_filter.go | 2 +-
.../measure/measure_plan_indexscan_local.go| 30 +-
pkg/query/logical/measure/topn_plan_localscan.go | 8 +-
test/cases/measure/data/want/all_only_fields.yaml | 66 ---
test/cases/measure/measure.go | 4 +-
33 files changed, 2202 insertions(+), 1104 deletions(-)
delete mode 100644 banyand/measure/measure_query.go
create mode 100644 banyand/measure/part_iter_test.go
create mode 100644 banyand/measure/part_test.go
create mode 100644 banyand/measure/query.go
create mode 100644 banyand/measure/query_test.go
rename banyand/measure/{measure_topn.go => topn.go} (98%)
create mode 100644 banyand/measure/tstable_test.go
rename banyand/measure/{measure_write.go => write.go} (92%)
(skywalking-banyandb) 01/07: Finish test block
This is an automated email from the ASF dual-hosted git repository.
hanahmily pushed a commit to branch storage-column
in repository https://gitbox.apache.org/repos/asf/skywalking-banyandb.git
commit 835b36fec113395d391771dca398f55e62ba05c7
Author: Gao Hongtao
AuthorDate: Wed Dec 13 05:53:43 2023 +0800
Finish test block
Signed-off-by: Gao Hongtao
---
banyand/measure/block.go | 71 +
banyand/measure/block_test.go | 158 +
banyand/measure/block_writer.go| 33
banyand/measure/column_metadata.go | 2 +-
4 files changed, 181 insertions(+), 83 deletions(-)
diff --git a/banyand/measure/block.go b/banyand/measure/block.go
index 98e8e811..2c0407ce 100644
--- a/banyand/measure/block.go
+++ b/banyand/measure/block.go
@@ -18,6 +18,7 @@
package measure
import (
+ "sort"
"sync"
"github.com/apache/skywalking-banyandb/api/common"
@@ -124,26 +125,25 @@ func (b *block) Len() int {
return len(b.timestamps)
}
-func (b *block) mustWriteTo(sid common.SeriesID, bh *blockMetadata, sw
*writers) {
+func (b *block) mustWriteTo(sid common.SeriesID, bm *blockMetadata, ww
*writers) {
b.validate()
- bh.reset()
+ bm.reset()
- bh.seriesID = sid
- bh.uncompressedSizeBytes = b.uncompressedSizeBytes()
- bh.count = uint64(b.Len())
+ bm.seriesID = sid
+ bm.uncompressedSizeBytes = b.uncompressedSizeBytes()
+ bm.count = uint64(b.Len())
- mustWriteTimestampsTo(, b.timestamps, sw.timestampsWriter)
+ mustWriteTimestampsTo(, b.timestamps, ww.timestampsWriter)
- tff := b.tagFamilies
- for ti := range tff {
- b.marshalTagFamily(tff[ti], bh, sw)
+ for ti := range b.tagFamilies {
+ b.marshalTagFamily(b.tagFamilies[ti], bm, ww)
}
f := b.field
cc := f.Columns
- chh := bh.field.resizeColumnMetadata(len(cc))
+ cmm := bm.field.resizeColumnMetadata(len(cc))
for i := range cc {
- cc[i].mustWriteTo([i], )
+ cc[i].mustWriteTo([i], )
}
}
@@ -173,30 +173,30 @@ func (b *block) validate() {
}
}
-func (b *block) marshalTagFamily(tf ColumnFamily, bh *blockMetadata, sw
*writers) {
- hw, w := sw.getColumnMetadataWriterAndColumnWriter(tf.Name)
+func (b *block) marshalTagFamily(tf ColumnFamily, bm *blockMetadata, ww
*writers) {
+ hw, w := ww.getColumnMetadataWriterAndColumnWriter(tf.Name)
cc := tf.Columns
- cfh := generateColumnFamilyMetadata()
- chh := cfh.resizeColumnMetadata(len(cc))
+ cfm := generateColumnFamilyMetadata()
+ cmm := cfm.resizeColumnMetadata(len(cc))
for i := range cc {
- cc[i].mustWriteTo([i], w)
+ cc[i].mustWriteTo([i], w)
}
bb := bigValuePool.Generate()
defer bigValuePool.Release(bb)
- bb.Buf = cfh.marshal(bb.Buf)
- releaseColumnFamilyMetadata(cfh)
- tfh := bh.getTagFamilyMetadata(tf.Name)
- tfh.offset = w.bytesWritten
- tfh.size = uint64(len(bb.Buf))
- if tfh.size > maxTagFamiliesMetadataSize {
- logger.Panicf("too big columnFamilyMetadataSize: %d bytes;
mustn't exceed %d bytes", tfh.size, maxTagFamiliesMetadataSize)
+ bb.Buf = cfm.marshal(bb.Buf)
+ releaseColumnFamilyMetadata(cfm)
+ tfm := bm.getTagFamilyMetadata(tf.Name)
+ tfm.offset = hw.bytesWritten
+ tfm.size = uint64(len(bb.Buf))
+ if tfm.size > maxTagFamiliesMetadataSize {
+ logger.Panicf("too big columnFamilyMetadataSize: %d bytes;
mustn't exceed %d bytes", tfm.size, maxTagFamiliesMetadataSize)
}
hw.MustWrite(bb.Buf)
}
func (b *block) unmarshalTagFamily(decoder *encoding.BytesBlockDecoder,
tfIndex int, name string, columnFamilyMetadataBlock *dataBlock, metaReader,
valueReader fs.Reader) {
bb := bigValuePool.Generate()
- bytes.ResizeExact(bb.Buf, int(columnFamilyMetadataBlock.size))
+ bb.Buf = bytes.ResizeExact(bb.Buf, int(columnFamilyMetadataBlock.size))
fs.MustReadData(metaReader, int64(columnFamilyMetadataBlock.offset),
bb.Buf)
cfm := generateColumnFamilyMetadata()
defer releaseColumnFamilyMetadata(cfm)
@@ -205,10 +205,10 @@ func (b *block) unmarshalTagFamily(decoder
*encoding.BytesBlockDecoder, tfIndex
logger.Panicf("%s: cannot unmarshal columnFamilyMetadata: %v",
metaReader.Path(), err)
}
bigValuePool.Release(bb)
- tf := b.tagFamilies[tfIndex]
- cc := tf.resizeColumns(len(cfm.columnMetadata))
- for i, c := range cc {
- c.mustReadValues(decoder, valueReader, cfm.columnMetadata[i],
uint64(b.Len()))
+ b.tagFamilies[tfIndex].Name = name
+ cc := b.tagFamilies[tfIndex].resizeColumns(len(cfm.columnMetadata))
+ for i := range cc {
+ cc[i].mustReadValues(decoder, valueReader,
(skywalking-banyandb) 05/07: Pass the smoke test of the integration test.
This is an automated email from the ASF dual-hosted git repository.
hanahmily pushed a commit to branch storage-column
in repository https://gitbox.apache.org/repos/asf/skywalking-banyandb.git
commit ad3f5c8e7a0efd6de4281085d542144164870325
Author: Gao Hongtao
AuthorDate: Sun Dec 17 08:30:40 2023 +0800
Pass the smoke test of the integration test.
Signed-off-by: Gao Hongtao
---
banyand/internal/storage/index.go | 2 +-
banyand/internal/storage/tsdb.go | 39 +-
banyand/liaison/grpc/measure.go| 9 ++---
banyand/measure/datapoints.go | 2 +-
banyand/measure/query.go | 13 ++--
banyand/measure/topn.go| 13 +---
banyand/measure/write.go | 21 ++--
banyand/queue/local.go | 21 ++--
pkg/pb/v1/value.go | 5 +++
.../measure/measure_plan_indexscan_local.go| 2 ++
10 files changed, 84 insertions(+), 43 deletions(-)
diff --git a/banyand/internal/storage/index.go
b/banyand/internal/storage/index.go
index 780a2fea..ecca0736 100644
--- a/banyand/internal/storage/index.go
+++ b/banyand/internal/storage/index.go
@@ -184,7 +184,7 @@ func (s *seriesIndex) Search(ctx context.Context, series
*pbv1.Series, filter in
}
}
- if order == nil && order.Index == nil {
+ if order == nil || order.Index == nil {
return filterSeriesList(seriesList, pl), nil
}
diff --git a/banyand/internal/storage/tsdb.go b/banyand/internal/storage/tsdb.go
index d6fdfcad..bb3e1169 100644
--- a/banyand/internal/storage/tsdb.go
+++ b/banyand/internal/storage/tsdb.go
@@ -83,7 +83,7 @@ func (d *database[T]) Close() error {
for _, s := range d.sLst {
s.closer()
}
- return nil
+ return d.index.Close()
}
// OpenTSDB returns a new tsdb runtime. This constructor will create a new
database if it's absent,
@@ -120,29 +120,38 @@ func (d *database[T]) Register(shardID common.ShardID,
series *pbv1.Series) (*pb
if series, err = d.index.createPrimary(series); err != nil {
return nil, err
}
- id := int(shardID)
- if id < int(atomic.LoadUint32()) {
- return series, nil
- }
- d.Lock()
- defer d.Unlock()
- if id < len(d.sLst) {
- return series, nil
- }
- d.logger.Info().Int("shard_id", id).Msg("creating a shard")
- if err = d.registerShard(id); err != nil {
- return nil, err
- }
return series, nil
}
func (d *database[T]) CreateTSTableIfNotExist(shardID common.ShardID, ts
time.Time) (TSTableWrapper[T], error) {
+ id := uint32(shardID)
+ if id >= atomic.LoadUint32() {
+ return func() (TSTableWrapper[T], error) {
+ d.Lock()
+ defer d.Unlock()
+ if int(id) >= len(d.sLst) {
+ for i := len(d.sLst); i <= int(id); i++ {
+ d.logger.Info().Int("shard_id",
i).Msg("creating a shard")
+ if err := d.registerShard(i); err !=
nil {
+ return nil, err
+ }
+ }
+ }
+ return d.createTSTTable(shardID, ts)
+ }()
+ }
+ d.RLock()
+ defer d.RUnlock()
+ return d.createTSTTable(shardID, ts)
+}
+
+func (d *database[T]) createTSTTable(shardID common.ShardID, ts time.Time)
(TSTableWrapper[T], error) {
timeRange := timestamp.NewInclusiveTimeRange(ts, ts)
ss := d.sLst[shardID].segmentController.selectTSTables(timeRange)
if len(ss) > 0 {
return ss[0], nil
}
- return d.sLst[shardID].segmentController.createTSTable(timeRange.Start)
+ return d.sLst[shardID].segmentController.createTSTable(ts)
}
func (d *database[T]) SelectTSTables(timeRange timestamp.TimeRange)
[]TSTableWrapper[T] {
diff --git a/banyand/liaison/grpc/measure.go b/banyand/liaison/grpc/measure.go
index ebb5c58f..afc07fab 100644
--- a/banyand/liaison/grpc/measure.go
+++ b/banyand/liaison/grpc/measure.go
@@ -113,10 +113,11 @@ func (ms *measureService) Write(measure
measurev1.MeasureService_WriteServer) er
}
}
iwr := {
- Request: writeRequest,
- ShardId: uint32(shardID),
- SeriesHash: tsdb.HashEntity(entity),
- EntityValues: tagValues.Encode(),
+ Request:writeRequest,
+ ShardId:uint32(shardID),
+ SeriesHash:
(skywalking-banyandb) 06/07: Add projection
This is an automated email from the ASF dual-hosted git repository.
hanahmily pushed a commit to branch storage-column
in repository https://gitbox.apache.org/repos/asf/skywalking-banyandb.git
commit c651c164c190db03bb153fe2e6fa724d63664da8
Author: Gao Hongtao
AuthorDate: Sun Dec 17 12:26:35 2023 +0800
Add projection
Signed-off-by: Gao Hongtao
---
banyand/measure/block.go | 89 --
banyand/measure/block_metadata.go | 4 +
banyand/measure/block_test.go | 35 +++--
banyand/measure/query.go | 4 +-
banyand/measure/query_test.go | 10 ++-
banyand/measure/tstable.go | 2 +-
banyand/measure/tstable_test.go| 18 -
pkg/pb/v1/metadata.go | 2 +-
.../measure/measure_plan_indexscan_local.go| 8 +-
pkg/query/logical/measure/topn_plan_localscan.go | 8 +-
test/cases/measure/data/want/all_only_fields.yaml | 66
test/cases/measure/measure.go | 2 +-
12 files changed, 133 insertions(+), 115 deletions(-)
diff --git a/banyand/measure/block.go b/banyand/measure/block.go
index adf1a5e0..fd0661e1 100644
--- a/banyand/measure/block.go
+++ b/banyand/measure/block.go
@@ -195,7 +195,10 @@ func (b *block) marshalTagFamily(tf ColumnFamily, bm
*blockMetadata, ww *writers
hw.MustWrite(bb.Buf)
}
-func (b *block) unmarshalTagFamily(decoder *encoding.BytesBlockDecoder,
tfIndex int, name string, columnFamilyMetadataBlock *dataBlock, metaReader,
valueReader fs.Reader) {
+func (b *block) unmarshalTagFamily(decoder *encoding.BytesBlockDecoder,
tfIndex int, name string, columnFamilyMetadataBlock *dataBlock, tagProjection
[]string, metaReader, valueReader fs.Reader) {
+ if len(tagProjection) < 1 {
+ return
+ }
bb := bigValuePool.Generate()
bb.Buf = bytes.ResizeExact(bb.Buf, int(columnFamilyMetadataBlock.size))
fs.MustReadData(metaReader, int64(columnFamilyMetadataBlock.offset),
bb.Buf)
@@ -207,9 +210,14 @@ func (b *block) unmarshalTagFamily(decoder
*encoding.BytesBlockDecoder, tfIndex
}
bigValuePool.Release(bb)
b.tagFamilies[tfIndex].Name = name
- cc := b.tagFamilies[tfIndex].resizeColumns(len(cfm.columnMetadata))
- for i := range cc {
- cc[i].mustReadValues(decoder, valueReader,
cfm.columnMetadata[i], uint64(b.Len()))
+ cc := b.tagFamilies[tfIndex].resizeColumns(len(tagProjection))
+
+ for j := range tagProjection {
+ for i := range cc {
+ if tagProjection[j] == cfm.columnMetadata[i].name {
+ cc[i].mustReadValues(decoder, valueReader,
cfm.columnMetadata[i], uint64(b.Len()))
+ }
+ }
}
}
@@ -245,16 +253,19 @@ func (b *block) uncompressedSizeBytes() uint64 {
return n
}
-func (b *block) mustReadFrom(decoder *encoding.BytesBlockDecoder, p *part, bm
*blockMetadata) {
+func (b *block) mustReadFrom(decoder *encoding.BytesBlockDecoder, p *part, bm
blockMetadata) {
b.reset()
b.timestamps = mustReadTimestampsFrom(b.timestamps, ,
int(bm.count), p.timestamps)
- _ = b.resizeTagFamilies(len(bm.tagFamilies))
- var i int
- for name, block := range bm.tagFamilies {
- b.unmarshalTagFamily(decoder, i, name, block,
p.tagFamilyMetadata[name], p.tagFamilies[name])
- i++
+ _ = b.resizeTagFamilies(len(bm.tagProjection))
+ for i := range bm.tagProjection {
+ name := bm.tagProjection[i].Family
+ block, ok := bm.tagFamilies[name]
+ if !ok {
+ continue
+ }
+ b.unmarshalTagFamily(decoder, i, name, block,
bm.tagProjection[i].Names, p.tagFamilyMetadata[name], p.tagFamilies[name])
}
cc := b.field.resizeColumns(len(bm.field.columnMetadata))
for i := range cc {
@@ -326,6 +337,8 @@ type blockCursor struct {
bm blockMetadata
minTimestampint64
maxTimestampint64
+ tagProjection []pbv1.TagProjection
+ fieldProjection []string
}
func (bc *blockCursor) reset() {
@@ -334,6 +347,8 @@ func (bc *blockCursor) reset() {
bc.bm = blockMetadata{}
bc.minTimestamp = 0
bc.maxTimestamp = 0
+ bc.tagProjection = bc.tagProjection[:0]
+ bc.fieldProjection = bc.fieldProjection[:0]
bc.timestamps = bc.timestamps[:0]
@@ -345,12 +360,14 @@ func (bc *blockCursor) reset() {
bc.fields.reset()
}
-func (bc *blockCursor) init(p *part, bm blockMetadata, minTimestamp,
maxTimestamp int64) {
+func (bc *blockCursor) init(p *part, bm blockMetadata, queryOpts QueryOptions)
{
bc.reset()
bc.p = p
bc.bm = bm
-
(skywalking-banyandb) 07/07: Pass integration test
This is an automated email from the ASF dual-hosted git repository.
hanahmily pushed a commit to branch storage-column
in repository https://gitbox.apache.org/repos/asf/skywalking-banyandb.git
commit 7f157e8040c33f310810132f8bcc5fccbf2a5271
Author: Gao Hongtao
AuthorDate: Mon Dec 18 05:45:33 2023 +0800
Pass integration test
Signed-off-by: Gao Hongtao
---
banyand/measure/block.go | 19 ---
banyand/measure/query.go | 4
pkg/index/inverted/inverted_series.go| 11 +--
pkg/index/inverted/inverted_series_test.go | 1 +
pkg/query/logical/index_filter.go| 2 +-
.../logical/measure/measure_plan_indexscan_local.go | 20 +++-
test/cases/measure/measure.go| 4 ++--
7 files changed, 44 insertions(+), 17 deletions(-)
diff --git a/banyand/measure/block.go b/banyand/measure/block.go
index fd0661e1..1730d022 100644
--- a/banyand/measure/block.go
+++ b/banyand/measure/block.go
@@ -213,9 +213,10 @@ func (b *block) unmarshalTagFamily(decoder
*encoding.BytesBlockDecoder, tfIndex
cc := b.tagFamilies[tfIndex].resizeColumns(len(tagProjection))
for j := range tagProjection {
- for i := range cc {
+ for i := range cfm.columnMetadata {
if tagProjection[j] == cfm.columnMetadata[i].name {
- cc[i].mustReadValues(decoder, valueReader,
cfm.columnMetadata[i], uint64(b.Len()))
+ cc[j].mustReadValues(decoder, valueReader,
cfm.columnMetadata[i], uint64(b.Len()))
+ break
}
}
}
@@ -372,7 +373,7 @@ func (bc *blockCursor) init(p *part, bm blockMetadata,
queryOpts QueryOptions) {
func (bc *blockCursor) copyAllTo(r *pbv1.Result) {
r.SID = bc.bm.seriesID
- r.Timestamps = bc.timestamps
+ r.Timestamps = append(r.Timestamps, bc.timestamps...)
for _, cf := range bc.tagFamilies {
tf := pbv1.TagFamily{
Name: cf.Name,
@@ -479,6 +480,12 @@ func (bc *blockCursor) loadData(tmpBlock *block) bool {
Name: cf.Columns[i].Name,
ValueType: cf.Columns[i].ValueType,
}
+ if len(cf.Columns[i].Values) == 0 {
+ continue
+ }
+ if len(cf.Columns[i].Values) !=
len(tmpBlock.timestamps) {
+ logger.Panicf("unexpected number of values for
tags %q: got %d; want %d", cf.Columns[i].Name, len(cf.Columns[i].Values),
len(tmpBlock.timestamps))
+ }
column.Values = append(column.Values,
cf.Columns[i].Values[start:end]...)
tf.Columns = append(tf.Columns, column)
}
@@ -486,6 +493,12 @@ func (bc *blockCursor) loadData(tmpBlock *block) bool {
}
bc.fields.Name = tmpBlock.field.Name
for i := range tmpBlock.field.Columns {
+ if len(tmpBlock.field.Columns[i].Values) == 0 {
+ continue
+ }
+ if len(tmpBlock.field.Columns[i].Values) !=
len(tmpBlock.timestamps) {
+ logger.Panicf("unexpected number of values for fields
%q: got %d; want %d", tmpBlock.field.Columns[i].Name,
len(tmpBlock.field.Columns[i].Values), len(tmpBlock.timestamps))
+ }
column := Column{
Name: tmpBlock.field.Columns[i].Name,
ValueType: tmpBlock.field.Columns[i].ValueType,
diff --git a/banyand/measure/query.go b/banyand/measure/query.go
index 15a9193f..6a41ca45 100644
--- a/banyand/measure/query.go
+++ b/banyand/measure/query.go
@@ -70,6 +70,9 @@ func (s *measure) Query(ctx context.Context, mqo
pbv1.MeasureQueryOptions) (pbv1
if mqo.TimeRange == nil || mqo.Entity == nil {
return nil, errors.New("invalid query options: timeRange and
series are required")
}
+ if len(mqo.TagProjection) == 0 && len(mqo.FieldProjection) == 0 {
+ return nil, errors.New("invalid query options: tagProjection or
fieldProjection is required")
+ }
tsdb := s.databaseSupplier.SupplyTSDB().(storage.TSDB[*tsTable])
tabWrappers := tsdb.SelectTSTables(*mqo.TimeRange)
sl, err := tsdb.IndexDB().Search(ctx, {Subject: mqo.Name,
EntityValues: mqo.Entity}, mqo.Filter, mqo.Order)
@@ -123,6 +126,7 @@ func (s *measure) Query(ctx context.Context, mqo
pbv1.MeasureQueryOptions) (pbv1
if mqo.Order.Sort == modelv1.Sort_SORT_ASC || mqo.Order.Sort ==
modelv1.Sort_SORT_UNSPECIFIED {
result.ascTS = true
}
+ return , nil
}
result.sidToIndex =
(skywalking-banyandb) 04/07: Test query result
This is an automated email from the ASF dual-hosted git repository.
hanahmily pushed a commit to branch storage-column
in repository https://gitbox.apache.org/repos/asf/skywalking-banyandb.git
commit 7cb0650cd0837e19c88c6042b30bd3254abc5261
Author: Gao Hongtao
AuthorDate: Sat Dec 16 09:46:06 2023 +0800
Test query result
Signed-off-by: Gao Hongtao
---
banyand/measure/block.go | 120 +-
banyand/measure/block_test.go | 88 +++-
banyand/measure/measure_query.go | 540 -
banyand/measure/part_test.go | 2 -
banyand/measure/query.go | 419 +++
banyand/measure/query_test.go | 439
banyand/measure/{measure_topn.go => topn.go} | 0
banyand/measure/tstable_test.go| 102 +
banyand/measure/{measure_write.go => write.go} | 0
pkg/pb/v1/metadata.go | 2 +
10 files changed, 1061 insertions(+), 651 deletions(-)
diff --git a/banyand/measure/block.go b/banyand/measure/block.go
index 8fd64781..adf1a5e0 100644
--- a/banyand/measure/block.go
+++ b/banyand/measure/block.go
@@ -26,6 +26,7 @@ import (
"github.com/apache/skywalking-banyandb/pkg/encoding"
"github.com/apache/skywalking-banyandb/pkg/fs"
"github.com/apache/skywalking-banyandb/pkg/logger"
+ pbv1 "github.com/apache/skywalking-banyandb/pkg/pb/v1"
)
type block struct {
@@ -323,14 +324,16 @@ type blockCursor struct {
columnValuesDecoder encoding.BytesBlockDecoder
p *part
bm blockMetadata
- opts*QueryOptions
+ minTimestampint64
+ maxTimestampint64
}
func (bc *blockCursor) reset() {
bc.idx = 0
bc.p = nil
bc.bm = blockMetadata{}
- bc.opts = nil
+ bc.minTimestamp = 0
+ bc.maxTimestamp = 0
bc.timestamps = bc.timestamps[:0]
@@ -342,47 +345,138 @@ func (bc *blockCursor) reset() {
bc.fields.reset()
}
-func (bc *blockCursor) init(p *part, bm blockMetadata, opts *QueryOptions) {
+func (bc *blockCursor) init(p *part, bm blockMetadata, minTimestamp,
maxTimestamp int64) {
bc.reset()
bc.p = p
bc.bm = bm
- bc.opts = opts
+ bc.minTimestamp = minTimestamp
+ bc.maxTimestamp = maxTimestamp
+}
+
+func (bc *blockCursor) copyAllTo(r *pbv1.Result) {
+ r.SID = bc.bm.seriesID
+ r.Timestamps = bc.timestamps
+ for _, cf := range bc.tagFamilies {
+ tf := pbv1.TagFamily{
+ Name: cf.Name,
+ }
+ for _, c := range cf.Columns {
+ t := pbv1.Tag{
+ Name: c.Name,
+ }
+ for _, v := range c.Values {
+ t.Values = append(t.Values,
mustDecodeTagValue(c.ValueType, v))
+ }
+ tf.Tags = append(tf.Tags, t)
+ }
+ r.TagFamilies = append(r.TagFamilies, tf)
+ }
+ for _, c := range bc.fields.Columns {
+ f := pbv1.Field{
+ Name: c.Name,
+ }
+ for _, v := range c.Values {
+ f.Values = append(f.Values,
mustDecodeFieldValue(c.ValueType, v))
+ }
+ r.Fields = append(r.Fields, f)
+ }
+}
+
+func (bc *blockCursor) copyTo(r *pbv1.Result) {
+ r.SID = bc.bm.seriesID
+ r.Timestamps = append(r.Timestamps, bc.timestamps[bc.idx])
+ if len(r.TagFamilies) != len(bc.tagFamilies) {
+ for _, cf := range bc.tagFamilies {
+ tf := pbv1.TagFamily{
+ Name: cf.Name,
+ }
+ for _, c := range cf.Columns {
+ t := pbv1.Tag{
+ Name: c.Name,
+ }
+ tf.Tags = append(tf.Tags, t)
+ }
+ r.TagFamilies = append(r.TagFamilies, tf)
+ }
+ }
+ for i, cf := range bc.tagFamilies {
+ for i2, c := range cf.Columns {
+ r.TagFamilies[i].Tags[i2].Values =
append(r.TagFamilies[i].Tags[i2].Values, mustDecodeTagValue(c.ValueType,
c.Values[bc.idx]))
+ }
+ }
+
+ if len(r.Fields) != len(bc.fields.Columns) {
+ for _, c := range bc.fields.Columns {
+ f := pbv1.Field{
+ Name: c.Name,
+ }
+ r.Fields = append(r.Fields, f)
+ }
+ }
+ for i, c := range bc.fields.Columns {
+ r.Fields[i].Values = append(r.Fields[i].Values,
mustDecodeFieldValue(c.ValueType,
(skywalking-banyandb) 03/07: Test tsTable
This is an automated email from the ASF dual-hosted git repository.
hanahmily pushed a commit to branch storage-column
in repository https://gitbox.apache.org/repos/asf/skywalking-banyandb.git
commit d91f5f26a0a1a2f07cdce0cac5a0545682146568
Author: Gao Hongtao
AuthorDate: Thu Dec 14 08:13:40 2023 +0800
Test tsTable
Signed-off-by: Gao Hongtao
---
banyand/measure/block_writer.go | 2 +
banyand/measure/measure_query.go | 66 ++-
banyand/measure/measure_write.go | 5 +-
banyand/measure/part_iter.go | 6 +-
banyand/measure/part_iter_test.go | 2 +-
banyand/measure/part_metadata.go | 2 +
banyand/measure/tstable.go| 20 +-
banyand/measure/tstable_test.go | 386 ++
8 files changed, 424 insertions(+), 65 deletions(-)
diff --git a/banyand/measure/block_writer.go b/banyand/measure/block_writer.go
index 4fd50190..546a8d32 100644
--- a/banyand/measure/block_writer.go
+++ b/banyand/measure/block_writer.go
@@ -19,6 +19,7 @@ package measure
import (
"sync"
+ "time"
"github.com/apache/skywalking-banyandb/api/common"
"github.com/apache/skywalking-banyandb/pkg/compress/zstd"
@@ -255,6 +256,7 @@ func (bsw *blockWriter) Flush(ph *partMetadata) {
ph.BlocksCount = bsw.totalBlocksCount
ph.MinTimestamp = bsw.totalMinTimestamp
ph.MaxTimestamp = bsw.totalMaxTimestamp
+ ph.Version = time.Now().UnixNano() // TODO: use a global version
bsw.mustFlushPrimaryBlock(bsw.primaryBlockData)
diff --git a/banyand/measure/measure_query.go b/banyand/measure/measure_query.go
index 4b2960c9..5b13923a 100644
--- a/banyand/measure/measure_query.go
+++ b/banyand/measure/measure_query.go
@@ -23,7 +23,6 @@ import (
"fmt"
"io"
"sort"
- "sync"
"time"
"github.com/pkg/errors"
@@ -54,7 +53,6 @@ type Query interface {
type Measure interface {
io.Closer
Query(ctx context.Context, opts pbv1.MeasureQueryOptions)
(pbv1.MeasureQueryResult, error)
- Shards(entity tsdb.Entity) ([]tsdb.Shard, error)
CompanionShards(metadata *commonv1.Metadata) ([]tsdb.Shard, error)
Shard(id common.ShardID) (tsdb.Shard, error)
ParseTagFamily(family string, item tsdb.Item) (*modelv1.TagFamily,
error)
@@ -89,8 +87,6 @@ type QueryResult struct {
ascbool
}
-const round = time.Millisecond
-
func (qr *QueryResult) Pull() *pbv1.Result {
if len(qr.data) == 0 {
return nil
@@ -111,7 +107,7 @@ func (qr *QueryResult) Pull() *pbv1.Result {
bc.copyAllTo(r)
return r
}
- return qr.merge(round)
+ return qr.merge()
}
originalData := qr.data
var lastSeriesID common.SeriesID
@@ -121,13 +117,13 @@ func (qr *QueryResult) Pull() *pbv1.Result {
qr.data = qr.data[:0]
qr.data = append(qr.data,
originalData[qr.lastIndex:i]...)
qr.lastIndex = i
- return qr.merge(round)
+ return qr.merge()
}
lastSeriesID = bc.bm.seriesID
}
if qr.lastIndex < len(originalData)-1 {
qr.data = originalData[qr.lastIndex:]
- return qr.merge(round)
+ return qr.merge()
}
return nil
}
@@ -171,23 +167,20 @@ func (qr *QueryResult) Pop() interface{} {
return x
}
-func (qr *QueryResult) merge(round time.Duration) *pbv1.Result {
+func (qr *QueryResult) merge() *pbv1.Result {
result := {}
- var lastOriginalTimestamp int64
- r := int64(round)
+ var lastPartVersion int64
for qr.Len() > 0 {
bc := heap.Pop(qr).(*blockCursor)
- roundedTimestamp := (bc.timestamps[bc.idx] / r) * r
-
- if len(result.Timestamps) > 0 && roundedTimestamp ==
result.Timestamps[len(result.Timestamps)-1] {
- if bc.timestamps[bc.idx] > lastOriginalTimestamp {
- lastOriginalTimestamp = bc.timestamps[bc.idx]
+ if len(result.Timestamps) > 0 && bc.timestamps[bc.idx] ==
result.Timestamps[len(result.Timestamps)-1] {
+ if bc.p.partMetadata.Version > lastPartVersion {
+ lastPartVersion = bc.p.partMetadata.Version
}
} else {
bc.copyTo(result)
- lastOriginalTimestamp = bc.timestamps[bc.idx]
+ lastPartVersion = bc.p.partMetadata.Version
}
bc.idx++
@@ -235,12 +228,12 @@ func (s *measure) Query(ctx context.Context, mqo
pbv1.MeasureQueryOptions) (pbv1
originalSids := make([]common.SeriesID, len(sids))
copy(originalSids, sids)
sort.Slice(sids, func(i, j int) bool { return
Re: [PR] Begin the process of `SWIP - SkyWalking Improvement Proposal`. [skywalking]
wu-sheng commented on PR #11668: URL: https://github.com/apache/skywalking/pull/11668#issuecomment-1859201464 FYI @apache/skywalking-committers Please comment if you have any concern. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: notifications-unsubscr...@skywalking.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Begin the process of `SWIP - SkyWalking Improvement Proposal`. [skywalking]
wu-sheng commented on code in PR #11668: URL: https://github.com/apache/skywalking/pull/11668#discussion_r1429154996 ## docs/en/swip/readme.md: ## @@ -0,0 +1,57 @@ +# SWIP - SkyWalking Improvement Proposal + +SWIP - SkyWalking Improvement Proposal, is an official document to propose a new feature and/or feature improvement, +which +are relative to end users and developers. + +SkyWalking has been very stable since v9.x. We are getting over the rapid changing stage. The core concepts, protocols for +reporting telemetry and query, 3rd party integration, and the streaming process kernel are very stable. From now(2024) on, +SkyWalking community would focus more on improvement and controllable improvement. All major changes should be evaluated +more seriously, and try as good as possible to avoid incompatible breaking changes. + +## What is considered a major change? + +The catalogs of a major change are listed as follows + +- New Feature. A feature doesn't exist for the latest version. +- Any change of the network Interfaces, especially + for [Query Protocol](https://github.com/apache/skywalking-query-protocol), + [Data Collect Protocols](https://github.com/apache/skywalking-data-collect-protocol), + Dynamic Configuration APIs, Exporting APIs, AI pipeline APIs. +- Any change of storage structure. + +**Q: Is Agent side feature or change considered a SWIP?** + +A: Right now, SWIP targets on OAP and UI side changes. All agent side changes are pending on the reviews from the +committers of those agents. + +## SWIP Template + +The purpose of this template should not be considered a hard requirement. The major purpose of SWIP is helping the PMC +and community member to understand the proposal better. + +* Title: SWIP-1234 + +- Motivation - the description of new feature or improvement. +- Architecture Graph - describing the relationship between your new proposal part and existing components. +- Imported Dependencies libs and their licenses. +- Compatibility - Whether breaking configuration, storage structure, protocols. +- General usage docs - This doesn't have to be final version, but help the reviews to understand how to use this new + feature. + +## SWIP Process + +Here is the process for starting a SWIP. + +1. Start a discussion at [GitHub Discussion Page](https://github.com/apache/skywalking/discussions) with + title `[DISCUSS] `. +2. Fill in the sections as described above in `SWIP Template`. +3. At least one SkyWalking committer commented on the discussion to show interests to adopt it. +4. This committer could update this page to grant a **SWIP ID**, and update the title to `[SWIP-ID NO.] [DISCUSS] `. +5. All further discussion could happen on the discussion page. +6. Once the consensus is made by enough committer supporters, and/or through a mail list vote, this SWIP should be + added [here](./) as `SWIP-ID NO.md` and listed in the below as `Known SWIPs`. + +## Known SWIPs + +Next SWIP Number: 2 Review Comment: I will use `SWIP-1` for https://github.com/apache/skywalking/discussions/11566 We have a consensus on this. @wankai123 @mrproliu As the new changes relative to #11566, you may need to update the SWIP-1 information. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: notifications-unsubscr...@skywalking.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
(skywalking) branch swip updated (5f9f7642c3 -> 1465223420)
This is an automated email from the ASF dual-hosted git repository. wusheng pushed a change to branch swip in repository https://gitbox.apache.org/repos/asf/skywalking.git from 5f9f7642c3 Fix format. add 1465223420 Fix a word. No new revisions were added by this update. Summary of changes: docs/en/swip/readme.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
[PR] Begin the process of `SWIP - SkyWalking Improvement Proposal`. [skywalking]
wu-sheng opened a new pull request, #11668: URL: https://github.com/apache/skywalking/pull/11668 - [ ] If this pull request closes/resolves/fixes an existing issue, replace the issue number. Closes #. - [x] Update the [`CHANGES` log](https://github.com/apache/skywalking/blob/master/docs/en/changes/changes.md). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: notifications-unsubscr...@skywalking.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
(skywalking) branch swip updated: Fix format.
This is an automated email from the ASF dual-hosted git repository. wusheng pushed a commit to branch swip in repository https://gitbox.apache.org/repos/asf/skywalking.git The following commit(s) were added to refs/heads/swip by this push: new 5f9f7642c3 Fix format. 5f9f7642c3 is described below commit 5f9f7642c3dbace70ac1b72f9b20a01e2a91d0f6 Author: Wu Sheng AuthorDate: Sun Dec 17 21:06:13 2023 +0800 Fix format. --- docs/en/swip/readme.md | 6 ++ 1 file changed, 2 insertions(+), 4 deletions(-) diff --git a/docs/en/swip/readme.md b/docs/en/swip/readme.md index 2ea559cab1..30d320201c 100644 --- a/docs/en/swip/readme.md +++ b/docs/en/swip/readme.md @@ -4,10 +4,8 @@ SWIP - SkyWalking Improvement Proposal, is an official document to propose a new which are relative to end users and developers. -SkyWalking has been very stable since v9.x. We are getting over the rapid changing stage. The core concepts, protocols -for -reporting telemetry and query, 3rd party integration, and the streaming process kernel are very stable. From now(2024) -on, +SkyWalking has been very stable since v9.x. We are getting over the rapid changing stage. The core concepts, protocols for +reporting telemetry and query, 3rd party integration, and the streaming process kernel are very stable. From now(2024) on, SkyWalking community would focus more on improvement and controllable improvement. All major changes should be evaluated more seriously, and try as good as possible to avoid incompatible breaking changes.
(skywalking) 01/01: Begin the process of `SWIP - SkyWalking Improvement Proposal`.
This is an automated email from the ASF dual-hosted git repository. wusheng pushed a commit to branch swip in repository https://gitbox.apache.org/repos/asf/skywalking.git commit 06510fcd4fc4e46a7f312a47ef1f047f7a954ef7 Author: Wu Sheng AuthorDate: Sun Dec 17 21:04:50 2023 +0800 Begin the process of `SWIP - SkyWalking Improvement Proposal`. --- docs/en/changes/changes.md | 1 + docs/en/swip/readme.md | 59 ++ docs/menu.yml | 2 ++ 3 files changed, 62 insertions(+) diff --git a/docs/en/changes/changes.md b/docs/en/changes/changes.md index 0a2b100a5f..d8410c1ee5 100644 --- a/docs/en/changes/changes.md +++ b/docs/en/changes/changes.md @@ -21,5 +21,6 @@ * Update the release notification mail a little. * Polish docs structure. Move customization docs separately from the introduction docs. * Add webhook/gRPC hooks settings example for `backend-alarm.md`. +* Begin the process of `SWIP - SkyWalking Improvement Proposal`. All issues and pull requests are [here](https://github.com/apache/skywalking/milestone/202?closed=1) diff --git a/docs/en/swip/readme.md b/docs/en/swip/readme.md new file mode 100644 index 00..2ea559cab1 --- /dev/null +++ b/docs/en/swip/readme.md @@ -0,0 +1,59 @@ +# SWIP - SkyWalking Improvement Proposal + +SWIP - SkyWalking Improvement Proposal, is an official document to propose a new feature and/or feature improvement, +which +are relative to end users and developers. + +SkyWalking has been very stable since v9.x. We are getting over the rapid changing stage. The core concepts, protocols +for +reporting telemetry and query, 3rd party integration, and the streaming process kernel are very stable. From now(2024) +on, +SkyWalking community would focus more on improvement and controllable improvement. All major changes should be evaluated +more seriously, and try as good as possible to avoid incompatible breaking changes. + +## What is considered a major change? + +The catalogs of a major change are listed as follows + +- New Feature. A feature doesn't exist for the latest version. +- Any change of the network Interfaces, especially + for [Query Protocol](https://github.com/apache/skywalking-query-protocol), + [Data Collect Protocols](https://github.com/apache/skywalking-data-collect-protocol), + Dynamic Configuration APIs, Exporting APIs, AI pipeline APIs. +- Any change of storage structure. + +**Q: Is Agent side feature or change considered a SWIP?** + +A: Right now, SWIP targets on OAP and UI side changes. All agent side changes are pending on the reviews from the +committers of those agents. + +## SWIP Template + +The purpose of this template should not be considered a hard requirement. The major purpose of SWIP is helping the PMC +and community member to understand the proposal better. + +* Title: SWIP-1234 + +- Motivation - the description of new feature or improvement. +- Architecture Graph - describing the relationship between your new proposal part and existing components. +- Imported Dependencies libs and their licenses. +- Compatibility - Whether breaking configuration, storage structure, protocols. +- General usage docs - This doesn't have to be final version, but help the reviews to understand how to use this new + feature. + +## SWIP Process + +Here is the process for starting a SWIP. + +1. Start a discussion at [GitHub Discussion Page](https://github.com/apache/skywalking/discussions) with + title `[DISCUSS] `. +2. Fill in the sections as described above in `SWIP Template`. +3. At least one SkyWalking committer commented on the discussion to show interests to adopt it. +4. This committer could update this page to grant a **SWIP ID**, and update the title to `[SWIP-ID NO.] [DISCUSS] `. +5. All further discussion could happen on the discussion page. +6. Once the consensus is made by enough committer supporters, and/or through a mail list vote, this SWIP should be + added [here](./) as `SWIP-ID NO.md` and listed in the below as `Known SWIPs`. + +## Known SWIPs + +Next FLIP Number: 2 diff --git a/docs/menu.yml b/docs/menu.yml index 6a6432919a..d88ad1cdbf 100644 --- a/docs/menu.yml +++ b/docs/menu.yml @@ -370,6 +370,8 @@ catalog: path: "/en/guides/how-to-bump-up-zipkin" - name: "I18n" path: "/en/guides/i18n" + - name: "SWIP" +path: "/en/swip/readme" - name: "Changelog" catalog: - name: "Current Version"
(skywalking) branch swip created (now 06510fcd4f)
This is an automated email from the ASF dual-hosted git repository. wusheng pushed a change to branch swip in repository https://gitbox.apache.org/repos/asf/skywalking.git at 06510fcd4f Begin the process of `SWIP - SkyWalking Improvement Proposal`. This branch includes the following new commits: new 06510fcd4f Begin the process of `SWIP - SkyWalking Improvement Proposal`. The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference.
Re: [I] [Bug] skywalking-client-js data is not present inside skywalking-ui browser dashboards [skywalking]
Fine0830 commented on issue #11662: URL: https://github.com/apache/skywalking/issues/11662#issuecomment-1859161225 If the `enableSPA` is true in the register method, it's essential to use `setPerformance` to monitor performance metrics. This won't affect tracing and logging functionalities. Regarding `autoTracePerf`, it doesn't directly relate to enabling SPA (`enableSPA`). It depends on whether performance metrics need to be monitored automatically. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: notifications-unsubscr...@skywalking.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] java argnet causes memory to grow infinitely [skywalking]
wu-sheng closed issue #11667: java argnet causes memory to grow infinitely URL: https://github.com/apache/skywalking/issues/11667 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: notifications-unsubscr...@skywalking.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] java argnet causes memory to grow infinitely [skywalking]
wu-sheng commented on issue #11667: URL: https://github.com/apache/skywalking/issues/11667#issuecomment-1859136144 Why do you repeated open things? I had replied to you. Locking. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: notifications-unsubscr...@skywalking.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[I] java argnet causes memory to grow infinitely [skywalking]
sanhe-qigai opened a new issue, #11667: URL: https://github.com/apache/skywalking/issues/11667 ### Search before asking - [X] I had searched in the [issues](https://github.com/apache/skywalking/issues?q=is%3Aissue) and found no similar feature requirement. ### Description argnet will cache the monitoring information in the machine memory and then report it asynchronously. Have you ever encountered a case where the machine memory grows infinitely due to this? ### Use case _No response_ ### Related issues _No response_ ### Are you willing to submit a pull request to implement this on your own? - [X] Yes I am willing to submit a pull request on my own! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: notifications-unsubscr...@skywalking.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] [Bug] skywalking-client-js data is not present inside skywalking-ui browser dashboards [skywalking]
marcanpilami commented on issue #11662: URL: https://github.com/apache/skywalking/issues/11662#issuecomment-1859134702 If I may, if you follow these guidelines you will encounter the issue I'm talking about. Part 1 "Enable SPA automatic parsing" actually only reports traces, not metrics nor errors. This is were I would add the mention I'm proposing above. (and I would also add it to the parameter table). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: notifications-unsubscr...@skywalking.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] [Bug] skywalking-client-js data is not present inside skywalking-ui browser dashboards [skywalking]
Fine0830 commented on issue #11662: URL: https://github.com/apache/skywalking/issues/11662#issuecomment-1859122994 Here is the guideline about the SPA scene. https://github.com/apache/skywalking-client-js?tab=readme-ov-file#spa-page -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: notifications-unsubscr...@skywalking.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
