This is an automated email from the ASF dual-hosted git repository.

klesh pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/incubator-devlake.git


The following commit(s) were added to refs/heads/main by this push:
     new e26c61b92 feat(github): Add incremental data collection (#8858)
e26c61b92 is described below

commit e26c61b9297dbc6bcb5e0e1609a8c91b6acdef29
Author: Leif Roger Frøysaa <[email protected]>
AuthorDate: Wed Apr 29 14:35:02 2026 +0200

    feat(github): Add incremental data collection (#8858)
    
    * fix(github): use RFC 3339 format for since= incremental API parameters
    
    The since= query parameter passed to the GitHub API in four collectors
    was formatted using Go's time.Time.String(), which produces a
    human-readable string (e.g. "2024-01-15 10:30:00 +0000 UTC") rather
    than the ISO 8601 / RFC 3339 format the GitHub API requires
    (e.g. "2024-01-15T10:30:00Z").
    
    The GitHub API silently ignores malformed date strings, causing these
    collectors to perform full re-scans on every incremental run despite
    appearing to filter correctly. Fix by using .UTC().Format(time.RFC3339)
    in all four affected collectors:
    - comment_collector.go
    - issue_collector.go
    - commit_collector.go
    - pr_review_comment_collector.go
    
    (cherry picked from commit cf3cb621462d8ae661df5fcb8e1b47c70564cd60)
    
    * feat(github): migrate event extractor to StatefulApiExtractor
    
    Switch Extract Events from the legacy full-scan NewApiExtractor to
    NewStatefulApiExtractor, which filters _raw_github_api_events by
    created_at >= last_run_start on incremental syncs. This table had
    13,828 rows in a representative production run and took ~117s to
    process on every run regardless of how few new events were collected.
    After this change incremental runs process only newly collected rows.
    
    (cherry picked from commit 84f324f65e8576cd2ff0ca8b74a2588ee2600e12)
    
    * feat(github): migrate PR extractor to StatefulApiExtractor
    
    Switch Extract Pull Requests from legacy full-scan NewApiExtractor to
    NewStatefulApiExtractor. The _raw_github_api_pull_requests table had
    12,448 rows in a representative production run, taking ~108s on every
    incremental sync. After this change only newly collected PR rows are
    processed.
    
    SubtaskConfig captures prType and prComponent regex strings so that a
    scope config change automatically triggers a full re-extract. BeforeExtract
    deletes GithubPrLabel rows for the current PR before re-inserting them
    in incremental mode, preventing stale labels from persisting when labels
    are removed upstream.
    
    (cherry picked from commit 80dc76d4e5f1ad6c65d48d42f60f50b87c9dad2d)
    
    * feat(github): migrate workflow run and PR commit extractors to 
StatefulApiExtractor
    
    Switch Extract Workflow Runs (~7,364 raw rows, ~65s) and Extract PR
    Commits (~6,982 raw rows, ~60s) from legacy full-scan NewApiExtractor
    to NewStatefulApiExtractor. Both extractors are simple mappings with
    no scope-config dependency, so no SubtaskConfig or BeforeExtract needed.
    
    (cherry picked from commit ea4f158a65ad0f7a2c23ba7bbc9932059e2ca408)
    
    * feat(github): migrate remaining high-volume extractors to 
StatefulApiExtractor
    
    Migrate Extract Jobs (~5,369 rows, ~48s), Extract PR Reviews (~3,073
    rows, ~27s), and Extract PR Review Comments (~1,820 rows, ~16s) from
    legacy full-scan NewApiExtractor to NewStatefulApiExtractor.
    
    Also moves prUrlRegex compilation in pr_review_comment_extractor.go
    from inside the Extract closure (recompiled on every raw row) to before
    the extractor is created, eliminating redundant regexp compilation.
    
    (cherry picked from commit 54d601587bc74ebd0f1103345c545571146739b5)
    
    * feat(github): migrate remaining low-volume extractors to 
StatefulApiExtractor
    
    Migrate the final seven GitHub extractors to NewStatefulApiExtractor:
    issue, comment, account, account_org, milestone, commit, commit_stats.
    
    issue_extractor gains SubtaskConfig (issue classification regex strings)
    so scope config changes trigger automatic full re-extraction, and
    BeforeExtract cleanup for GithubIssueLabel and GithubIssueAssignee rows
    in incremental mode to prevent stale labels/assignees persisting after
    upstream removal.
    
    All other extractors in this commit are simple migrations with no
    config-sensitivity or child record cleanup needed.
    
    With this commit all 14 GitHub plugin extractors are now incremental.
    Combined with the collector fixes in earlier commits, incremental
    collection runs that previously took 9+ minutes in the extract phase
    will now complete in seconds when few or no new records were collected.
    
    (cherry picked from commit 606acea88caef63662733162faa47a6c6d3155cc)
    
    * feat(github): migrate CICD converters to StatefulDataConverter
    
    Convert Workflow Runs and Convert Jobs now use NewStatefulDataConverter,
    skipping records unchanged since last run on incremental pipelines.
    Jobs are filtered via JOIN on _tool_github_runs.github_updated_at.
    
    (cherry picked from commit 75a909efccab1d04bfae4058f4674663b00762a6)
    
    * feat(github): migrate PR supporting data converters to 
StatefulDataConverter
    
    Convert PR Commits, Convert PR Comments, and Convert PR Reviews now use
    NewStatefulDataConverter. Child-of-PR records are filtered incrementally
    via JOIN on _tool_github_pull_requests.github_updated_at; PR comments
    additionally filter on their own github_updated_at.
    
    (cherry picked from commit 17cbc2e4caff7349fa2d3389d7cafcc5d0edee71)
    
    * feat(github): migrate PR main and cross converters to 
StatefulDataConverter
    
    Convert Pull Requests filters by GithubPullRequest.github_updated_at.
    Convert Reviews and Convert PR Issues filter via JOIN on pull_requests
    github_updated_at since reviewers and pr_issues have no own timestamp.
    
    (cherry picked from commit 47437ae49d30b967b443c964a4b1baca34344acc)
    
    * feat(github): migrate remaining data converters to StatefulDataConverter
    
    Migrate the last 9 converters from DataConverter to StatefulDataConverter
    so they skip already-processed records on incremental runs:
    
    - issue_convertor: filter on github_updated_at
    - issue_comment_convertor: filter on github_updated_at
    - issue_label_convertor: JOIN to issues, filter on issues.github_updated_at
    - issue_assignee_convertor: JOIN to issues, filter on 
issues.github_updated_at
    - pr_label_convertor: JOIN to pull_requests, filter on pr.github_updated_at
    - account_convertor: filter on updated_at
    - release_convertor: filter on updated_at
    - repo_convertor: filter on updated_at; retain GithubApiRepo struct (used 
by pr_extractor)
    - commit_convertor: filter on authored_date
    
    (cherry picked from commit 7bcc9da4676d34d9cd24e41a68765cf85723ac81)
    
    * fix(github): keep incremental commit and PR-issue processing accurate
    
    * fix(github): bootstrap workflow runs incremental window from tool state
    
    * feat(runner): include processed record count in subtask finish logs
    
    * fix(stateful): bootstrap subtask state from collector checkpoints
    
    * fix(github): optimize incremental Convert Jobs query
    
    * fix(github): resolve lint and staticcheck issues in converters
    
    * fix(stateful): tolerate missing collector state table in bootstrap
    
    * fix(github): address CI failures in state bootstrap and run tests
    
    * fix(server): use interface{} for store handler swag annotations (#8859)
    
    Signed-off-by: yamoyamoto <[email protected]>
    
    * fix(github): stabilize e2e state bootstrap and issue assignee join
    
    * fix(stateful): skip collector bootstrap lookup when state table is absent
    
    * test(stateful): use generic scope params in bootstrap tests
    
    * fix(github): keep issue assignee conversion full-sync safe
    
    ---------
    
    Signed-off-by: yamoyamoto <[email protected]>
    Co-authored-by: Tomoya Kawaguchi 
<[email protected]>
---
 backend/core/plugin/plugin_task.go                 |   1 +
 backend/core/runner/run_task.go                    |   2 +-
 .../pluginhelper/api/subtask_state_manager.go      |  55 +++++++
 .../pluginhelper/api/subtask_state_manager_test.go | 169 +++++++++++++++++++++
 backend/impls/context/default_exec_context.go      |   4 +
 backend/plugins/github/tasks/account_convertor.go  |  62 ++++----
 backend/plugins/github/tasks/account_extractor.go  |  38 ++---
 .../plugins/github/tasks/account_org_extractor.go  |  21 +--
 backend/plugins/github/tasks/cicd_job_convertor.go |  45 +++---
 backend/plugins/github/tasks/cicd_job_extractor.go |  19 +--
 backend/plugins/github/tasks/cicd_run_collector.go |  48 +++++-
 .../github/tasks/cicd_run_collector_test.go        |  52 +++++++
 backend/plugins/github/tasks/cicd_run_convertor.go |  39 ++---
 backend/plugins/github/tasks/cicd_run_extractor.go |  25 ++-
 backend/plugins/github/tasks/comment_collector.go  |   3 +-
 backend/plugins/github/tasks/comment_extractor.go  |  65 ++++----
 backend/plugins/github/tasks/commit_collector.go   |   3 +-
 backend/plugins/github/tasks/commit_convertor.go   |  46 +++---
 backend/plugins/github/tasks/commit_extractor.go   |  60 +++-----
 .../plugins/github/tasks/commit_stats_extractor.go |  35 +----
 backend/plugins/github/tasks/event_extractor.go    |  22 +--
 .../github/tasks/issue_assignee_convertor.go       |  40 ++---
 backend/plugins/github/tasks/issue_collector.go    |   3 +-
 .../github/tasks/issue_comment_convertor.go        |  41 +++--
 backend/plugins/github/tasks/issue_convertor.go    |  37 +++--
 backend/plugins/github/tasks/issue_extractor.go    |  50 +++---
 .../plugins/github/tasks/issue_label_convertor.go  |  40 ++---
 .../plugins/github/tasks/milestone_extractor.go    |  19 +--
 .../plugins/github/tasks/pr_comment_convertor.go   |  42 ++---
 .../plugins/github/tasks/pr_commit_convertor.go    |  41 +++--
 .../plugins/github/tasks/pr_commit_extractor.go    |  34 +----
 backend/plugins/github/tasks/pr_convertor.go       |  37 +++--
 backend/plugins/github/tasks/pr_extractor.go       |  57 ++++---
 backend/plugins/github/tasks/pr_issue_convertor.go |  39 ++---
 backend/plugins/github/tasks/pr_issue_enricher.go  |  48 +++---
 backend/plugins/github/tasks/pr_label_convertor.go |  40 ++---
 .../github/tasks/pr_review_comment_collector.go    |   3 +-
 .../github/tasks/pr_review_comment_extractor.go    |  81 +++++-----
 .../plugins/github/tasks/pr_review_convertor.go    |  41 +++--
 .../plugins/github/tasks/pr_review_extractor.go    |  58 +++----
 backend/plugins/github/tasks/release_convertor.go  |  58 +++----
 backend/plugins/github/tasks/repo_convertor.go     |  47 +++---
 backend/plugins/github/tasks/review_convertor.go   |  41 +++--
 backend/server/services/remote/bridge/context.go   |   7 +
 44 files changed, 955 insertions(+), 763 deletions(-)

diff --git a/backend/core/plugin/plugin_task.go 
b/backend/core/plugin/plugin_task.go
index e1404ff08..e3eea470a 100644
--- a/backend/core/plugin/plugin_task.go
+++ b/backend/core/plugin/plugin_task.go
@@ -53,6 +53,7 @@ type ExecContext interface {
        GetData() interface{}
        SetProgress(current int, total int)
        IncProgress(quantity int)
+       GetProgress() int
 }
 
 // SubTaskContext This interface define all resources that needed for subtask 
execution
diff --git a/backend/core/runner/run_task.go b/backend/core/runner/run_task.go
index 2b848ea9b..961725bce 100644
--- a/backend/core/runner/run_task.go
+++ b/backend/core/runner/run_task.go
@@ -336,7 +336,7 @@ func RunPluginSubTasks(
                        logger.Info("executing subtask %s", subtaskMeta.Name)
                        start := time.Now()
                        err = runSubtask(basicRes, subtaskCtx, task.ID, 
subtaskNumber, subtaskMeta.EntryPoint)
-                       logger.Info("subtask %s finished in %d ms", 
subtaskMeta.Name, time.Since(start).Milliseconds())
+                       logger.Info("subtask %s finished in %d ms, records 
processed: %d", subtaskMeta.Name, time.Since(start).Milliseconds(), 
subtaskCtx.GetProgress())
                        if err != nil {
                                err = errors.SubtaskErr.Wrap(err, 
fmt.Sprintf("subtask %s ended unexpectedly", subtaskMeta.Name), 
errors.WithData(&subtaskMeta))
                                logger.Error(err, "")
diff --git a/backend/helpers/pluginhelper/api/subtask_state_manager.go 
b/backend/helpers/pluginhelper/api/subtask_state_manager.go
index 7dd7d2370..ff1c8d7f4 100644
--- a/backend/helpers/pluginhelper/api/subtask_state_manager.go
+++ b/backend/helpers/pluginhelper/api/subtask_state_manager.go
@@ -20,6 +20,7 @@ package api
 import (
        "fmt"
        "reflect"
+       "strings"
        "time"
 
        "github.com/apache/incubator-devlake/core/dal"
@@ -88,6 +89,10 @@ func NewSubtaskStateManager(args *SubtaskCommonArgs) 
(stateManager *SubtaskState
        if err != nil {
                return
        }
+       preState, err = bootstrapStateFromCollectorStateIfNeeded(db, preState, 
args)
+       if err != nil {
+               return
+       }
 
        isIncremental, since := 
calculateStateManagerIncrementalMode(syncPolicy, preState, 
utils.ToJsonString(args.SubtaskConfig))
 
@@ -127,6 +132,56 @@ func loadPreviousState(db dal.Dal, plugin, subtask, params 
string) (*models.Subt
        return preState, nil
 }
 
+func bootstrapStateFromCollectorStateIfNeeded(db dal.Dal, preState 
*models.SubtaskState, args *SubtaskCommonArgs) (*models.SubtaskState, 
errors.Error) {
+       if preState == nil || preState.PrevStartedAt != nil {
+               return preState, nil
+       }
+       if args == nil || args.Table == "" {
+               return preState, nil
+       }
+       if !db.HasTable(&models.CollectorLatestState{}) {
+               return preState, nil
+       }
+
+       rawTable := args.GetRawDataTable()
+       if rawTable == "" {
+               return preState, nil
+       }
+
+       collectorState := &models.CollectorLatestState{}
+       err := db.First(
+               collectorState,
+               dal.Where("raw_data_table = ? AND raw_data_params = ?", 
rawTable, preState.Params),
+       )
+       if err != nil {
+               if db.IsErrorNotFound(err) || isStateTableNotReadyError(err) {
+                       return preState, nil
+               }
+               return nil, errors.Default.Wrap(err, "failed to load collector 
state for subtask bootstrap")
+       }
+
+       if collectorState.LatestSuccessStart != nil {
+               preState.PrevStartedAt = collectorState.LatestSuccessStart
+       }
+       if preState.TimeAfter == nil && collectorState.TimeAfter != nil {
+               preState.TimeAfter = collectorState.TimeAfter
+       }
+
+       return preState, nil
+}
+
+func isStateTableNotReadyError(err error) bool {
+       if err == nil {
+               return false
+       }
+       msg := strings.ToLower(err.Error())
+       return strings.Contains(msg, "_devlake_collector_latest_state") &&
+               (strings.Contains(msg, "doesn't exist") ||
+                       strings.Contains(msg, "does not exist") ||
+                       strings.Contains(msg, "unknown table") ||
+                       strings.Contains(msg, "no such table"))
+}
+
 // calculateStateManagerIncrementalMode tries to calculate whether state 
manager should run in incremental mode and returns the state manager's 'since' 
time.
 func calculateStateManagerIncrementalMode(syncPolicy *models.SyncPolicy, 
preState *models.SubtaskState, newSubtaskConfig string) (bool, *time.Time) {
        if preState == nil || syncPolicy == nil {
diff --git a/backend/helpers/pluginhelper/api/subtask_state_manager_test.go 
b/backend/helpers/pluginhelper/api/subtask_state_manager_test.go
index 06eeaa433..f4ba733ea 100644
--- a/backend/helpers/pluginhelper/api/subtask_state_manager_test.go
+++ b/backend/helpers/pluginhelper/api/subtask_state_manager_test.go
@@ -30,6 +30,8 @@ import (
        "github.com/stretchr/testify/mock"
 )
 
+const testGithubScopeParams = `{"ConnectionId":1,"Name":"test/repo"}`
+
 func TestSubtaskStateManager(t *testing.T) {
        time0 := errors.Must1(time.Parse(time.RFC3339, "2020-01-01T00:00:00Z"))
        time1 := errors.Must1(time.Parse(time.RFC3339, "2021-01-01T00:00:00Z"))
@@ -187,3 +189,170 @@ func TestSubtaskStateManager(t *testing.T) {
                })
        }
 }
+
+func TestBootstrapStateFromCollectorStateIfNeeded(t *testing.T) {
+       latest := errors.Must1(time.Parse(time.RFC3339, "2026-04-27T18:05:31Z"))
+       timeAfter := errors.Must1(time.Parse(time.RFC3339, 
"2024-12-20T00:00:00Z"))
+
+       t.Run("bootstrap prev_started_at and time_after from collector state", 
func(t *testing.T) {
+               mockDal := new(mockdal.Dal)
+               mockDal.On("HasTable", mock.Anything).Return(true).Once()
+               mockDal.On("First", mock.Anything, mock.Anything).Run(func(args 
mock.Arguments) {
+                       dst := args.Get(0).(*models.CollectorLatestState)
+                       dst.LatestSuccessStart = &latest
+                       dst.TimeAfter = &timeAfter
+               }).Return(nil).Once()
+
+               state := &models.SubtaskState{
+                       Plugin:  "github",
+                       Subtask: "Convert Jobs",
+                       Params:  testGithubScopeParams,
+               }
+               args := &SubtaskCommonArgs{Table: "github_api_jobs"}
+
+               bootstrapped, err := 
bootstrapStateFromCollectorStateIfNeeded(mockDal, state, args)
+               assert.Nil(t, err)
+               if assert.NotNil(t, bootstrapped.PrevStartedAt) {
+                       assert.True(t, bootstrapped.PrevStartedAt.Equal(latest))
+               }
+               if assert.NotNil(t, bootstrapped.TimeAfter) {
+                       assert.True(t, bootstrapped.TimeAfter.Equal(timeAfter))
+               }
+               mockDal.AssertExpectations(t)
+       })
+
+       t.Run("do not overwrite existing prev_started_at", func(t *testing.T) {
+               mockDal := new(mockdal.Dal)
+
+               existing := errors.Must1(time.Parse(time.RFC3339, 
"2026-04-27T18:06:37Z"))
+               state := &models.SubtaskState{
+                       Plugin:        "github",
+                       Subtask:       "Convert Jobs",
+                       Params:        testGithubScopeParams,
+                       PrevStartedAt: &existing,
+               }
+               args := &SubtaskCommonArgs{Table: "github_api_jobs"}
+
+               bootstrapped, err := 
bootstrapStateFromCollectorStateIfNeeded(mockDal, state, args)
+               assert.Nil(t, err)
+               if assert.NotNil(t, bootstrapped.PrevStartedAt) {
+                       assert.True(t, 
bootstrapped.PrevStartedAt.Equal(existing))
+               }
+               mockDal.AssertNotCalled(t, "HasTable", mock.Anything)
+               mockDal.AssertNotCalled(t, "First", mock.Anything, 
mock.Anything)
+       })
+
+       t.Run("ignore not found collector state", func(t *testing.T) {
+               mockDal := new(mockdal.Dal)
+               notFoundErr := errors.Default.New("record not found")
+               mockDal.On("HasTable", mock.Anything).Return(true).Once()
+               mockDal.On("First", mock.Anything, 
mock.Anything).Return(notFoundErr).Once()
+               mockDal.On("IsErrorNotFound", notFoundErr).Return(true).Once()
+
+               state := &models.SubtaskState{
+                       Plugin:  "github",
+                       Subtask: "Convert Jobs",
+                       Params:  testGithubScopeParams,
+               }
+               args := &SubtaskCommonArgs{Table: "github_api_jobs"}
+
+               bootstrapped, err := 
bootstrapStateFromCollectorStateIfNeeded(mockDal, state, args)
+               assert.Nil(t, err)
+               assert.Nil(t, bootstrapped.PrevStartedAt)
+               assert.Nil(t, bootstrapped.TimeAfter)
+               mockDal.AssertExpectations(t)
+       })
+
+       t.Run("return error when collector state query fails", func(t 
*testing.T) {
+               mockDal := new(mockdal.Dal)
+               dbErr := errors.Default.New("db unavailable")
+               mockDal.On("HasTable", mock.Anything).Return(true).Once()
+               mockDal.On("First", mock.Anything, 
mock.Anything).Return(dbErr).Once()
+               mockDal.On("IsErrorNotFound", dbErr).Return(false).Once()
+
+               state := &models.SubtaskState{
+                       Plugin:  "github",
+                       Subtask: "Convert Jobs",
+                       Params:  testGithubScopeParams,
+               }
+               args := &SubtaskCommonArgs{Table: "github_api_jobs"}
+
+               bootstrapped, err := 
bootstrapStateFromCollectorStateIfNeeded(mockDal, state, args)
+               assert.Nil(t, bootstrapped)
+               assert.NotNil(t, err)
+               assert.Contains(t, err.Error(), "failed to load collector state 
for subtask bootstrap")
+               mockDal.AssertExpectations(t)
+       })
+
+       t.Run("ignore missing collector table errors", func(t *testing.T) {
+               mockDal := new(mockdal.Dal)
+               mockDal.On("HasTable", mock.Anything).Return(true).Once()
+               tableErr := errors.Default.New("Error 1146 (42S02): Table 
'lake._devlake_collector_latest_state' doesn't exist")
+               mockDal.On("First", mock.Anything, 
mock.Anything).Return(tableErr).Once()
+               mockDal.On("IsErrorNotFound", tableErr).Return(false).Once()
+
+               state := &models.SubtaskState{
+                       Plugin:  "github",
+                       Subtask: "Convert Jobs",
+                       Params:  testGithubScopeParams,
+               }
+               args := &SubtaskCommonArgs{Table: "github_api_jobs"}
+
+               bootstrapped, err := 
bootstrapStateFromCollectorStateIfNeeded(mockDal, state, args)
+               assert.Nil(t, err)
+               assert.NotNil(t, bootstrapped)
+               assert.Nil(t, bootstrapped.PrevStartedAt)
+               assert.Nil(t, bootstrapped.TimeAfter)
+               mockDal.AssertExpectations(t)
+       })
+
+       t.Run("do not ignore missing unrelated table errors", func(t 
*testing.T) {
+               mockDal := new(mockdal.Dal)
+               mockDal.On("HasTable", mock.Anything).Return(true).Once()
+               tableErr := errors.Default.New("Error 1146 (42S02): Table 
'lake._tool_github_issues' doesn't exist")
+               mockDal.On("First", mock.Anything, 
mock.Anything).Return(tableErr).Once()
+               mockDal.On("IsErrorNotFound", tableErr).Return(false).Once()
+
+               state := &models.SubtaskState{
+                       Plugin:  "github",
+                       Subtask: "Convert Jobs",
+                       Params:  testGithubScopeParams,
+               }
+               args := &SubtaskCommonArgs{Table: "github_api_jobs"}
+
+               bootstrapped, err := 
bootstrapStateFromCollectorStateIfNeeded(mockDal, state, args)
+               assert.Nil(t, bootstrapped)
+               assert.NotNil(t, err)
+               assert.Contains(t, err.Error(), "failed to load collector state 
for subtask bootstrap")
+               mockDal.AssertExpectations(t)
+       })
+
+       t.Run("skip collector lookup when collector state table does not 
exist", func(t *testing.T) {
+               mockDal := new(mockdal.Dal)
+               mockDal.On("HasTable", mock.Anything).Return(false).Once()
+
+               state := &models.SubtaskState{
+                       Plugin:  "github",
+                       Subtask: "Convert Jobs",
+                       Params:  testGithubScopeParams,
+               }
+               args := &SubtaskCommonArgs{Table: "github_api_jobs"}
+
+               bootstrapped, err := 
bootstrapStateFromCollectorStateIfNeeded(mockDal, state, args)
+               assert.Nil(t, err)
+               assert.NotNil(t, bootstrapped)
+               assert.Nil(t, bootstrapped.PrevStartedAt)
+               assert.Nil(t, bootstrapped.TimeAfter)
+               mockDal.AssertNotCalled(t, "First", mock.Anything, 
mock.Anything)
+               mockDal.AssertExpectations(t)
+       })
+}
+
+func TestIsStateTableNotReadyError(t *testing.T) {
+       assert.False(t, isStateTableNotReadyError(nil))
+       assert.True(t, isStateTableNotReadyError(errors.Default.New("Error 1146 
(42S02): Table 'lake._devlake_collector_latest_state' doesn't exist")))
+       assert.True(t, isStateTableNotReadyError(errors.Default.New("pq: 
relation \"_devlake_collector_latest_state\" does not exist")))
+       assert.True(t, isStateTableNotReadyError(errors.Default.New("no such 
table: _devlake_collector_latest_state")))
+       assert.False(t, isStateTableNotReadyError(errors.Default.New("Error 
1146 (42S02): Table 'lake._tool_github_issues' doesn't exist")))
+       assert.False(t, isStateTableNotReadyError(errors.Default.New("db 
unavailable")))
+}
diff --git a/backend/impls/context/default_exec_context.go 
b/backend/impls/context/default_exec_context.go
index d1a2ceda0..2d372f5b9 100644
--- a/backend/impls/context/default_exec_context.go
+++ b/backend/impls/context/default_exec_context.go
@@ -96,6 +96,10 @@ func (c *defaultExecContext) IncProgress(progressType 
plugin.ProgressType, quant
        }
 }
 
+func (c *defaultExecContext) GetProgress() int {
+       return int(atomic.LoadInt64(&c.current))
+}
+
 func (c *defaultExecContext) fork(name string) *defaultExecContext {
        return newDefaultExecContext(
                c.ctx,
diff --git a/backend/plugins/github/tasks/account_convertor.go 
b/backend/plugins/github/tasks/account_convertor.go
index dfd9a929e..e86780d27 100644
--- a/backend/plugins/github/tasks/account_convertor.go
+++ b/backend/plugins/github/tasks/account_convertor.go
@@ -18,12 +18,10 @@ limitations under the License.
 package tasks
 
 import (
-       "reflect"
        "strings"
 
        "github.com/apache/incubator-devlake/core/dal"
        "github.com/apache/incubator-devlake/core/errors"
-       "github.com/apache/incubator-devlake/core/models/common"
        "github.com/apache/incubator-devlake/core/models/domainlayer"
        
"github.com/apache/incubator-devlake/core/models/domainlayer/crossdomain"
        "github.com/apache/incubator-devlake/core/models/domainlayer/didgen"
@@ -49,50 +47,44 @@ var ConvertAccountsMeta = plugin.SubTaskMeta{
        ProductTables: []string{crossdomain.Account{}.TableName()},
 }
 
-type GithubAccountWithOrg struct {
-       models.GithubAccount
-       Login string `json:"login" gorm:"type:varchar(255)"`
-       common.NoPKModel
-}
-
 func ConvertAccounts(taskCtx plugin.SubTaskContext) errors.Error {
        db := taskCtx.GetDal()
        data := taskCtx.GetData().(*GithubTaskData)
 
-       cursor, err := db.Cursor(
-               dal.Select("_tool_github_accounts.*"),
-               dal.From(&models.GithubAccount{}),
-               dal.Where(
-                       "repo_github_id = ? and 
_tool_github_accounts.connection_id=?",
-                       data.Options.GithubId,
-                       data.Options.ConnectionId,
-               ),
-               dal.Join(`left join _tool_github_repo_accounts gra on (
-                       _tool_github_accounts.connection_id = gra.connection_id
-                       AND _tool_github_accounts.id = gra.account_id
-               )`),
-       )
-       if err != nil {
-               return err
-       }
-       defer cursor.Close()
-
        accountIdGen := didgen.NewDomainIdGenerator(&models.GithubAccount{})
 
-       converter, err := api.NewDataConverter(api.DataConverterArgs{
-               InputRowType: reflect.TypeOf(models.GithubAccount{}),
-               Input:        cursor,
-               RawDataSubTaskArgs: api.RawDataSubTaskArgs{
-                       Ctx: taskCtx,
+       converter, err := 
api.NewStatefulDataConverter(&api.StatefulDataConverterArgs[models.GithubAccount]{
+               SubtaskCommonArgs: &api.SubtaskCommonArgs{
+                       SubTaskContext: taskCtx,
+                       Table:          RAW_ACCOUNT_TABLE,
                        Params: GithubApiParams{
                                ConnectionId: data.Options.ConnectionId,
                                Name:         data.Options.Name,
                        },
-                       Table: RAW_ACCOUNT_TABLE,
                },
-               Convert: func(inputRow interface{}) ([]interface{}, 
errors.Error) {
-                       githubUser := inputRow.(*models.GithubAccount)
-
+               Input: func(stateManager *api.SubtaskStateManager) (dal.Rows, 
errors.Error) {
+                       clauses := []dal.Clause{
+                               dal.Select("_tool_github_accounts.*"),
+                               dal.From(&models.GithubAccount{}),
+                               dal.Where(
+                                       "repo_github_id = ? and 
_tool_github_accounts.connection_id=?",
+                                       data.Options.GithubId,
+                                       data.Options.ConnectionId,
+                               ),
+                               dal.Join(`left join _tool_github_repo_accounts 
gra on (
+                                       _tool_github_accounts.connection_id = 
gra.connection_id
+                                       AND _tool_github_accounts.id = 
gra.account_id
+                               )`),
+                       }
+                       if stateManager.IsIncremental() {
+                               since := stateManager.GetSince()
+                               if since != nil {
+                                       clauses = append(clauses, 
dal.Where("_tool_github_accounts.updated_at >= ?", since))
+                               }
+                       }
+                       return db.Cursor(clauses...)
+               },
+               Convert: func(githubUser *models.GithubAccount) ([]interface{}, 
errors.Error) {
                        // query related orgs
                        var orgs []string
                        err := db.Pluck(`org_login`, &orgs,
diff --git a/backend/plugins/github/tasks/account_extractor.go 
b/backend/plugins/github/tasks/account_extractor.go
index 88e4e8efb..27500402b 100644
--- a/backend/plugins/github/tasks/account_extractor.go
+++ b/backend/plugins/github/tasks/account_extractor.go
@@ -18,7 +18,6 @@ limitations under the License.
 package tasks
 
 import (
-       "encoding/json"
        "time"
 
        "github.com/apache/incubator-devlake/core/errors"
@@ -61,39 +60,32 @@ type DetailGithubAccountResponse struct {
 
 func ExtractAccounts(taskCtx plugin.SubTaskContext) errors.Error {
        data := taskCtx.GetData().(*GithubTaskData)
-       extractor, err := api.NewApiExtractor(api.ApiExtractorArgs{
-               RawDataSubTaskArgs: api.RawDataSubTaskArgs{
-                       Ctx: taskCtx,
+       extractor, err := 
api.NewStatefulApiExtractor(&api.StatefulApiExtractorArgs[DetailGithubAccountResponse]{
+               SubtaskCommonArgs: &api.SubtaskCommonArgs{
+                       SubTaskContext: taskCtx,
                        Params: GithubApiParams{
                                ConnectionId: data.Options.ConnectionId,
                                Name:         data.Options.Name,
                        },
                        Table: RAW_ACCOUNT_TABLE,
                },
-               Extract: func(row *api.RawData) ([]interface{}, errors.Error) {
-                       apiAccount := &DetailGithubAccountResponse{}
-                       err := errors.Convert(json.Unmarshal(row.Data, 
apiAccount))
-                       if err != nil {
-                               return nil, err
-                       }
-                       results := make([]interface{}, 0, 1)
-                       if apiAccount.Id == 0 {
+               Extract: func(body *DetailGithubAccountResponse, row 
*api.RawData) ([]any, errors.Error) {
+                       if body.Id == 0 {
                                return nil, nil
                        }
                        githubAccount := &models.GithubAccount{
                                ConnectionId: data.Options.ConnectionId,
-                               Id:           apiAccount.Id,
-                               Login:        apiAccount.Login,
-                               Name:         apiAccount.Name,
-                               Company:      apiAccount.Company,
-                               Email:        apiAccount.Email,
-                               AvatarUrl:    apiAccount.AvatarUrl,
-                               Url:          apiAccount.Url,
-                               HtmlUrl:      apiAccount.HtmlUrl,
-                               Type:         apiAccount.Type,
+                               Id:           body.Id,
+                               Login:        body.Login,
+                               Name:         body.Name,
+                               Company:      body.Company,
+                               Email:        body.Email,
+                               AvatarUrl:    body.AvatarUrl,
+                               Url:          body.Url,
+                               HtmlUrl:      body.HtmlUrl,
+                               Type:         body.Type,
                        }
-                       results = append(results, githubAccount)
-                       return results, nil
+                       return []any{githubAccount}, nil
                },
        })
 
diff --git a/backend/plugins/github/tasks/account_org_extractor.go 
b/backend/plugins/github/tasks/account_org_extractor.go
index 8aa69b664..91f9a7b8c 100644
--- a/backend/plugins/github/tasks/account_org_extractor.go
+++ b/backend/plugins/github/tasks/account_org_extractor.go
@@ -50,28 +50,23 @@ type GithubAccountOrgsResponse struct {
 
 func ExtractAccountOrg(taskCtx plugin.SubTaskContext) errors.Error {
        data := taskCtx.GetData().(*GithubTaskData)
-       extractor, err := api.NewApiExtractor(api.ApiExtractorArgs{
-               RawDataSubTaskArgs: api.RawDataSubTaskArgs{
-                       Ctx: taskCtx,
+       extractor, err := 
api.NewStatefulApiExtractor(&api.StatefulApiExtractorArgs[[]GithubAccountOrgsResponse]{
+               SubtaskCommonArgs: &api.SubtaskCommonArgs{
+                       SubTaskContext: taskCtx,
                        Params: GithubApiParams{
                                ConnectionId: data.Options.ConnectionId,
                                Name:         data.Options.Name,
                        },
                        Table: RAW_ACCOUNT_ORG_TABLE,
                },
-               Extract: func(row *api.RawData) ([]interface{}, errors.Error) {
-                       apiAccountOrgs := &[]GithubAccountOrgsResponse{}
-                       err := json.Unmarshal(row.Data, apiAccountOrgs)
-                       if err != nil {
-                               return nil, errors.Convert(err)
-                       }
+               Extract: func(body *[]GithubAccountOrgsResponse, row 
*api.RawData) ([]any, errors.Error) {
                        simpleAccount := &SimpleAccountWithId{}
-                       err = json.Unmarshal(row.Input, simpleAccount)
+                       err := errors.Convert(json.Unmarshal(row.Input, 
simpleAccount))
                        if err != nil {
-                               return nil, errors.Convert(err)
+                               return nil, err
                        }
-                       results := make([]interface{}, 0, len(*apiAccountOrgs))
-                       for _, apiAccountOrg := range *apiAccountOrgs {
+                       results := make([]interface{}, 0, len(*body))
+                       for _, apiAccountOrg := range *body {
                                githubAccount := &models.GithubAccountOrg{
                                        ConnectionId: data.Options.ConnectionId,
                                        AccountId:    simpleAccount.AccountId,
diff --git a/backend/plugins/github/tasks/cicd_job_convertor.go 
b/backend/plugins/github/tasks/cicd_job_convertor.go
index 81ef9cd4d..2dcfbce37 100644
--- a/backend/plugins/github/tasks/cicd_job_convertor.go
+++ b/backend/plugins/github/tasks/cicd_job_convertor.go
@@ -18,8 +18,6 @@ limitations under the License.
 package tasks
 
 import (
-       "reflect"
-
        "github.com/apache/incubator-devlake/core/dal"
        "github.com/apache/incubator-devlake/core/errors"
        "github.com/apache/incubator-devlake/core/models/domainlayer"
@@ -55,37 +53,38 @@ type SimpleBranch struct {
 
 func ConvertJobs(taskCtx plugin.SubTaskContext) (err errors.Error) {
        db := taskCtx.GetDal()
+       logger := taskCtx.GetLogger()
        data := taskCtx.GetData().(*GithubTaskData)
        repoId := data.Options.GithubId
-       if err != nil {
-               return err
-       }
-       job := &models.GithubJob{}
-       cursor, err := db.Cursor(
-               dal.From(job),
-               dal.Where("repo_id = ? and connection_id=?", repoId, 
data.Options.ConnectionId),
-       )
-       if err != nil {
-               return err
-       }
-       defer cursor.Close()
+
        jobIdGen := didgen.NewDomainIdGenerator(&models.GithubJob{})
        runIdGen := didgen.NewDomainIdGenerator(&models.GithubRun{})
        repoIdGen := didgen.NewDomainIdGenerator(&models.GithubRepo{})
-       converter, err := api.NewDataConverter(api.DataConverterArgs{
-               RawDataSubTaskArgs: api.RawDataSubTaskArgs{
-                       Ctx: taskCtx,
+
+       converter, err := 
api.NewStatefulDataConverter(&api.StatefulDataConverterArgs[models.GithubJob]{
+               SubtaskCommonArgs: &api.SubtaskCommonArgs{
+                       SubTaskContext: taskCtx,
+                       Table:          RAW_JOB_TABLE,
                        Params: GithubApiParams{
                                ConnectionId: data.Options.ConnectionId,
                                Name:         data.Options.Name,
                        },
-                       Table: RAW_JOB_TABLE,
                },
-               InputRowType: reflect.TypeOf(models.GithubJob{}),
-               Input:        cursor,
-               Convert: func(inputRow interface{}) ([]interface{}, 
errors.Error) {
-                       line := inputRow.(*models.GithubJob)
-
+               Input: func(stateManager *api.SubtaskStateManager) (dal.Rows, 
errors.Error) {
+                       clauses := []dal.Clause{
+                               dal.From(&models.GithubJob{}),
+                               dal.Where("_tool_github_jobs.repo_id = ? and 
_tool_github_jobs.connection_id = ?", repoId, data.Options.ConnectionId),
+                       }
+                       if stateManager.IsIncremental() {
+                               since := stateManager.GetSince()
+                               if since != nil {
+                                       logger.Info("[Convert Jobs] incremental 
mode using _tool_github_jobs.updated_at >= %v", since)
+                                       clauses = append(clauses, 
dal.Where("_tool_github_jobs.updated_at >= ?", since))
+                               }
+                       }
+                       return db.Cursor(clauses...)
+               },
+               Convert: func(line *models.GithubJob) ([]interface{}, 
errors.Error) {
                        // Skip jobs with no started_at value (workaround for 
https://github.com/apache/incubator-devlake/issues/8442)
                        if line.StartedAt == nil {
                                return nil, nil
diff --git a/backend/plugins/github/tasks/cicd_job_extractor.go 
b/backend/plugins/github/tasks/cicd_job_extractor.go
index 5701476bc..6cfab02f3 100644
--- a/backend/plugins/github/tasks/cicd_job_extractor.go
+++ b/backend/plugins/github/tasks/cicd_job_extractor.go
@@ -18,7 +18,6 @@ limitations under the License.
 package tasks
 
 import (
-       "encoding/json"
        "strings"
 
        "github.com/apache/incubator-devlake/core/errors"
@@ -46,23 +45,16 @@ func ExtractJobs(taskCtx plugin.SubTaskContext) 
errors.Error {
        data := taskCtx.GetData().(*GithubTaskData)
        repoId := data.Options.GithubId
 
-       extractor, err := api.NewApiExtractor(api.ApiExtractorArgs{
-               RawDataSubTaskArgs: api.RawDataSubTaskArgs{
-                       Ctx: taskCtx,
+       extractor, err := 
api.NewStatefulApiExtractor(&api.StatefulApiExtractorArgs[models.GithubJob]{
+               SubtaskCommonArgs: &api.SubtaskCommonArgs{
+                       SubTaskContext: taskCtx,
                        Params: GithubApiParams{
                                ConnectionId: data.Options.ConnectionId,
                                Name:         data.Options.Name,
                        },
                        Table: RAW_JOB_TABLE,
                },
-               Extract: func(row *api.RawData) ([]interface{}, errors.Error) {
-                       githubJob := &models.GithubJob{}
-                       err := errors.Convert(json.Unmarshal(row.Data, 
githubJob))
-                       if err != nil {
-                               return nil, err
-                       }
-
-                       results := make([]interface{}, 0, 1)
+               Extract: func(githubJob *models.GithubJob, row *api.RawData) 
([]any, errors.Error) {
                        githubJobResult := &models.GithubJob{
                                ConnectionId:  data.Options.ConnectionId,
                                RepoId:        repoId,
@@ -87,8 +79,7 @@ func ExtractJobs(taskCtx plugin.SubTaskContext) errors.Error {
                                Type:          
data.RegexEnricher.ReturnNameIfMatched(devops.DEPLOYMENT, githubJob.Name),
                                Environment:   
data.RegexEnricher.ReturnNameIfOmittedOrMatched(devops.PRODUCTION, 
githubJob.Name),
                        }
-                       results = append(results, githubJobResult)
-                       return results, nil
+                       return []any{githubJobResult}, nil
                },
        })
 
diff --git a/backend/plugins/github/tasks/cicd_run_collector.go 
b/backend/plugins/github/tasks/cicd_run_collector.go
index 44c816ca2..9359b98de 100644
--- a/backend/plugins/github/tasks/cicd_run_collector.go
+++ b/backend/plugins/github/tasks/cicd_run_collector.go
@@ -24,10 +24,12 @@ import (
        "net/url"
        "time"
 
+       "github.com/apache/incubator-devlake/core/dal"
        "github.com/apache/incubator-devlake/core/errors"
        "github.com/apache/incubator-devlake/core/log"
        "github.com/apache/incubator-devlake/core/plugin"
        helper "github.com/apache/incubator-devlake/helpers/pluginhelper/api"
+       "github.com/apache/incubator-devlake/plugins/github/models"
 )
 
 func init() {
@@ -89,6 +91,24 @@ func CollectRuns(taskCtx plugin.SubTaskContext) errors.Error 
{
        // `windowStart` past the previously collected second 
(inclusive-both-ends), while
        // fullsync + TimeAfter keeps the user-specified bound inclusive.
        createdAfter := manager.GetSince()
+       sinceSource := "state_since"
+       syncPolicy := taskCtx.TaskContext().SyncPolicy()
+       if createdAfter == nil {
+               sinceSource = "none"
+       }
+       if createdAfter == nil && (syncPolicy == nil || !syncPolicy.FullSync) {
+               fallbackSince, err := loadLatestRunUpdatedAt(taskCtx, 
data.Options.ConnectionId, data.Options.GithubId)
+               if err != nil {
+                       return err
+               }
+               if fallbackSince != nil {
+                       createdAfter = fallbackSince
+                       sinceSource = "tool_runs_fallback"
+                       logger.Info("cicd_run_collector: collector state 
missing; bootstrapping since from existing _tool_github_runs at %s", 
fallbackSince.UTC().Format(time.RFC3339))
+               } else {
+                       logger.Debug("cicd_run_collector: collector state 
missing and no _tool_github_runs timestamp found")
+               }
+       }
        untilPtr := manager.GetUntil()
        *untilPtr = untilPtr.Truncate(time.Second)
        until := *untilPtr
@@ -102,12 +122,14 @@ func CollectRuns(taskCtx plugin.SubTaskContext) 
errors.Error {
        } else {
                // 2018-01-01 conservatively predates GitHub Actions' late-2019 
GA.
                windowStart = time.Date(2018, 1, 1, 0, 0, 0, 0, time.UTC)
+               sinceSource = "epoch_fullsync"
        }
 
-       logger.Info("cicd_run_collector: collecting workflow runs in [%s, %s] 
(incremental=%v)",
+       logger.Info("cicd_run_collector: collecting workflow runs in [%s, %s] 
(incremental=%v, since_source=%s)",
                windowStart.Format(githubTimeLayout),
                until.Format(githubTimeLayout),
-               manager.IsIncremental())
+               manager.IsIncremental(),
+               sinceSource)
 
        leafWindows, err := newLeafWindowBuilder(taskCtx, 
data).build(windowStart, until)
        if err != nil {
@@ -122,6 +144,28 @@ func CollectRuns(taskCtx plugin.SubTaskContext) 
errors.Error {
        return manager.Execute()
 }
 
+func loadLatestRunUpdatedAt(taskCtx plugin.SubTaskContext, connectionId 
uint64, repoId int) (*time.Time, errors.Error) {
+       db := taskCtx.GetDal()
+       latest := &models.GithubRun{}
+       err := db.First(
+               latest,
+               dal.Where("connection_id = ? AND repo_id = ? AND 
github_updated_at IS NOT NULL", connectionId, repoId),
+               dal.Orderby("github_updated_at DESC"),
+               dal.Limit(1),
+       )
+       if err != nil {
+               if db.IsErrorNotFound(err) {
+                       return nil, nil
+               }
+               return nil, err
+       }
+       if latest.GithubUpdatedAt == nil {
+               return nil, nil
+       }
+       fallback := latest.GithubUpdatedAt.UTC()
+       return &fallback, nil
+}
+
 // buildRunsQuery assembles the filtered-mode query for a single leaf 
TimeWindow.
 // Shared between registerCollectorForLeafWindows and tests.
 func buildRunsQuery(reqData *helper.RequestData) (url.Values, errors.Error) {
diff --git a/backend/plugins/github/tasks/cicd_run_collector_test.go 
b/backend/plugins/github/tasks/cicd_run_collector_test.go
index d69b5c73b..a7a12abe6 100644
--- a/backend/plugins/github/tasks/cicd_run_collector_test.go
+++ b/backend/plugins/github/tasks/cicd_run_collector_test.go
@@ -34,12 +34,64 @@ import (
        "github.com/apache/incubator-devlake/helpers/unithelper"
        mockdal "github.com/apache/incubator-devlake/mocks/core/dal"
        mockapi 
"github.com/apache/incubator-devlake/mocks/helpers/pluginhelper/api"
+       "github.com/apache/incubator-devlake/plugins/github/models"
 
        "github.com/stretchr/testify/assert"
        "github.com/stretchr/testify/mock"
        "github.com/stretchr/testify/require"
 )
 
+func TestCicdRunLoadLatestRunUpdatedAt_ReturnsLatestTimestamp(t *testing.T) {
+       connectionId := uint64(1)
+       repoId := 101
+       latestTs := time.Date(2025, 4, 1, 10, 11, 12, 0, time.UTC)
+
+       mockDal := new(mockdal.Dal)
+       mockDal.On("First", mock.Anything, mock.Anything, mock.Anything, 
mock.Anything).Run(func(args mock.Arguments) {
+               dst := args.Get(0).(*models.GithubRun)
+               dst.ConnectionId = connectionId
+               dst.RepoId = repoId
+               dst.GithubUpdatedAt = &latestTs
+       }).Return(nil).Once()
+
+       ctx := unithelper.DummySubTaskContext(mockDal)
+       since, err := loadLatestRunUpdatedAt(ctx, connectionId, repoId)
+
+       require.Nil(t, err)
+       require.NotNil(t, since)
+       assert.True(t, since.Equal(latestTs))
+       mockDal.AssertExpectations(t)
+}
+
+func TestCicdRunLoadLatestRunUpdatedAt_NotFoundReturnsNil(t *testing.T) {
+       mockDal := new(mockdal.Dal)
+       notFoundErr := errors.Default.New("record not found")
+       mockDal.On("First", mock.Anything, mock.Anything, mock.Anything, 
mock.Anything).Return(notFoundErr).Once()
+       mockDal.On("IsErrorNotFound", notFoundErr).Return(true).Once()
+
+       ctx := unithelper.DummySubTaskContext(mockDal)
+       since, err := loadLatestRunUpdatedAt(ctx, 1, 101)
+
+       require.Nil(t, err)
+       assert.Nil(t, since)
+       mockDal.AssertExpectations(t)
+}
+
+func TestCicdRunLoadLatestRunUpdatedAt_PropagatesError(t *testing.T) {
+       mockDal := new(mockdal.Dal)
+       dbErr := errors.Default.New("db unavailable")
+       mockDal.On("First", mock.Anything, mock.Anything, mock.Anything, 
mock.Anything).Return(dbErr).Once()
+       mockDal.On("IsErrorNotFound", dbErr).Return(false).Once()
+
+       ctx := unithelper.DummySubTaskContext(mockDal)
+       since, err := loadLatestRunUpdatedAt(ctx, 1, 101)
+
+       assert.Nil(t, since)
+       require.NotNil(t, err)
+       assert.Contains(t, err.Error(), "db unavailable")
+       mockDal.AssertExpectations(t)
+}
+
 // newTestBuilder constructs a leafWindowBuilder with a stubbed probe for unit 
testing.
 func newTestBuilder(probe probeFunc) *leafWindowBuilder {
        mockDal := new(mockdal.Dal)
diff --git a/backend/plugins/github/tasks/cicd_run_convertor.go 
b/backend/plugins/github/tasks/cicd_run_convertor.go
index 1cdf9c6da..69911482e 100644
--- a/backend/plugins/github/tasks/cicd_run_convertor.go
+++ b/backend/plugins/github/tasks/cicd_run_convertor.go
@@ -18,7 +18,6 @@ limitations under the License.
 package tasks
 
 import (
-       "reflect"
        "time"
 
        "github.com/apache/incubator-devlake/core/dal"
@@ -63,31 +62,33 @@ func ConvertRuns(taskCtx plugin.SubTaskContext) 
errors.Error {
                return err
        }
 
-       pipeline := &models.GithubRun{}
-       cursor, err := db.Cursor(
-               dal.Select("*"),
-               dal.From(pipeline),
-               dal.Where("repo_id = ? and connection_id=?", repoId, 
data.Options.ConnectionId),
-       )
-       if err != nil {
-               return err
-       }
-       defer cursor.Close()
        repoIdGen := didgen.NewDomainIdGenerator(&models.GithubRepo{})
        runIdGen := didgen.NewDomainIdGenerator(&models.GithubRun{})
-       converter, err := api.NewDataConverter(api.DataConverterArgs{
-               RawDataSubTaskArgs: api.RawDataSubTaskArgs{
-                       Ctx: taskCtx,
+
+       converter, err := 
api.NewStatefulDataConverter(&api.StatefulDataConverterArgs[models.GithubRun]{
+               SubtaskCommonArgs: &api.SubtaskCommonArgs{
+                       SubTaskContext: taskCtx,
+                       Table:          RAW_RUN_TABLE,
                        Params: GithubApiParams{
                                ConnectionId: data.Options.ConnectionId,
                                Name:         data.Options.Name,
                        },
-                       Table: RAW_RUN_TABLE,
                },
-               InputRowType: reflect.TypeOf(models.GithubRun{}),
-               Input:        cursor,
-               Convert: func(inputRow interface{}) ([]interface{}, 
errors.Error) {
-                       line := inputRow.(*models.GithubRun)
+               Input: func(stateManager *api.SubtaskStateManager) (dal.Rows, 
errors.Error) {
+                       clauses := []dal.Clause{
+                               dal.Select("*"),
+                               dal.From(&models.GithubRun{}),
+                               dal.Where("repo_id = ? and connection_id = ?", 
repoId, data.Options.ConnectionId),
+                       }
+                       if stateManager.IsIncremental() {
+                               since := stateManager.GetSince()
+                               if since != nil {
+                                       clauses = append(clauses, 
dal.Where("github_updated_at >= ?", since))
+                               }
+                       }
+                       return db.Cursor(clauses...)
+               },
+               Convert: func(line *models.GithubRun) ([]interface{}, 
errors.Error) {
                        createdAt := time.Now()
                        if line.GithubCreatedAt != nil {
                                createdAt = *line.GithubCreatedAt
diff --git a/backend/plugins/github/tasks/cicd_run_extractor.go 
b/backend/plugins/github/tasks/cicd_run_extractor.go
index 1e6a3f41a..3a474c64c 100644
--- a/backend/plugins/github/tasks/cicd_run_extractor.go
+++ b/backend/plugins/github/tasks/cicd_run_extractor.go
@@ -18,8 +18,6 @@ limitations under the License.
 package tasks
 
 import (
-       "encoding/json"
-
        "github.com/apache/incubator-devlake/core/errors"
        "github.com/apache/incubator-devlake/core/models/domainlayer/devops"
        "github.com/apache/incubator-devlake/core/plugin"
@@ -45,26 +43,21 @@ func ExtractRuns(taskCtx plugin.SubTaskContext) 
errors.Error {
        data := taskCtx.GetData().(*GithubTaskData)
        repoId := data.Options.GithubId
 
-       extractor, err := api.NewApiExtractor(api.ApiExtractorArgs{
-               RawDataSubTaskArgs: api.RawDataSubTaskArgs{
-                       Ctx: taskCtx,
+       extractor, err := 
api.NewStatefulApiExtractor(&api.StatefulApiExtractorArgs[models.GithubRun]{
+               SubtaskCommonArgs: &api.SubtaskCommonArgs{
+                       SubTaskContext: taskCtx,
                        Params: GithubApiParams{
                                ConnectionId: data.Options.ConnectionId,
                                Name:         data.Options.Name,
                        },
                        Table: RAW_RUN_TABLE,
                },
-               Extract: func(row *api.RawData) ([]interface{}, errors.Error) {
-                       githubRun := &models.GithubRun{}
-                       err := errors.Convert(json.Unmarshal(row.Data, 
githubRun))
-                       if err != nil {
-                               return nil, err
-                       }
-                       githubRun.RepoId = repoId
-                       githubRun.ConnectionId = data.Options.ConnectionId
-                       githubRun.Type = 
data.RegexEnricher.ReturnNameIfMatched(devops.DEPLOYMENT, githubRun.Name)
-                       githubRun.Environment = 
data.RegexEnricher.ReturnNameIfOmittedOrMatched(devops.PRODUCTION, 
githubRun.Name, githubRun.HeadBranch)
-                       return []interface{}{githubRun}, nil
+               Extract: func(body *models.GithubRun, row *api.RawData) ([]any, 
errors.Error) {
+                       body.RepoId = repoId
+                       body.ConnectionId = data.Options.ConnectionId
+                       body.Type = 
data.RegexEnricher.ReturnNameIfMatched(devops.DEPLOYMENT, body.Name)
+                       body.Environment = 
data.RegexEnricher.ReturnNameIfOmittedOrMatched(devops.PRODUCTION, body.Name, 
body.HeadBranch)
+                       return []any{body}, nil
                },
        })
 
diff --git a/backend/plugins/github/tasks/comment_collector.go 
b/backend/plugins/github/tasks/comment_collector.go
index 9f40e7736..303ecf7b1 100644
--- a/backend/plugins/github/tasks/comment_collector.go
+++ b/backend/plugins/github/tasks/comment_collector.go
@@ -22,6 +22,7 @@ import (
        "fmt"
        "net/http"
        "net/url"
+       "time"
 
        "github.com/apache/incubator-devlake/core/errors"
        "github.com/apache/incubator-devlake/core/plugin"
@@ -66,7 +67,7 @@ func CollectApiComments(taskCtx plugin.SubTaskContext) 
errors.Error {
                        query := url.Values{}
                        query.Set("state", "all")
                        if apiCollector.GetSince() != nil {
-                               query.Set("since", 
apiCollector.GetSince().String())
+                               query.Set("since", 
apiCollector.GetSince().UTC().Format(time.RFC3339))
                        }
                        query.Set("page", fmt.Sprintf("%v", reqData.Pager.Page))
                        query.Set("direction", "asc")
diff --git a/backend/plugins/github/tasks/comment_extractor.go 
b/backend/plugins/github/tasks/comment_extractor.go
index 8fc52a18b..c383e532e 100644
--- a/backend/plugins/github/tasks/comment_extractor.go
+++ b/backend/plugins/github/tasks/comment_extractor.go
@@ -59,28 +59,21 @@ type IssueComment struct {
 func ExtractApiComments(taskCtx plugin.SubTaskContext) errors.Error {
        data := taskCtx.GetData().(*GithubTaskData)
 
-       extractor, err := helper.NewApiExtractor(helper.ApiExtractorArgs{
-               RawDataSubTaskArgs: helper.RawDataSubTaskArgs{
-                       Ctx: taskCtx,
+       extractor, err := 
helper.NewStatefulApiExtractor(&helper.StatefulApiExtractorArgs[IssueComment]{
+               SubtaskCommonArgs: &helper.SubtaskCommonArgs{
+                       SubTaskContext: taskCtx,
                        Params: GithubApiParams{
                                ConnectionId: data.Options.ConnectionId,
                                Name:         data.Options.Name,
                        },
                        Table: RAW_COMMENTS_TABLE,
                },
-               Extract: func(row *helper.RawData) ([]interface{}, 
errors.Error) {
-                       apiComment := &IssueComment{}
-                       err := errors.Convert(json.Unmarshal(row.Data, 
apiComment))
-                       if err != nil {
-                               return nil, err
-                       }
-                       // need to extract 2 kinds of entities here
+               Extract: func(body *IssueComment, row *helper.RawData) ([]any, 
errors.Error) {
                        results := make([]interface{}, 0, 2)
-                       if apiComment.GithubId == 0 {
+                       if body.GithubId == 0 {
                                return nil, nil
                        }
-                       //If this is a pr, ignore
-                       issueINumber, err := 
errors.Convert01(githubUtils.GetIssueIdByIssueUrl(apiComment.IssueUrl))
+                       issueINumber, err := 
errors.Convert01(githubUtils.GetIssueIdByIssueUrl(body.IssueUrl))
                        if err != nil {
                                return nil, err
                        }
@@ -90,7 +83,7 @@ func ExtractApiComments(taskCtx plugin.SubTaskContext) 
errors.Error {
                        if err != nil && !db.IsErrorNotFound(err) {
                                return nil, err
                        }
-                       //if we can not find issues with issue number above, 
move the comments to github_pull_request_comments
+                       // if we can not find issues with issue number above, 
move the comments to github_pull_request_comments
                        if issue.GithubId == 0 {
                                pr := &models.GithubPullRequest{}
                                err = db.First(pr, dal.Where("connection_id = ? 
and number = ? and repo_id = ?", data.Options.ConnectionId, issueINumber, 
data.Options.GithubId))
@@ -99,23 +92,21 @@ func ExtractApiComments(taskCtx plugin.SubTaskContext) 
errors.Error {
                                }
                                githubPrComment := &models.GithubPrComment{
                                        ConnectionId:    
data.Options.ConnectionId,
-                                       GithubId:        apiComment.GithubId,
+                                       GithubId:        body.GithubId,
                                        PullRequestId:   pr.GithubId,
-                                       Body:            
string(apiComment.Body),
-                                       GithubCreatedAt: 
apiComment.GithubCreatedAt.ToTime(),
-                                       GithubUpdatedAt: 
apiComment.GithubUpdatedAt.ToTime(),
+                                       Body:            string(body.Body),
+                                       GithubCreatedAt: 
body.GithubCreatedAt.ToTime(),
+                                       GithubUpdatedAt: 
body.GithubUpdatedAt.ToTime(),
                                        Type:            "NORMAL",
                                }
-                               if apiComment.User != nil {
-                                       // Filter bot comments by username
-                                       if 
shouldSkipByUsername(apiComment.User.Login) {
-                                               
taskCtx.GetLogger().Debug("Skipping PR comment #%d from bot user: %s", 
apiComment.GithubId, apiComment.User.Login)
+                               if body.User != nil {
+                                       if 
shouldSkipByUsername(body.User.Login) {
+                                               
taskCtx.GetLogger().Debug("Skipping PR comment #%d from bot user: %s", 
body.GithubId, body.User.Login)
                                                return nil, nil
                                        }
-                                       githubPrComment.AuthorUsername = 
apiComment.User.Login
-                                       githubPrComment.AuthorUserId = 
apiComment.User.Id
-
-                                       githubAccount, err := 
convertAccount(apiComment.User, data.Options.GithubId, 
data.Options.ConnectionId)
+                                       githubPrComment.AuthorUsername = 
body.User.Login
+                                       githubPrComment.AuthorUserId = 
body.User.Id
+                                       githubAccount, err := 
convertAccount(body.User, data.Options.GithubId, data.Options.ConnectionId)
                                        if err != nil {
                                                return nil, err
                                        }
@@ -125,22 +116,20 @@ func ExtractApiComments(taskCtx plugin.SubTaskContext) 
errors.Error {
                        } else {
                                githubIssueComment := 
&models.GithubIssueComment{
                                        ConnectionId:    
data.Options.ConnectionId,
-                                       GithubId:        apiComment.GithubId,
+                                       GithubId:        body.GithubId,
                                        IssueId:         issue.GithubId,
-                                       Body:            
string(apiComment.Body),
-                                       GithubCreatedAt: 
apiComment.GithubCreatedAt.ToTime(),
-                                       GithubUpdatedAt: 
apiComment.GithubUpdatedAt.ToTime(),
+                                       Body:            string(body.Body),
+                                       GithubCreatedAt: 
body.GithubCreatedAt.ToTime(),
+                                       GithubUpdatedAt: 
body.GithubUpdatedAt.ToTime(),
                                }
-                               if apiComment.User != nil {
-                                       // Filter bot comments by username
-                                       if 
shouldSkipByUsername(apiComment.User.Login) {
-                                               
taskCtx.GetLogger().Debug("Skipping issue comment #%d from bot user: %s", 
apiComment.GithubId, apiComment.User.Login)
+                               if body.User != nil {
+                                       if 
shouldSkipByUsername(body.User.Login) {
+                                               
taskCtx.GetLogger().Debug("Skipping issue comment #%d from bot user: %s", 
body.GithubId, body.User.Login)
                                                return nil, nil
                                        }
-                                       githubIssueComment.AuthorUsername = 
apiComment.User.Login
-                                       githubIssueComment.AuthorUserId = 
apiComment.User.Id
-
-                                       githubAccount, err := 
convertAccount(apiComment.User, data.Options.GithubId, 
data.Options.ConnectionId)
+                                       githubIssueComment.AuthorUsername = 
body.User.Login
+                                       githubIssueComment.AuthorUserId = 
body.User.Id
+                                       githubAccount, err := 
convertAccount(body.User, data.Options.GithubId, data.Options.ConnectionId)
                                        if err != nil {
                                                return nil, err
                                        }
diff --git a/backend/plugins/github/tasks/commit_collector.go 
b/backend/plugins/github/tasks/commit_collector.go
index 019060c17..3376f2486 100644
--- a/backend/plugins/github/tasks/commit_collector.go
+++ b/backend/plugins/github/tasks/commit_collector.go
@@ -22,6 +22,7 @@ import (
        "fmt"
        "net/http"
        "net/url"
+       "time"
 
        "github.com/apache/incubator-devlake/core/errors"
        "github.com/apache/incubator-devlake/core/plugin"
@@ -78,7 +79,7 @@ func CollectApiCommits(taskCtx plugin.SubTaskContext) 
errors.Error {
                        query := url.Values{}
                        query.Set("state", "all")
                        if apiCollector.GetSince() != nil {
-                               query.Set("since", 
apiCollector.GetSince().String())
+                               query.Set("since", 
apiCollector.GetSince().UTC().Format(time.RFC3339))
                        }
                        query.Set("direction", "asc")
                        query.Set("page", fmt.Sprintf("%v", reqData.Pager.Page))
diff --git a/backend/plugins/github/tasks/commit_convertor.go 
b/backend/plugins/github/tasks/commit_convertor.go
index 4dc3fa43c..14448d48b 100644
--- a/backend/plugins/github/tasks/commit_convertor.go
+++ b/backend/plugins/github/tasks/commit_convertor.go
@@ -18,8 +18,6 @@ limitations under the License.
 package tasks
 
 import (
-       "reflect"
-
        "github.com/apache/incubator-devlake/core/dal"
        "github.com/apache/incubator-devlake/core/errors"
        "github.com/apache/incubator-devlake/core/models/domainlayer/code"
@@ -54,36 +52,36 @@ func ConvertCommits(taskCtx plugin.SubTaskContext) 
errors.Error {
        data := taskCtx.GetData().(*GithubTaskData)
        repoId := data.Options.GithubId
 
-       cursor, err := db.Cursor(
-               dal.From("_tool_github_commits gc"),
-               dal.Join(`left join _tool_github_repo_commits grc on (
-                       grc.commit_sha = gc.sha
-               )`),
-               dal.Select("gc.*"),
-               dal.Where("grc.repo_id = ? AND grc.connection_id = ?", repoId, 
data.Options.ConnectionId),
-       )
-       if err != nil {
-               return err
-       }
-       defer cursor.Close()
-
        repoDidGen := didgen.NewDomainIdGenerator(&models.GithubRepo{})
        domainRepoId := repoDidGen.Generate(data.Options.ConnectionId, repoId)
 
-       converter, err := api.NewDataConverter(api.DataConverterArgs{
-               RawDataSubTaskArgs: api.RawDataSubTaskArgs{
-                       Ctx: taskCtx,
+       converter, err := 
api.NewStatefulDataConverter(&api.StatefulDataConverterArgs[models.GithubCommit]{
+               SubtaskCommonArgs: &api.SubtaskCommonArgs{
+                       SubTaskContext: taskCtx,
+                       Table:          RAW_COMMIT_TABLE,
                        Params: GithubApiParams{
                                ConnectionId: data.Options.ConnectionId,
                                Name:         data.Options.Name,
                        },
-                       Table: RAW_COMMIT_TABLE,
                },
-               InputRowType: reflect.TypeOf(models.GithubCommit{}),
-               Input:        cursor,
-
-               Convert: func(inputRow interface{}) ([]interface{}, 
errors.Error) {
-                       githubCommit := inputRow.(*models.GithubCommit)
+               Input: func(stateManager *api.SubtaskStateManager) (dal.Rows, 
errors.Error) {
+                       clauses := []dal.Clause{
+                               dal.From("_tool_github_commits gc"),
+                               dal.Join(`left join _tool_github_repo_commits 
grc on (
+                                       grc.commit_sha = gc.sha
+                               )`),
+                               dal.Select("gc.*"),
+                               dal.Where("grc.repo_id = ? AND 
grc.connection_id = ?", repoId, data.Options.ConnectionId),
+                       }
+                       if stateManager.IsIncremental() {
+                               since := stateManager.GetSince()
+                               if since != nil {
+                                       clauses = append(clauses, 
dal.Where("(gc.updated_at >= ? OR grc.updated_at >= ?)", since, since))
+                               }
+                       }
+                       return db.Cursor(clauses...)
+               },
+               Convert: func(githubCommit *models.GithubCommit) 
([]interface{}, errors.Error) {
                        domainCommit := &code.Commit{
                                Sha:            githubCommit.Sha,
                                Message:        githubCommit.Message,
diff --git a/backend/plugins/github/tasks/commit_extractor.go 
b/backend/plugins/github/tasks/commit_extractor.go
index a0ed8d11d..72a29454d 100644
--- a/backend/plugins/github/tasks/commit_extractor.go
+++ b/backend/plugins/github/tasks/commit_extractor.go
@@ -18,8 +18,6 @@ limitations under the License.
 package tasks
 
 import (
-       "encoding/json"
-
        "github.com/apache/incubator-devlake/core/errors"
        "github.com/apache/incubator-devlake/core/models/common"
        "github.com/apache/incubator-devlake/core/plugin"
@@ -68,60 +66,44 @@ type Commit struct {
 func ExtractApiCommits(taskCtx plugin.SubTaskContext) errors.Error {
        data := taskCtx.GetData().(*GithubTaskData)
 
-       extractor, err := api.NewApiExtractor(api.ApiExtractorArgs{
-               RawDataSubTaskArgs: api.RawDataSubTaskArgs{
-                       Ctx: taskCtx,
-                       /*
-                               This struct will be JSONEncoded and stored into 
database along with raw data itself, to identity minimal
-                               set of data to be process, for example, we 
process JiraCommits by Board
-                       */
+       extractor, err := 
api.NewStatefulApiExtractor(&api.StatefulApiExtractorArgs[CommitsResponse]{
+               SubtaskCommonArgs: &api.SubtaskCommonArgs{
+                       SubTaskContext: taskCtx,
                        Params: GithubApiParams{
                                ConnectionId: data.Options.ConnectionId,
                                Name:         data.Options.Name,
                        },
-                       /*
-                               Table store raw data
-                       */
                        Table: RAW_COMMIT_TABLE,
                },
-               Extract: func(row *api.RawData) ([]interface{}, errors.Error) {
-                       commit := &CommitsResponse{}
-                       err := errors.Convert(json.Unmarshal(row.Data, commit))
-                       if err != nil {
-                               return nil, err
-                       }
-                       if commit.Sha == "" {
+               Extract: func(body *CommitsResponse, row *api.RawData) ([]any, 
errors.Error) {
+                       if body.Sha == "" {
                                return nil, nil
                        }
-
                        results := make([]interface{}, 0, 4)
-
                        githubCommit := &models.GithubCommit{
-                               Sha:            commit.Sha,
-                               Message:        commit.Commit.Message,
-                               AuthorName:     commit.Commit.Author.Name,
-                               AuthorEmail:    commit.Commit.Author.Email,
-                               AuthoredDate:   
commit.Commit.Author.Date.ToTime(),
-                               CommitterName:  commit.Commit.Committer.Name,
-                               CommitterEmail: commit.Commit.Committer.Email,
-                               CommittedDate:  
commit.Commit.Committer.Date.ToTime(),
-                               Url:            commit.Url,
+                               Sha:            body.Sha,
+                               Message:        body.Commit.Message,
+                               AuthorName:     body.Commit.Author.Name,
+                               AuthorEmail:    body.Commit.Author.Email,
+                               AuthoredDate:   
body.Commit.Author.Date.ToTime(),
+                               CommitterName:  body.Commit.Committer.Name,
+                               CommitterEmail: body.Commit.Committer.Email,
+                               CommittedDate:  
body.Commit.Committer.Date.ToTime(),
+                               Url:            body.Url,
                        }
-                       if commit.Author != nil {
-                               githubCommit.AuthorId = commit.Author.Id
-                               results = append(results, commit.Author)
+                       if body.Author != nil {
+                               githubCommit.AuthorId = body.Author.Id
+                               results = append(results, body.Author)
                        }
-                       if commit.Committer != nil {
-                               githubCommit.CommitterId = commit.Committer.Id
-                               results = append(results, commit.Committer)
+                       if body.Committer != nil {
+                               githubCommit.CommitterId = body.Committer.Id
+                               results = append(results, body.Committer)
                        }
-
                        githubRepoCommit := &models.GithubRepoCommit{
                                ConnectionId: data.Options.ConnectionId,
                                RepoId:       data.Options.GithubId,
-                               CommitSha:    commit.Sha,
+                               CommitSha:    body.Sha,
                        }
-
                        results = append(results, githubCommit)
                        results = append(results, githubRepoCommit)
                        return results, nil
diff --git a/backend/plugins/github/tasks/commit_stats_extractor.go 
b/backend/plugins/github/tasks/commit_stats_extractor.go
index 5f99b1ed5..431f2d28e 100644
--- a/backend/plugins/github/tasks/commit_stats_extractor.go
+++ b/backend/plugins/github/tasks/commit_stats_extractor.go
@@ -18,8 +18,6 @@ limitations under the License.
 package tasks
 
 import (
-       "encoding/json"
-
        "github.com/apache/incubator-devlake/core/dal"
        "github.com/apache/incubator-devlake/core/errors"
        "github.com/apache/incubator-devlake/core/models/common"
@@ -62,42 +60,27 @@ type ApiSingleCommitResponse struct {
 func ExtractApiCommitStats(taskCtx plugin.SubTaskContext) errors.Error {
        data := taskCtx.GetData().(*GithubTaskData)
 
-       extractor, err := api.NewApiExtractor(api.ApiExtractorArgs{
-               RawDataSubTaskArgs: api.RawDataSubTaskArgs{
-                       Ctx: taskCtx,
-                       /*
-                               This struct will be JSONEncoded and stored into 
database along with raw data itself, to identity minimal
-                               set of data to be process, for example, we 
process JiraCommits by Board
-                       */
+       extractor, err := 
api.NewStatefulApiExtractor(&api.StatefulApiExtractorArgs[ApiSingleCommitResponse]{
+               SubtaskCommonArgs: &api.SubtaskCommonArgs{
+                       SubTaskContext: taskCtx,
                        Params: GithubApiParams{
                                ConnectionId: data.Options.ConnectionId,
                                Name:         data.Options.Name,
                        },
-                       /*
-                               Table store raw data
-                       */
                        Table: RAW_COMMIT_STATS_TABLE,
                },
-               Extract: func(row *api.RawData) ([]interface{}, errors.Error) {
-                       body := &ApiSingleCommitResponse{}
-                       err := errors.Convert(json.Unmarshal(row.Data, body))
-                       if err != nil {
-                               return nil, err
-                       }
+               Extract: func(body *ApiSingleCommitResponse, row *api.RawData) 
([]any, errors.Error) {
                        if body.Sha == "" {
                                return nil, nil
                        }
-
                        db := taskCtx.GetDal()
                        commit := &models.GithubCommit{}
-                       err = db.First(commit, dal.Where("sha = ?", body.Sha), 
dal.Limit(1))
+                       err := db.First(commit, dal.Where("sha = ?", body.Sha), 
dal.Limit(1))
                        if err != nil && !db.IsErrorNotFound(err) {
                                return nil, err
                        }
-
                        commit.Additions = body.Stats.Additions
                        commit.Deletions = body.Stats.Deletions
-
                        commitStat := &models.GithubCommitStat{
                                ConnectionId:  data.Options.ConnectionId,
                                Additions:     body.Stats.Additions,
@@ -105,13 +88,7 @@ func ExtractApiCommitStats(taskCtx plugin.SubTaskContext) 
errors.Error {
                                CommittedDate: 
body.Commit.Committer.Date.ToTime(),
                                Sha:           body.Sha,
                        }
-
-                       results := make([]interface{}, 0, 2)
-
-                       results = append(results, commit)
-                       results = append(results, commitStat)
-
-                       return results, nil
+                       return []any{commit, commitStat}, nil
                },
        })
 
diff --git a/backend/plugins/github/tasks/event_extractor.go 
b/backend/plugins/github/tasks/event_extractor.go
index 133527d16..01b5495be 100644
--- a/backend/plugins/github/tasks/event_extractor.go
+++ b/backend/plugins/github/tasks/event_extractor.go
@@ -18,8 +18,6 @@ limitations under the License.
 package tasks
 
 import (
-       "encoding/json"
-
        "github.com/apache/incubator-devlake/core/errors"
        "github.com/apache/incubator-devlake/core/models/common"
        "github.com/apache/incubator-devlake/core/plugin"
@@ -54,21 +52,16 @@ type IssueEvent struct {
 func ExtractApiEvents(taskCtx plugin.SubTaskContext) errors.Error {
        data := taskCtx.GetData().(*GithubTaskData)
 
-       extractor, err := api.NewApiExtractor(api.ApiExtractorArgs{
-               RawDataSubTaskArgs: api.RawDataSubTaskArgs{
-                       Ctx: taskCtx,
+       extractor, err := 
api.NewStatefulApiExtractor(&api.StatefulApiExtractorArgs[IssueEvent]{
+               SubtaskCommonArgs: &api.SubtaskCommonArgs{
+                       SubTaskContext: taskCtx,
                        Params: GithubApiParams{
                                ConnectionId: data.Options.ConnectionId,
                                Name:         data.Options.Name,
                        },
                        Table: RAW_EVENTS_TABLE,
                },
-               Extract: func(row *api.RawData) ([]interface{}, errors.Error) {
-                       body := &IssueEvent{}
-                       err := errors.Convert(json.Unmarshal(row.Data, body))
-                       if err != nil {
-                               return nil, err
-                       }
+               Extract: func(body *IssueEvent, row *api.RawData) ([]any, 
errors.Error) {
                        results := make([]interface{}, 0, 1)
                        if body.GithubId == 0 || body.Actor == nil {
                                return nil, nil
@@ -80,22 +73,15 @@ func ExtractApiEvents(taskCtx plugin.SubTaskContext) 
errors.Error {
                                Type:            body.Event,
                                GithubCreatedAt: body.GithubCreatedAt.ToTime(),
                        }
-
                        if body.Actor != nil {
                                githubIssueEvent.AuthorUsername = 
body.Actor.Login
-
                                githubAccount, err := 
convertAccount(body.Actor, data.Options.GithubId, data.Options.ConnectionId)
                                if err != nil {
                                        return nil, err
                                }
                                results = append(results, githubAccount)
                        }
-
-                       if err != nil {
-                               return nil, err
-                       }
                        results = append(results, githubIssueEvent)
-
                        return results, nil
                },
        })
diff --git a/backend/plugins/github/tasks/issue_assignee_convertor.go 
b/backend/plugins/github/tasks/issue_assignee_convertor.go
index 5f8c883b8..f632db8a6 100644
--- a/backend/plugins/github/tasks/issue_assignee_convertor.go
+++ b/backend/plugins/github/tasks/issue_assignee_convertor.go
@@ -18,8 +18,6 @@ limitations under the License.
 package tasks
 
 import (
-       "reflect"
-
        "github.com/apache/incubator-devlake/core/dal"
        "github.com/apache/incubator-devlake/core/errors"
        "github.com/apache/incubator-devlake/core/models/domainlayer/didgen"
@@ -53,31 +51,35 @@ func ConvertIssueAssignee(taskCtx plugin.SubTaskContext) 
errors.Error {
        data := taskCtx.GetData().(*GithubTaskData)
        repoId := data.Options.GithubId
 
-       cursor, err := db.Cursor(
-               dal.From(&models.GithubIssueAssignee{}),
-               dal.Where("repo_id = ? and connection_id=?", repoId, 
data.Options.ConnectionId),
-       )
-       if err != nil {
-               return err
-       }
-       defer cursor.Close()
-
        issueIdGen := didgen.NewDomainIdGenerator(&models.GithubIssue{})
        accountIdGen := didgen.NewDomainIdGenerator(&models.GithubAccount{})
 
-       converter, err := api.NewDataConverter(api.DataConverterArgs{
-               RawDataSubTaskArgs: api.RawDataSubTaskArgs{
-                       Ctx: taskCtx,
+       converter, err := 
api.NewStatefulDataConverter(&api.StatefulDataConverterArgs[models.GithubIssueAssignee]{
+               SubtaskCommonArgs: &api.SubtaskCommonArgs{
+                       SubTaskContext: taskCtx,
+                       Table:          RAW_ISSUE_TABLE,
                        Params: GithubApiParams{
                                ConnectionId: data.Options.ConnectionId,
                                Name:         data.Options.Name,
                        },
-                       Table: RAW_ISSUE_TABLE,
                },
-               InputRowType: reflect.TypeOf(models.GithubIssueAssignee{}),
-               Input:        cursor,
-               Convert: func(inputRow interface{}) ([]interface{}, 
errors.Error) {
-                       githubIssueAssignee := 
inputRow.(*models.GithubIssueAssignee)
+               Input: func(stateManager *api.SubtaskStateManager) (dal.Rows, 
errors.Error) {
+                       clauses := []dal.Clause{
+                               dal.From(&models.GithubIssueAssignee{}),
+                               dal.Where("_tool_github_issue_assignees.repo_id 
= ? and _tool_github_issue_assignees.connection_id = ?", repoId, 
data.Options.ConnectionId),
+                       }
+                       if stateManager.IsIncremental() {
+                               since := stateManager.GetSince()
+                               if since != nil {
+                                       clauses = append(clauses,
+                                               dal.Join(`left join 
_tool_github_issues on _tool_github_issues.github_id = 
_tool_github_issue_assignees.issue_id and _tool_github_issues.connection_id = 
_tool_github_issue_assignees.connection_id`),
+                                       )
+                                       clauses = append(clauses, 
dal.Where("_tool_github_issues.github_updated_at >= ?", since))
+                               }
+                       }
+                       return db.Cursor(clauses...)
+               },
+               Convert: func(githubIssueAssignee *models.GithubIssueAssignee) 
([]interface{}, errors.Error) {
                        issueAssignee := &ticket.IssueAssignee{
                                IssueId:      
issueIdGen.Generate(data.Options.ConnectionId, githubIssueAssignee.IssueId),
                                AssigneeId:   
accountIdGen.Generate(data.Options.ConnectionId, 
githubIssueAssignee.AssigneeId),
diff --git a/backend/plugins/github/tasks/issue_collector.go 
b/backend/plugins/github/tasks/issue_collector.go
index 5558da697..4b38d4be6 100644
--- a/backend/plugins/github/tasks/issue_collector.go
+++ b/backend/plugins/github/tasks/issue_collector.go
@@ -22,6 +22,7 @@ import (
        "fmt"
        "net/http"
        "net/url"
+       "time"
 
        "github.com/apache/incubator-devlake/core/errors"
        "github.com/apache/incubator-devlake/core/plugin"
@@ -78,7 +79,7 @@ func CollectApiIssues(taskCtx plugin.SubTaskContext) 
errors.Error {
                        query := url.Values{}
                        query.Set("state", "all")
                        if apiCollector.GetSince() != nil {
-                               query.Set("since", 
apiCollector.GetSince().String())
+                               query.Set("since", 
apiCollector.GetSince().UTC().Format(time.RFC3339))
                        }
                        query.Set("direction", "asc")
                        query.Set("page", fmt.Sprintf("%v", reqData.Pager.Page))
diff --git a/backend/plugins/github/tasks/issue_comment_convertor.go 
b/backend/plugins/github/tasks/issue_comment_convertor.go
index 07ddeec65..2510e32d2 100644
--- a/backend/plugins/github/tasks/issue_comment_convertor.go
+++ b/backend/plugins/github/tasks/issue_comment_convertor.go
@@ -18,8 +18,6 @@ limitations under the License.
 package tasks
 
 import (
-       "reflect"
-
        "github.com/apache/incubator-devlake/core/dal"
        "github.com/apache/incubator-devlake/core/errors"
        "github.com/apache/incubator-devlake/core/models/domainlayer"
@@ -53,33 +51,34 @@ func ConvertIssueComments(taskCtx plugin.SubTaskContext) 
errors.Error {
        data := taskCtx.GetData().(*GithubTaskData)
        repoId := data.Options.GithubId
 
-       cursor, err := db.Cursor(
-               dal.From(&models.GithubIssueComment{}),
-               dal.Join("left join _tool_github_issues "+
-                       "on _tool_github_issues.github_id = 
_tool_github_issue_comments.issue_id"),
-               dal.Where("repo_id = ? and _tool_github_issues.connection_id = 
?", repoId, data.Options.ConnectionId),
-       )
-       if err != nil {
-               return err
-       }
-       defer cursor.Close()
-
        issueIdGen := didgen.NewDomainIdGenerator(&models.GithubIssue{})
        accountIdGen := didgen.NewDomainIdGenerator(&models.GithubAccount{})
 
-       converter, err := api.NewDataConverter(api.DataConverterArgs{
-               InputRowType: reflect.TypeOf(models.GithubIssueComment{}),
-               Input:        cursor,
-               RawDataSubTaskArgs: api.RawDataSubTaskArgs{
-                       Ctx: taskCtx,
+       converter, err := 
api.NewStatefulDataConverter(&api.StatefulDataConverterArgs[models.GithubIssueComment]{
+               SubtaskCommonArgs: &api.SubtaskCommonArgs{
+                       SubTaskContext: taskCtx,
+                       Table:          RAW_COMMENTS_TABLE,
                        Params: GithubApiParams{
                                ConnectionId: data.Options.ConnectionId,
                                Name:         data.Options.Name,
                        },
-                       Table: RAW_COMMENTS_TABLE,
                },
-               Convert: func(inputRow interface{}) ([]interface{}, 
errors.Error) {
-                       githubIssueComment := 
inputRow.(*models.GithubIssueComment)
+               Input: func(stateManager *api.SubtaskStateManager) (dal.Rows, 
errors.Error) {
+                       clauses := []dal.Clause{
+                               dal.From(&models.GithubIssueComment{}),
+                               dal.Join("left join _tool_github_issues " +
+                                       "on _tool_github_issues.github_id = 
_tool_github_issue_comments.issue_id"),
+                               dal.Where("repo_id = ? and 
_tool_github_issues.connection_id = ?", repoId, data.Options.ConnectionId),
+                       }
+                       if stateManager.IsIncremental() {
+                               since := stateManager.GetSince()
+                               if since != nil {
+                                       clauses = append(clauses, 
dal.Where("_tool_github_issue_comments.github_updated_at >= ?", since))
+                               }
+                       }
+                       return db.Cursor(clauses...)
+               },
+               Convert: func(githubIssueComment *models.GithubIssueComment) 
([]interface{}, errors.Error) {
                        domainIssueComment := &ticket.IssueComment{
                                DomainEntity: domainlayer.DomainEntity{
                                        Id: 
issueIdGen.Generate(data.Options.ConnectionId, githubIssueComment.GithubId),
diff --git a/backend/plugins/github/tasks/issue_convertor.go 
b/backend/plugins/github/tasks/issue_convertor.go
index e50f6f22a..13e79529a 100644
--- a/backend/plugins/github/tasks/issue_convertor.go
+++ b/backend/plugins/github/tasks/issue_convertor.go
@@ -18,7 +18,6 @@ limitations under the License.
 package tasks
 
 import (
-       "reflect"
        "strconv"
        "strings"
 
@@ -57,33 +56,33 @@ func ConvertIssues(taskCtx plugin.SubTaskContext) 
errors.Error {
        data := taskCtx.GetData().(*GithubTaskData)
        repoId := data.Options.GithubId
 
-       issue := &models.GithubIssue{}
-       cursor, err := db.Cursor(
-               dal.From(issue),
-               dal.Where("repo_id = ? and connection_id=?", repoId, 
data.Options.ConnectionId),
-       )
-       if err != nil {
-               return err
-       }
-       defer cursor.Close()
-
        issueIdGen := didgen.NewDomainIdGenerator(&models.GithubIssue{})
        accountIdGen := didgen.NewDomainIdGenerator(&models.GithubAccount{})
        boardIdGen := didgen.NewDomainIdGenerator(&models.GithubRepo{})
 
-       converter, err := api.NewDataConverter(api.DataConverterArgs{
-               RawDataSubTaskArgs: api.RawDataSubTaskArgs{
-                       Ctx: taskCtx,
+       converter, err := 
api.NewStatefulDataConverter(&api.StatefulDataConverterArgs[models.GithubIssue]{
+               SubtaskCommonArgs: &api.SubtaskCommonArgs{
+                       SubTaskContext: taskCtx,
+                       Table:          RAW_ISSUE_TABLE,
                        Params: GithubApiParams{
                                ConnectionId: data.Options.ConnectionId,
                                Name:         data.Options.Name,
                        },
-                       Table: RAW_ISSUE_TABLE,
                },
-               InputRowType: reflect.TypeOf(models.GithubIssue{}),
-               Input:        cursor,
-               Convert: func(inputRow interface{}) ([]interface{}, 
errors.Error) {
-                       issue := inputRow.(*models.GithubIssue)
+               Input: func(stateManager *api.SubtaskStateManager) (dal.Rows, 
errors.Error) {
+                       clauses := []dal.Clause{
+                               dal.From(&models.GithubIssue{}),
+                               dal.Where("repo_id = ? and connection_id=?", 
repoId, data.Options.ConnectionId),
+                       }
+                       if stateManager.IsIncremental() {
+                               since := stateManager.GetSince()
+                               if since != nil {
+                                       clauses = append(clauses, 
dal.Where("github_updated_at >= ?", since))
+                               }
+                       }
+                       return db.Cursor(clauses...)
+               },
+               Convert: func(issue *models.GithubIssue) ([]interface{}, 
errors.Error) {
                        domainIssue := &ticket.Issue{
                                DomainEntity:    domainlayer.DomainEntity{Id: 
issueIdGen.Generate(data.Options.ConnectionId, issue.GithubId)},
                                IssueKey:        strconv.Itoa(issue.Number),
diff --git a/backend/plugins/github/tasks/issue_extractor.go 
b/backend/plugins/github/tasks/issue_extractor.go
index 101b1e5b9..43a276f3e 100644
--- a/backend/plugins/github/tasks/issue_extractor.go
+++ b/backend/plugins/github/tasks/issue_extractor.go
@@ -22,6 +22,7 @@ import (
        "regexp"
        "strings"
 
+       "github.com/apache/incubator-devlake/core/dal"
        "github.com/apache/incubator-devlake/core/errors"
        "github.com/apache/incubator-devlake/core/models/common"
        "github.com/apache/incubator-devlake/core/models/domainlayer/ticket"
@@ -84,44 +85,53 @@ type IssueRegexes struct {
 
 func ExtractApiIssues(taskCtx plugin.SubTaskContext) errors.Error {
        data := taskCtx.GetData().(*GithubTaskData)
+       db := taskCtx.GetDal()
 
        config := data.Options.ScopeConfig
        issueRegexes, err := NewIssueRegexes(config)
        if err != nil {
                return nil
        }
-       extractor, err := api.NewApiExtractor(api.ApiExtractorArgs{
-               RawDataSubTaskArgs: api.RawDataSubTaskArgs{
-                       Ctx: taskCtx,
-                       /*
-                               This struct will be JSONEncoded and stored into 
database along with raw data itself, to identity minimal
-                               set of data to be process, for example, we 
process JiraIssues by Board
-                       */
+
+       extractor, extErr := 
api.NewStatefulApiExtractor(&api.StatefulApiExtractorArgs[IssuesResponse]{
+               SubtaskCommonArgs: &api.SubtaskCommonArgs{
+                       SubTaskContext: taskCtx,
                        Params: GithubApiParams{
                                ConnectionId: data.Options.ConnectionId,
                                Name:         data.Options.Name,
                        },
-                       /*
-                               Table store raw data
-                       */
                        Table: RAW_ISSUE_TABLE,
+                       SubtaskConfig: map[string]string{
+                               "issueSeverity":        config.IssueSeverity,
+                               "issueComponent":       config.IssueComponent,
+                               "issuePriority":        config.IssuePriority,
+                               "issueTypeBug":         config.IssueTypeBug,
+                               "issueTypeRequirement": 
config.IssueTypeRequirement,
+                               "issueTypeIncident":    
config.IssueTypeIncident,
+                       },
                },
-               Extract: func(row *api.RawData) ([]interface{}, errors.Error) {
-                       body := &IssuesResponse{}
-                       err := errors.Convert(json.Unmarshal(row.Data, body))
-                       if err != nil {
-                               return nil, err
+               BeforeExtract: func(body *IssuesResponse, stateManager 
*api.SubtaskStateManager) errors.Error {
+                       if stateManager.IsIncremental() {
+                               if err := db.Delete(&models.GithubIssueLabel{},
+                                       dal.Where("connection_id = ? AND 
issue_id = ?", data.Options.ConnectionId, body.GithubId),
+                               ); err != nil {
+                                       return errors.Convert(err)
+                               }
+                               return 
errors.Convert(db.Delete(&models.GithubIssueAssignee{},
+                                       dal.Where("connection_id = ? AND 
issue_id = ?", data.Options.ConnectionId, body.GithubId),
+                               ))
                        }
-                       // need to extract 2 kinds of entities here
+                       return nil
+               },
+               Extract: func(body *IssuesResponse, row *api.RawData) ([]any, 
errors.Error) {
                        if body.GithubId == 0 {
                                return nil, nil
                        }
-                       //If this is a pr, ignore
+                       // If this is a pr, ignore
                        if body.PullRequest.Url != "" {
                                return nil, nil
                        }
                        results := make([]interface{}, 0, 2)
-
                        githubIssue, err := convertGithubIssue(body, 
data.Options.ConnectionId, data.Options.GithubId)
                        if err != nil {
                                return nil, err
@@ -163,8 +173,8 @@ func ExtractApiIssues(taskCtx plugin.SubTaskContext) 
errors.Error {
                        return results, nil
                },
        })
-       if err != nil {
-               return err
+       if extErr != nil {
+               return extErr
        }
 
        return extractor.Execute()
diff --git a/backend/plugins/github/tasks/issue_label_convertor.go 
b/backend/plugins/github/tasks/issue_label_convertor.go
index bff0c0143..6ef940b06 100644
--- a/backend/plugins/github/tasks/issue_label_convertor.go
+++ b/backend/plugins/github/tasks/issue_label_convertor.go
@@ -18,8 +18,6 @@ limitations under the License.
 package tasks
 
 import (
-       "reflect"
-
        "github.com/apache/incubator-devlake/core/dal"
        "github.com/apache/incubator-devlake/core/errors"
        "github.com/apache/incubator-devlake/core/models/domainlayer/didgen"
@@ -51,31 +49,33 @@ func ConvertIssueLabels(taskCtx plugin.SubTaskContext) 
errors.Error {
        data := taskCtx.GetData().(*GithubTaskData)
        repoId := data.Options.GithubId
 
-       cursor, err := db.Cursor(
-               dal.From(&models.GithubIssueLabel{}),
-               dal.Join(`left join _tool_github_issues on 
_tool_github_issues.github_id = _tool_github_issue_labels.issue_id`),
-               dal.Where("_tool_github_issues.repo_id = ? and 
_tool_github_issues.connection_id = ?", repoId, data.Options.ConnectionId),
-               dal.Orderby("issue_id ASC"),
-       )
-       if err != nil {
-               return err
-       }
-       defer cursor.Close()
        issueIdGen := didgen.NewDomainIdGenerator(&models.GithubIssue{})
 
-       converter, err := api.NewDataConverter(api.DataConverterArgs{
-               RawDataSubTaskArgs: api.RawDataSubTaskArgs{
-                       Ctx: taskCtx,
+       converter, err := 
api.NewStatefulDataConverter(&api.StatefulDataConverterArgs[models.GithubIssueLabel]{
+               SubtaskCommonArgs: &api.SubtaskCommonArgs{
+                       SubTaskContext: taskCtx,
+                       Table:          RAW_ISSUE_TABLE,
                        Params: GithubApiParams{
                                ConnectionId: data.Options.ConnectionId,
                                Name:         data.Options.Name,
                        },
-                       Table: RAW_ISSUE_TABLE,
                },
-               InputRowType: reflect.TypeOf(models.GithubIssueLabel{}),
-               Input:        cursor,
-               Convert: func(inputRow interface{}) ([]interface{}, 
errors.Error) {
-                       issueLabel := inputRow.(*models.GithubIssueLabel)
+               Input: func(stateManager *api.SubtaskStateManager) (dal.Rows, 
errors.Error) {
+                       clauses := []dal.Clause{
+                               dal.From(&models.GithubIssueLabel{}),
+                               dal.Join(`left join _tool_github_issues on 
_tool_github_issues.github_id = _tool_github_issue_labels.issue_id`),
+                               dal.Where("_tool_github_issues.repo_id = ? and 
_tool_github_issues.connection_id = ?", repoId, data.Options.ConnectionId),
+                               dal.Orderby("issue_id ASC"),
+                       }
+                       if stateManager.IsIncremental() {
+                               since := stateManager.GetSince()
+                               if since != nil {
+                                       clauses = append(clauses, 
dal.Where("_tool_github_issues.github_updated_at >= ?", since))
+                               }
+                       }
+                       return db.Cursor(clauses...)
+               },
+               Convert: func(issueLabel *models.GithubIssueLabel) 
([]interface{}, errors.Error) {
                        domainIssueLabel := &ticket.IssueLabel{
                                IssueId:   
issueIdGen.Generate(data.Options.ConnectionId, issueLabel.IssueId),
                                LabelName: issueLabel.LabelName,
diff --git a/backend/plugins/github/tasks/milestone_extractor.go 
b/backend/plugins/github/tasks/milestone_extractor.go
index 7828b2c26..4737bd47e 100644
--- a/backend/plugins/github/tasks/milestone_extractor.go
+++ b/backend/plugins/github/tasks/milestone_extractor.go
@@ -18,8 +18,6 @@ limitations under the License.
 package tasks
 
 import (
-       "encoding/json"
-
        "github.com/apache/incubator-devlake/core/errors"
        "github.com/apache/incubator-devlake/core/models/common"
        "github.com/apache/incubator-devlake/core/plugin"
@@ -81,24 +79,17 @@ type MilestonesResponse struct {
 
 func ExtractMilestones(taskCtx plugin.SubTaskContext) errors.Error {
        data := taskCtx.GetData().(*GithubTaskData)
-       extractor, err := api.NewApiExtractor(api.ApiExtractorArgs{
-               RawDataSubTaskArgs: api.RawDataSubTaskArgs{
-                       Ctx: taskCtx,
+       extractor, err := 
api.NewStatefulApiExtractor(&api.StatefulApiExtractorArgs[MilestonesResponse]{
+               SubtaskCommonArgs: &api.SubtaskCommonArgs{
+                       SubTaskContext: taskCtx,
                        Params: GithubApiParams{
                                ConnectionId: data.Options.ConnectionId,
                                Name:         data.Options.Name,
                        },
                        Table: RAW_MILESTONE_TABLE,
                },
-               Extract: func(row *api.RawData) ([]interface{}, errors.Error) {
-                       response := &MilestonesResponse{}
-                       err := errors.Convert(json.Unmarshal(row.Data, 
response))
-                       if err != nil {
-                               return nil, err
-                       }
-                       results := make([]interface{}, 0, 1)
-                       results = append(results, 
convertGithubMilestone(response, data.Options.ConnectionId, 
data.Options.GithubId))
-                       return results, nil
+               Extract: func(body *MilestonesResponse, row *api.RawData) 
([]any, errors.Error) {
+                       return []any{convertGithubMilestone(body, 
data.Options.ConnectionId, data.Options.GithubId)}, nil
                },
        })
        if err != nil {
diff --git a/backend/plugins/github/tasks/pr_comment_convertor.go 
b/backend/plugins/github/tasks/pr_comment_convertor.go
index 43e0e73ba..b02342e2b 100644
--- a/backend/plugins/github/tasks/pr_comment_convertor.go
+++ b/backend/plugins/github/tasks/pr_comment_convertor.go
@@ -18,8 +18,6 @@ limitations under the License.
 package tasks
 
 import (
-       "reflect"
-
        "github.com/apache/incubator-devlake/core/dal"
        "github.com/apache/incubator-devlake/core/errors"
        "github.com/apache/incubator-devlake/core/models/domainlayer"
@@ -54,34 +52,36 @@ func ConvertPullRequestComments(taskCtx 
plugin.SubTaskContext) errors.Error {
        data := taskCtx.GetData().(*GithubTaskData)
        repoId := data.Options.GithubId
 
-       cursor, err := db.Cursor(
-               dal.From(&models.GithubPrComment{}),
-               dal.Join("left join _tool_github_pull_requests "+
-                       "on _tool_github_pull_requests.github_id = 
_tool_github_pull_request_comments.pull_request_id"),
-               dal.Where("repo_id = ? and 
_tool_github_pull_requests.connection_id = ?", repoId, 
data.Options.ConnectionId),
-       )
-       if err != nil {
-               return err
-       }
-       defer cursor.Close()
-
        prCommentIdGen := didgen.NewDomainIdGenerator(&models.GithubPrComment{})
        prIdGen := didgen.NewDomainIdGenerator(&models.GithubPullRequest{})
        accountIdGen := didgen.NewDomainIdGenerator(&models.GithubAccount{})
        prReviewIdGen := didgen.NewDomainIdGenerator(&models.GithubPrReview{})
-       converter, err := api.NewDataConverter(api.DataConverterArgs{
-               InputRowType: reflect.TypeOf(models.GithubPrComment{}),
-               Input:        cursor,
-               RawDataSubTaskArgs: api.RawDataSubTaskArgs{
-                       Ctx: taskCtx,
+
+       converter, err := 
api.NewStatefulDataConverter(&api.StatefulDataConverterArgs[models.GithubPrComment]{
+               SubtaskCommonArgs: &api.SubtaskCommonArgs{
+                       SubTaskContext: taskCtx,
+                       Table:          RAW_COMMENTS_TABLE,
                        Params: GithubApiParams{
                                ConnectionId: data.Options.ConnectionId,
                                Name:         data.Options.Name,
                        },
-                       Table: RAW_COMMENTS_TABLE,
                },
-               Convert: func(inputRow interface{}) ([]interface{}, 
errors.Error) {
-                       githubPullRequestComment := 
inputRow.(*models.GithubPrComment)
+               Input: func(stateManager *api.SubtaskStateManager) (dal.Rows, 
errors.Error) {
+                       clauses := []dal.Clause{
+                               dal.From(&models.GithubPrComment{}),
+                               dal.Join("left join _tool_github_pull_requests 
" +
+                                       "on 
_tool_github_pull_requests.github_id = 
_tool_github_pull_request_comments.pull_request_id"),
+                               dal.Where("repo_id = ? and 
_tool_github_pull_requests.connection_id = ?", repoId, 
data.Options.ConnectionId),
+                       }
+                       if stateManager.IsIncremental() {
+                               since := stateManager.GetSince()
+                               if since != nil {
+                                       clauses = append(clauses, 
dal.Where("_tool_github_pull_request_comments.github_updated_at >= ?", since))
+                               }
+                       }
+                       return db.Cursor(clauses...)
+               },
+               Convert: func(githubPullRequestComment *models.GithubPrComment) 
([]interface{}, errors.Error) {
                        domainPrComment := &code.PullRequestComment{
                                DomainEntity: domainlayer.DomainEntity{
                                        Id: 
prCommentIdGen.Generate(data.Options.ConnectionId, 
githubPullRequestComment.GithubId),
diff --git a/backend/plugins/github/tasks/pr_commit_convertor.go 
b/backend/plugins/github/tasks/pr_commit_convertor.go
index b9cbbbbf5..d21c343dc 100644
--- a/backend/plugins/github/tasks/pr_commit_convertor.go
+++ b/backend/plugins/github/tasks/pr_commit_convertor.go
@@ -18,8 +18,6 @@ limitations under the License.
 package tasks
 
 import (
-       "reflect"
-
        "github.com/apache/incubator-devlake/core/dal"
        "github.com/apache/incubator-devlake/core/errors"
        "github.com/apache/incubator-devlake/core/models/domainlayer/code"
@@ -54,30 +52,31 @@ func ConvertPullRequestCommits(taskCtx 
plugin.SubTaskContext) (err errors.Error)
 
        pullIdGen := didgen.NewDomainIdGenerator(&models.GithubPullRequest{})
 
-       cursor, err := db.Cursor(
-               dal.From(&models.GithubPrCommit{}),
-               dal.Join(`left join _tool_github_pull_requests on 
_tool_github_pull_requests.github_id = 
_tool_github_pull_request_commits.pull_request_id`),
-               dal.Where("_tool_github_pull_requests.repo_id = ? and 
_tool_github_pull_requests.connection_id = ?", repoId, 
data.Options.ConnectionId),
-               dal.Orderby("pull_request_id ASC"),
-       )
-       if err != nil {
-               return err
-       }
-       defer cursor.Close()
-
-       converter, err := api.NewDataConverter(api.DataConverterArgs{
-               InputRowType: reflect.TypeOf(models.GithubPrCommit{}),
-               Input:        cursor,
-               RawDataSubTaskArgs: api.RawDataSubTaskArgs{
-                       Ctx: taskCtx,
+       converter, err := 
api.NewStatefulDataConverter(&api.StatefulDataConverterArgs[models.GithubPrCommit]{
+               SubtaskCommonArgs: &api.SubtaskCommonArgs{
+                       SubTaskContext: taskCtx,
+                       Table:          RAW_PR_COMMIT_TABLE,
                        Params: GithubApiParams{
                                ConnectionId: data.Options.ConnectionId,
                                Name:         data.Options.Name,
                        },
-                       Table: RAW_PR_COMMIT_TABLE,
                },
-               Convert: func(inputRow interface{}) ([]interface{}, 
errors.Error) {
-                       githubPullRequestCommit := 
inputRow.(*models.GithubPrCommit)
+               Input: func(stateManager *api.SubtaskStateManager) (dal.Rows, 
errors.Error) {
+                       clauses := []dal.Clause{
+                               dal.From(&models.GithubPrCommit{}),
+                               dal.Join(`left join _tool_github_pull_requests 
on _tool_github_pull_requests.github_id = 
_tool_github_pull_request_commits.pull_request_id`),
+                               dal.Where("_tool_github_pull_requests.repo_id = 
? and _tool_github_pull_requests.connection_id = ?", repoId, 
data.Options.ConnectionId),
+                               dal.Orderby("pull_request_id ASC"),
+                       }
+                       if stateManager.IsIncremental() {
+                               since := stateManager.GetSince()
+                               if since != nil {
+                                       clauses = append(clauses, 
dal.Where("_tool_github_pull_requests.github_updated_at >= ?", since))
+                               }
+                       }
+                       return db.Cursor(clauses...)
+               },
+               Convert: func(githubPullRequestCommit *models.GithubPrCommit) 
([]interface{}, errors.Error) {
                        domainPrCommit := &code.PullRequestCommit{
                                CommitSha:          
githubPullRequestCommit.CommitSha,
                                PullRequestId:      
pullIdGen.Generate(data.Options.ConnectionId, 
githubPullRequestCommit.PullRequestId),
diff --git a/backend/plugins/github/tasks/pr_commit_extractor.go 
b/backend/plugins/github/tasks/pr_commit_extractor.go
index 569e39aba..536af4d26 100644
--- a/backend/plugins/github/tasks/pr_commit_extractor.go
+++ b/backend/plugins/github/tasks/pr_commit_extractor.go
@@ -70,62 +70,44 @@ type PullRequestCommit struct {
 func ExtractApiPullRequestCommits(taskCtx plugin.SubTaskContext) errors.Error {
        data := taskCtx.GetData().(*GithubTaskData)
        repoId := data.Options.GithubId
-       extractor, err := helper.NewApiExtractor(helper.ApiExtractorArgs{
-               RawDataSubTaskArgs: helper.RawDataSubTaskArgs{
-                       Ctx: taskCtx,
-                       /*
-                               This struct will be JSONEncoded and stored into 
database along with raw data itself, to identity minimal
-                               set of data to be process, for example, we 
process JiraIssues by Board
-                       */
+       extractor, err := 
helper.NewStatefulApiExtractor(&helper.StatefulApiExtractorArgs[PrCommitsResponse]{
+               SubtaskCommonArgs: &helper.SubtaskCommonArgs{
+                       SubTaskContext: taskCtx,
                        Params: GithubApiParams{
                                ConnectionId: data.Options.ConnectionId,
                                Name:         data.Options.Name,
                        },
-                       /*
-                               Table store raw data
-                       */
                        Table: RAW_PR_COMMIT_TABLE,
                },
-               Extract: func(row *helper.RawData) ([]interface{}, 
errors.Error) {
-                       apiPullRequestCommit := &PrCommitsResponse{}
+               Extract: func(body *PrCommitsResponse, row *helper.RawData) 
([]any, errors.Error) {
                        if strings.HasPrefix(string(row.Data), "{\"message\": 
\"Not Found\"") {
                                return nil, nil
                        }
-                       err := errors.Convert(json.Unmarshal(row.Data, 
apiPullRequestCommit))
-                       if err != nil {
-                               return nil, err
-                       }
                        pull := &SimplePr{}
-                       err = errors.Convert(json.Unmarshal(row.Input, pull))
+                       err := errors.Convert(json.Unmarshal(row.Input, pull))
                        if err != nil {
                                return nil, err
                        }
-                       // need to extract 2 kinds of entities here
                        results := make([]interface{}, 0, 3)
                        githubRepoCommit := &models.GithubRepoCommit{
                                ConnectionId: data.Options.ConnectionId,
                                RepoId:       repoId,
-                               CommitSha:    apiPullRequestCommit.Sha,
+                               CommitSha:    body.Sha,
                        }
                        results = append(results, githubRepoCommit)
-
-                       githubCommit, err := 
convertPullRequestCommit(apiPullRequestCommit, data.Options.ConnectionId)
+                       githubCommit, err := convertPullRequestCommit(body, 
data.Options.ConnectionId)
                        if err != nil {
                                return nil, err
                        }
                        results = append(results, githubCommit)
-
                        githubPullRequestCommit := &models.GithubPrCommit{
                                ConnectionId:       data.Options.ConnectionId,
-                               CommitSha:          apiPullRequestCommit.Sha,
+                               CommitSha:          body.Sha,
                                PullRequestId:      pull.GithubId,
                                CommitAuthorName:   githubCommit.AuthorName,
                                CommitAuthorEmail:  githubCommit.AuthorEmail,
                                CommitAuthoredDate: githubCommit.AuthoredDate,
                        }
-                       if err != nil {
-                               return nil, err
-                       }
                        results = append(results, githubPullRequestCommit)
                        return results, nil
                },
diff --git a/backend/plugins/github/tasks/pr_convertor.go 
b/backend/plugins/github/tasks/pr_convertor.go
index 4a94d17df..48a40cc4e 100644
--- a/backend/plugins/github/tasks/pr_convertor.go
+++ b/backend/plugins/github/tasks/pr_convertor.go
@@ -18,8 +18,6 @@ limitations under the License.
 package tasks
 
 import (
-       "reflect"
-
        "github.com/apache/incubator-devlake/core/dal"
        "github.com/apache/incubator-devlake/core/errors"
        "github.com/apache/incubator-devlake/core/models/domainlayer"
@@ -53,32 +51,33 @@ func ConvertPullRequests(taskCtx plugin.SubTaskContext) 
errors.Error {
        data := taskCtx.GetData().(*GithubTaskData)
        repoId := data.Options.GithubId
 
-       cursor, err := db.Cursor(
-               dal.From(&models.GithubPullRequest{}),
-               dal.Where("repo_id = ? and connection_id = ?", repoId, 
data.Options.ConnectionId),
-       )
-       if err != nil {
-               return err
-       }
-       defer cursor.Close()
-
        prIdGen := didgen.NewDomainIdGenerator(&models.GithubPullRequest{})
        repoIdGen := didgen.NewDomainIdGenerator(&models.GithubRepo{})
        accountIdGen := didgen.NewDomainIdGenerator(&models.GithubAccount{})
 
-       converter, err := api.NewDataConverter(api.DataConverterArgs{
-               InputRowType: reflect.TypeOf(models.GithubPullRequest{}),
-               Input:        cursor,
-               RawDataSubTaskArgs: api.RawDataSubTaskArgs{
-                       Ctx: taskCtx,
+       converter, err := 
api.NewStatefulDataConverter(&api.StatefulDataConverterArgs[models.GithubPullRequest]{
+               SubtaskCommonArgs: &api.SubtaskCommonArgs{
+                       SubTaskContext: taskCtx,
+                       Table:          RAW_PULL_REQUEST_TABLE,
                        Params: GithubApiParams{
                                ConnectionId: data.Options.ConnectionId,
                                Name:         data.Options.Name,
                        },
-                       Table: RAW_PULL_REQUEST_TABLE,
                },
-               Convert: func(inputRow interface{}) ([]interface{}, 
errors.Error) {
-                       pr := inputRow.(*models.GithubPullRequest)
+               Input: func(stateManager *api.SubtaskStateManager) (dal.Rows, 
errors.Error) {
+                       clauses := []dal.Clause{
+                               dal.From(&models.GithubPullRequest{}),
+                               dal.Where("repo_id = ? and connection_id = ?", 
repoId, data.Options.ConnectionId),
+                       }
+                       if stateManager.IsIncremental() {
+                               since := stateManager.GetSince()
+                               if since != nil {
+                                       clauses = append(clauses, 
dal.Where("github_updated_at >= ?", since))
+                               }
+                       }
+                       return db.Cursor(clauses...)
+               },
+               Convert: func(pr *models.GithubPullRequest) ([]interface{}, 
errors.Error) {
                        domainPr := &code.PullRequest{
                                DomainEntity: domainlayer.DomainEntity{
                                        Id: 
prIdGen.Generate(data.Options.ConnectionId, pr.GithubId),
diff --git a/backend/plugins/github/tasks/pr_extractor.go 
b/backend/plugins/github/tasks/pr_extractor.go
index ea7858385..802846f30 100644
--- a/backend/plugins/github/tasks/pr_extractor.go
+++ b/backend/plugins/github/tasks/pr_extractor.go
@@ -21,6 +21,7 @@ import (
        "encoding/json"
        "regexp"
 
+       "github.com/apache/incubator-devlake/core/dal"
        "github.com/apache/incubator-devlake/core/errors"
        "github.com/apache/incubator-devlake/core/models/common"
        "github.com/apache/incubator-devlake/core/plugin"
@@ -85,6 +86,7 @@ type GithubApiPullRequest struct {
 
 func ExtractApiPullRequests(taskCtx plugin.SubTaskContext) errors.Error {
        data := taskCtx.GetData().(*GithubTaskData)
+       db := taskCtx.GetDal()
        config := data.Options.ScopeConfig
        var labelTypeRegex *regexp.Regexp
        var labelComponentRegex *regexp.Regexp
@@ -104,45 +106,44 @@ func ExtractApiPullRequests(taskCtx 
plugin.SubTaskContext) errors.Error {
                }
        }
 
-       extractor, err := api.NewApiExtractor(api.ApiExtractorArgs{
-               RawDataSubTaskArgs: api.RawDataSubTaskArgs{
-                       Ctx: taskCtx,
-                       /*
-                               This struct will be JSONEncoded and stored into 
database along with raw data itself, to identity minimal
-                               set of data to be process, for example, we 
process JiraIssues by Board
-                       */
+       extractor, extErr := 
api.NewStatefulApiExtractor(&api.StatefulApiExtractorArgs[GithubApiPullRequest]{
+               SubtaskCommonArgs: &api.SubtaskCommonArgs{
+                       SubTaskContext: taskCtx,
                        Params: GithubApiParams{
                                ConnectionId: data.Options.ConnectionId,
                                Name:         data.Options.Name,
                        },
-                       /*
-                               Table store raw data
-                       */
                        Table: RAW_PULL_REQUEST_TABLE,
+                       SubtaskConfig: map[string]string{
+                               "prType":      prType,
+                               "prComponent": prComponent,
+                       },
                },
-               Extract: func(row *api.RawData) ([]interface{}, errors.Error) {
-                       rawL := &GithubApiPullRequest{}
-                       err := errors.Convert(json.Unmarshal(row.Data, rawL))
-                       if err != nil {
-                               return nil, err
+               BeforeExtract: func(body *GithubApiPullRequest, stateManager 
*api.SubtaskStateManager) errors.Error {
+                       if stateManager.IsIncremental() {
+                               return errors.Convert(db.Delete(
+                                       &models.GithubPrLabel{},
+                                       dal.Where("connection_id = ? AND 
pull_id = ?", data.Options.ConnectionId, body.GithubId),
+                               ))
                        }
-                       // need to extract 2 kinds of entities here
+                       return nil
+               },
+               Extract: func(body *GithubApiPullRequest, row *api.RawData) 
([]any, errors.Error) {
                        results := make([]interface{}, 0, 1)
-                       if rawL.GithubId == 0 {
+                       if body.GithubId == 0 {
                                return nil, nil
                        }
                        // Filter bot PRs by username
-                       if rawL.User != nil && 
shouldSkipByUsername(rawL.User.Login) {
-                               taskCtx.GetLogger().Debug("Skipping PR #%d from 
bot user: %s", rawL.Number, rawL.User.Login)
+                       if body.User != nil && 
shouldSkipByUsername(body.User.Login) {
+                               taskCtx.GetLogger().Debug("Skipping PR #%d from 
bot user: %s", body.Number, body.User.Login)
                                return nil, nil
                        }
-                       //If this is a pr, ignore
-                       githubPr, err := convertGithubPullRequest(rawL, 
data.Options.ConnectionId, data.Options.GithubId)
+                       githubPr, err := convertGithubPullRequest(body, 
data.Options.ConnectionId, data.Options.GithubId)
                        if err != nil {
                                return nil, err
                        }
-                       if rawL.User != nil {
-                               githubUser, err := convertAccount(rawL.User, 
data.Options.GithubId, data.Options.ConnectionId)
+                       if body.User != nil {
+                               githubUser, err := convertAccount(body.User, 
data.Options.GithubId, data.Options.ConnectionId)
                                if err != nil {
                                        return nil, err
                                }
@@ -150,33 +151,31 @@ func ExtractApiPullRequests(taskCtx 
plugin.SubTaskContext) errors.Error {
                                githubPr.AuthorName = githubUser.Login
                                githubPr.AuthorId = githubUser.AccountId
                        }
-                       for _, label := range rawL.Labels {
+                       for _, label := range body.Labels {
                                results = append(results, &models.GithubPrLabel{
                                        ConnectionId: data.Options.ConnectionId,
                                        PullId:       githubPr.GithubId,
                                        LabelName:    label.Name,
                                })
-                               // if pr.Type has not been set and prType is 
set in .env, process the below
                                if labelTypeRegex != nil && 
labelTypeRegex.MatchString(label.Name) {
                                        githubPr.Type = label.Name
                                }
-                               // if pr.Component has not been set and 
prComponent is set in .env, process
                                if labelComponentRegex != nil && 
labelComponentRegex.MatchString(label.Name) {
                                        githubPr.Component = label.Name
                                }
                        }
                        results = append(results, githubPr)
-
                        return results, nil
                },
        })
 
-       if err != nil {
-               return errors.Default.Wrap(err, "error initializing Github PR 
extractor")
+       if extErr != nil {
+               return errors.Default.Wrap(extErr, "error initializing Github 
PR extractor")
        }
 
        return extractor.Execute()
 }
+
 func convertGithubPullRequest(pull *GithubApiPullRequest, connId uint64, 
repoId int) (*models.GithubPullRequest, errors.Error) {
        githubPull := &models.GithubPullRequest{
                ConnectionId:    connId,
diff --git a/backend/plugins/github/tasks/pr_issue_convertor.go 
b/backend/plugins/github/tasks/pr_issue_convertor.go
index 979a8b128..0d724224f 100644
--- a/backend/plugins/github/tasks/pr_issue_convertor.go
+++ b/backend/plugins/github/tasks/pr_issue_convertor.go
@@ -19,7 +19,6 @@ package tasks
 
 import (
        "github.com/spf13/cast"
-       "reflect"
 
        "github.com/apache/incubator-devlake/core/dal"
        "github.com/apache/incubator-devlake/core/errors"
@@ -52,32 +51,34 @@ func ConvertPullRequestIssues(taskCtx 
plugin.SubTaskContext) errors.Error {
        data := taskCtx.GetData().(*GithubTaskData)
        repoId := data.Options.GithubId
 
-       cursor, err := db.Cursor(
-               dal.From(&models.GithubPrIssue{}),
-               dal.Join(`left join _tool_github_pull_requests on 
_tool_github_pull_requests.github_id = 
_tool_github_pull_request_issues.pull_request_id`),
-               dal.Where("_tool_github_pull_requests.repo_id = ? and 
_tool_github_pull_requests.connection_id = ?", repoId, 
data.Options.ConnectionId),
-               dal.Orderby("pull_request_id ASC"),
-       )
-       if err != nil {
-               return err
-       }
-       defer cursor.Close()
        prIdGen := didgen.NewDomainIdGenerator(&models.GithubPullRequest{})
        issueIdGen := didgen.NewDomainIdGenerator(&models.GithubIssue{})
 
-       converter, err := api.NewDataConverter(api.DataConverterArgs{
-               InputRowType: reflect.TypeOf(models.GithubPrIssue{}),
-               Input:        cursor,
-               RawDataSubTaskArgs: api.RawDataSubTaskArgs{
-                       Ctx: taskCtx,
+       converter, err := 
api.NewStatefulDataConverter(&api.StatefulDataConverterArgs[models.GithubPrIssue]{
+               SubtaskCommonArgs: &api.SubtaskCommonArgs{
+                       SubTaskContext: taskCtx,
+                       Table:          RAW_PULL_REQUEST_TABLE,
                        Params: GithubApiParams{
                                ConnectionId: data.Options.ConnectionId,
                                Name:         data.Options.Name,
                        },
-                       Table: RAW_PULL_REQUEST_TABLE,
                },
-               Convert: func(inputRow interface{}) ([]interface{}, 
errors.Error) {
-                       githubPrIssue := inputRow.(*models.GithubPrIssue)
+               Input: func(stateManager *api.SubtaskStateManager) (dal.Rows, 
errors.Error) {
+                       clauses := []dal.Clause{
+                               dal.From(&models.GithubPrIssue{}),
+                               dal.Join(`left join _tool_github_pull_requests 
on _tool_github_pull_requests.github_id = 
_tool_github_pull_request_issues.pull_request_id`),
+                               dal.Where("_tool_github_pull_requests.repo_id = 
? and _tool_github_pull_requests.connection_id = ?", repoId, 
data.Options.ConnectionId),
+                               dal.Orderby("pull_request_id ASC"),
+                       }
+                       if stateManager.IsIncremental() {
+                               since := stateManager.GetSince()
+                               if since != nil {
+                                       clauses = append(clauses, 
dal.Where("_tool_github_pull_requests.github_updated_at >= ?", since))
+                               }
+                       }
+                       return db.Cursor(clauses...)
+               },
+               Convert: func(githubPrIssue *models.GithubPrIssue) 
([]interface{}, errors.Error) {
                        pullRequestIssue := &crossdomain.PullRequestIssue{
                                PullRequestId:  
prIdGen.Generate(data.Options.ConnectionId, githubPrIssue.PullRequestId),
                                IssueId:        
issueIdGen.Generate(data.Options.ConnectionId, githubPrIssue.IssueId),
diff --git a/backend/plugins/github/tasks/pr_issue_enricher.go 
b/backend/plugins/github/tasks/pr_issue_enricher.go
index bf4d2837d..cf8228b79 100644
--- a/backend/plugins/github/tasks/pr_issue_enricher.go
+++ b/backend/plugins/github/tasks/pr_issue_enricher.go
@@ -18,7 +18,6 @@ limitations under the License.
 package tasks
 
 import (
-       "reflect"
        "regexp"
        "strconv"
        "strings"
@@ -62,27 +61,42 @@ func EnrichPullRequestIssues(taskCtx plugin.SubTaskContext) 
(err errors.Error) {
                }
        }
        charPattern := regexp.MustCompile(`[\/a-zA-Z\s,]+`)
-       cursor, err := db.Cursor(dal.From(&models.GithubPullRequest{}),
-               dal.Where("repo_id = ? and connection_id = ?", repoId, 
data.Options.ConnectionId))
-       if err != nil {
-               return err
-       }
-       defer cursor.Close()
-       // iterate all rows
-
-       converter, err := api.NewDataConverter(api.DataConverterArgs{
-               InputRowType: reflect.TypeOf(models.GithubPullRequest{}),
-               Input:        cursor,
-               RawDataSubTaskArgs: api.RawDataSubTaskArgs{
-                       Ctx: taskCtx,
+
+       converter, err := 
api.NewStatefulDataConverter(&api.StatefulDataConverterArgs[models.GithubPullRequest]{
+               SubtaskCommonArgs: &api.SubtaskCommonArgs{
+                       SubTaskContext: taskCtx,
+                       Table:          RAW_PULL_REQUEST_TABLE,
                        Params: GithubApiParams{
                                ConnectionId: data.Options.ConnectionId,
                                Name:         data.Options.Name,
                        },
-                       Table: RAW_PULL_REQUEST_TABLE,
+                       SubtaskConfig: map[string]string{
+                               "prBodyClosePattern": prBodyClosePattern,
+                       },
+               },
+               Input: func(stateManager *api.SubtaskStateManager) (dal.Rows, 
errors.Error) {
+                       clauses := []dal.Clause{
+                               dal.From(&models.GithubPullRequest{}),
+                               dal.Where("repo_id = ? and connection_id = ?", 
repoId, data.Options.ConnectionId),
+                       }
+                       if stateManager.IsIncremental() {
+                               since := stateManager.GetSince()
+                               if since != nil {
+                                       clauses = append(clauses, 
dal.Where("github_updated_at >= ?", since))
+                               }
+                       }
+                       return db.Cursor(clauses...)
+               },
+               BeforeConvert: func(githubPullRequest 
*models.GithubPullRequest, stateManager *api.SubtaskStateManager) errors.Error {
+                       if !stateManager.IsIncremental() {
+                               return nil
+                       }
+                       return db.Delete(
+                               &models.GithubPrIssue{},
+                               dal.Where("connection_id = ? AND 
pull_request_id = ?", data.Options.ConnectionId, githubPullRequest.GithubId),
+                       )
                },
-               Convert: func(inputRow interface{}) ([]interface{}, 
errors.Error) {
-                       githubPullRequst := inputRow.(*models.GithubPullRequest)
+               Convert: func(githubPullRequst *models.GithubPullRequest) 
([]interface{}, errors.Error) {
                        results := make([]interface{}, 0, 1)
 
                        //find the matched string in body
diff --git a/backend/plugins/github/tasks/pr_label_convertor.go 
b/backend/plugins/github/tasks/pr_label_convertor.go
index 3351218a7..432a7308f 100644
--- a/backend/plugins/github/tasks/pr_label_convertor.go
+++ b/backend/plugins/github/tasks/pr_label_convertor.go
@@ -18,8 +18,6 @@ limitations under the License.
 package tasks
 
 import (
-       "reflect"
-
        "github.com/apache/incubator-devlake/core/dal"
        "github.com/apache/incubator-devlake/core/errors"
        "github.com/apache/incubator-devlake/core/models/domainlayer/code"
@@ -51,31 +49,33 @@ func ConvertPullRequestLabels(taskCtx 
plugin.SubTaskContext) errors.Error {
        data := taskCtx.GetData().(*GithubTaskData)
        repoId := data.Options.GithubId
 
-       cursor, err := db.Cursor(
-               dal.From(&models.GithubPrLabel{}),
-               dal.Join(`left join _tool_github_pull_requests on 
_tool_github_pull_requests.github_id = 
_tool_github_pull_request_labels.pull_id`),
-               dal.Where("_tool_github_pull_requests.repo_id = ? and 
_tool_github_pull_requests.connection_id = ?", repoId, 
data.Options.ConnectionId),
-               dal.Orderby("pull_id ASC"),
-       )
-       if err != nil {
-               return err
-       }
-       defer cursor.Close()
        prIdGen := didgen.NewDomainIdGenerator(&models.GithubPullRequest{})
 
-       converter, err := api.NewDataConverter(api.DataConverterArgs{
-               InputRowType: reflect.TypeOf(models.GithubPrLabel{}),
-               Input:        cursor,
-               RawDataSubTaskArgs: api.RawDataSubTaskArgs{
-                       Ctx: taskCtx,
+       converter, err := 
api.NewStatefulDataConverter(&api.StatefulDataConverterArgs[models.GithubPrLabel]{
+               SubtaskCommonArgs: &api.SubtaskCommonArgs{
+                       SubTaskContext: taskCtx,
+                       Table:          RAW_PULL_REQUEST_TABLE,
                        Params: GithubApiParams{
                                ConnectionId: data.Options.ConnectionId,
                                Name:         data.Options.Name,
                        },
-                       Table: RAW_PULL_REQUEST_TABLE,
                },
-               Convert: func(inputRow interface{}) ([]interface{}, 
errors.Error) {
-                       prLabel := inputRow.(*models.GithubPrLabel)
+               Input: func(stateManager *api.SubtaskStateManager) (dal.Rows, 
errors.Error) {
+                       clauses := []dal.Clause{
+                               dal.From(&models.GithubPrLabel{}),
+                               dal.Join(`left join _tool_github_pull_requests 
on _tool_github_pull_requests.github_id = 
_tool_github_pull_request_labels.pull_id`),
+                               dal.Where("_tool_github_pull_requests.repo_id = 
? and _tool_github_pull_requests.connection_id = ?", repoId, 
data.Options.ConnectionId),
+                               dal.Orderby("pull_id ASC"),
+                       }
+                       if stateManager.IsIncremental() {
+                               since := stateManager.GetSince()
+                               if since != nil {
+                                       clauses = append(clauses, 
dal.Where("_tool_github_pull_requests.github_updated_at >= ?", since))
+                               }
+                       }
+                       return db.Cursor(clauses...)
+               },
+               Convert: func(prLabel *models.GithubPrLabel) ([]interface{}, 
errors.Error) {
                        domainPrLabel := &code.PullRequestLabel{
                                PullRequestId: 
prIdGen.Generate(data.Options.ConnectionId, prLabel.PullId),
                                LabelName:     prLabel.LabelName,
diff --git a/backend/plugins/github/tasks/pr_review_comment_collector.go 
b/backend/plugins/github/tasks/pr_review_comment_collector.go
index d6dfb296f..ac29938cf 100644
--- a/backend/plugins/github/tasks/pr_review_comment_collector.go
+++ b/backend/plugins/github/tasks/pr_review_comment_collector.go
@@ -22,6 +22,7 @@ import (
        "fmt"
        "net/http"
        "net/url"
+       "time"
 
        "github.com/apache/incubator-devlake/core/errors"
        "github.com/apache/incubator-devlake/core/plugin"
@@ -75,7 +76,7 @@ func CollectPrReviewComments(taskCtx plugin.SubTaskContext) 
errors.Error {
                Query: func(reqData *helper.RequestData) (url.Values, 
errors.Error) {
                        query := url.Values{}
                        if apiCollector.GetSince() != nil {
-                               query.Set("since", 
apiCollector.GetSince().String())
+                               query.Set("since", 
apiCollector.GetSince().UTC().Format(time.RFC3339))
                        }
                        query.Set("page", fmt.Sprintf("%v", reqData.Pager.Page))
                        query.Set("direction", "asc")
diff --git a/backend/plugins/github/tasks/pr_review_comment_extractor.go 
b/backend/plugins/github/tasks/pr_review_comment_extractor.go
index ed8aa1825..cc39806cd 100644
--- a/backend/plugins/github/tasks/pr_review_comment_extractor.go
+++ b/backend/plugins/github/tasks/pr_review_comment_extractor.go
@@ -48,86 +48,75 @@ var ExtractApiPrReviewCommentsMeta = plugin.SubTaskMeta{
                models.GithubPrCommit{}.TableName()},
 }
 
+type GithubPrReviewCommentBody struct {
+       GithubId        int `json:"id"`
+       Body            json.RawMessage
+       User            *GithubAccountResponse
+       PrUrl           string             `json:"pull_request_url"`
+       GithubCreatedAt common.Iso8601Time `json:"created_at"`
+       GithubUpdatedAt common.Iso8601Time `json:"updated_at"`
+       CommitId        string             `json:"commit_id"`
+       PrReviewId      int                `json:"pull_request_review_id"`
+}
+
 func ExtractApiPrReviewComments(taskCtx plugin.SubTaskContext) errors.Error {
        data := taskCtx.GetData().(*GithubTaskData)
        db := taskCtx.GetDal()
+
        prUrlPattern := 
fmt.Sprintf(`https\:\/\/api\.github\.com\/repos\/%s\/pulls\/(\d+)`, 
data.Options.Name)
+       prUrlRegex, err := regexp.Compile(prUrlPattern)
+       if err != nil {
+               return errors.Default.Wrap(err, "regexp Compile prUrlPattern 
failed")
+       }
 
-       extractor, err := helper.NewApiExtractor(helper.ApiExtractorArgs{
-               RawDataSubTaskArgs: helper.RawDataSubTaskArgs{
-                       Ctx: taskCtx,
+       extractor, extErr := 
helper.NewStatefulApiExtractor(&helper.StatefulApiExtractorArgs[GithubPrReviewCommentBody]{
+               SubtaskCommonArgs: &helper.SubtaskCommonArgs{
+                       SubTaskContext: taskCtx,
                        Params: GithubApiParams{
                                ConnectionId: data.Options.ConnectionId,
                                Name:         data.Options.Name,
                        },
                        Table: RAW_PR_REVIEW_COMMENTS_TABLE,
                },
-               Extract: func(row *helper.RawData) ([]interface{}, 
errors.Error) {
-                       var prReviewComment struct {
-                               GithubId        int `json:"id"`
-                               Body            json.RawMessage
-                               User            *GithubAccountResponse
-                               PrUrl           string             
`json:"pull_request_url"`
-                               GithubCreatedAt common.Iso8601Time 
`json:"created_at"`
-                               GithubUpdatedAt common.Iso8601Time 
`json:"updated_at"`
-                               CommitId        string             
`json:"commit_id"`
-                               PrReviewId      int                
`json:"pull_request_review_id"`
-                       }
-                       err := errors.Convert(json.Unmarshal(row.Data, 
&prReviewComment))
-                       if err != nil {
-                               return nil, err
-                       }
-
+               Extract: func(body *GithubPrReviewCommentBody, row 
*helper.RawData) ([]any, errors.Error) {
                        results := make([]interface{}, 0, 1)
                        githubPrComment := &models.GithubPrComment{
                                ConnectionId:    data.Options.ConnectionId,
-                               GithubId:        prReviewComment.GithubId,
-                               Body:            string(prReviewComment.Body),
-                               CommitSha:       prReviewComment.CommitId,
-                               ReviewId:        prReviewComment.PrReviewId,
-                               GithubCreatedAt: 
prReviewComment.GithubCreatedAt.ToTime(),
-                               GithubUpdatedAt: 
prReviewComment.GithubUpdatedAt.ToTime(),
+                               GithubId:        body.GithubId,
+                               Body:            string(body.Body),
+                               CommitSha:       body.CommitId,
+                               ReviewId:        body.PrReviewId,
+                               GithubCreatedAt: body.GithubCreatedAt.ToTime(),
+                               GithubUpdatedAt: body.GithubUpdatedAt.ToTime(),
                                Type:            "DIFF",
                        }
-
-                       prUrlRegex, err := 
errors.Convert01(regexp.Compile(prUrlPattern))
-                       if err != nil {
-                               return nil, errors.Default.Wrap(err, "regexp 
Compile prUrlPattern failed")
-                       }
-                       prId, err := enrichGithubPrComment(data, db, 
prUrlRegex, prReviewComment.PrUrl)
+                       prId, err := enrichGithubPrComment(data, db, 
prUrlRegex, body.PrUrl)
                        if err != nil {
                                return nil, errors.Default.Wrap(err, "parse 
prId failed")
                        }
                        if prId != 0 {
                                githubPrComment.PullRequestId = prId
                        }
-
-                       if prReviewComment.User != nil {
-                               // Filter bot comments by username
-                               if 
shouldSkipByUsername(prReviewComment.User.Login) {
-                                       taskCtx.GetLogger().Debug("Skipping PR 
review comment #%d from bot user: %s", prReviewComment.GithubId, 
prReviewComment.User.Login)
+                       if body.User != nil {
+                               if shouldSkipByUsername(body.User.Login) {
+                                       taskCtx.GetLogger().Debug("Skipping PR 
review comment #%d from bot user: %s", body.GithubId, body.User.Login)
                                        return nil, nil
                                }
-
-                               githubPrComment.AuthorUserId = 
prReviewComment.User.Id
-                               githubPrComment.AuthorUsername = 
prReviewComment.User.Login
-
-                               githubAccount, err := 
convertAccount(prReviewComment.User, data.Options.GithubId, 
data.Options.ConnectionId)
+                               githubPrComment.AuthorUserId = body.User.Id
+                               githubPrComment.AuthorUsername = body.User.Login
+                               githubAccount, err := convertAccount(body.User, 
data.Options.GithubId, data.Options.ConnectionId)
                                if err != nil {
                                        return nil, err
                                }
                                results = append(results, githubAccount)
                        }
-
                        results = append(results, githubPrComment)
                        return results, nil
                },
        })
-
-       if err != nil {
-               return err
+       if extErr != nil {
+               return extErr
        }
-
        return extractor.Execute()
 }
 
diff --git a/backend/plugins/github/tasks/pr_review_convertor.go 
b/backend/plugins/github/tasks/pr_review_convertor.go
index 4ff2f3531..f266b130a 100644
--- a/backend/plugins/github/tasks/pr_review_convertor.go
+++ b/backend/plugins/github/tasks/pr_review_convertor.go
@@ -18,8 +18,6 @@ limitations under the License.
 package tasks
 
 import (
-       "reflect"
-
        "github.com/apache/incubator-devlake/core/dal"
        "github.com/apache/incubator-devlake/core/errors"
        "github.com/apache/incubator-devlake/core/models/domainlayer"
@@ -53,34 +51,35 @@ func ConvertPullRequestReviews(taskCtx 
plugin.SubTaskContext) errors.Error {
        data := taskCtx.GetData().(*GithubTaskData)
        repoId := data.Options.GithubId
 
-       cursor, err := db.Cursor(
-               dal.From(&models.GithubPrReview{}),
-               dal.Join("left join _tool_github_pull_requests "+
-                       "on _tool_github_pull_requests.github_id = 
_tool_github_pull_request_reviews.pull_request_id"),
-               dal.Where("repo_id = ? and 
_tool_github_pull_requests.connection_id = ?", repoId, 
data.Options.ConnectionId),
-       )
-       if err != nil {
-               return err
-       }
-       defer cursor.Close()
-
        prReviewUIdGen := didgen.NewDomainIdGenerator(&models.GithubPrReview{})
        prIdGen := didgen.NewDomainIdGenerator(&models.GithubPullRequest{})
        accountIdGen := didgen.NewDomainIdGenerator(&models.GithubAccount{})
 
-       converter, err := api.NewDataConverter(api.DataConverterArgs{
-               InputRowType: reflect.TypeOf(models.GithubPrReview{}),
-               Input:        cursor,
-               RawDataSubTaskArgs: api.RawDataSubTaskArgs{
-                       Ctx: taskCtx,
+       converter, err := 
api.NewStatefulDataConverter(&api.StatefulDataConverterArgs[models.GithubPrReview]{
+               SubtaskCommonArgs: &api.SubtaskCommonArgs{
+                       SubTaskContext: taskCtx,
+                       Table:          RAW_PR_REVIEW_TABLE,
                        Params: GithubApiParams{
                                ConnectionId: data.Options.ConnectionId,
                                Name:         data.Options.Name,
                        },
-                       Table: RAW_PR_REVIEW_TABLE,
                },
-               Convert: func(inputRow interface{}) ([]interface{}, 
errors.Error) {
-                       githubPullRequestReview := 
inputRow.(*models.GithubPrReview)
+               Input: func(stateManager *api.SubtaskStateManager) (dal.Rows, 
errors.Error) {
+                       clauses := []dal.Clause{
+                               dal.From(&models.GithubPrReview{}),
+                               dal.Join("left join _tool_github_pull_requests 
" +
+                                       "on 
_tool_github_pull_requests.github_id = 
_tool_github_pull_request_reviews.pull_request_id"),
+                               dal.Where("repo_id = ? and 
_tool_github_pull_requests.connection_id = ?", repoId, 
data.Options.ConnectionId),
+                       }
+                       if stateManager.IsIncremental() {
+                               since := stateManager.GetSince()
+                               if since != nil {
+                                       clauses = append(clauses, 
dal.Where("_tool_github_pull_requests.github_updated_at >= ?", since))
+                               }
+                       }
+                       return db.Cursor(clauses...)
+               },
+               Convert: func(githubPullRequestReview *models.GithubPrReview) 
([]interface{}, errors.Error) {
                        domainPrReview := &code.PullRequestComment{
                                DomainEntity: domainlayer.DomainEntity{
                                        Id: 
prReviewUIdGen.Generate(data.Options.ConnectionId, 
githubPullRequestReview.GithubId),
diff --git a/backend/plugins/github/tasks/pr_review_extractor.go 
b/backend/plugins/github/tasks/pr_review_extractor.go
index 2af41b60a..ff8015411 100644
--- a/backend/plugins/github/tasks/pr_review_extractor.go
+++ b/backend/plugins/github/tasks/pr_review_extractor.go
@@ -56,79 +56,59 @@ type PullRequestReview struct {
 
 func ExtractApiPullRequestReviews(taskCtx plugin.SubTaskContext) errors.Error {
        data := taskCtx.GetData().(*GithubTaskData)
-       extractor, err := api.NewApiExtractor(api.ApiExtractorArgs{
-               RawDataSubTaskArgs: api.RawDataSubTaskArgs{
-                       Ctx: taskCtx,
-                       /*
-                               This struct will be JSONEncoded and stored into 
database along with raw data itself, to identity minimal
-                               set of data to be process, for example, we 
process JiraIssues by Board
-                       */
+       extractor, err := 
api.NewStatefulApiExtractor(&api.StatefulApiExtractorArgs[PullRequestReview]{
+               SubtaskCommonArgs: &api.SubtaskCommonArgs{
+                       SubTaskContext: taskCtx,
                        Params: GithubApiParams{
                                ConnectionId: data.Options.ConnectionId,
                                Name:         data.Options.Name,
                        },
-                       /*
-                               Table store raw data
-                       */
                        Table: RAW_PR_REVIEW_TABLE,
                },
-               Extract: func(row *api.RawData) ([]interface{}, errors.Error) {
-                       apiPullRequestReview := &PullRequestReview{}
+               Extract: func(body *PullRequestReview, row *api.RawData) 
([]any, errors.Error) {
                        if strings.HasPrefix(string(row.Data), "{\"message\": 
\"Not Found\"") {
                                return nil, nil
                        }
-                       err := errors.Convert(json.Unmarshal(row.Data, 
apiPullRequestReview))
-                       if err != nil {
-                               return nil, err
-                       }
-                       if apiPullRequestReview.State == "PENDING" || 
apiPullRequestReview.User == nil {
+                       if body.State == "PENDING" || body.User == nil {
                                return nil, nil
                        }
                        // Filter bot reviews by username
-                       if 
shouldSkipByUsername(apiPullRequestReview.User.Login) {
-                               taskCtx.GetLogger().Debug("Skipping review #%d 
from bot user: %s", apiPullRequestReview.GithubId, 
apiPullRequestReview.User.Login)
+                       if shouldSkipByUsername(body.User.Login) {
+                               taskCtx.GetLogger().Debug("Skipping review #%d 
from bot user: %s", body.GithubId, body.User.Login)
                                return nil, nil
                        }
                        pull := &SimplePr{}
-                       err = errors.Convert(json.Unmarshal(row.Input, pull))
+                       err := errors.Convert(json.Unmarshal(row.Input, pull))
                        if err != nil {
                                return nil, err
                        }
-                       // need to extract 2 kinds of entities here
                        results := make([]interface{}, 0, 1)
-
                        githubReviewer := &models.GithubReviewer{
                                ConnectionId:  data.Options.ConnectionId,
                                PullRequestId: pull.GithubId,
                        }
-
                        githubPrReview := &models.GithubPrReview{
                                ConnectionId:   data.Options.ConnectionId,
-                               GithubId:       apiPullRequestReview.GithubId,
-                               Body:           apiPullRequestReview.Body,
-                               State:          apiPullRequestReview.State,
-                               CommitSha:      apiPullRequestReview.CommitId,
-                               GithubSubmitAt: 
apiPullRequestReview.SubmittedAt.ToNullableTime(),
+                               GithubId:       body.GithubId,
+                               Body:           body.Body,
+                               State:          body.State,
+                               CommitSha:      body.CommitId,
+                               GithubSubmitAt: 
body.SubmittedAt.ToNullableTime(),
                                PullRequestId:  pull.GithubId,
                        }
-
-                       if apiPullRequestReview.User != nil {
-                               githubReviewer.ReviewerId = 
apiPullRequestReview.User.Id
-                               githubReviewer.Username = 
apiPullRequestReview.User.Login
-
-                               githubPrReview.AuthorUserId = 
apiPullRequestReview.User.Id
-                               githubPrReview.AuthorUsername = 
apiPullRequestReview.User.Login
-
-                               githubUser, err := 
convertAccount(apiPullRequestReview.User, data.Options.GithubId, 
data.Options.ConnectionId)
+                       if body.User != nil {
+                               githubReviewer.ReviewerId = body.User.Id
+                               githubReviewer.Username = body.User.Login
+                               githubPrReview.AuthorUserId = body.User.Id
+                               githubPrReview.AuthorUsername = body.User.Login
+                               githubUser, err := convertAccount(body.User, 
data.Options.GithubId, data.Options.ConnectionId)
                                if err != nil {
                                        return nil, err
                                }
                                results = append(results, githubUser)
                        }
-
                        results = append(results, githubReviewer)
                        results = append(results, githubPrReview)
-
                        return results, nil
                },
        })
diff --git a/backend/plugins/github/tasks/release_convertor.go 
b/backend/plugins/github/tasks/release_convertor.go
index be49141eb..a5c2f941d 100644
--- a/backend/plugins/github/tasks/release_convertor.go
+++ b/backend/plugins/github/tasks/release_convertor.go
@@ -18,8 +18,6 @@ limitations under the License.
 package tasks
 
 import (
-       "reflect"
-
        "github.com/apache/incubator-devlake/core/dal"
        "github.com/apache/incubator-devlake/core/errors"
        "github.com/apache/incubator-devlake/core/models/domainlayer"
@@ -50,28 +48,38 @@ var ConvertReleasesMeta = plugin.SubTaskMeta{
 
 func ConvertRelease(taskCtx plugin.SubTaskContext) errors.Error {
        db := taskCtx.GetDal()
-       rawDataSubTaskArgs, data := CreateRawDataSubTaskArgs(taskCtx, 
RAW_RELEASE_TABLE)
-       cursor, err := db.Cursor(
-               dal.From(&models.GithubRelease{}),
-               dal.Where(
-                       "published_at IS NOT NULL AND connection_id = ? AND 
github_id = ?",
-                       data.Options.ConnectionId, data.Options.GithubId,
-               ),
-       )
-       if err != nil {
-               return err
+       data := taskCtx.GetData().(*GithubTaskData)
+       params := GithubApiParams{
+               ConnectionId: data.Options.ConnectionId,
+               Name:         data.Options.Name,
        }
-       defer cursor.Close()
 
        releaseIdGen := didgen.NewDomainIdGenerator(&models.GithubRelease{})
        releaseScopeIdGen := didgen.NewDomainIdGenerator(&models.GithubRepo{})
 
-       converter, err := api.NewDataConverter(api.DataConverterArgs{
-               InputRowType:       reflect.TypeOf(models.GithubRelease{}),
-               Input:              cursor,
-               RawDataSubTaskArgs: *rawDataSubTaskArgs,
-               Convert: func(inputRow interface{}) ([]interface{}, 
errors.Error) {
-                       githubRelease := inputRow.(*models.GithubRelease)
+       converter, err := 
api.NewStatefulDataConverter(&api.StatefulDataConverterArgs[models.GithubRelease]{
+               SubtaskCommonArgs: &api.SubtaskCommonArgs{
+                       SubTaskContext: taskCtx,
+                       Table:          RAW_RELEASE_TABLE,
+                       Params:         params,
+               },
+               Input: func(stateManager *api.SubtaskStateManager) (dal.Rows, 
errors.Error) {
+                       clauses := []dal.Clause{
+                               dal.From(&models.GithubRelease{}),
+                               dal.Where(
+                                       "published_at IS NOT NULL AND 
connection_id = ? AND github_id = ?",
+                                       data.Options.ConnectionId, 
data.Options.GithubId,
+                               ),
+                       }
+                       if stateManager.IsIncremental() {
+                               since := stateManager.GetSince()
+                               if since != nil {
+                                       clauses = append(clauses, 
dal.Where("updated_at >= ?", since))
+                               }
+                       }
+                       return db.Cursor(clauses...)
+               },
+               Convert: func(githubRelease *models.GithubRelease) 
([]interface{}, errors.Error) {
                        release := &devops.CicdRelease{
                                DomainEntity: domainlayer.DomainEntity{
                                        Id: 
releaseIdGen.Generate(githubRelease.ConnectionId, githubRelease.Id),
@@ -87,18 +95,12 @@ func ConvertRelease(taskCtx plugin.SubTaskContext) 
errors.Error {
                                IsPrerelease: githubRelease.IsPrerelease,
                                TagName:      githubRelease.TagName,
                                CommitSha:    githubRelease.CommitSha,
-
-                               AuthorID: githubRelease.AuthorID,
-
-                               RepoId: 
releaseScopeIdGen.Generate(githubRelease.ConnectionId, githubRelease.GithubId),
+                               AuthorID:     githubRelease.AuthorID,
+                               RepoId:       
releaseScopeIdGen.Generate(githubRelease.ConnectionId, githubRelease.GithubId),
                        }
-
-                       return []interface{}{
-                               release,
-                       }, nil
+                       return []interface{}{release}, nil
                },
        })
-
        if err != nil {
                return err
        }
diff --git a/backend/plugins/github/tasks/repo_convertor.go 
b/backend/plugins/github/tasks/repo_convertor.go
index b9c711ad9..c6bff20b3 100644
--- a/backend/plugins/github/tasks/repo_convertor.go
+++ b/backend/plugins/github/tasks/repo_convertor.go
@@ -19,7 +19,6 @@ package tasks
 
 import (
        "fmt"
-       "reflect"
 
        "github.com/apache/incubator-devlake/core/dal"
        "github.com/apache/incubator-devlake/core/errors"
@@ -35,10 +34,6 @@ import (
        "github.com/apache/incubator-devlake/plugins/github/models"
 )
 
-func init() {
-       RegisterSubtaskMeta(&ConvertRepoMeta)
-}
-
 type GithubApiRepo struct {
        Name        string `json:"name"`
        FullName    string `json:"full_name"`
@@ -53,6 +48,10 @@ type GithubApiRepo struct {
        CloneUrl    string              `json:"clone_url"`
 }
 
+func init() {
+       RegisterSubtaskMeta(&ConvertRepoMeta)
+}
+
 var ConvertRepoMeta = plugin.SubTaskMeta{
        Name:             "Convert Repos",
        EntryPoint:       ConvertRepo,
@@ -80,30 +79,31 @@ func ConvertRepo(taskCtx plugin.SubTaskContext) 
errors.Error {
        data := taskCtx.GetData().(*GithubTaskData)
        repoId := data.Options.GithubId
 
-       cursor, err := db.Cursor(
-               dal.From(&models.GithubRepo{}),
-               dal.Where("github_id = ? and connection_id = ?", repoId, 
data.Options.ConnectionId),
-       )
-       if err != nil {
-               return err
-       }
-       defer cursor.Close()
-
        repoIdGen := didgen.NewDomainIdGenerator(&models.GithubRepo{})
 
-       converter, err := api.NewDataConverter(api.DataConverterArgs{
-               InputRowType: reflect.TypeOf(models.GithubRepo{}),
-               Input:        cursor,
-               RawDataSubTaskArgs: api.RawDataSubTaskArgs{
-                       Ctx: taskCtx,
+       converter, err := 
api.NewStatefulDataConverter(&api.StatefulDataConverterArgs[models.GithubRepo]{
+               SubtaskCommonArgs: &api.SubtaskCommonArgs{
+                       SubTaskContext: taskCtx,
+                       Table:          models.GithubRepo{}.TableName(),
                        Params: GithubApiParams{
                                ConnectionId: data.Options.ConnectionId,
                                Name:         data.Options.Name,
                        },
-                       Table: models.GithubRepo{}.TableName(),
                },
-               Convert: func(inputRow interface{}) ([]interface{}, 
errors.Error) {
-                       repository := inputRow.(*models.GithubRepo)
+               Input: func(stateManager *api.SubtaskStateManager) (dal.Rows, 
errors.Error) {
+                       clauses := []dal.Clause{
+                               dal.From(&models.GithubRepo{}),
+                               dal.Where("github_id = ? and connection_id = 
?", repoId, data.Options.ConnectionId),
+                       }
+                       if stateManager.IsIncremental() {
+                               since := stateManager.GetSince()
+                               if since != nil {
+                                       clauses = append(clauses, 
dal.Where("updated_at >= ?", since))
+                               }
+                       }
+                       return db.Cursor(clauses...)
+               },
+               Convert: func(repository *models.GithubRepo) ([]interface{}, 
errors.Error) {
                        domainRepository := &code.Repo{
                                DomainEntity: domainlayer.DomainEntity{
                                        Id: 
repoIdGen.Generate(data.Options.ConnectionId, repository.GithubId),
@@ -125,12 +125,10 @@ func ConvertRepo(taskCtx plugin.SubTaskContext) 
errors.Error {
                                Description: repository.Description,
                                CreatedDate: repository.CreatedDate,
                        }
-
                        domainBoardRepo := &crossdomain.BoardRepo{
                                BoardId: 
repoIdGen.Generate(data.Options.ConnectionId, repository.GithubId),
                                RepoId:  
repoIdGen.Generate(data.Options.ConnectionId, repository.GithubId),
                        }
-
                        domainCicdScope := &devops.CicdScope{
                                DomainEntity: domainlayer.DomainEntity{
                                        Id: 
repoIdGen.Generate(data.Options.ConnectionId, repository.GithubId),
@@ -141,7 +139,6 @@ func ConvertRepo(taskCtx plugin.SubTaskContext) 
errors.Error {
                                CreatedDate: repository.CreatedDate,
                                UpdatedDate: repository.UpdatedDate,
                        }
-
                        return []interface{}{
                                domainRepository,
                                domainBoard,
diff --git a/backend/plugins/github/tasks/review_convertor.go 
b/backend/plugins/github/tasks/review_convertor.go
index d4d101042..18a0de211 100644
--- a/backend/plugins/github/tasks/review_convertor.go
+++ b/backend/plugins/github/tasks/review_convertor.go
@@ -18,8 +18,6 @@ limitations under the License.
 package tasks
 
 import (
-       "reflect"
-
        "github.com/apache/incubator-devlake/core/dal"
        "github.com/apache/incubator-devlake/core/errors"
        "github.com/apache/incubator-devlake/core/models/domainlayer/code"
@@ -52,33 +50,34 @@ func ConvertReviews(taskCtx plugin.SubTaskContext) 
errors.Error {
        data := taskCtx.GetData().(*GithubTaskData)
        repoId := data.Options.GithubId
 
-       cursor, err := db.Cursor(
-               dal.From(&models.GithubReviewer{}),
-               dal.Join("left join _tool_github_pull_requests "+
-                       "on _tool_github_pull_requests.github_id = 
_tool_github_reviewers.pull_request_id"),
-               dal.Where("repo_id = ? and 
_tool_github_pull_requests.connection_id = ?", repoId, 
data.Options.ConnectionId),
-       )
-       if err != nil {
-               return err
-       }
-       defer cursor.Close()
-
        prIdGen := didgen.NewDomainIdGenerator(&models.GithubPullRequest{})
        accountIdGen := didgen.NewDomainIdGenerator(&models.GithubAccount{})
 
-       converter, err := api.NewDataConverter(api.DataConverterArgs{
-               InputRowType: reflect.TypeOf(models.GithubReviewer{}),
-               Input:        cursor,
-               RawDataSubTaskArgs: api.RawDataSubTaskArgs{
-                       Ctx: taskCtx,
+       converter, err := 
api.NewStatefulDataConverter(&api.StatefulDataConverterArgs[models.GithubReviewer]{
+               SubtaskCommonArgs: &api.SubtaskCommonArgs{
+                       SubTaskContext: taskCtx,
+                       Table:          RAW_PR_REVIEW_TABLE,
                        Params: GithubApiParams{
                                ConnectionId: data.Options.ConnectionId,
                                Name:         data.Options.Name,
                        },
-                       Table: RAW_PR_REVIEW_TABLE,
                },
-               Convert: func(inputRow interface{}) ([]interface{}, 
errors.Error) {
-                       githubReview := inputRow.(*models.GithubReviewer)
+               Input: func(stateManager *api.SubtaskStateManager) (dal.Rows, 
errors.Error) {
+                       clauses := []dal.Clause{
+                               dal.From(&models.GithubReviewer{}),
+                               dal.Join("left join _tool_github_pull_requests 
" +
+                                       "on 
_tool_github_pull_requests.github_id = _tool_github_reviewers.pull_request_id"),
+                               dal.Where("repo_id = ? and 
_tool_github_pull_requests.connection_id = ?", repoId, 
data.Options.ConnectionId),
+                       }
+                       if stateManager.IsIncremental() {
+                               since := stateManager.GetSince()
+                               if since != nil {
+                                       clauses = append(clauses, 
dal.Where("_tool_github_pull_requests.github_updated_at >= ?", since))
+                               }
+                       }
+                       return db.Cursor(clauses...)
+               },
+               Convert: func(githubReview *models.GithubReviewer) 
([]interface{}, errors.Error) {
                        domainReview := &code.PullRequestReviewer{
                                PullRequestId: 
prIdGen.Generate(data.Options.ConnectionId, githubReview.PullRequestId),
                                ReviewerId:    
accountIdGen.Generate(data.Options.ConnectionId, githubReview.ReviewerId),
diff --git a/backend/server/services/remote/bridge/context.go 
b/backend/server/services/remote/bridge/context.go
index e66d1e66e..08393dd46 100644
--- a/backend/server/services/remote/bridge/context.go
+++ b/backend/server/services/remote/bridge/context.go
@@ -138,4 +138,11 @@ func (r remoteContextImpl) IncProgress(quantity int) {
        }
 }
 
+func (r remoteContextImpl) GetProgress() int {
+       if r.parent != nil {
+               return r.parent.GetProgress()
+       }
+       return 0
+}
+
 var _ RemoteContext = (*remoteContextImpl)(nil)

Reply via email to