Copilot commented on code in PR #19273:
URL: https://github.com/apache/iotdb/pull/19273#discussion_r4572720996


##########
iotdb-core/ainode/iotdb/ainode/core/model/tirex2/pipeline_tirex2.py:
##########
@@ -0,0 +1,130 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+import torch
+
+from iotdb.ainode.core.inference.pipeline.basic_pipeline import 
ForecastPipeline
+from iotdb.ainode.core.model.model_info import ModelInfo
+from iotdb.ainode.core.model.tirex2.types import TimeseriesType
+
+
+class TiRex2Pipeline(ForecastPipeline):
+    """
+    Forecasting pipeline of TiRex-2.
+
+    Each input is one series of one or more targets, forecast jointly, with 
optional past
+    covariates and future-known covariates. The model predicts nine quantiles 
(0.1 to 0.9) of up
+    to 320 future steps in one pass; the pipeline returns the median.
+    """
+
+    def __init__(self, model_info: ModelInfo, **model_kwargs):
+        super().__init__(model_info, **model_kwargs)
+
+    @property
+    def max_output_length(self) -> int:
+        return self.model.future_len
+
+    def _preprocess(self, inputs, **infer_kwargs) -> list[TimeseriesType]:
+        """
+        Convert the validated inputs to TiRex-2 series.
+
+        Parameters
+        ----------
+        inputs : list of dict
+            Each dictionary contains:
+            - 'targets': A tensor of shape (input_length,) or (target_count, 
input_length).
+            - 'past_covariates' (optional): A dictionary of tensors of shape 
(input_length,).
+            - 'future_covariates' (optional): A dictionary of tensors of shape 
(output_length,),
+              whose keys are a subset of 'past_covariates'.
+
+        infer_kwargs:
+            - `output_length` (int): The number of steps to forecast, at most 
320.
+
+        Returns
+        -------
+        list of TimeseriesType
+            One series per input: the targets as a (target_count, 
input_length) tensor, the
+            covariates known only in the past as a (count, input_length) 
tensor, and the
+            covariates also known in the future as a (count, input_length + 
output_length)
+            tensor of their past and future values.
+        """
+        output_length = infer_kwargs.get("output_length", 96)
+        if output_length > self.max_output_length:
+            raise ValueError(

Review Comment:
   With `tirex2` loaded into an inference pool, requesting 400 steps raises 
this exception in `_step()` and terminates the execution thread because 
`_requests_execute_loop()` does not catch it. The caller waits indefinitely in 
`InferenceRequestProxy.wait_for_result()`, and subsequent requests to that pool 
stop progressing. Validate the model-specific horizon before pool dispatch, or 
propagate worker errors to waiting requests while keeping the execution loop 
alive.



##########
iotdb-core/ainode/iotdb/ainode/core/model/tirex2/component/kernel/__init__.py:
##########
@@ -0,0 +1,49 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+# Copyright (c) NXAI GmbH.
+# SPDX-License-Identifier: Apache-2.0
+
+"""sLSTM and mLSTM recurrence kernels. Import the modules directly; the triton 
ones need triton."""
+
+from functools import cache
+
+import torch
+
+
+@cache
+def triton_available() -> bool:
+    """Whether the Triton kernels can be used; PyTorch's Windows wheels, for 
one, ship without Triton.
+
+    Triton compiles a kernel from its Python source, so the kernel modules are 
imported here as well:
+    they fail to import where Triton is installed but the source is not, as in 
a frozen executable.
+    """
+    try:
+        import triton  # noqa: F401
+
+        from iotdb.ainode.core.model.tirex2.component.kernel import (  # noqa: 
F401
+            mlstm_triton,
+            slstm_triton,
+        )
+    except Exception:
+        return False
+    return True
+
+
+def use_triton(x: torch.Tensor) -> bool:
+    """Whether to run the recurrence of ``x`` on the Triton kernels: on CUDA, 
when they are available."""
+    return x.is_cuda and triton_available()

Review Comment:
   On Volta or Turing GPUs, this predicate selects Triton even though the sLSTM 
kernel's BF16 `tl.dot` operations require compute capability 8.0 or newer, so 
forecasting fails during kernel compilation. AINode explicitly preserves Volta 
support (`pyproject.toml:82`). Check the input device's compute capability 
before selecting Triton so these GPUs use the PyTorch fallback.



##########
iotdb-core/ainode/iotdb/ainode/core/model/tirex2/component/kernel/slstm_torch.py:
##########
@@ -0,0 +1,135 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+# Copyright (c) NXAI GmbH.
+# SPDX-License-Identifier: Apache-2.0
+
+"""PyTorch sLSTM forward, used for the recurrence on CPU and MPS.
+
+Vendored from ``slstm_kernels/torch_kernel/slstm_torch.py`` in
+https://github.com/NX-AI/slstm-kernels at commit ``b2bf942``. Keep the two in
+sync; changes belong upstream first. Like the Triton kernel it keeps the states
+in float32 and rounds only the hidden state to bfloat16 for the recurrent 
matmul.
+"""
+
+import torch
+import torch.nn.functional as F
+
+NG_GATES = 4
+
+
+def check_tensors(x, states, weight, bias, cu_seqlens):
+    Bx, S, NH, NG, HD = x.shape
+    B = states.shape[0]
+
+    assert states.shape == (
+        B,
+        NH,
+        NG,
+        HD,
+    ), f"states must have shape [B, NH, NG, HD] = {(B, NH, NG, HD)}, got 
{tuple(states.shape)}!"
+    assert weight.shape == (
+        NH,
+        NG,
+        HD,
+        HD,
+    ), f"weight must have shape [NH, NG, HD, HD] = {(NH, NG, HD, HD)}, got 
{tuple(weight.shape)}!"
+    assert bias.shape == (
+        NH,
+        NG,
+        HD,
+    ), f"bias must have shape [NH, NG, HD] = {(NH, NG, HD)}, got 
{tuple(bias.shape)}!"
+
+    assert NG == NG_GATES, f"slstm requires NG={NG_GATES} gates (i, f, z, o), 
got {NG}!"
+
+    assert x.dtype in (torch.bfloat16, torch.float32), "x must be bfloat16!"
+    assert states.dtype == torch.float32, "states must be float32!"
+    assert weight.dtype in (torch.bfloat16, torch.float32), "weight must be 
bfloat16!"
+    assert bias.dtype == torch.float32, "bias must be float32!"
+
+    assert x.is_contiguous(), "x must be contiguous!"
+    assert states.is_contiguous(), "states must be contiguous!"
+    assert weight.is_contiguous(), "weight must be contiguous!"
+    assert bias.is_contiguous(), "bias must be contiguous!"
+
+    assert states.device == x.device, "states must be on the same device as x!"
+    assert weight.device == x.device, "weight must be on the same device as x!"
+    assert bias.device == x.device, "bias must be on the same device as x!"
+
+    if cu_seqlens is None:
+        assert Bx == B, "x and states must have the same batch size!"
+    else:
+        assert cu_seqlens.shape == (B + 1,), "cu_seqlens must have shape 
[B+1]!"
+        assert cu_seqlens.dtype == torch.int64, "cu_seqlens must be int64!"
+        assert (
+            cu_seqlens.device == x.device
+        ), "cu_seqlens must be on the same device as x!"
+        assert cu_seqlens.is_contiguous(), "cu_seqlens must be contiguous!"
+
+
+def slstm_torch(
+    x: torch.Tensor,  # [B, S, NH, NG, HD]
+    states: torch.Tensor,  # [B, NH, NG, HD]
+    weight: torch.Tensor,  # [NH, NG, HD, HD]
+    bias: torch.Tensor,  # [NH, NG, HD]
+    cu_seqlens: torch.Tensor = None,  # [B+1]
+) -> tuple[torch.Tensor, torch.Tensor]:  # [B, S, NH, HD], [B, NH, NG, HD]
+    check_tensors(x, states, weight, bias, cu_seqlens)
+    assert cu_seqlens is None, "cu_seqlens isn't implementent for torch 
backend!"
+    B, S, NH, NG, HD = x.shape
+    assert NG == NG_GATES, f"torch backend requires NG={NG_GATES} gates, got 
{NG}!"
+    assert states.shape == (B, NH, NG, HD)
+    states = states.unbind(dim=-2)
+
+    output = []
+    for i in range(S):
+        y_bf16, w_t = (
+            states[0].bfloat16().float(),
+            weight.view(NH, NG * HD, HD).transpose(-1, -2).float(),
+        )

Review Comment:
   `sLSTMLayer.slstm_kernels()` passes bfloat16 recurrent weights here, so 
`.float()` allocates and converts the entire recurrent matrix on every 
timestep, despite the matrix being unchanged throughout the sequence. This 
repeats the same conversion S times per sLSTM layer on every CPU forecast and 
CUDA fallback. Hoist the weight layout/conversion outside the loop, leaving 
only the hidden-state conversion inside.



##########
iotdb-core/ainode/iotdb/ainode/core/model/tirex2/component/mlstm_block.py:
##########
@@ -0,0 +1,197 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+# Copyright (c) NXAI GmbH.
+# SPDX-License-Identifier: Apache-2.0
+
+"""mLSTM layer copied from xLSTM Large, with TiRex-specific tweaks."""
+
+from dataclasses import dataclass
+from typing import Literal
+
+import torch
+import torch.nn as nn
+
+from iotdb.ainode.core.model.tirex2.component import kernel
+from iotdb.ainode.core.model.tirex2.component.norm import MultiHeadLayerNorm
+
+# soft_cap, bias_linspace_init_ and the mLSTMLayerConfig fields are copied 
from xlstm.
+
+
+def soft_cap(
+    values: torch.Tensor, cap_value: float | torch.Tensor | None
+) -> torch.Tensor:
+    """Soft caps a tensor to cap_value with a scaled tanh; no cap when 
cap_value is None."""
+    if cap_value is None:
+        return values
+    return cap_value * torch.tanh(values / cap_value)
+
+
+def bias_linspace_init_(
+    param: torch.Tensor, start: float = 3.4, end: float = 6.0
+) -> torch.Tensor:
+    """Linearly spaced bias init across dimensions."""
+    assert (
+        param.dim() == 1
+    ), f"param must be 1-dimensional (typically a bias), got {param.dim()}"
+    n_dims = param.shape[0]
+    init_vals = torch.linspace(start, end, n_dims)
+    with torch.no_grad():
+        param.copy_(init_vals)
+    return param
+
+
+@dataclass
+class mLSTMLayerConfig:
+    """Configuration of the xLSTM Large mLSTM layer."""
+
+    embedding_dim: int
+    num_heads: int
+    use_bias: bool = False
+    norm_eps: float = 1e-6
+    qk_dim_factor: float = 0.5
+    v_dim_factor: float = 1.0
+    gate_soft_cap: float = 15.0
+    # "auto" runs the Triton kernel for CUDA inputs when Triton is available 
and the native one otherwise.
+    mlstm_kernel: Literal["auto", "native", "triton"] = "auto"
+
+
+class mLSTMLayer(nn.Module):
+    """mLSTM implementation copied from xLSTM 7B."""
+
+    def __init__(self, config: mLSTMLayerConfig):
+        super().__init__()
+        self.config = config
+
+        self.v_dim = int(config.embedding_dim * config.v_dim_factor)
+        self.qk_dim = int(config.embedding_dim * config.qk_dim_factor)
+
+        # Fused input projection of q, k, v and the o, i, f gate 
pre-activations, in that order.
+        # The i and f gates always have a bias; without use_bias the rest of 
the bias starts at zero.
+        self.in_proj_split = (
+            self.qk_dim,
+            self.qk_dim,
+            self.v_dim,
+            self.v_dim,
+            self.config.num_heads,
+            self.config.num_heads,
+        )
+        self.in_proj = nn.Linear(
+            self.config.embedding_dim, sum(self.in_proj_split), bias=True
+        )
+        if not self.config.use_bias:
+            nn.init.zeros_(self.in_proj.bias)
+
+        self.ogate_act_fn = nn.Sigmoid()
+        if config.mlstm_kernel not in ("auto", "native", "triton"):
+            raise ValueError(
+                f"mLSTM kernel must be 'auto', 'native' or 'triton', got 
{config.mlstm_kernel!r}."
+            )
+
+        self.multihead_norm = MultiHeadLayerNorm(
+            num_heads=self.config.num_heads,
+            head_dim=self.v_dim // self.config.num_heads,
+            eps=self.config.norm_eps,
+        )
+        self.out_proj = nn.Linear(
+            in_features=self.v_dim,
+            out_features=self.config.embedding_dim,
+            bias=self.config.use_bias,
+        )
+
+        # Match mLSTMBlock.reset_parameters gate initialisation; i and f are 
the last rows of in_proj.
+        i_rows = slice(-2 * config.num_heads, -config.num_heads)
+        f_rows = slice(-config.num_heads, None)
+        with torch.no_grad():
+            torch.nn.init.zeros_(self.in_proj.weight[f_rows])
+            bias_linspace_init_(self.in_proj.bias[f_rows], start=3.0, end=6.0)
+            torch.nn.init.zeros_(self.in_proj.weight[i_rows])
+            torch.nn.init.normal_(self.in_proj.bias[i_rows], mean=0.0, std=0.1)
+
+    def _load_from_state_dict(self, state_dict, prefix, *args, **kwargs):
+        # Older checkpoints store a separate linear layer per projection
+        legacy = ("q", "k", "v", "ogate_preact", "igate_preact", 
"fgate_preact")
+        if all(f"{prefix}{name}.weight" in state_dict for name in legacy):

Review Comment:
   A checkpoint missing `igate_preact.bias` or `fgate_preact.bias` is silently 
accepted: this conversion substitutes zeros and creates a complete 
`in_proj.bias`, so the later `load_state_dict(strict=True)` cannot detect the 
missing learned parameter. Both biases are present in the published layout, 
even when `use_bias=False`, and replacing them changes forecasts. Require these 
gate biases (and all projection biases when `use_bias=True`) before performing 
the conversion, so incomplete checkpoints are rejected.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to