This is an automated email from the ASF dual-hosted git repository.
aglinxinyuan pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/texera.git
The following commit(s) were added to refs/heads/main by this push:
new b9bbf0db79 fix: admin-adjustable max columns with a clear overflow
message (#4669)
b9bbf0db79 is described below
commit b9bbf0db799c0867594e20181dd5a7a6bee38826
Author: Matthew B. <[email protected]>
AuthorDate: Thu May 7 00:08:09 2026 -0700
fix: admin-adjustable max columns with a clear overflow message (#4669)
<!--
Thanks for sending a pull request (PR)! Here are some tips for you:
1. If this is your first time, please read our contributor guidelines:
[Contributing to
Texera](https://github.com/apache/texera/blob/main/CONTRIBUTING.md)
2. Ensure you have added or run the appropriate tests for your PR
3. If the PR is work in progress, mark it a draft on GitHub.
4. Please write your PR title to summarize what this PR proposes, we
are following Conventional Commits style for PR titles as well.
5. Be sure to keep the PR description updated to reflect all changes.
-->
### What changes were proposed in this PR?
<!--
Please clarify what changes you are proposing. The purpose of this
section
is to outline the changes. Here are some tips for you:
1. If you propose a new API, clarify the use case for a new API.
2. If you fix a bug, you can clarify why it is a bug.
3. If it is a refactoring, clarify what has been changed.
3. It would be helpful to include a before-and-after comparison using
screenshots or GIFs.
4. Please consider writing useful notes for better and faster reviews.
-->
- **Admin-adjustable max columns.** Adds `csv_parser_max_columns` to
Admin → Settings → Result Panel. The CSV scan operator reads this from
`site_settings` at runtime.
- **Readable overflow error.** When a CSV row exceeds the limit, the
operator now throws `RuntimeException("Max columns of N exceeded.")`
Instead of letting Univocity's raw stack trace through. Detection looks
at the cause
(`ArrayIndexOutOfBoundsException`) and parses the index from both the
Java 8 and Java 9+ message formats — needed because Univocity's own hint
string is silently dropped on Java 9+.
- **Refactor.** Extracts the repeated "read int/long from
`site_settings` with fallback" pattern into `common/dao/SiteSettings`,
replacing inline jOOQ in `CSVScanSourceOpExec` and `DatasetResource`.
### Any related issues, documentation, discussions?
<!--
Please use this section to link other resources if not mentioned
already.
1. If this PR fixes an issue, please include `Fixes #1234`, `Resolves
#1234`
or `Closes #1234`. If it is only related, simply mention the issue
number.
2. If there is design documentation, please add the link.
3. If there is a discussion in the mailing list, please add the link.
-->
closes #4589
Related #3825
### How was this PR tested?
<!--
If tests were added, say they were added here. Or simply mention that if
the PR
is tested with existing test cases. Make sure to include/update test
cases that
check the changes thoroughly including negative and positive cases if
possible.
If it was tested in a way different from regular unit tests, please
clarify how
you tested step by step, ideally copy and paste-able, so that other
reviewers can
test and check, and descendants can verify in the future. If tests were
not added,
please describe why they were not added and/or why it was difficult to
add.
-->
- New `CSVScanSourceOpExecSpec`: happy path, end-of-input, overflow
translation against a real parser, and `isColumnOverflow` cases for both
AIOOBE message formats.
- Expanded `admin-settings.component.spec.ts` for load/save/reset of the
max columns setting.
- `sbt Test/compile` clean across DAO, ConfigService, WorkflowOperator,
FileService. Frontend specs need Node ≥ 20.19 to run.
- Manual: lowered the limit, ran a workflow on a CSV that exceeded it,
confirmed the result panel shows `Max columns of N exceeded.`
### Was this PR authored or co-authored using generative AI tooling?
<!--
If generative AI tooling has been used in the process of authoring this
PR,
please include the phrase: 'Generated-by: ' followed by the name of the
tool
and its version. If no, write 'No'.
Please refer to the [ASF Generative Tooling
Guidance](https://www.apache.org/legal/generative-tooling.html) for
details.
-->
Co-authored with: Claude Opus 4.7
---------
Co-authored-by: Chen Li <[email protected]>
Co-authored-by: Xinyuan Lin <[email protected]>
---
.../scala/org/apache/texera/dao/SiteSettings.scala | 57 +++++++++++
.../org/apache/texera/dao/SiteSettingsSpec.scala | 45 +++++++++
.../source/scan/csv/CSVScanSourceOpDesc.scala | 7 +-
.../source/scan/csv/CSVScanSourceOpExec.scala | 46 ++++++++-
.../source/scan/csv/CSVScanSourceOpExecSpec.scala | 110 +++++++++++++++++++++
.../texera/service/resource/DatasetResource.scala | 16 +--
.../admin/settings/admin-settings.component.html | 33 +++++++
.../admin/settings/admin-settings.component.ts | 37 ++++++-
.../result-table-frame.component.spec.ts | 19 ++--
9 files changed, 346 insertions(+), 24 deletions(-)
diff --git a/common/dao/src/main/scala/org/apache/texera/dao/SiteSettings.scala
b/common/dao/src/main/scala/org/apache/texera/dao/SiteSettings.scala
new file mode 100644
index 0000000000..a1d70ea841
--- /dev/null
+++ b/common/dao/src/main/scala/org/apache/texera/dao/SiteSettings.scala
@@ -0,0 +1,57 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.texera.dao
+
+import org.jooq.impl.DSL
+
+import scala.util.Try
+
+/**
+ * Read-side accessor for the `site_settings` key/value table that admin pages
+ * write through. Centralises the "look up by key, parse, fall back on any
+ * failure" pattern that previously lived inline in ConfigResource,
+ * CSVScanSourceOpExec, and DatasetResource.
+ *
+ * Failures swallowed by the outer Try include: SqlServer not initialised
+ * (e.g. on workers in distributed mode), no row for the key, and value that
+ * can't be parsed. In all of these cases the caller's default takes over.
+ */
+object SiteSettings {
+
+ def getInt(key: String, default: => Int): Int =
+ readAndParse(key, default)(_.toInt)
+
+ def getLong(key: String, default: => Long): Long =
+ readAndParse(key, default)(_.toLong)
+
+ private[dao] def parseOrDefault[T](raw: Option[String], default: T)(parse:
String => T): T =
+ raw.flatMap(s => Try(parse(s.trim)).toOption).getOrElse(default)
+
+ private def readAndParse[T](key: String, default: => T)(parse: String => T):
T =
+ Try {
+ val raw = SqlServer
+ .getInstance()
+ .createDSLContext()
+ .select(DSL.field("value", classOf[String]))
+ .from(DSL.table(DSL.name("texera_db", "site_settings")))
+ .where(DSL.field("key", classOf[String]).eq(key))
+ .fetchOneInto(classOf[String])
+ parseOrDefault(Option(raw), default)(parse)
+ }.getOrElse(default)
+}
diff --git
a/common/dao/src/test/scala/org/apache/texera/dao/SiteSettingsSpec.scala
b/common/dao/src/test/scala/org/apache/texera/dao/SiteSettingsSpec.scala
new file mode 100644
index 0000000000..00f7cd359d
--- /dev/null
+++ b/common/dao/src/test/scala/org/apache/texera/dao/SiteSettingsSpec.scala
@@ -0,0 +1,45 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.texera.dao
+
+import org.scalatest.flatspec.AnyFlatSpec
+import org.scalatest.matchers.should.Matchers
+
+class SiteSettingsSpec extends AnyFlatSpec with Matchers {
+
+ "parseOrDefault" should "return the parsed value when the raw string is
present and valid" in {
+ SiteSettings.parseOrDefault(Some("42"), 0)(_.toInt) shouldBe 42
+ }
+
+ it should "return the default when the Option is None" in {
+ SiteSettings.parseOrDefault(None, 99)(_.toInt) shouldBe 99
+ }
+
+ it should "return the default when the string cannot be parsed" in {
+ SiteSettings.parseOrDefault(Some("not-a-number"), 7)(_.toInt) shouldBe 7
+ }
+
+ it should "trim whitespace before parsing" in {
+ SiteSettings.parseOrDefault(Some(" 100 "), 0)(_.toInt) shouldBe 100
+ }
+
+ it should "work for Long values" in {
+ SiteSettings.parseOrDefault(Some("9999999999"), 0L)(_.toLong) shouldBe
9999999999L
+ }
+}
diff --git
a/common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/source/scan/csv/CSVScanSourceOpDesc.scala
b/common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/source/scan/csv/CSVScanSourceOpDesc.scala
index a44e2765d5..57b173583e 100644
---
a/common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/source/scan/csv/CSVScanSourceOpDesc.scala
+++
b/common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/source/scan/csv/CSVScanSourceOpDesc.scala
@@ -29,6 +29,7 @@ import org.apache.texera.amber.core.tuple.{AttributeType,
Schema}
import org.apache.texera.amber.core.virtualidentity.{ExecutionIdentity,
WorkflowIdentity}
import org.apache.texera.amber.core.workflow.{PhysicalOp,
SchemaPropagationFunc}
import org.apache.texera.amber.operator.source.scan.ScanSourceOpDesc
+import org.apache.texera.amber.operator.source.scan.csv.CSVScanSourceOpExec
import org.apache.texera.amber.util.JSONUtils.objectMapper
import java.io.{IOException, InputStreamReader}
@@ -89,6 +90,8 @@ class CSVScanSourceOpDesc extends ScanSourceOpDesc {
csvFormat.setLineSeparator("\n")
val csvSetting = new CsvParserSettings()
csvSetting.setMaxCharsPerColumn(-1)
+ val maxColumns = CSVScanSourceOpExec.getMaxColumns
+ csvSetting.setMaxColumns(maxColumns)
csvSetting.setFormat(csvFormat)
csvSetting.setHeaderExtractionEnabled(hasHeader)
csvSetting.setNullValue("")
@@ -97,8 +100,8 @@ class CSVScanSourceOpDesc extends ScanSourceOpDesc {
var data: Array[Array[String]] = Array()
val readLimit = limit.getOrElse(INFER_READ_LIMIT).min(INFER_READ_LIMIT)
- for (i <- 0 until readLimit) {
- val row = parser.parseNext()
+ for (_ <- 0 until readLimit) {
+ val row = CSVScanSourceOpExec.parseNextRow(parser, maxColumns)
if (row != null) {
data = data :+ row
}
diff --git
a/common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/source/scan/csv/CSVScanSourceOpExec.scala
b/common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/source/scan/csv/CSVScanSourceOpExec.scala
index c3fbbe9bb5..147238536d 100644
---
a/common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/source/scan/csv/CSVScanSourceOpExec.scala
+++
b/common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/source/scan/csv/CSVScanSourceOpExec.scala
@@ -19,15 +19,18 @@
package org.apache.texera.amber.operator.source.scan.csv
+import com.univocity.parsers.common.TextParsingException
import com.univocity.parsers.csv.{CsvFormat, CsvParser, CsvParserSettings}
import org.apache.texera.amber.core.executor.SourceOperatorExecutor
import org.apache.texera.amber.core.storage.DocumentFactory
import org.apache.texera.amber.core.tuple.{AttributeTypeUtils, Schema,
TupleLike}
import org.apache.texera.amber.util.JSONUtils.objectMapper
+import org.apache.texera.dao.SiteSettings
import java.io.InputStreamReader
import java.net.URI
import scala.collection.immutable.ArraySeq
+import scala.util.Try
class CSVScanSourceOpExec private[csv] (descString: String) extends
SourceOperatorExecutor {
val desc: CSVScanSourceOpDesc = objectMapper.readValue(descString,
classOf[CSVScanSourceOpDesc])
@@ -35,6 +38,7 @@ class CSVScanSourceOpExec private[csv] (descString: String)
extends SourceOperat
var parser: CsvParser = _
var nextRow: Array[String] = _
var numRowGenerated = 0
+ private var maxColumns: Int = CSVScanSourceOpExec.DEFAULT_MAX_COLUMNS
private val schema: Schema = desc.sourceSchema()
override def produceTuple(): Iterator[TupleLike] = {
@@ -44,7 +48,7 @@ class CSVScanSourceOpExec private[csv] (descString: String)
extends SourceOperat
if (nextRow != null) {
return true
}
- nextRow = parser.parseNext()
+ nextRow = CSVScanSourceOpExec.parseNextRow(parser, maxColumns)
nextRow != null
}
@@ -90,6 +94,8 @@ class CSVScanSourceOpExec private[csv] (descString: String)
extends SourceOperat
) // disable skipping lines starting with # (default comment character)
val csvSetting = new CsvParserSettings()
csvSetting.setMaxCharsPerColumn(-1)
+ maxColumns = CSVScanSourceOpExec.getMaxColumns
+ csvSetting.setMaxColumns(maxColumns)
csvSetting.setFormat(csvFormat)
csvSetting.setHeaderExtractionEnabled(desc.hasHeader)
@@ -106,3 +112,41 @@ class CSVScanSourceOpExec private[csv] (descString:
String) extends SourceOperat
}
}
}
+
+object CSVScanSourceOpExec {
+ val DEFAULT_MAX_COLUMNS = 512
+
+ def getMaxColumns: Int =
+ SiteSettings.getInt("csv_parser_max_columns", DEFAULT_MAX_COLUMNS)
+
+ /**
+ * Wraps `parser.parseNext()` so a column-count overflow is reported to the
user
+ * as a clear instruction rather than a deep Univocity stack trace. Other
parser
+ * failures are rethrown unchanged.
+ *
+ * The thrown RuntimeException's message bubbles up through
DataProcessor.handleExecutorException
+ * and becomes the title of the console message that drives the top-of-page
toast.
+ */
+ def parseNextRow(parser: CsvParser, maxColumns: Int): Array[String] = {
+ try parser.parseNext()
+ catch {
+ case e: TextParsingException if isColumnOverflow(e, maxColumns) =>
+ throw new RuntimeException(columnOverflowMessage(maxColumns), e)
+ }
+ }
+
+ private[csv] def isColumnOverflow(e: TextParsingException, maxColumns: Int):
Boolean =
+ Option(e.getCause)
+ .collect { case aioobe: ArrayIndexOutOfBoundsException => aioobe }
+ .exists(aioobe => aioobeIndex(aioobe).exists(_ == maxColumns))
+
+ private def aioobeIndex(aioobe: ArrayIndexOutOfBoundsException): Option[Int]
= {
+ val msg = Option(aioobe.getMessage).getOrElse("")
+ Try(msg.trim.toInt).toOption.orElse {
+ raw"Index (\d+) out of
bounds".r.findFirstMatchIn(msg).map(_.group(1).toInt)
+ }
+ }
+
+ private[csv] def columnOverflowMessage(maxColumns: Int): String =
+ s"Max columns of $maxColumns exceeded."
+}
diff --git
a/common/workflow-operator/src/test/scala/org/apache/texera/amber/operator/source/scan/csv/CSVScanSourceOpExecSpec.scala
b/common/workflow-operator/src/test/scala/org/apache/texera/amber/operator/source/scan/csv/CSVScanSourceOpExecSpec.scala
new file mode 100644
index 0000000000..ec4b9d6244
--- /dev/null
+++
b/common/workflow-operator/src/test/scala/org/apache/texera/amber/operator/source/scan/csv/CSVScanSourceOpExecSpec.scala
@@ -0,0 +1,110 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.texera.amber.operator.source.scan.csv
+
+import com.univocity.parsers.common.TextParsingException
+import com.univocity.parsers.csv.{CsvParser, CsvParserSettings}
+import org.scalatest.flatspec.AnyFlatSpec
+
+import java.io.StringReader
+
+/**
+ * Verifies the column-overflow translation in
[[CSVScanSourceOpExec.parseNextRow]]
+ * — the path that turns a deep Univocity stack trace into a single-sentence
message
+ * the workflow user can act on.
+ */
+class CSVScanSourceOpExecSpec extends AnyFlatSpec {
+
+ private def parserWithMaxColumns(max: Int): CsvParser = {
+ val settings = new CsvParserSettings()
+ settings.setMaxColumns(max)
+ settings.setMaxCharsPerColumn(-1)
+ new CsvParser(settings)
+ }
+
+ "parseNextRow" should "return the parsed row when the input is within the
column limit" in {
+ val parser = parserWithMaxColumns(10)
+ parser.beginParsing(new StringReader("a,b,c\n"))
+
+ val row = CSVScanSourceOpExec.parseNextRow(parser, 10)
+
+ assert(row.toSeq == Seq("a", "b", "c"))
+ }
+
+ it should "return null at end of input (so the iterator can terminate
cleanly)" in {
+ val parser = parserWithMaxColumns(10)
+ parser.beginParsing(new StringReader(""))
+
+ assert(CSVScanSourceOpExec.parseNextRow(parser, 10) == null)
+ }
+
+ it should "translate a column-overflow TextParsingException into a clear
user message" in {
+ val maxColumns = 2
+ val parser = parserWithMaxColumns(maxColumns)
+ parser.beginParsing(new StringReader("a,b,c,d,e\n"))
+
+ val ex = intercept[RuntimeException] {
+ CSVScanSourceOpExec.parseNextRow(parser, maxColumns)
+ }
+
+ // The message must mention the configured limit so the user knows what
was hit.
+ assert(ex.getMessage.contains(maxColumns.toString))
+ assert(ex.getMessage.toLowerCase.contains("max columns"))
+ assert(ex.getMessage.toLowerCase.contains("exceeded"))
+ // The original Univocity exception is preserved as the cause so developers
+ // can still inspect the underlying parser state if needed.
+ assert(ex.getCause.isInstanceOf[TextParsingException])
+ }
+
+ "isColumnOverflow" should "detect AIOOBE causes from Java 8's plain-integer
message" in {
+ val cause = new ArrayIndexOutOfBoundsException("5")
+ val ex = new TextParsingException(null, "wrapper", cause)
+ assert(CSVScanSourceOpExec.isColumnOverflow(ex, maxColumns = 5))
+ assert(!CSVScanSourceOpExec.isColumnOverflow(ex, maxColumns = 6))
+ }
+
+ it should "detect AIOOBE causes from Java 9+'s 'Index N out of bounds for
length M' message" in {
+ val cause = new ArrayIndexOutOfBoundsException("Index 5 out of bounds for
length 5")
+ val ex = new TextParsingException(null, "wrapper", cause)
+ assert(CSVScanSourceOpExec.isColumnOverflow(ex, maxColumns = 5))
+ assert(!CSVScanSourceOpExec.isColumnOverflow(ex, maxColumns = 6))
+ }
+
+ it should "ignore TextParsingExceptions whose cause is unrelated" in {
+ val unrelated = new TextParsingException(null, "Some other parsing
problem")
+ val withDifferentCause =
+ new TextParsingException(null, "wrapper", new
IllegalStateException("nope"))
+ assert(!CSVScanSourceOpExec.isColumnOverflow(unrelated, maxColumns = 5))
+ assert(!CSVScanSourceOpExec.isColumnOverflow(withDifferentCause,
maxColumns = 5))
+ }
+
+ it should "ignore an AIOOBE whose message cannot be parsed as an index" in {
+ val unparseable = new ArrayIndexOutOfBoundsException("something went
wrong")
+ val ex = new TextParsingException(null, "wrapper", unparseable)
+ assert(!CSVScanSourceOpExec.isColumnOverflow(ex, maxColumns = 5))
+ }
+
+ "columnOverflowMessage" should "include the configured maximum so the user
knows the current limit" in {
+ val msg = CSVScanSourceOpExec.columnOverflowMessage(750)
+ assert(msg.contains("750"))
+ assert(msg.toLowerCase.contains("max columns"))
+ assert(msg.toLowerCase.contains("exceeded"))
+ }
+}
diff --git
a/file-service/src/main/scala/org/apache/texera/service/resource/DatasetResource.scala
b/file-service/src/main/scala/org/apache/texera/service/resource/DatasetResource.scala
index 7bcd3bb77c..46457c9454 100644
---
a/file-service/src/main/scala/org/apache/texera/service/resource/DatasetResource.scala
+++
b/file-service/src/main/scala/org/apache/texera/service/resource/DatasetResource.scala
@@ -28,6 +28,7 @@ import org.apache.texera.amber.core.storage.model.OnDataset
import org.apache.texera.amber.core.storage.util.LakeFSStorageClient
import org.apache.texera.amber.core.storage.{DocumentFactory, FileResolver}
import org.apache.texera.auth.SessionUser
+import org.apache.texera.dao.SiteSettings
import org.apache.texera.dao.SqlServer
import org.apache.texera.dao.SqlServer.withTransaction
import org.apache.texera.dao.jooq.generated.enums.PrivilegeEnum
@@ -87,15 +88,8 @@ object DatasetResource {
.getInstance()
.createDSLContext()
- private def singleFileUploadMaxBytes(ctx: DSLContext, defaultMiB: Long =
20L): Long = {
- val limit = ctx
- .select(DSL.field("value", classOf[String]))
- .from(DSL.table(DSL.name("texera_db", "site_settings")))
- .where(DSL.field("key",
classOf[String]).eq("single_file_upload_max_size_mib"))
- .fetchOneInto(classOf[String])
- Try(Option(limit).getOrElse(defaultMiB.toString).trim.toLong)
- .getOrElse(defaultMiB) * 1024L * 1024L
- }
+ private def singleFileUploadMaxBytes(defaultMiB: Long = 20L): Long =
+ SiteSettings.getLong("single_file_upload_max_size_mib", defaultMiB) *
1024L * 1024L
/**
* Helper function to get the dataset from DB using did
@@ -1577,7 +1571,7 @@ class DatasetResource {
if (fileSizeBytesValue <= 0L) throw new
BadRequestException("fileSizeBytes must be > 0")
if (partSizeBytesValue <= 0L) throw new
BadRequestException("partSizeBytes must be > 0")
- val totalMaxBytes: Long = singleFileUploadMaxBytes(ctx)
+ val totalMaxBytes: Long = singleFileUploadMaxBytes()
if (totalMaxBytes <= 0L) {
throw new WebApplicationException(
"singleFileUploadMaxBytes must be > 0",
@@ -1969,7 +1963,7 @@ class DatasetResource {
)
}
- val maxBytes = singleFileUploadMaxBytes(ctx)
+ val maxBytes = singleFileUploadMaxBytes()
val tooLarge = actualSizeBytes > maxBytes
if (tooLarge) {
diff --git
a/frontend/src/app/dashboard/component/admin/settings/admin-settings.component.html
b/frontend/src/app/dashboard/component/admin/settings/admin-settings.component.html
index 319e13c345..01ab1e4ceb 100644
---
a/frontend/src/app/dashboard/component/admin/settings/admin-settings.component.html
+++
b/frontend/src/app/dashboard/component/admin/settings/admin-settings.component.html
@@ -349,3 +349,36 @@
</button>
</div>
</nz-card>
+
+<nz-card nzTitle="Result Panel">
+ <div class="settings-row">
+ <span>Max Columns:</span>
+ <nz-input-number
+ [(ngModel)]="csvMaxColumns"
+ [nzMin]="MIN_CSV_MAX_COLUMNS"
+ [nzMax]="MAX_CSV_MAX_COLUMNS"
+ [nzStep]="100"
+ [nzPrecision]="0">
+ </nz-input-number>
+ </div>
+ <div class="help-text-number">
+ Maximum number of columns the CSV parser will accept per row. The
Univocity parser default is 512. Increase this
+ value if your CSV files have more than 512 columns. (Range: {{
MIN_CSV_MAX_COLUMNS }} - {{ MAX_CSV_MAX_COLUMNS |
+ number }})
+ </div>
+
+ <div class="button-row">
+ <button
+ nz-button
+ nzType="primary"
+ (click)="saveCsvSettings()">
+ Save
+ </button>
+ <button
+ nz-button
+ nzType="default"
+ (click)="resetCsvSettings()">
+ Reset
+ </button>
+ </div>
+</nz-card>
diff --git
a/frontend/src/app/dashboard/component/admin/settings/admin-settings.component.ts
b/frontend/src/app/dashboard/component/admin/settings/admin-settings.component.ts
index 3d0e759065..691ae79f6f 100644
---
a/frontend/src/app/dashboard/component/admin/settings/admin-settings.component.ts
+++
b/frontend/src/app/dashboard/component/admin/settings/admin-settings.component.ts
@@ -20,6 +20,7 @@
import { Component, OnInit } from "@angular/core";
import { AdminSettingsService } from
"../../../service/admin/settings/admin-settings.service";
import { NzMessageService } from "ng-zorro-antd/message";
+import { NotificationService } from
"../../../../common/service/notification/notification.service";
import { UntilDestroy, untilDestroyed } from "@ngneat/until-destroy";
import { SidebarTabs } from "../../../../common/type/gui-config";
import { forkJoin } from "rxjs";
@@ -79,22 +80,29 @@ export class AdminSettingsComponent implements OnInit {
maxConcurrentChunks: number = 10;
chunkSizeMiB: number = 50;
+ csvMaxColumns: number = 512;
+
// S3 Multipart Upload Constraints
readonly MIN_PART_SIZE_MiB = 5; // 5 MiB minimum for parts (except last part)
readonly MAX_PART_SIZE_MiB = 5120; // 5 GiB maximum per part (5 * 1024 MiB)
readonly MAX_FILE_SIZE_MiB = 5242880; // 5 TiB maximum object size (5 * 1024
* 1024 MiB)
readonly MAX_TOTAL_PARTS = 10000; // S3 maximum parts per upload
+ readonly MIN_CSV_MAX_COLUMNS = 1;
+ readonly MAX_CSV_MAX_COLUMNS = 100000;
+
private readonly RELOAD_DELAY = 1000;
constructor(
private adminSettingsService: AdminSettingsService,
- private message: NzMessageService
+ private message: NzMessageService,
+ private notificationService: NotificationService
) {}
ngOnInit(): void {
this.loadBranding();
this.loadTabs();
this.loadDatasetSettings();
+ this.loadCsvSettings();
}
private loadBranding(): void {
@@ -284,4 +292,31 @@ export class AdminSettingsComponent implements OnInit {
this.message.info("Resetting dataset settings...");
setTimeout(() => window.location.reload(), this.RELOAD_DELAY);
}
+
+ private loadCsvSettings(): void {
+ this.adminSettingsService
+ .getSetting("csv_parser_max_columns")
+ .pipe(untilDestroyed(this))
+ .subscribe(value => (this.csvMaxColumns = parseInt(value) || 512));
+ }
+
+ saveCsvSettings(): void {
+ const saveRequests = [
+ this.adminSettingsService.updateSetting("csv_parser_max_columns",
this.csvMaxColumns.toString()),
+ ];
+
+ forkJoin(saveRequests)
+ .pipe(untilDestroyed(this))
+ .subscribe({
+ next: () => this.notificationService.success("Result panel settings
saved."),
+ error: () => this.notificationService.error("Could not save result
panel settings."),
+ });
+ }
+
+ resetCsvSettings(): void {
+
this.adminSettingsService.resetSetting("csv_parser_max_columns").pipe(untilDestroyed(this)).subscribe({});
+
+ this.notificationService.info("Resetting result panel settings...");
+ setTimeout(() => window.location.reload(), this.RELOAD_DELAY);
+ }
}
diff --git
a/frontend/src/app/workspace/component/result-panel/result-table-frame/result-table-frame.component.spec.ts
b/frontend/src/app/workspace/component/result-panel/result-table-frame/result-table-frame.component.spec.ts
index 36d94fd137..21dd4db5b0 100644
---
a/frontend/src/app/workspace/component/result-panel/result-table-frame/result-table-frame.component.spec.ts
+++
b/frontend/src/app/workspace/component/result-panel/result-table-frame/result-table-frame.component.spec.ts
@@ -24,6 +24,8 @@ import { OperatorMetadataService } from
"../../../service/operator-metadata/oper
import { StubOperatorMetadataService } from
"../../../service/operator-metadata/stub-operator-metadata.service";
import { HttpClientTestingModule } from "@angular/common/http/testing";
import { NzModalModule } from "ng-zorro-antd/modal";
+import { NzTableModule } from "ng-zorro-antd/table";
+import { NoopAnimationsModule } from "@angular/platform-browser/animations";
import { commonTestProviders } from "../../../../common/testing/test-utils";
import { GuiConfigService } from
"../../../../common/service/gui-config.service";
@@ -31,9 +33,11 @@ describe("ResultTableFrameComponent", () => {
let component: ResultTableFrameComponent;
let fixture: ComponentFixture<ResultTableFrameComponent>;
+ const GUI_CONFIG_LIMIT = 15;
+
beforeEach(async () => {
await TestBed.configureTestingModule({
- imports: [ResultTableFrameComponent, HttpClientTestingModule,
NzModalModule],
+ imports: [ResultTableFrameComponent, HttpClientTestingModule,
NzModalModule, NzTableModule, NoopAnimationsModule],
providers: [
{
provide: OperatorMetadataService,
@@ -43,16 +47,13 @@ describe("ResultTableFrameComponent", () => {
provide: GuiConfigService,
useValue: {
env: {
- limitColumns: 15,
+ limitColumns: GUI_CONFIG_LIMIT,
},
},
},
...commonTestProviders,
],
}).compileComponents();
- });
-
- beforeEach(() => {
fixture = TestBed.createComponent(ResultTableFrameComponent);
component = fixture.componentInstance;
fixture.detectChanges();
@@ -62,14 +63,14 @@ describe("ResultTableFrameComponent", () => {
expect(component).toBeTruthy();
});
- it("currentResult should not be modified if setupResultTable is called with
empty (zero-length) execution result ", () => {
+ it("currentResult should not be modified if setupResultTable is called with
empty (zero-length) execution result", () => {
component.currentResult = [{ test: "property" }];
- (component as any).setupResultTable([]);
+ (component as any).setupResultTable([], 0);
expect(component.currentResult).toEqual([{ test: "property" }]);
});
- it("should set columnLimit from config", () => {
- expect(component.columnLimit).toEqual(15);
+ it("should set columnLimit from gui-config", () => {
+ expect(component.columnLimit).toEqual(GUI_CONFIG_LIMIT);
});
});
