bitflicker64 commented on code in PR #839: URL: https://github.com/apache/hugegraph-toolchain/pull/839#discussion_r4353257981
########## hugegraph-dist/scripts/dependency/known-dependencies.txt: ########## @@ -43,6 +43,7 @@ avro-1.7.7.jar bolt-1.6.4.jar cache-api-1.1.1.jar caffeine-2.8.0.jar +caffeine-3.2.4.jar Review Comment: Important, blocking: this list no longer matches the runtime jars, so `check_dependencies.sh` fails. Moving Caffeine from 2.8.0 to the Boot-managed 3.2.4 also drops `checker-qual-2.10.0` and `error_prone_annotations-2.3.3`, and brings in `error_prone_annotations-2.49.0`. Only `caffeine-3.2.4.jar` was added. A full-reactor runtime `dependency:list` gives this diff against the file: ``` -caffeine-2.8.0.jar -checker-qual-2.10.0.jar -error_prone_annotations-2.3.3.jar +error_prone_annotations-2.49.0.jar ``` The org mirror PR (hugegraph/hugegraph-toolchain#85) fails `check-dependency-license` with the same four lines at b5b63d71. At the merge base the list matches exactly. Please rerun `regenerate_known_dependencies.sh`. `release-docs/NOTICE` also still names `error_prone_annotations:jar:2.3.3`. ########## hugegraph-hubble/hubble-be/src/main/java/org/apache/hugegraph/service/load/FilePreviewReader.java: ########## @@ -0,0 +1,125 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with this + * work for additional information regarding copyright ownership. The ASF + * licenses this file to You under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT + * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the + * License for the specific language governing permissions and limitations + * under the License. + */ + +package org.apache.hugegraph.service.load; + +import java.io.BufferedReader; +import java.io.IOException; +import java.io.InputStreamReader; +import java.nio.charset.Charset; +import java.nio.file.Files; +import java.nio.file.Path; +import java.util.ArrayList; +import java.util.Arrays; +import java.util.LinkedHashMap; +import java.util.List; +import java.util.Map; +import java.util.regex.Pattern; + +import org.apache.hugegraph.entity.load.FileSetting; +import org.apache.hugegraph.exception.InternalException; +import org.apache.hugegraph.loader.parser.CsvLineParser; +import org.apache.hugegraph.loader.parser.LineParser; +import org.apache.hugegraph.loader.parser.TextLineParser; +import org.apache.hugegraph.loader.source.file.FileFormat; +import org.apache.hugegraph.util.Ex; + +import com.fasterxml.jackson.core.JsonParser; +import com.fasterxml.jackson.core.type.TypeReference; +import com.fasterxml.jackson.databind.ObjectMapper; + +/** + * Reads a header and one physical data line using the loader's delimiter semantics. + */ +public final class FilePreviewReader { + + private static final ObjectMapper JSON_MAPPER = + new ObjectMapper().disable(JsonParser.Feature.INCLUDE_SOURCE_IN_LOCATION); + + private FilePreviewReader() { + } + + public static Columns read(Path file, FileSetting setting) { + Charset charset = Charset.forName(setting.getCharset()); + Pattern skippedLine = Pattern.compile(setting.getSkippedLine()); + FileFormat format = FileFormat.valueOf(setting.getFormat()); + // Upload previews run before users can select the encoding. Preserve undecodable + // bytes as replacement characters so the file settings can still be corrected. + try (BufferedReader reader = new BufferedReader( + new InputStreamReader(Files.newInputStream(file), charset))) { + String firstLine = nextDataLine(reader, skippedLine); + Ex.check(firstLine != null, "The file has no data line can treat as header"); + return switch (format) { + case CSV -> readDelimited(reader, firstLine, skippedLine, + new CsvLineParser(), setting.isHasHeader()); + case TEXT -> readDelimited(reader, firstLine, skippedLine, + new TextLineParser(setting.getDelimiter()), setting.isHasHeader()); + case JSON -> readJson(firstLine, file); + }; + } catch (IOException e) { + throw new InternalException("Failed to read columns from file %s", e, file); + } + } + + private static Columns readDelimited(BufferedReader reader, String firstLine, + Pattern skippedLine, LineParser parser, + boolean hasHeader) throws IOException { + List<String> firstValues = Arrays.asList(parser.split(firstLine)); + if (hasHeader) { + String sample = nextDataLine(reader, skippedLine); + List<String> values = sample == null ? List.of() : Arrays.asList(parser.split(sample)); + return new Columns(firstValues, values); + } + List<String> names = new ArrayList<>(firstValues.size()); + for (int i = 1; i <= firstValues.size(); i++) { + names.add("col-" + i); + } + return new Columns(names, firstValues); + } + + private static String nextDataLine(BufferedReader reader, Pattern skippedLine) throws IOException { + String line; + while ((line = reader.readLine()) != null) { + if (!skippedLine.matcher(line).matches()) { + return line; Review Comment: Important, not blocking: a CSV whose line after the header is blank now fails to preview. `nextDataLine` returns the empty line, and `CsvLineParser.split("")` throws `ReadException: Parse line '' error`. For `a,b\n\n1,2\n` the base returned names `[a, b]` with an empty sample. At this head, `extractColumns` throws during upload in `FileUploadController`, so the mapping never reaches COMPLETED. The ingest `/schemas` endpoint and task creation without a supplied header fail the same way. Hubble's default `skipped_line` does not match empty lines, unlike the loader's `(^#|^//).*|`. The simplest fix is to skip empty lines in `nextDataLine`, for example `if (!line.isEmpty() && !skippedLine.matcher(line).matches())`. A `FilePreviewReaderTest` case for `a,b\n\n1,2\n` would cover it. ########## hugegraph-hubble/hubble-be/src/main/java/org/apache/hugegraph/util/HubbleUtil.java: ########## @@ -56,149 +68,84 @@ public static <T> T uncheckedCast(Object value) { } public static String dateFormat() { - return DATE_FORMAT.format(new Date()); + return DATE_TIME_FORMAT.format(CLOCK.instant()); } public static String dateFormatMillis() { Review Comment: Minor: nothing under `src/main` calls `dateFormatMillis`, `dateFormatMonth`, `dateFormatLastMonth`, `getTimestampsBefore7Days`, `getWeekTimestamps` or `getMonthTimestamps`, at the base or at this head. Four are called only from tests, and `dateFormatMillis` and `getTimestampsBefore7Days` have no callers at all. The week and month boundary fixes therefore change nothing at runtime. Deleting these methods, along with `MILLIS_FORMAT`, `MONTH_FORMAT`, `calendarRange` and their tests, is simpler than rewriting them. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
