bitflicker64 opened a new pull request, #478:
URL: https://github.com/apache/hugegraph-doc/pull/478
## Purpose of the PR
Sync the hugegraph-loader docs (en + cn) with hugegraph-toolchain master
(3b385c3d).
The two quickstart pages had drifted from the code: several mapping-file
defaults were wrong, a number of input-source keys were never documented, the
breakpoint and failure-file section described a layout the loader no longer
uses, and the flink-cdc loader had no section at all. The English page also had
a table row with a heading glued onto it, which broke the rest of the parameter
table.
The performance pages
(`content/{en,cn}/docs/performance/hugegraph-loader-performance.md`) state no
options or defaults that master contradicts, so they are untouched.
| Page | What was wrong | What changed | Source on master |
|------|----------------|--------------|------------------|
| en + cn quickstart | Sections 1 and 3.2 listed only files, HDFS and
relational databases as data sources | Added the Kafka topic and the GRAPH
input source |
hugegraph-loader/src/main/java/org/apache/hugegraph/loader/source/SourceType.java:22
|
| en + cn quickstart | FILE `format` was marked required with no default |
Marked optional, default CSV |
hugegraph-loader/src/main/java/org/apache/hugegraph/loader/source/file/FileSource.java:110
|
| en + cn quickstart | FILE `delimiter` was documented as defaulting to a
comma for every format | Default is now given per format, a comma for CSV and a
tab for TEXT, and CSV accepts no other delimiter |
hugegraph-loader/src/main/java/org/apache/hugegraph/loader/source/file/FileFormat.java:24,
hugegraph-loader/src/main/java/org/apache/hugegraph/loader/source/file/FileSource.java:111
|
| en + cn quickstart | FILE `has_header` was undocumented | Documented the
repeated-header check and how to switch it off |
hugegraph-loader/src/main/java/org/apache/hugegraph/loader/source/file/FileSource.java:63,
hugegraph-loader/src/main/java/org/apache/hugegraph/loader/reader/file/FileLineFetcher.java:227
|
| en + cn quickstart | FILE `extra_date_formats` was undocumented |
Documented as a fallback list for `date_format` |
hugegraph-loader/src/main/java/org/apache/hugegraph/loader/source/file/FileSource.java:49,
hugegraph-loader/src/main/java/org/apache/hugegraph/loader/util/DataTypeUtil.java:145
|
| en + cn quickstart | `skipped_line` was said to skip nothing by default |
Gave the real default regex `(^#\|^//).*\|` and how to disable it |
hugegraph-loader/src/main/java/org/apache/hugegraph/loader/source/file/SkippedLine.java:36,
hugegraph-loader/src/main/java/org/apache/hugegraph/loader/constant/Constants.java:51
|
| en + cn quickstart | `list_format` start and end symbols were documented
as `[` and `]` | Corrected to empty strings, noted that `elem_delimiter` must
differ from `delimiter`, added `ignored_elems` |
hugegraph-loader/src/main/java/org/apache/hugegraph/loader/source/file/ListFormat.java:30,
hugegraph-loader/src/main/java/org/apache/hugegraph/loader/source/file/FileSource.java:150
|
| en + cn quickstart | Nothing said that ORC and PARQUET match the header
case-insensitively | Added to the `compression` entry |
hugegraph-loader/src/main/java/org/apache/hugegraph/loader/source/file/FileSource.java:124
|
| en + cn quickstart | The HDFS input source listed only `core_site_path`,
and did not mark it required | Marked it required and added `hdfs_site_path`,
`dir_filter` and `kerberos_config` |
hugegraph-loader/src/main/java/org/apache/hugegraph/loader/source/hdfs/HDFSSource.java:36,
hugegraph-loader/src/main/java/org/apache/hugegraph/loader/source/hdfs/KerberosConfig.java:33,
hugegraph-loader/src/main/java/org/apache/hugegraph/loader/reader/hdfs/HDFSFileReader.java:215
|
| en + cn quickstart | JDBC `driver` was marked required | Marked optional,
since the vendor default is filled in when it is left out |
hugegraph-loader/src/main/java/org/apache/hugegraph/loader/source/jdbc/JDBCSource.java:71
|
| en + cn quickstart | JDBC `where` was undocumented | Documented as an
extra condition on the generated select |
hugegraph-loader/src/main/java/org/apache/hugegraph/loader/source/jdbc/JDBCSource.java:45,
hugegraph-loader/src/main/java/org/apache/hugegraph/loader/reader/jdbc/JDBCFetcher.java:135
|
| en + cn quickstart | The Oracle schema default was described as the
username | Corrected to the upper-cased username |
hugegraph-loader/src/main/java/org/apache/hugegraph/loader/source/jdbc/JDBCVendor.java:140
|
| en + cn quickstart | The Kafka input source was described in terms of
files: header read from a first line, comma default delimiter, no `batch_size`,
no required keys | Rewritten for messages: `bootstrap_server`, `topic`, `group`
and `format` required, `from_beginning` mapped to the offset reset, `header`
needed for CSV and TEXT, `delimiter` used by TEXT only, `batch_size` mapped to
`max.poll.records` |
hugegraph-loader/src/main/java/org/apache/hugegraph/loader/source/kafka/KafkaSource.java:37,
hugegraph-loader/src/main/java/org/apache/hugegraph/loader/reader/kafka/KafkaReader.java:128
|
| en + cn quickstart | GRAPH `graphspace` was said to default to `DEFAULT`,
and the mode and fallback rules were missing | `graphspace` marked required,
added the fallback to the matching command-line options, the rule that all
input sources must then be GRAPH, the switch to `RESTORING` mode, and the shape
of the selected and ignored entries |
hugegraph-loader/src/main/java/org/apache/hugegraph/loader/source/graph/GraphSource.java:82,
hugegraph-loader/src/main/java/org/apache/hugegraph/loader/HugeGraphLoader.java:187,
hugegraph-loader/src/main/java/org/apache/hugegraph/loader/HugeGraphLoader.java:311
|
| en + cn quickstart | The vertex and edge mapping `skip` node was
undocumented | Documented in the shared nodes list |
hugegraph-loader/src/main/java/org/apache/hugegraph/loader/mapping/ElementMapping.java:43
|
| en + cn quickstart | `--max-conn` and `--max-conn-per-route` were
described only as something to tune by hand | Documented that a default value
is raised automatically from `--batch-insert-threads` |
hugegraph-loader/src/main/java/org/apache/hugegraph/loader/executor/LoadOptions.java:437
|
| en + cn quickstart | `--short-id` had no value format | Documented
`label:field:type` and the accepted types |
hugegraph-loader/src/main/java/org/apache/hugegraph/loader/filter/util/ShortIdConfig.java:66
|
| en + cn quickstart | `--sink-type` and the HBase options carried no
context | Scoped `--sink-type` to spark-loader, described both sinks, and
marked the HBase options as belonging to `--sink-type false` |
hugegraph-loader/src/main/java/org/apache/hugegraph/loader/spark/HugeGraphSparkLoader.java:154,
hugegraph-loader/src/main/java/org/apache/hugegraph/loader/direct/loader/HBaseDirectLoader.java:137
|
| en + cn quickstart | The progress file was named `load-progress ${date}` |
Corrected to `load-progress_yyyyMMdd-HHmmss` and added that a resume reads the
newest file in the directory |
hugegraph-loader/src/main/java/org/apache/hugegraph/loader/constant/Constants.java:58,
hugegraph-loader/src/main/java/org/apache/hugegraph/loader/progress/LoadProgress.java:167,
hugegraph-loader/src/main/java/org/apache/hugegraph/loader/executor/LoadContext.java:68
|
| en + cn quickstart | Failure handling referred to `--reload-failure` and
to `.parse-error` and `.insert-error` files under `${struct}/current` |
Rewritten for `--failure-mode` and for one `${id}.error` file per input source
under `${struct}/failure-data`, with the tip-line format, the sibling
`${id}.header`, the deletion of empty files, and append versus rewrite |
hugegraph-loader/src/main/java/org/apache/hugegraph/loader/executor/LoadOptions.java:156,
hugegraph-loader/src/main/java/org/apache/hugegraph/loader/failure/FailLogger.java:67,
hugegraph-loader/src/main/java/org/apache/hugegraph/loader/constant/Constants.java:53
|
| en + cn quickstart | `--max-read-errors` was missing from the list of
limits that trigger a failure file | Added, together with the note that failure
mode lifts all three limits |
hugegraph-loader/src/main/java/org/apache/hugegraph/loader/executor/LoadOptions.java:423
|
| en + cn quickstart | The run command was given as `bin/hugegraph-loader` |
Corrected to `bin/hugegraph-loader.sh` and documented how the script picks the
JVM and passes `JVM_OPTS` |
hugegraph-loader/assembly/static/bin/hugegraph-loader.sh:45 |
| en + cn quickstart | Nothing said that the loader needs at least three
arguments | Added a note after the parameter table |
hugegraph-loader/src/main/java/org/apache/hugegraph/loader/executor/LoadOptions.java:384
|
| en + cn quickstart | `utf8-bom-to-utf8.sh` was not mentioned | Documented
next to `mapping-convert.sh` |
hugegraph-loader/assembly/static/bin/utf8-bom-to-utf8.sh:18 |
| en + cn quickstart | The spark-loader section did not explain how the
command line is split, which input sources work, or what the two sinks do |
Added `SPARK_HOME` and `APP_NAME`, the exact option list that reaches the
loader, the `--file` handling under `--deploy-mode cluster`, the supported
input sources, and the two sink modes |
hugegraph-loader/assembly/static/bin/hugegraph-spark-loader.sh:30,
hugegraph-loader/assembly/static/bin/get-params.sh:25,
hugegraph-loader/src/main/java/org/apache/hugegraph/loader/spark/HugeGraphSparkLoader.java:279
|
| en + cn quickstart | The flink-cdc loader had no section | Added section
4.7 with `FLINK_HOME`, the MySQL JDBC mapping requirement and the two cdc
options | hugegraph-loader/assembly/static/bin/hugegraph-flinkcdc-loader.sh:29,
hugegraph-loader/src/main/java/org/apache/hugegraph/loader/flink/HugeGraphFlinkCDCLoader.java:84
|
| en quickstart | The `--batch-failure-fallback` row had the 3.4.2 heading
appended to it and the heading then appeared twice, so the table and the
section both rendered wrong | Closed the table row and dropped the duplicate
heading; also fixed the stray four-backtick fence that closed the spark example
|
hugegraph-loader/src/main/java/org/apache/hugegraph/loader/executor/LoadOptions.java:343
|
| en quickstart | The docker-compose sample pinned 1.3.0 while the text
around it said 1.5.0, and it left out hubble, unlike the cn page | Aligned the
tags with the surrounding text and added the hubble service, matching the
sample compose file shipped with the loader |
hugegraph-loader/docker/example/docker-compose.yml:26 |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
