bitflicker64 opened a new pull request, #474:
URL: https://github.com/apache/hugegraph-doc/pull/474
## Purpose of the PR
Sync the hugegraph-spark-connector docs (en + cn) with hugegraph-toolchain
master (3b385c3d).
Every row below applies to both
`content/en/docs/quickstart/toolchain/hugegraph-spark-connector.md` and
`content/cn/docs/quickstart/toolchain/hugegraph-spark-connector.md`; the two
pages are kept in sync.
| Page | What was wrong | What changed | Source on master |
|------|----------------|--------------|------------------|
| content/{en,cn}/docs/quickstart/toolchain/hugegraph-spark-connector.md |
Overview did not say that reading is unsupported or which id strategies work |
Added a note: the table only implements `SupportsWrite`; `CUSTOMIZE` and
`PRIMARY_KEY` supported, `AUTOMATIC` rejected |
hugegraph-spark-connector/src/main/scala/org/apache/hugegraph/spark/connector/HGTable.scala:30,
hugegraph-spark-connector/src/main/java/org/apache/hugegraph/spark/connector/builder/VertexBuilder.java:79
|
| content/{en,cn}/docs/quickstart/toolchain/hugegraph-spark-connector.md |
Environment listed a vague "Spark 3.x" and "Scala 2.12" | Pinned to Spark 3.2.2
(`provided` scope) and Scala 2.12.11 | hugegraph-spark-connector/pom.xml:43,
hugegraph-spark-connector/pom.xml:44, hugegraph-spark-connector/pom.xml:48 |
| content/{en,cn}/docs/quickstart/toolchain/hugegraph-spark-connector.md |
Build section never said what artifact the build produces | Documented the
`*-jar-with-dependencies.jar` output for `spark-submit --jars` |
hugegraph-spark-connector/pom.xml:207, hugegraph-spark-connector/pom.xml:210 |
| content/{en,cn}/docs/quickstart/toolchain/hugegraph-spark-connector.md |
Usage did not state that `format` must be the full class name, nor how to pass
auth | Added a sentence on the full class name (no `DataSourceRegister` service
file under `src/main/resources`) and on `username` / `token` options |
hugegraph-spark-connector/src/test/scala/org/apache/hugegraph/spark/connector/SinkExampleTest.scala:59,
hugegraph-spark-connector/src/test/scala/org/apache/hugegraph/spark/connector/SinkExampleTest.scala:82
|
| content/{en,cn}/docs/quickstart/toolchain/hugegraph-spark-connector.md |
No example for a PRIMARY_KEY vertex label or for `ignored-fields` | Added
section 4.4 (software vertex, no `id`, `ignored-fields=ISBN`) |
hugegraph-spark-connector/src/test/scala/org/apache/hugegraph/spark/connector/SinkExampleTest.scala:96
|
| content/{en,cn}/docs/quickstart/toolchain/hugegraph-spark-connector.md |
No example of an edge whose source and target use different id strategies |
Added section 4.5 (created edge, customize source + primary key target) |
hugegraph-spark-connector/src/test/scala/org/apache/hugegraph/spark/connector/SinkExampleTest.scala:157
|
| content/{en,cn}/docs/quickstart/toolchain/hugegraph-spark-connector.md |
Nothing said what `SaveMode.Overwrite` does | Added a note: overwrite returns a
plain writer and does not delete existing data |
hugegraph-spark-connector/src/main/scala/org/apache/hugegraph/spark/connector/writer/HGWriterBuilder.scala:35
|
| content/{en,cn}/docs/quickstart/toolchain/hugegraph-spark-connector.md |
Required options and key matching rules were not stated | Added an intro to
section 5: keys lowercased and trimmed; `data-type`, `label` required;
`source-name`, `target-name` required for edges; marked those rows "Required" |
hugegraph-spark-connector/src/main/java/org/apache/hugegraph/spark/connector/options/HGOptions.java:73,
hugegraph-spark-connector/src/main/java/org/apache/hugegraph/spark/connector/options/HGOptions.java:84,
hugegraph-spark-connector/src/main/java/org/apache/hugegraph/spark/connector/options/HGOptions.java:89,
hugegraph-spark-connector/src/main/java/org/apache/hugegraph/spark/connector/options/HGOptions.java:93,
hugegraph-spark-connector/src/main/java/org/apache/hugegraph/spark/connector/options/HGOptions.java:97
|
| content/{en,cn}/docs/quickstart/toolchain/hugegraph-spark-connector.md |
`host` row did not mention that a scheme prefix is accepted | Noted that `host`
may carry an `http://` or `https://` prefix |
hugegraph-spark-connector/src/main/java/org/apache/hugegraph/spark/connector/client/HGClientHolder.java:47
|
| content/{en,cn}/docs/quickstart/toolchain/hugegraph-spark-connector.md |
`username` row omitted the fallback | Noted that the graph name is used when
`username` is unset |
hugegraph-spark-connector/src/main/java/org/apache/hugegraph/spark/connector/client/HGClientHolder.java:54
|
| content/{en,cn}/docs/quickstart/toolchain/hugegraph-spark-connector.md |
`trust-store-file` and `trust-store-token` rows omitted the https fallbacks |
Documented `${connector.home.path}/conf/hugegraph.truststore` (system property
required) and the `hugegraph` default password |
hugegraph-spark-connector/src/main/java/org/apache/hugegraph/spark/connector/client/HGClientHolder.java:65,
hugegraph-spark-connector/src/main/java/org/apache/hugegraph/spark/connector/client/HGClientHolder.java:72,
hugegraph-spark-connector/src/main/java/org/apache/hugegraph/spark/connector/constant/Constants.java:32,
hugegraph-spark-connector/src/main/java/org/apache/hugegraph/spark/connector/constant/Constants.java:52
|
| content/{en,cn}/docs/quickstart/toolchain/hugegraph-spark-connector.md |
`id` row did not say AUTOMATIC is unsupported | Added that the AUTOMATIC id
policy is not supported |
hugegraph-spark-connector/src/main/java/org/apache/hugegraph/spark/connector/builder/VertexBuilder.java:79
|
| content/{en,cn}/docs/quickstart/toolchain/hugegraph-spark-connector.md |
`batch-size` row did not say the batch is per Spark task | Explained
per-partition buffering and the flush at commit |
hugegraph-spark-connector/src/main/scala/org/apache/hugegraph/spark/connector/writer/HGVertexWriter.scala:56,
hugegraph-spark-connector/src/main/scala/org/apache/hugegraph/spark/connector/writer/HGEdgeWriter.scala:56
|
| content/{en,cn}/docs/quickstart/toolchain/hugegraph-spark-connector.md |
`delimiter` row claimed it also splits `source-name` and `target-name`; the
code always splits those on `,` | Limited `delimiter` to `selected-fields` and
`ignored-fields`; noted `,` for source/target |
hugegraph-spark-connector/src/main/java/org/apache/hugegraph/spark/connector/options/HGOptions.java:228,
hugegraph-spark-connector/src/main/java/org/apache/hugegraph/spark/connector/options/HGOptions.java:240,
hugegraph-spark-connector/src/main/java/org/apache/hugegraph/spark/connector/options/HGOptions.java:248,
hugegraph-spark-connector/src/main/java/org/apache/hugegraph/spark/connector/options/HGOptions.java:260
|
| content/{en,cn}/docs/quickstart/toolchain/hugegraph-spark-connector.md |
No notes on runtime behaviour and limits | Added section 6 "Notes and
Limitations": per-task HugeClient and LOADING mode, 128 byte vertex id limit,
SET/LIST cardinality unsupported, date format `yyyy-MM-dd HH:mm:ss` in GMT+8,
accepted boolean strings, empty vs null id handling; License moved to section 7
|
hugegraph-spark-connector/src/main/java/org/apache/hugegraph/spark/connector/client/HGLoadContext.java:66,
hugegraph-spark-connector/src/main/java/org/apache/hugegraph/spark/connector/client/HGLoadContext.java:81,
hugegraph-spark-connector/src/main/java/org/apache/hugegraph/spark/connector/builder/ElementBuilder.java:211,
hugegraph-spark-connector/src/main/java/org/apache/hugegraph/spark/connector/utils/DataTypeUtils.java:54,
hugegraph-spark-connector/src/main/java/org/apache/hugegraph/spark/connector/constant/Constants.java:48,
hugegraph-spark-connector/src/main/java/org/apache/hugegraph/spark/connector/
constant/Constants.java:50,
hugegraph-spark-connector/src/main/java/org/apache/hugegraph/spark/connector/utils/DataTypeUtils.java:35,
hugegraph-spark-connector/src/main/java/org/apache/hugegraph/spark/connector/utils/DataTypeUtils.java:39
|
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
