Jackie-Jiang commented on a change in pull request #4646: Enabling alternative
controller rest API classes in preprocess
URL: https://github.com/apache/incubator-pinot/pull/4646#discussion_r328403744
##########
File path:
pinot-hadoop/src/main/java/org/apache/pinot/hadoop/job/SegmentPreprocessingJob.java
##########
@@ -375,13 +377,37 @@ private void addHashCodeField(Set<Schema.Field>
fieldSet) {
fieldSet.add(hashCodeField);
}
+ @Override
+ protected org.apache.pinot.common.data.Schema getSchema()
+ throws IOException {
+ try (ControllerRestApi controllerRestApi = getControllerRestApi()) {
+ if (controllerRestApi != null) {
+ return controllerRestApi.getSchema();
+ } else {
+ try (InputStream inputStream = _fileSystem.open(_schemaFile)) {
+ return
org.apache.pinot.common.data.Schema.fromInputSteam(inputStream);
+ }
+ }
+ }
+ }
+
+ /**
+ * Can be overridden to set additional job properties.
+ */
+ @SuppressWarnings("unused")
+ protected void addAdditionalJobProperties(Job job) {
+ }
+
private void setTableConfigAndSchema() throws IOException {
// If push locations, table config, and schema are not configured, this
does not necessarily mean that segments
// cannot be created. We should allow the user to go to the next step
rather than failing the job.
- Preconditions.checkState(!_pushLocations.isEmpty(), "Push locations cannot
be empty.");
- try(ControllerRestApi controllerRestApi = new
DefaultControllerRestApi(_pushLocations, _rawTableName)) {
- _tableConfig = controllerRestApi.getTableConfig();
- _pinotTableSchema = controllerRestApi.getSchema();
+ try(ControllerRestApi controllerRestApi = getControllerRestApi()) {
Review comment:
Here we don't allow using `_schemaFile`?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
