imay commented on a change in pull request #1695: Refactor alter job
URL: https://github.com/apache/incubator-doris/pull/1695#discussion_r323563026
##########
File path: fe/src/main/java/org/apache/doris/load/Load.java
##########
@@ -807,6 +860,327 @@ public static void checkAndCreateSource(Database db,
DataDescription dataDescrip
}
}
+ /*
+ * This function will do followings:
+ * 1. fill the column exprs if user does not specify any column or column
mapping.
+ * 2. For not specified columns, check if they have default value.
+ * 3. Add any shadow columns if have.
+ * 4. validate hadoop functions
+ * 5. init slot descs and expr map for load plan
+ *
+ * This function should be used for broker load v2 and stream load.
+ * And it must be called in same db lock when planing.
+ */
+ public static void initColumns(Table tbl, boolean specifyFileFieldNames,
List<ImportColumnDesc> columnExprs,
+ Map<String, Pair<String, List<String>>> columnToHadoopFunction,
+ Map<String, Expr> exprsByName, Analyzer analyzer, TupleDescriptor
srcTupleDesc,
+ Map<String, SlotDescriptor> slotDescByName, TBrokerScanRangeParams
params) throws UserException {
+ // If user does not specify the file field names, generate it by using
base schema of table.
+ // So that the following process can be unified
+ if (!specifyFileFieldNames) {
+ List<Column> columns = tbl.getBaseSchema();
+ for (Column column : columns) {
+ ImportColumnDesc columnDesc = new
ImportColumnDesc(column.getName());
+ LOG.debug("add base column {} to stream load task",
column.getName());
+ columnExprs.add(columnDesc);
+ }
+ }
+ // generate a map for checking easily
+ Map<String, Expr> columnExprMap =
Maps.newTreeMap(String.CASE_INSENSITIVE_ORDER);
+ for (ImportColumnDesc importColumnDesc : columnExprs) {
+ columnExprMap.put(importColumnDesc.getColumnName(),
importColumnDesc.getExpr());
+ }
+
+ // check default value
+ for (Column column : tbl.getBaseSchema()) {
+ String columnName = column.getName();
+ if (columnExprMap.containsKey(columnName)) {
+ continue;
+ }
+ if (column.getDefaultValue() != null || column.isAllowNull()) {
+ continue;
+ }
+ throw new DdlException("Column has no default value. column: " +
columnName);
+ }
+
+ // When doing schema change, there may have some 'shadow' columns,
with prefix '__doris_shadow_' in
+ // their names. These columns are invisible to user, but we need to
generate data for these columns.
+ // So we add column mappings for these column.
+ // eg1:
+ // base schema is (A, B, C), and B is under schema change, so there
will be a shadow column: '__doris_shadow_B'
+ // So the final column mapping should looks like: (A, B, C,
__doris_shadow_B = substitute(B));
+ for (Column column : tbl.getFullSchema()) {
+ if
(column.isNameWithPrefix(SchemaChangeHandler.SHADOW_NAME_PRFIX)) {
Review comment:
```suggestion
if
(!column.isNameWithPrefix(SchemaChangeHandler.SHADOW_NAME_PRFIX)) {
continue;
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
