[jira] [Work logged] (HDDS-1469) Generate default configuration fragments based on annotations
[ https://issues.apache.org/jira/browse/HDDS-1469?focusedWorklogId=236248=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-236248 ] ASF GitHub Bot logged work on HDDS-1469: Author: ASF GitHub Bot Created on: 02/May/19 12:24 Start Date: 02/May/19 12:24 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on issue #773: HDDS-1469. Generate default configuration fragments based on annotations URL: https://github.com/apache/hadoop/pull/773#issuecomment-488652122 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 45 | Docker mode activated. | ||| _ Prechecks _ | | +1 | @author | 0 | The patch does not contain any @author tags. | | +1 | test4tests | 0 | The patch appears to include 8 new or modified test files. | ||| _ trunk Compile Tests _ | | 0 | mvndep | 76 | Maven dependency ordering for branch | | +1 | mvninstall | 1200 | trunk passed | | +1 | compile | 1002 | trunk passed | | +1 | checkstyle | 143 | trunk passed | | -1 | mvnsite | 43 | server-scm in trunk failed. | | +1 | shadedclient | 1166 | branch has no errors when building and testing our client artifacts. | | 0 | findbugs | 0 | Skipped patched modules with no Java source: hadoop-hdds hadoop-ozone/integration-test | | -1 | findbugs | 30 | server-scm in trunk failed. | | +1 | javadoc | 172 | trunk passed | ||| _ Patch Compile Tests _ | | 0 | mvndep | 22 | Maven dependency ordering for patch | | +1 | mvninstall | 240 | the patch passed | | +1 | compile | 941 | the patch passed | | +1 | javac | 941 | the patch passed | | +1 | checkstyle | 143 | the patch passed | | +1 | mvnsite | 251 | the patch passed | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | xml | 8 | The patch has no ill-formed XML file. | | +1 | shadedclient | 739 | patch has no errors when building and testing our client artifacts. | | 0 | findbugs | 0 | Skipped patched modules with no Java source: hadoop-hdds hadoop-ozone/integration-test | | +1 | findbugs | 179 | the patch passed | | +1 | javadoc | 70 | hadoop-hdds generated 0 new + 14 unchanged - 6 fixed = 14 total (was 20) | | +1 | javadoc | 47 | common in the patch passed. | | +1 | javadoc | 28 | config in the patch passed. | | +1 | javadoc | 30 | hadoop-hdds_server-scm generated 0 new + 5 unchanged - 6 fixed = 5 total (was 11) | | +1 | javadoc | 26 | integration-test in the patch passed. | ||| _ Other Tests _ | | -1 | unit | 161 | hadoop-hdds in the patch failed. | | +1 | unit | 84 | common in the patch passed. | | +1 | unit | 31 | config in the patch passed. | | +1 | unit | 128 | server-scm in the patch passed. | | -1 | unit | 828 | integration-test in the patch failed. | | +1 | asflicense | 49 | The patch does not generate ASF License warnings. | | | | 8045 | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.ozone.client.rpc.TestOzoneClientRetriesOnException | | | hadoop.hdds.scm.safemode.TestSCMSafeModeWithPipelineRules | | | hadoop.ozone.scm.TestXceiverClientMetrics | | | hadoop.ozone.web.TestOzoneVolumes | | | hadoop.ozone.om.TestOmInit | | | hadoop.ozone.web.client.TestKeys | | | hadoop.ozone.scm.pipeline.TestSCMPipelineMetrics | | | hadoop.ozone.web.client.TestOzoneClient | | | hadoop.ozone.om.TestOmAcls | | | hadoop.hdds.scm.container.TestContainerStateManagerIntegration | | | hadoop.ozone.ozShell.TestOzoneDatanodeShell | | | hadoop.ozone.TestStorageContainerManager | | | hadoop.ozone.scm.pipeline.TestPipelineManagerMXBean | | | hadoop.ozone.om.TestOzoneManagerHA | | | hadoop.ozone.container.TestContainerReplication | | | hadoop.ozone.ozShell.TestS3Shell | | | hadoop.ozone.scm.TestSCMNodeManagerMXBean | | | hadoop.ozone.scm.TestSCMMXBean | | | hadoop.ozone.ozShell.TestOzoneShell | | | hadoop.ozone.web.TestOzoneWebAccess | | | hadoop.ozone.client.rpc.TestCommitWatcher | | | hadoop.ozone.om.TestOmMetrics | | | hadoop.ozone.om.TestContainerReportWithKeys | | | hadoop.ozone.om.TestOzoneManager | | | hadoop.ozone.client.rpc.TestOzoneRpcClientWithRatis | | | hadoop.ozone.client.rpc.TestReadRetries | | | hadoop.ozone.client.rpc.TestBlockOutputStreamWithFailures | | | hadoop.ozone.client.rpc.TestOzoneRpcClient | | | hadoop.ozone.scm.TestContainerSmallFile | | | hadoop.ozone.scm.node.TestQueryNode | | | hadoop.hdds.scm.pipeline.TestRatisPipelineUtils | | | hadoop.ozone.scm.node.TestSCMNodeMetrics | | | hadoop.ozone.client.rpc.TestBCSID | | | hadoop.ozone.om.TestOMDbCheckpointServlet | | | hadoop.hdds.scm.pipeline.TestNodeFailure | | |
[jira] [Work logged] (HDDS-1469) Generate default configuration fragments based on annotations
[ https://issues.apache.org/jira/browse/HDDS-1469?focusedWorklogId=236209=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-236209 ] ASF GitHub Bot logged work on HDDS-1469: Author: ASF GitHub Bot Created on: 02/May/19 10:19 Start Date: 02/May/19 10:19 Worklog Time Spent: 10m Work Description: elek commented on pull request #773: HDDS-1469. Generate default configuration fragments based on annotations URL: https://github.com/apache/hadoop/pull/773 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 236209) Time Spent: 5h (was: 4h 50m) > Generate default configuration fragments based on annotations > - > > Key: HDDS-1469 > URL: https://issues.apache.org/jira/browse/HDDS-1469 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Elek, Marton >Assignee: Elek, Marton >Priority: Major > Labels: pull-request-available > Time Spent: 5h > Remaining Estimate: 0h > > See the design doc in the parent jira for more details. > In this jira I introduce a new annotation processor which can generate > ozone-default.xml fragments based on the annotations which are introduced by > HDDS-1468. > The ozone-default-generated.xml fragments can be used directly by the > OzoneConfiguration as I added a small code to the constructor to check ALL > the available ozone-default-generated.xml files and add them to the available > resources. > With this approach we don't need to edit ozone-default.xml as all the > configuration can be defined in java code. > As a side effect each service will see only the available configuration keys > and values based on the classpath. (If the ozone-default-generated.xml file > of OzoneManager is not on the classpath of the SCM, SCM doesn't see the > available configs.) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1469) Generate default configuration fragments based on annotations
[ https://issues.apache.org/jira/browse/HDDS-1469?focusedWorklogId=236208=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-236208 ] ASF GitHub Bot logged work on HDDS-1469: Author: ASF GitHub Bot Created on: 02/May/19 10:18 Start Date: 02/May/19 10:18 Worklog Time Spent: 10m Work Description: elek commented on issue #773: HDDS-1469. Generate default configuration fragments based on annotations URL: https://github.com/apache/hadoop/pull/773#issuecomment-488621848 Thanks the review @anuengineer I am merging it right now. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 236208) Time Spent: 4h 50m (was: 4h 40m) > Generate default configuration fragments based on annotations > - > > Key: HDDS-1469 > URL: https://issues.apache.org/jira/browse/HDDS-1469 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Elek, Marton >Assignee: Elek, Marton >Priority: Major > Labels: pull-request-available > Time Spent: 4h 50m > Remaining Estimate: 0h > > See the design doc in the parent jira for more details. > In this jira I introduce a new annotation processor which can generate > ozone-default.xml fragments based on the annotations which are introduced by > HDDS-1468. > The ozone-default-generated.xml fragments can be used directly by the > OzoneConfiguration as I added a small code to the constructor to check ALL > the available ozone-default-generated.xml files and add them to the available > resources. > With this approach we don't need to edit ozone-default.xml as all the > configuration can be defined in java code. > As a side effect each service will see only the available configuration keys > and values based on the classpath. (If the ozone-default-generated.xml file > of OzoneManager is not on the classpath of the SCM, SCM doesn't see the > available configs.) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1469) Generate default configuration fragments based on annotations
[ https://issues.apache.org/jira/browse/HDDS-1469?focusedWorklogId=235438=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-235438 ] ASF GitHub Bot logged work on HDDS-1469: Author: ASF GitHub Bot Created on: 30/Apr/19 17:52 Start Date: 30/Apr/19 17:52 Worklog Time Spent: 10m Work Description: anuengineer commented on issue #773: HDDS-1469. Generate default configuration fragments based on annotations URL: https://github.com/apache/hadoop/pull/773#issuecomment-488050243 +1 , thanks LGTM. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 235438) Time Spent: 4h 40m (was: 4.5h) > Generate default configuration fragments based on annotations > - > > Key: HDDS-1469 > URL: https://issues.apache.org/jira/browse/HDDS-1469 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Elek, Marton >Assignee: Elek, Marton >Priority: Major > Labels: pull-request-available > Time Spent: 4h 40m > Remaining Estimate: 0h > > See the design doc in the parent jira for more details. > In this jira I introduce a new annotation processor which can generate > ozone-default.xml fragments based on the annotations which are introduced by > HDDS-1468. > The ozone-default-generated.xml fragments can be used directly by the > OzoneConfiguration as I added a small code to the constructor to check ALL > the available ozone-default-generated.xml files and add them to the available > resources. > With this approach we don't need to edit ozone-default.xml as all the > configuration can be defined in java code. > As a side effect each service will see only the available configuration keys > and values based on the classpath. (If the ozone-default-generated.xml file > of OzoneManager is not on the classpath of the SCM, SCM doesn't see the > available configs.) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1469) Generate default configuration fragments based on annotations
[ https://issues.apache.org/jira/browse/HDDS-1469?focusedWorklogId=233578=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-233578 ] ASF GitHub Bot logged work on HDDS-1469: Author: ASF GitHub Bot Created on: 26/Apr/19 14:53 Start Date: 26/Apr/19 14:53 Worklog Time Spent: 10m Work Description: anuengineer commented on issue #773: HDDS-1469. Generate default configuration fragments based on annotations URL: https://github.com/apache/hadoop/pull/773#issuecomment-487086122 Thank you for your comments and explanations. +1. Please feel free to commit this. Thanks for getting this done. We can now add more features into the processor, hopefully generating code for get/set and validation methods. At some point, it would be nice to have a validation method too. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 233578) Time Spent: 4.5h (was: 4h 20m) > Generate default configuration fragments based on annotations > - > > Key: HDDS-1469 > URL: https://issues.apache.org/jira/browse/HDDS-1469 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Elek, Marton >Assignee: Elek, Marton >Priority: Major > Labels: pull-request-available > Time Spent: 4.5h > Remaining Estimate: 0h > > See the design doc in the parent jira for more details. > In this jira I introduce a new annotation processor which can generate > ozone-default.xml fragments based on the annotations which are introduced by > HDDS-1468. > The ozone-default-generated.xml fragments can be used directly by the > OzoneConfiguration as I added a small code to the constructor to check ALL > the available ozone-default-generated.xml files and add them to the available > resources. > With this approach we don't need to edit ozone-default.xml as all the > configuration can be defined in java code. > As a side effect each service will see only the available configuration keys > and values based on the classpath. (If the ozone-default-generated.xml file > of OzoneManager is not on the classpath of the SCM, SCM doesn't see the > available configs.) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1469) Generate default configuration fragments based on annotations
[ https://issues.apache.org/jira/browse/HDDS-1469?focusedWorklogId=233429=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-233429 ] ASF GitHub Bot logged work on HDDS-1469: Author: ASF GitHub Bot Created on: 26/Apr/19 10:35 Start Date: 26/Apr/19 10:35 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on issue #773: HDDS-1469. Generate default configuration fragments based on annotations URL: https://github.com/apache/hadoop/pull/773#issuecomment-487011937 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 0 | Docker mode activated. | | -1 | patch | 7 | https://github.com/apache/hadoop/pull/773 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. | | Subsystem | Report/Notes | |--:|:-| | GITHUB PR | https://github.com/apache/hadoop/pull/773 | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-773/2/console | | Powered by | Apache Yetus 0.9.0 http://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 233429) Time Spent: 4h 20m (was: 4h 10m) > Generate default configuration fragments based on annotations > - > > Key: HDDS-1469 > URL: https://issues.apache.org/jira/browse/HDDS-1469 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Elek, Marton >Assignee: Elek, Marton >Priority: Major > Labels: pull-request-available > Time Spent: 4h 20m > Remaining Estimate: 0h > > See the design doc in the parent jira for more details. > In this jira I introduce a new annotation processor which can generate > ozone-default.xml fragments based on the annotations which are introduced by > HDDS-1468. > The ozone-default-generated.xml fragments can be used directly by the > OzoneConfiguration as I added a small code to the constructor to check ALL > the available ozone-default-generated.xml files and add them to the available > resources. > With this approach we don't need to edit ozone-default.xml as all the > configuration can be defined in java code. > As a side effect each service will see only the available configuration keys > and values based on the classpath. (If the ozone-default-generated.xml file > of OzoneManager is not on the classpath of the SCM, SCM doesn't see the > available configs.) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1469) Generate default configuration fragments based on annotations
[ https://issues.apache.org/jira/browse/HDDS-1469?focusedWorklogId=233412=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-233412 ] ASF GitHub Bot logged work on HDDS-1469: Author: ASF GitHub Bot Created on: 26/Apr/19 10:02 Start Date: 26/Apr/19 10:02 Worklog Time Spent: 10m Work Description: elek commented on issue #773: HDDS-1469. Generate default configuration fragments based on annotations URL: https://github.com/apache/hadoop/pull/773#issuecomment-487003326 > I am ok with that, but some of the old school people might like a single file, and in the deployment, phase don't we need a single file ? or should we move away since the code already has the default? It's a very good question and I don't know the final answer. In fact we use standard hadoop Configuration features to load all the fragments, so it should be fine. I would prefer to try out this approach (with independent config fragments), but based on the feedback, experiences, we can improve/refactor it. My arguments: 1. First of all, it's easier to implement. We don't need a final merge. 2. It's way easier to test. To generate the final ozone-default.xml we need a project which depends on all the others with config fragments. But in the mean time we need merged ozone-default.xml to test the different components. With fragments it just works based on classpath. 3. The biggest argument to use one ozone-default.xml (IMHO) is that it can be used as a documentation. But I think we can provide better documentation page (with better structures). But it can be true: we may need to generate a static doc page about all the configuration settings. 4. It's very interesting that the source of a key is recorded in the Configuration class. With using fragments we will have a source information out of the box: ```XML hdds.scm.replication.event.timeout 10m false jar:file:/opt/hadoop/share/ozone/lib/hadoop-hdds-server-scm-0.5.0-SNAPSHOT.jar!/ozone-default-generated.xml ``` It also means that we don't need to use SCM, HDDS, OZONE tags any more as they can be added based on the source. And with this approach we can print out the configuration based on the components (eg. SCM configs, common configs, etc.). Would be great to add an other information, too: which class defined the specific configuration key. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 233412) Time Spent: 4h 10m (was: 4h) > Generate default configuration fragments based on annotations > - > > Key: HDDS-1469 > URL: https://issues.apache.org/jira/browse/HDDS-1469 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Elek, Marton >Assignee: Elek, Marton >Priority: Major > Labels: pull-request-available > Time Spent: 4h 10m > Remaining Estimate: 0h > > See the design doc in the parent jira for more details. > In this jira I introduce a new annotation processor which can generate > ozone-default.xml fragments based on the annotations which are introduced by > HDDS-1468. > The ozone-default-generated.xml fragments can be used directly by the > OzoneConfiguration as I added a small code to the constructor to check ALL > the available ozone-default-generated.xml files and add them to the available > resources. > With this approach we don't need to edit ozone-default.xml as all the > configuration can be defined in java code. > As a side effect each service will see only the available configuration keys > and values based on the classpath. (If the ozone-default-generated.xml file > of OzoneManager is not on the classpath of the SCM, SCM doesn't see the > available configs.) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1469) Generate default configuration fragments based on annotations
[
https://issues.apache.org/jira/browse/HDDS-1469?focusedWorklogId=233403=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-233403
]
ASF GitHub Bot logged work on HDDS-1469:
Author: ASF GitHub Bot
Created on: 26/Apr/19 09:52
Start Date: 26/Apr/19 09:52
Worklog Time Spent: 10m
Work Description: elek commented on pull request #773: HDDS-1469.
Generate default configuration fragments based on annotations
URL: https://github.com/apache/hadoop/pull/773#discussion_r278882360
##
File path:
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/ReplicationManager.java
##
@@ -754,4 +747,64 @@ private InflightAction(final DatanodeDetails datanode,
this.time = time;
}
}
+
+ /**
+ * Configuration used by the Replication Manager.
+ */
+ @ConfigGroup(prefix = "hdds.scm.replication")
+ public static class ReplicationManagerConfiguration {
+/**
+ * The frequency in which ReplicationMonitor thread should run.
+ */
+private long interval = 5 * 60 * 1000;
+
+/**
+ * Timeout for container replication & deletion command issued by
+ * ReplicationManager.
+ */
+private long eventTimeout = 10 * 60 * 1000;
+
+@Config(key = "thread.interval",
+type = ConfigType.TIME,
+defaultValue = "3s",
+tags = {SCM, OZONE},
+description = "When a heartbeat from the data node arrives on SCM, "
++ "It is queued for processing with the time stamp of when the "
Review comment:
We use it (conf.getInterval())
```
/**
* ReplicationMonitor thread runnable. This wakes up at configured
* interval and processes all the containers in the system.
*/
private synchronized void run() {
try {
while (running) {
final long start = Time.monotonicNow();
final Set containerIds =
containerManager.getContainerIDs();
containerIds.forEach(this::processContainer);
LOG.info("Replication Monitor Thread took {} milliseconds for" +
" processing {} containers.", Time.monotonicNow() - start,
containerIds.size());
wait(conf.getInterval());
}
} catch (Throwable t) {
// When we get runtime exception, we should terminate SCM.
LOG.error("Exception in Replication Monitor Thread.", t);
ExitUtil.terminate(1, t);
}
}
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 233403)
Time Spent: 4h (was: 3h 50m)
> Generate default configuration fragments based on annotations
> -
>
> Key: HDDS-1469
> URL: https://issues.apache.org/jira/browse/HDDS-1469
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
>Reporter: Elek, Marton
>Assignee: Elek, Marton
>Priority: Major
> Labels: pull-request-available
> Time Spent: 4h
> Remaining Estimate: 0h
>
> See the design doc in the parent jira for more details.
> In this jira I introduce a new annotation processor which can generate
> ozone-default.xml fragments based on the annotations which are introduced by
> HDDS-1468.
> The ozone-default-generated.xml fragments can be used directly by the
> OzoneConfiguration as I added a small code to the constructor to check ALL
> the available ozone-default-generated.xml files and add them to the available
> resources.
> With this approach we don't need to edit ozone-default.xml as all the
> configuration can be defined in java code.
> As a side effect each service will see only the available configuration keys
> and values based on the classpath. (If the ozone-default-generated.xml file
> of OzoneManager is not on the classpath of the SCM, SCM doesn't see the
> available configs.)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1469) Generate default configuration fragments based on annotations
[
https://issues.apache.org/jira/browse/HDDS-1469?focusedWorklogId=233402=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-233402
]
ASF GitHub Bot logged work on HDDS-1469:
Author: ASF GitHub Bot
Created on: 26/Apr/19 09:51
Start Date: 26/Apr/19 09:51
Worklog Time Spent: 10m
Work Description: elek commented on pull request #773: HDDS-1469.
Generate default configuration fragments based on annotations
URL: https://github.com/apache/hadoop/pull/773#discussion_r278881892
##
File path:
hadoop-hdds/config/src/main/java/org/apache/hadoop/hdds/conf/ConfigFileGenerator.java
##
@@ -0,0 +1,115 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hdds.conf;
+
+import javax.annotation.processing.AbstractProcessor;
+import javax.annotation.processing.Filer;
+import javax.annotation.processing.RoundEnvironment;
+import javax.annotation.processing.SupportedAnnotationTypes;
+import javax.lang.model.element.Element;
+import javax.lang.model.element.ElementKind;
+import javax.lang.model.element.TypeElement;
+import javax.tools.Diagnostic.Kind;
+import javax.tools.FileObject;
+import javax.tools.StandardLocation;
+import java.io.FileNotFoundException;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStreamWriter;
+import java.io.Writer;
+import java.nio.charset.StandardCharsets;
+import java.util.Set;
+
+/**
+ * Annotation processor to generate ozone-site-generated fragments from
+ * ozone-site.xml.
+ */
+@SupportedAnnotationTypes("org.apache.hadoop.hdds.conf.ConfigGroup")
+public class ConfigFileGenerator extends AbstractProcessor {
+
+ public static final String OUTPUT_FILE_NAME = "ozone-default-generated.xml";
+
+ @Override
+ public boolean process(Set annotations,
+ RoundEnvironment roundEnv) {
+if (roundEnv.processingOver()) {
+ return false;
+}
+
+Filer filer = processingEnv.getFiler();
+System.out.println("round");
+
+try {
+
+ //load existing generated config (if exists)
+ ConfigFileAppender appender = new ConfigFileAppender();
+ try (InputStream input = filer
+ .getResource(StandardLocation.CLASS_OUTPUT, "",
+ OUTPUT_FILE_NAME).openInputStream()) {
+appender.load(input);
+ } catch (FileNotFoundException ex) {
+appender.init();
+ }
+
+ Set annotatedElements =
+ roundEnv.getElementsAnnotatedWith(ConfigGroup.class);
+ for (Element annotatedElement : annotatedElements) {
+TypeElement configGroup = (TypeElement) annotatedElement;
+
+//check if any of the setters are annotated with @Config
+for (Element element : configGroup.getEnclosedElements()) {
+ if (element.getKind() == ElementKind.METHOD) {
+processingEnv.getMessager()
+.printMessage(Kind.WARNING,
element.getSimpleName().toString());
+if (element.getSimpleName().toString().startsWith("set")
Review comment:
Yes, we can implement a lot of validation. For example if you have a class
with `@ConfigGroup` annotation but without annotated setters.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 233402)
Time Spent: 3h 50m (was: 3h 40m)
> Generate default configuration fragments based on annotations
> -
>
> Key: HDDS-1469
> URL: https://issues.apache.org/jira/browse/HDDS-1469
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
>Reporter: Elek, Marton
>Assignee: Elek, Marton
>Priority: Major
> Labels: pull-request-available
> Time Spent: 3h 50m
> Remaining Estimate: 0h
>
> See the design doc in
[jira] [Work logged] (HDDS-1469) Generate default configuration fragments based on annotations
[ https://issues.apache.org/jira/browse/HDDS-1469?focusedWorklogId=233400=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-233400 ] ASF GitHub Bot logged work on HDDS-1469: Author: ASF GitHub Bot Created on: 26/Apr/19 09:50 Start Date: 26/Apr/19 09:50 Worklog Time Spent: 10m Work Description: elek commented on pull request #773: HDDS-1469. Generate default configuration fragments based on annotations URL: https://github.com/apache/hadoop/pull/773#discussion_r278881502 ## File path: hadoop-hdds/config/src/main/java/org/apache/hadoop/hdds/conf/ConfigFileGenerator.java ## @@ -0,0 +1,115 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.hdds.conf; + +import javax.annotation.processing.AbstractProcessor; +import javax.annotation.processing.Filer; +import javax.annotation.processing.RoundEnvironment; +import javax.annotation.processing.SupportedAnnotationTypes; +import javax.lang.model.element.Element; +import javax.lang.model.element.ElementKind; +import javax.lang.model.element.TypeElement; +import javax.tools.Diagnostic.Kind; +import javax.tools.FileObject; +import javax.tools.StandardLocation; +import java.io.FileNotFoundException; +import java.io.IOException; +import java.io.InputStream; +import java.io.OutputStreamWriter; +import java.io.Writer; +import java.nio.charset.StandardCharsets; +import java.util.Set; + +/** + * Annotation processor to generate ozone-site-generated fragments from + * ozone-site.xml. Review comment: Yep. Thanks. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 233400) Time Spent: 3.5h (was: 3h 20m) > Generate default configuration fragments based on annotations > - > > Key: HDDS-1469 > URL: https://issues.apache.org/jira/browse/HDDS-1469 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Elek, Marton >Assignee: Elek, Marton >Priority: Major > Labels: pull-request-available > Time Spent: 3.5h > Remaining Estimate: 0h > > See the design doc in the parent jira for more details. > In this jira I introduce a new annotation processor which can generate > ozone-default.xml fragments based on the annotations which are introduced by > HDDS-1468. > The ozone-default-generated.xml fragments can be used directly by the > OzoneConfiguration as I added a small code to the constructor to check ALL > the available ozone-default-generated.xml files and add them to the available > resources. > With this approach we don't need to edit ozone-default.xml as all the > configuration can be defined in java code. > As a side effect each service will see only the available configuration keys > and values based on the classpath. (If the ozone-default-generated.xml file > of OzoneManager is not on the classpath of the SCM, SCM doesn't see the > available configs.) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1469) Generate default configuration fragments based on annotations
[
https://issues.apache.org/jira/browse/HDDS-1469?focusedWorklogId=233401=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-233401
]
ASF GitHub Bot logged work on HDDS-1469:
Author: ASF GitHub Bot
Created on: 26/Apr/19 09:50
Start Date: 26/Apr/19 09:50
Worklog Time Spent: 10m
Work Description: elek commented on pull request #773: HDDS-1469.
Generate default configuration fragments based on annotations
URL: https://github.com/apache/hadoop/pull/773#discussion_r278881642
##
File path:
hadoop-hdds/config/src/main/java/org/apache/hadoop/hdds/conf/ConfigFileGenerator.java
##
@@ -0,0 +1,115 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hdds.conf;
+
+import javax.annotation.processing.AbstractProcessor;
+import javax.annotation.processing.Filer;
+import javax.annotation.processing.RoundEnvironment;
+import javax.annotation.processing.SupportedAnnotationTypes;
+import javax.lang.model.element.Element;
+import javax.lang.model.element.ElementKind;
+import javax.lang.model.element.TypeElement;
+import javax.tools.Diagnostic.Kind;
+import javax.tools.FileObject;
+import javax.tools.StandardLocation;
+import java.io.FileNotFoundException;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStreamWriter;
+import java.io.Writer;
+import java.nio.charset.StandardCharsets;
+import java.util.Set;
+
+/**
+ * Annotation processor to generate ozone-site-generated fragments from
+ * ozone-site.xml.
+ */
+@SupportedAnnotationTypes("org.apache.hadoop.hdds.conf.ConfigGroup")
+public class ConfigFileGenerator extends AbstractProcessor {
+
+ public static final String OUTPUT_FILE_NAME = "ozone-default-generated.xml";
+
+ @Override
+ public boolean process(Set annotations,
+ RoundEnvironment roundEnv) {
+if (roundEnv.processingOver()) {
+ return false;
+}
+
+Filer filer = processingEnv.getFiler();
+System.out.println("round");
Review comment:
Ups.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 233401)
Time Spent: 3h 40m (was: 3.5h)
> Generate default configuration fragments based on annotations
> -
>
> Key: HDDS-1469
> URL: https://issues.apache.org/jira/browse/HDDS-1469
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
>Reporter: Elek, Marton
>Assignee: Elek, Marton
>Priority: Major
> Labels: pull-request-available
> Time Spent: 3h 40m
> Remaining Estimate: 0h
>
> See the design doc in the parent jira for more details.
> In this jira I introduce a new annotation processor which can generate
> ozone-default.xml fragments based on the annotations which are introduced by
> HDDS-1468.
> The ozone-default-generated.xml fragments can be used directly by the
> OzoneConfiguration as I added a small code to the constructor to check ALL
> the available ozone-default-generated.xml files and add them to the available
> resources.
> With this approach we don't need to edit ozone-default.xml as all the
> configuration can be defined in java code.
> As a side effect each service will see only the available configuration keys
> and values based on the classpath. (If the ozone-default-generated.xml file
> of OzoneManager is not on the classpath of the SCM, SCM doesn't see the
> available configs.)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1469) Generate default configuration fragments based on annotations
[ https://issues.apache.org/jira/browse/HDDS-1469?focusedWorklogId=233399=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-233399 ] ASF GitHub Bot logged work on HDDS-1469: Author: ASF GitHub Bot Created on: 26/Apr/19 09:49 Start Date: 26/Apr/19 09:49 Worklog Time Spent: 10m Work Description: elek commented on pull request #773: HDDS-1469. Generate default configuration fragments based on annotations URL: https://github.com/apache/hadoop/pull/773#discussion_r278881241 ## File path: hadoop-hdds/config/pom.xml ## @@ -0,0 +1,66 @@ + + +http://maven.apache.org/POM/4.0.0; + xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance; + xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 +http://maven.apache.org/xsd/maven-4.0.0.xsd;> + 4.0.0 + +org.apache.hadoop +hadoop-hdds +0.5.0-SNAPSHOT Review comment: This is some kind of chicken egg problem. You should have an exact version for your parent pom as you don't know which parent should be downloaded to check the current version. What you can remove is the normal `` (not the `` of the parent) which is by default the same as the `parent.version` (AFAIK). But as a solution the maven version plugin is introduced to search and replace both version and parent version in a consistent way so it's not a problem. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 233399) Time Spent: 3h 20m (was: 3h 10m) > Generate default configuration fragments based on annotations > - > > Key: HDDS-1469 > URL: https://issues.apache.org/jira/browse/HDDS-1469 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Elek, Marton >Assignee: Elek, Marton >Priority: Major > Labels: pull-request-available > Time Spent: 3h 20m > Remaining Estimate: 0h > > See the design doc in the parent jira for more details. > In this jira I introduce a new annotation processor which can generate > ozone-default.xml fragments based on the annotations which are introduced by > HDDS-1468. > The ozone-default-generated.xml fragments can be used directly by the > OzoneConfiguration as I added a small code to the constructor to check ALL > the available ozone-default-generated.xml files and add them to the available > resources. > With this approach we don't need to edit ozone-default.xml as all the > configuration can be defined in java code. > As a side effect each service will see only the available configuration keys > and values based on the classpath. (If the ozone-default-generated.xml file > of OzoneManager is not on the classpath of the SCM, SCM doesn't see the > available configs.) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1469) Generate default configuration fragments based on annotations
[ https://issues.apache.org/jira/browse/HDDS-1469?focusedWorklogId=233398=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-233398 ] ASF GitHub Bot logged work on HDDS-1469: Author: ASF GitHub Bot Created on: 26/Apr/19 09:46 Start Date: 26/Apr/19 09:46 Worklog Time Spent: 10m Work Description: elek commented on pull request #773: HDDS-1469. Generate default configuration fragments based on annotations URL: https://github.com/apache/hadoop/pull/773#discussion_r278880210 ## File path: hadoop-hdds/config/src/main/java/org/apache/hadoop/hdds/conf/ConfigFileAppender.java ## @@ -0,0 +1,127 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.hdds.conf; + +import javax.xml.parsers.DocumentBuilder; +import javax.xml.parsers.DocumentBuilderFactory; +import javax.xml.transform.OutputKeys; +import javax.xml.transform.Transformer; +import javax.xml.transform.TransformerException; +import javax.xml.transform.TransformerFactory; +import javax.xml.transform.dom.DOMSource; +import javax.xml.transform.stream.StreamResult; +import java.io.InputStream; +import java.io.Writer; +import java.util.Arrays; +import java.util.stream.Collectors; + +import org.w3c.dom.Document; +import org.w3c.dom.Element; + +/** + * Simple DOM based config file writer. + * + * This class can init/load existing ozone-site.xml fragments and append + * new entries and write to the file system. Review comment: You are right. More preciously it generates ozone-default-*.xml fragments. I updated the java doc. It works exactly as before (and as you described) we are generating ozone-default (but multiple one instead of one big one) and you can use your own ozone-site.xml. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 233398) Time Spent: 3h 10m (was: 3h) > Generate default configuration fragments based on annotations > - > > Key: HDDS-1469 > URL: https://issues.apache.org/jira/browse/HDDS-1469 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Elek, Marton >Assignee: Elek, Marton >Priority: Major > Labels: pull-request-available > Time Spent: 3h 10m > Remaining Estimate: 0h > > See the design doc in the parent jira for more details. > In this jira I introduce a new annotation processor which can generate > ozone-default.xml fragments based on the annotations which are introduced by > HDDS-1468. > The ozone-default-generated.xml fragments can be used directly by the > OzoneConfiguration as I added a small code to the constructor to check ALL > the available ozone-default-generated.xml files and add them to the available > resources. > With this approach we don't need to edit ozone-default.xml as all the > configuration can be defined in java code. > As a side effect each service will see only the available configuration keys > and values based on the classpath. (If the ozone-default-generated.xml file > of OzoneManager is not on the classpath of the SCM, SCM doesn't see the > available configs.) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1469) Generate default configuration fragments based on annotations
[
https://issues.apache.org/jira/browse/HDDS-1469?focusedWorklogId=233396=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-233396
]
ASF GitHub Bot logged work on HDDS-1469:
Author: ASF GitHub Bot
Created on: 26/Apr/19 09:43
Start Date: 26/Apr/19 09:43
Worklog Time Spent: 10m
Work Description: elek commented on pull request #773: HDDS-1469.
Generate default configuration fragments based on annotations
URL: https://github.com/apache/hadoop/pull/773#discussion_r278879227
##
File path:
hadoop-hdds/config/src/main/java/org/apache/hadoop/hdds/conf/Config.java
##
@@ -0,0 +1,59 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hdds.conf;
+
+import java.lang.annotation.ElementType;
+import java.lang.annotation.Retention;
+import java.lang.annotation.RetentionPolicy;
+import java.lang.annotation.Target;
+import java.util.concurrent.TimeUnit;
+
+/**
+ * Mark field to be configurable from ozone-site.xml.
+ */
+@Retention(RetentionPolicy.RUNTIME)
+@Target(ElementType.METHOD)
+public @interface Config {
+
+ /**
+ * Configuration fragment relative to the prefix defined with @ConfigGroup.
+ */
+ String key();
+
+ /**
+ * Default value to use if not set.
+ */
+ String defaultValue();
+
+ /**
+ * Custom description as a help.
+ */
+ String description() default "";
+
+ /**
+ * Type of configuration. Use AUTO to decide it based on the java type.
+ */
+ ConfigType type() default ConfigType.AUTO;
+
+ /**
+ * If type == TIME the unit should be defined with this attribute.
+ */
+ TimeUnit timeUnit() default TimeUnit.MILLISECONDS;
+
+ ConfigTag[] tags() default {ConfigTag.OZONE};
Review comment:
Good point. I removed the 'default's for tags and description.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 233396)
Time Spent: 3h (was: 2h 50m)
> Generate default configuration fragments based on annotations
> -
>
> Key: HDDS-1469
> URL: https://issues.apache.org/jira/browse/HDDS-1469
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
>Reporter: Elek, Marton
>Assignee: Elek, Marton
>Priority: Major
> Labels: pull-request-available
> Time Spent: 3h
> Remaining Estimate: 0h
>
> See the design doc in the parent jira for more details.
> In this jira I introduce a new annotation processor which can generate
> ozone-default.xml fragments based on the annotations which are introduced by
> HDDS-1468.
> The ozone-default-generated.xml fragments can be used directly by the
> OzoneConfiguration as I added a small code to the constructor to check ALL
> the available ozone-default-generated.xml files and add them to the available
> resources.
> With this approach we don't need to edit ozone-default.xml as all the
> configuration can be defined in java code.
> As a side effect each service will see only the available configuration keys
> and values based on the classpath. (If the ozone-default-generated.xml file
> of OzoneManager is not on the classpath of the SCM, SCM doesn't see the
> available configs.)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1469) Generate default configuration fragments based on annotations
[
https://issues.apache.org/jira/browse/HDDS-1469?focusedWorklogId=233392=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-233392
]
ASF GitHub Bot logged work on HDDS-1469:
Author: ASF GitHub Bot
Created on: 26/Apr/19 09:34
Start Date: 26/Apr/19 09:34
Worklog Time Spent: 10m
Work Description: elek commented on pull request #773: HDDS-1469.
Generate default configuration fragments based on annotations
URL: https://github.com/apache/hadoop/pull/773#discussion_r278875723
##
File path:
hadoop-hdds/common/src/test/java/org/apache/hadoop/hdds/conf/SimpleConfiguration.java
##
@@ -0,0 +1,83 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hdds.conf;
+
+import java.util.concurrent.TimeUnit;
+
+/**
+ * Example configuration to test the configuration injection.
+ */
+@ConfigGroup(prefix = "ozone.scm.client")
+public class SimpleConfiguration {
+
+ private String clientAddress;
+
+ private String bindHost;
+
+ private boolean enabled;
+
+ private int port = 1234;
+
+ private long waitTime = 1;
+
+ @Config(key = "address", defaultValue = "localhost")
+ public void setClientAddress(String clientAddress) {
+this.clientAddress = clientAddress;
+ }
+
+ @Config(key = "bind.host", defaultValue = "0.0.0.0")
+ public void setBindHost(String bindHost) {
+this.bindHost = bindHost;
+ }
+
+ @Config(key = "enabled", defaultValue = "true")
+ public void setEnabled(boolean enabled) {
+this.enabled = enabled;
Review comment:
This is just an example configuration (for unit tests) I invented a random
boolean configuration. Name doesn't matter but I can use a more meaningful one.
For example client.compression.enabled.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 233392)
Time Spent: 2h 50m (was: 2h 40m)
> Generate default configuration fragments based on annotations
> -
>
> Key: HDDS-1469
> URL: https://issues.apache.org/jira/browse/HDDS-1469
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
>Reporter: Elek, Marton
>Assignee: Elek, Marton
>Priority: Major
> Labels: pull-request-available
> Time Spent: 2h 50m
> Remaining Estimate: 0h
>
> See the design doc in the parent jira for more details.
> In this jira I introduce a new annotation processor which can generate
> ozone-default.xml fragments based on the annotations which are introduced by
> HDDS-1468.
> The ozone-default-generated.xml fragments can be used directly by the
> OzoneConfiguration as I added a small code to the constructor to check ALL
> the available ozone-default-generated.xml files and add them to the available
> resources.
> With this approach we don't need to edit ozone-default.xml as all the
> configuration can be defined in java code.
> As a side effect each service will see only the available configuration keys
> and values based on the classpath. (If the ozone-default-generated.xml file
> of OzoneManager is not on the classpath of the SCM, SCM doesn't see the
> available configs.)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1469) Generate default configuration fragments based on annotations
[
https://issues.apache.org/jira/browse/HDDS-1469?focusedWorklogId=233391=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-233391
]
ASF GitHub Bot logged work on HDDS-1469:
Author: ASF GitHub Bot
Created on: 26/Apr/19 09:32
Start Date: 26/Apr/19 09:32
Worklog Time Spent: 10m
Work Description: elek commented on pull request #773: HDDS-1469.
Generate default configuration fragments based on annotations
URL: https://github.com/apache/hadoop/pull/773#discussion_r278874474
##
File path:
hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/conf/OzoneConfiguration.java
##
@@ -161,14 +285,14 @@ public static void activate() {
Configuration.addDefaultResource("hdfs-default.xml");
Configuration.addDefaultResource("hdfs-site.xml");
Configuration.addDefaultResource("ozone-default.xml");
-Configuration.addDefaultResource("ozone-site.xml");
Review comment:
We should. But in fact I didn't remove the possibility to use ozone-site.xml
I just _moved_ it.
The problem here is that the hadoop Configuration is not designed to be
reused. We can use multiple `Configuration.addDefaultResource(String)` but
there is no `Configuration.addDefaultResource(URL)` (and we can't create one as
all the required methods are private :-( ).
With using configuration fragments we have multiple files with the same name
(String) but with different URL (eg.
jar://hadoop-ozone-ozone-manager.jar!/ozone-default-generated.xml).
To support this, but _keep_ the same precedence I modified the configuration
loading:
```
Configuration.addDefaultResource("hdfs-default.xml");
Configuration.addDefaultResource("hdfs-site.xml");
Configuration.addDefaultResource("ozone-default.xml");
conf.addResource("jar://ozone-manager.jar!/ozone-site-generated.xml");
conf.addResource("jar://storage-container-manager.jar!/ozone-site-generated.xml");
conf.addResource("ozone-site.xml");
```
The last three lines are in the constructor of OzoneConfiguration and as I
would like to load `ozone-site.xml ` at the end I moved it to there.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 233391)
Time Spent: 2h 40m (was: 2.5h)
> Generate default configuration fragments based on annotations
> -
>
> Key: HDDS-1469
> URL: https://issues.apache.org/jira/browse/HDDS-1469
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
>Reporter: Elek, Marton
>Assignee: Elek, Marton
>Priority: Major
> Labels: pull-request-available
> Time Spent: 2h 40m
> Remaining Estimate: 0h
>
> See the design doc in the parent jira for more details.
> In this jira I introduce a new annotation processor which can generate
> ozone-default.xml fragments based on the annotations which are introduced by
> HDDS-1468.
> The ozone-default-generated.xml fragments can be used directly by the
> OzoneConfiguration as I added a small code to the constructor to check ALL
> the available ozone-default-generated.xml files and add them to the available
> resources.
> With this approach we don't need to edit ozone-default.xml as all the
> configuration can be defined in java code.
> As a side effect each service will see only the available configuration keys
> and values based on the classpath. (If the ozone-default-generated.xml file
> of OzoneManager is not on the classpath of the SCM, SCM doesn't see the
> available configs.)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1469) Generate default configuration fragments based on annotations
[
https://issues.apache.org/jira/browse/HDDS-1469?focusedWorklogId=233390=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-233390
]
ASF GitHub Bot logged work on HDDS-1469:
Author: ASF GitHub Bot
Created on: 26/Apr/19 09:31
Start Date: 26/Apr/19 09:31
Worklog Time Spent: 10m
Work Description: elek commented on pull request #773: HDDS-1469.
Generate default configuration fragments based on annotations
URL: https://github.com/apache/hadoop/pull/773#discussion_r278874474
##
File path:
hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/conf/OzoneConfiguration.java
##
@@ -161,14 +285,14 @@ public static void activate() {
Configuration.addDefaultResource("hdfs-default.xml");
Configuration.addDefaultResource("hdfs-site.xml");
Configuration.addDefaultResource("ozone-default.xml");
-Configuration.addDefaultResource("ozone-site.xml");
Review comment:
We should. But in fact I didn't remove the possibility to use ozone-site.xml
I just _moved_ it.
The problem here is that the hadoop Configuration is not designed to be
reused. We can use multiple `Configuration.addDefaultResource(String)` but
there is no Configuration.addDefaultResource(URL) (and we can't add it as all
of the methods are private).
With using configuration fragment we have multiple files with the same name
(String) but with different URL (eg.
jar://hadoop-ozone-ozone-manager.jar!/ozone-default-generated.xml).
To support this, but _keep_ the same precedence I modified the configuration
loading:
```
Configuration.addDefaultResource("hdfs-default.xml");
Configuration.addDefaultResource("hdfs-site.xml");
Configuration.addDefaultResource("ozone-default.xml");
conf.addResource("jar://ozone-manager.jar!/ozone-site-generated.xml");
conf.addResource("jar://storage-container-manager.jar!/ozone-site-generated.xml");
conf.addResource("ozone-site.xml");
```
The last three lines are in the constructor of OzoneConfiguration and as I
would like to load ozone-site.xml at the end I moved it to there.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 233390)
Time Spent: 2.5h (was: 2h 20m)
> Generate default configuration fragments based on annotations
> -
>
> Key: HDDS-1469
> URL: https://issues.apache.org/jira/browse/HDDS-1469
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
>Reporter: Elek, Marton
>Assignee: Elek, Marton
>Priority: Major
> Labels: pull-request-available
> Time Spent: 2.5h
> Remaining Estimate: 0h
>
> See the design doc in the parent jira for more details.
> In this jira I introduce a new annotation processor which can generate
> ozone-default.xml fragments based on the annotations which are introduced by
> HDDS-1468.
> The ozone-default-generated.xml fragments can be used directly by the
> OzoneConfiguration as I added a small code to the constructor to check ALL
> the available ozone-default-generated.xml files and add them to the available
> resources.
> With this approach we don't need to edit ozone-default.xml as all the
> configuration can be defined in java code.
> As a side effect each service will see only the available configuration keys
> and values based on the classpath. (If the ozone-default-generated.xml file
> of OzoneManager is not on the classpath of the SCM, SCM doesn't see the
> available configs.)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1469) Generate default configuration fragments based on annotations
[ https://issues.apache.org/jira/browse/HDDS-1469?focusedWorklogId=233073=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-233073 ] ASF GitHub Bot logged work on HDDS-1469: Author: ASF GitHub Bot Created on: 25/Apr/19 19:39 Start Date: 25/Apr/19 19:39 Worklog Time Spent: 10m Work Description: anuengineer commented on issue #773: HDDS-1469. Generate default configuration fragments based on annotations URL: https://github.com/apache/hadoop/pull/773#issuecomment-486810661 > /cc @anuengineer This is the annotation processor. With a small modification in the OzoneConfiguration (to load all the generated fragments) we don't need to merge all the generated config files to one big ozone-default.xml I am ok with that, but some of the old school people might like a single file, and in the deployment, phase don't we need a single file ? or should we move away since the code already has the default? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 233073) Time Spent: 2h 20m (was: 2h 10m) > Generate default configuration fragments based on annotations > - > > Key: HDDS-1469 > URL: https://issues.apache.org/jira/browse/HDDS-1469 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Elek, Marton >Assignee: Elek, Marton >Priority: Major > Labels: pull-request-available > Time Spent: 2h 20m > Remaining Estimate: 0h > > See the design doc in the parent jira for more details. > In this jira I introduce a new annotation processor which can generate > ozone-default.xml fragments based on the annotations which are introduced by > HDDS-1468. > The ozone-default-generated.xml fragments can be used directly by the > OzoneConfiguration as I added a small code to the constructor to check ALL > the available ozone-default-generated.xml files and add them to the available > resources. > With this approach we don't need to edit ozone-default.xml as all the > configuration can be defined in java code. > As a side effect each service will see only the available configuration keys > and values based on the classpath. (If the ozone-default-generated.xml file > of OzoneManager is not on the classpath of the SCM, SCM doesn't see the > available configs.) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1469) Generate default configuration fragments based on annotations
[ https://issues.apache.org/jira/browse/HDDS-1469?focusedWorklogId=233068=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-233068 ] ASF GitHub Bot logged work on HDDS-1469: Author: ASF GitHub Bot Created on: 25/Apr/19 19:34 Start Date: 25/Apr/19 19:34 Worklog Time Spent: 10m Work Description: anuengineer commented on pull request #773: HDDS-1469. Generate default configuration fragments based on annotations URL: https://github.com/apache/hadoop/pull/773#discussion_r278697747 ## File path: hadoop-hdds/config/src/main/java/org/apache/hadoop/hdds/conf/ConfigFileGenerator.java ## @@ -0,0 +1,115 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.hdds.conf; + +import javax.annotation.processing.AbstractProcessor; +import javax.annotation.processing.Filer; +import javax.annotation.processing.RoundEnvironment; +import javax.annotation.processing.SupportedAnnotationTypes; +import javax.lang.model.element.Element; +import javax.lang.model.element.ElementKind; +import javax.lang.model.element.TypeElement; +import javax.tools.Diagnostic.Kind; +import javax.tools.FileObject; +import javax.tools.StandardLocation; +import java.io.FileNotFoundException; +import java.io.IOException; +import java.io.InputStream; +import java.io.OutputStreamWriter; +import java.io.Writer; +import java.nio.charset.StandardCharsets; +import java.util.Set; + +/** + * Annotation processor to generate ozone-site-generated fragments from + * ozone-site.xml. Review comment: wrong comment? from config classes? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 233068) Time Spent: 2h 10m (was: 2h) > Generate default configuration fragments based on annotations > - > > Key: HDDS-1469 > URL: https://issues.apache.org/jira/browse/HDDS-1469 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Elek, Marton >Assignee: Elek, Marton >Priority: Major > Labels: pull-request-available > Time Spent: 2h 10m > Remaining Estimate: 0h > > See the design doc in the parent jira for more details. > In this jira I introduce a new annotation processor which can generate > ozone-default.xml fragments based on the annotations which are introduced by > HDDS-1468. > The ozone-default-generated.xml fragments can be used directly by the > OzoneConfiguration as I added a small code to the constructor to check ALL > the available ozone-default-generated.xml files and add them to the available > resources. > With this approach we don't need to edit ozone-default.xml as all the > configuration can be defined in java code. > As a side effect each service will see only the available configuration keys > and values based on the classpath. (If the ozone-default-generated.xml file > of OzoneManager is not on the classpath of the SCM, SCM doesn't see the > available configs.) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1469) Generate default configuration fragments based on annotations
[ https://issues.apache.org/jira/browse/HDDS-1469?focusedWorklogId=233065=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-233065 ] ASF GitHub Bot logged work on HDDS-1469: Author: ASF GitHub Bot Created on: 25/Apr/19 19:34 Start Date: 25/Apr/19 19:34 Worklog Time Spent: 10m Work Description: anuengineer commented on pull request #773: HDDS-1469. Generate default configuration fragments based on annotations URL: https://github.com/apache/hadoop/pull/773#discussion_r278696810 ## File path: hadoop-hdds/config/pom.xml ## @@ -0,0 +1,66 @@ + + +http://maven.apache.org/POM/4.0.0; + xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance; + xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 +http://maven.apache.org/xsd/maven-4.0.0.xsd;> + 4.0.0 + +org.apache.hadoop +hadoop-hdds +0.5.0-SNAPSHOT Review comment: is it possible to inherit this value from the parent POM ? more a question for my understanding.. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 233065) Time Spent: 1h 40m (was: 1.5h) > Generate default configuration fragments based on annotations > - > > Key: HDDS-1469 > URL: https://issues.apache.org/jira/browse/HDDS-1469 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Elek, Marton >Assignee: Elek, Marton >Priority: Major > Labels: pull-request-available > Time Spent: 1h 40m > Remaining Estimate: 0h > > See the design doc in the parent jira for more details. > In this jira I introduce a new annotation processor which can generate > ozone-default.xml fragments based on the annotations which are introduced by > HDDS-1468. > The ozone-default-generated.xml fragments can be used directly by the > OzoneConfiguration as I added a small code to the constructor to check ALL > the available ozone-default-generated.xml files and add them to the available > resources. > With this approach we don't need to edit ozone-default.xml as all the > configuration can be defined in java code. > As a side effect each service will see only the available configuration keys > and values based on the classpath. (If the ozone-default-generated.xml file > of OzoneManager is not on the classpath of the SCM, SCM doesn't see the > available configs.) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1469) Generate default configuration fragments based on annotations
[
https://issues.apache.org/jira/browse/HDDS-1469?focusedWorklogId=233066=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-233066
]
ASF GitHub Bot logged work on HDDS-1469:
Author: ASF GitHub Bot
Created on: 25/Apr/19 19:34
Start Date: 25/Apr/19 19:34
Worklog Time Spent: 10m
Work Description: anuengineer commented on pull request #773: HDDS-1469.
Generate default configuration fragments based on annotations
URL: https://github.com/apache/hadoop/pull/773#discussion_r278692062
##
File path:
hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/conf/OzoneConfiguration.java
##
@@ -161,14 +285,14 @@ public static void activate() {
Configuration.addDefaultResource("hdfs-default.xml");
Configuration.addDefaultResource("hdfs-site.xml");
Configuration.addDefaultResource("ozone-default.xml");
-Configuration.addDefaultResource("ozone-site.xml");
Review comment:
Shouldn't we still allow this over-ride?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 233066)
Time Spent: 1h 50m (was: 1h 40m)
> Generate default configuration fragments based on annotations
> -
>
> Key: HDDS-1469
> URL: https://issues.apache.org/jira/browse/HDDS-1469
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
>Reporter: Elek, Marton
>Assignee: Elek, Marton
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 50m
> Remaining Estimate: 0h
>
> See the design doc in the parent jira for more details.
> In this jira I introduce a new annotation processor which can generate
> ozone-default.xml fragments based on the annotations which are introduced by
> HDDS-1468.
> The ozone-default-generated.xml fragments can be used directly by the
> OzoneConfiguration as I added a small code to the constructor to check ALL
> the available ozone-default-generated.xml files and add them to the available
> resources.
> With this approach we don't need to edit ozone-default.xml as all the
> configuration can be defined in java code.
> As a side effect each service will see only the available configuration keys
> and values based on the classpath. (If the ozone-default-generated.xml file
> of OzoneManager is not on the classpath of the SCM, SCM doesn't see the
> available configs.)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1469) Generate default configuration fragments based on annotations
[
https://issues.apache.org/jira/browse/HDDS-1469?focusedWorklogId=233060=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-233060
]
ASF GitHub Bot logged work on HDDS-1469:
Author: ASF GitHub Bot
Created on: 25/Apr/19 19:34
Start Date: 25/Apr/19 19:34
Worklog Time Spent: 10m
Work Description: anuengineer commented on pull request #773: HDDS-1469.
Generate default configuration fragments based on annotations
URL: https://github.com/apache/hadoop/pull/773#discussion_r278698781
##
File path:
hadoop-hdds/config/src/main/java/org/apache/hadoop/hdds/conf/ConfigFileGenerator.java
##
@@ -0,0 +1,115 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hdds.conf;
+
+import javax.annotation.processing.AbstractProcessor;
+import javax.annotation.processing.Filer;
+import javax.annotation.processing.RoundEnvironment;
+import javax.annotation.processing.SupportedAnnotationTypes;
+import javax.lang.model.element.Element;
+import javax.lang.model.element.ElementKind;
+import javax.lang.model.element.TypeElement;
+import javax.tools.Diagnostic.Kind;
+import javax.tools.FileObject;
+import javax.tools.StandardLocation;
+import java.io.FileNotFoundException;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStreamWriter;
+import java.io.Writer;
+import java.nio.charset.StandardCharsets;
+import java.util.Set;
+
+/**
+ * Annotation processor to generate ozone-site-generated fragments from
+ * ozone-site.xml.
+ */
+@SupportedAnnotationTypes("org.apache.hadoop.hdds.conf.ConfigGroup")
+public class ConfigFileGenerator extends AbstractProcessor {
+
+ public static final String OUTPUT_FILE_NAME = "ozone-default-generated.xml";
+
+ @Override
+ public boolean process(Set annotations,
+ RoundEnvironment roundEnv) {
+if (roundEnv.processingOver()) {
+ return false;
+}
+
+Filer filer = processingEnv.getFiler();
+System.out.println("round");
+
+try {
+
+ //load existing generated config (if exists)
+ ConfigFileAppender appender = new ConfigFileAppender();
+ try (InputStream input = filer
+ .getResource(StandardLocation.CLASS_OUTPUT, "",
+ OUTPUT_FILE_NAME).openInputStream()) {
+appender.load(input);
+ } catch (FileNotFoundException ex) {
+appender.init();
+ }
+
+ Set annotatedElements =
+ roundEnv.getElementsAnnotatedWith(ConfigGroup.class);
+ for (Element annotatedElement : annotatedElements) {
+TypeElement configGroup = (TypeElement) annotatedElement;
+
+//check if any of the setters are annotated with @Config
+for (Element element : configGroup.getEnclosedElements()) {
+ if (element.getKind() == ElementKind.METHOD) {
+processingEnv.getMessager()
+.printMessage(Kind.WARNING,
element.getSimpleName().toString());
+if (element.getSimpleName().toString().startsWith("set")
Review comment:
In future, we might want to emit a warning if you have "Set" for example,
assuming that was a mistake that user is making, and letting them know we are
ignoring it.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 233060)
> Generate default configuration fragments based on annotations
> -
>
> Key: HDDS-1469
> URL: https://issues.apache.org/jira/browse/HDDS-1469
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
>Reporter: Elek, Marton
>Assignee: Elek, Marton
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h
> Remaining Estimate: 0h
>
> See the design doc in the
[jira] [Work logged] (HDDS-1469) Generate default configuration fragments based on annotations
[
https://issues.apache.org/jira/browse/HDDS-1469?focusedWorklogId=233067=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-233067
]
ASF GitHub Bot logged work on HDDS-1469:
Author: ASF GitHub Bot
Created on: 25/Apr/19 19:34
Start Date: 25/Apr/19 19:34
Worklog Time Spent: 10m
Work Description: anuengineer commented on pull request #773: HDDS-1469.
Generate default configuration fragments based on annotations
URL: https://github.com/apache/hadoop/pull/773#discussion_r278693655
##
File path:
hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/conf/OzoneConfiguration.java
##
@@ -45,12 +48,31 @@
public OzoneConfiguration() {
OzoneConfiguration.activate();
+loadDefaults();
}
public OzoneConfiguration(Configuration conf) {
super(conf);
//load the configuration from the classloader of the original conf.
setClassLoader(conf.getClassLoader());
+loadDefaults();
+ }
+
+ private void loadDefaults() {
Review comment:
Can I be a greedy pig and request that we also sort these keys (in another
JIRA of course) before we write out the XML?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 233067)
Time Spent: 2h (was: 1h 50m)
> Generate default configuration fragments based on annotations
> -
>
> Key: HDDS-1469
> URL: https://issues.apache.org/jira/browse/HDDS-1469
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
>Reporter: Elek, Marton
>Assignee: Elek, Marton
>Priority: Major
> Labels: pull-request-available
> Time Spent: 2h
> Remaining Estimate: 0h
>
> See the design doc in the parent jira for more details.
> In this jira I introduce a new annotation processor which can generate
> ozone-default.xml fragments based on the annotations which are introduced by
> HDDS-1468.
> The ozone-default-generated.xml fragments can be used directly by the
> OzoneConfiguration as I added a small code to the constructor to check ALL
> the available ozone-default-generated.xml files and add them to the available
> resources.
> With this approach we don't need to edit ozone-default.xml as all the
> configuration can be defined in java code.
> As a side effect each service will see only the available configuration keys
> and values based on the classpath. (If the ozone-default-generated.xml file
> of OzoneManager is not on the classpath of the SCM, SCM doesn't see the
> available configs.)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1469) Generate default configuration fragments based on annotations
[
https://issues.apache.org/jira/browse/HDDS-1469?focusedWorklogId=233061=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-233061
]
ASF GitHub Bot logged work on HDDS-1469:
Author: ASF GitHub Bot
Created on: 25/Apr/19 19:34
Start Date: 25/Apr/19 19:34
Worklog Time Spent: 10m
Work Description: anuengineer commented on pull request #773: HDDS-1469.
Generate default configuration fragments based on annotations
URL: https://github.com/apache/hadoop/pull/773#discussion_r278694440
##
File path:
hadoop-hdds/common/src/test/java/org/apache/hadoop/hdds/conf/SimpleConfiguration.java
##
@@ -0,0 +1,83 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hdds.conf;
+
+import java.util.concurrent.TimeUnit;
+
+/**
+ * Example configuration to test the configuration injection.
+ */
+@ConfigGroup(prefix = "ozone.scm.client")
+public class SimpleConfiguration {
+
+ private String clientAddress;
+
+ private String bindHost;
+
+ private boolean enabled;
+
+ private int port = 1234;
+
+ private long waitTime = 1;
+
+ @Config(key = "address", defaultValue = "localhost")
+ public void setClientAddress(String clientAddress) {
+this.clientAddress = clientAddress;
+ }
+
+ @Config(key = "bind.host", defaultValue = "0.0.0.0")
+ public void setBindHost(String bindHost) {
+this.bindHost = bindHost;
+ }
+
+ @Config(key = "enabled", defaultValue = "true")
+ public void setEnabled(boolean enabled) {
+this.enabled = enabled;
+ }
+
+ @Config(key = "port", defaultValue = "9878")
+ public void setPort(int port) {
+this.port = port;
+ }
+
+ @Config(key = "wait", type = ConfigType.TIME, timeUnit =
+ TimeUnit.SECONDS, defaultValue = "10m")
+ public void setWaitTime(long waitTime) {
Review comment:
Nice
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 233061)
> Generate default configuration fragments based on annotations
> -
>
> Key: HDDS-1469
> URL: https://issues.apache.org/jira/browse/HDDS-1469
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
>Reporter: Elek, Marton
>Assignee: Elek, Marton
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h
> Remaining Estimate: 0h
>
> See the design doc in the parent jira for more details.
> In this jira I introduce a new annotation processor which can generate
> ozone-default.xml fragments based on the annotations which are introduced by
> HDDS-1468.
> The ozone-default-generated.xml fragments can be used directly by the
> OzoneConfiguration as I added a small code to the constructor to check ALL
> the available ozone-default-generated.xml files and add them to the available
> resources.
> With this approach we don't need to edit ozone-default.xml as all the
> configuration can be defined in java code.
> As a side effect each service will see only the available configuration keys
> and values based on the classpath. (If the ozone-default-generated.xml file
> of OzoneManager is not on the classpath of the SCM, SCM doesn't see the
> available configs.)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1469) Generate default configuration fragments based on annotations
[
https://issues.apache.org/jira/browse/HDDS-1469?focusedWorklogId=233063=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-233063
]
ASF GitHub Bot logged work on HDDS-1469:
Author: ASF GitHub Bot
Created on: 25/Apr/19 19:34
Start Date: 25/Apr/19 19:34
Worklog Time Spent: 10m
Work Description: anuengineer commented on pull request #773: HDDS-1469.
Generate default configuration fragments based on annotations
URL: https://github.com/apache/hadoop/pull/773#discussion_r278695268
##
File path:
hadoop-hdds/config/src/main/java/org/apache/hadoop/hdds/conf/Config.java
##
@@ -0,0 +1,59 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hdds.conf;
+
+import java.lang.annotation.ElementType;
+import java.lang.annotation.Retention;
+import java.lang.annotation.RetentionPolicy;
+import java.lang.annotation.Target;
+import java.util.concurrent.TimeUnit;
+
+/**
+ * Mark field to be configurable from ozone-site.xml.
+ */
+@Retention(RetentionPolicy.RUNTIME)
+@Target(ElementType.METHOD)
+public @interface Config {
+
+ /**
+ * Configuration fragment relative to the prefix defined with @ConfigGroup.
+ */
+ String key();
+
+ /**
+ * Default value to use if not set.
+ */
+ String defaultValue();
+
+ /**
+ * Custom description as a help.
+ */
+ String description() default "";
+
+ /**
+ * Type of configuration. Use AUTO to decide it based on the java type.
+ */
+ ConfigType type() default ConfigType.AUTO;
+
+ /**
+ * If type == TIME the unit should be defined with this attribute.
+ */
+ TimeUnit timeUnit() default TimeUnit.MILLISECONDS;
+
+ ConfigTag[] tags() default {ConfigTag.OZONE};
Review comment:
We should enforce this and description, so the code will error out during
compile.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 233063)
Time Spent: 1h 20m (was: 1h 10m)
> Generate default configuration fragments based on annotations
> -
>
> Key: HDDS-1469
> URL: https://issues.apache.org/jira/browse/HDDS-1469
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
>Reporter: Elek, Marton
>Assignee: Elek, Marton
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> See the design doc in the parent jira for more details.
> In this jira I introduce a new annotation processor which can generate
> ozone-default.xml fragments based on the annotations which are introduced by
> HDDS-1468.
> The ozone-default-generated.xml fragments can be used directly by the
> OzoneConfiguration as I added a small code to the constructor to check ALL
> the available ozone-default-generated.xml files and add them to the available
> resources.
> With this approach we don't need to edit ozone-default.xml as all the
> configuration can be defined in java code.
> As a side effect each service will see only the available configuration keys
> and values based on the classpath. (If the ozone-default-generated.xml file
> of OzoneManager is not on the classpath of the SCM, SCM doesn't see the
> available configs.)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1469) Generate default configuration fragments based on annotations
[ https://issues.apache.org/jira/browse/HDDS-1469?focusedWorklogId=233058=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-233058 ] ASF GitHub Bot logged work on HDDS-1469: Author: ASF GitHub Bot Created on: 25/Apr/19 19:34 Start Date: 25/Apr/19 19:34 Worklog Time Spent: 10m Work Description: anuengineer commented on pull request #773: HDDS-1469. Generate default configuration fragments based on annotations URL: https://github.com/apache/hadoop/pull/773#discussion_r278696010 ## File path: hadoop-hdds/config/src/main/java/org/apache/hadoop/hdds/conf/ConfigFileAppender.java ## @@ -0,0 +1,127 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.hdds.conf; + +import javax.xml.parsers.DocumentBuilder; +import javax.xml.parsers.DocumentBuilderFactory; +import javax.xml.transform.OutputKeys; +import javax.xml.transform.Transformer; +import javax.xml.transform.TransformerException; +import javax.xml.transform.TransformerFactory; +import javax.xml.transform.dom.DOMSource; +import javax.xml.transform.stream.StreamResult; +import java.io.InputStream; +import java.io.Writer; +import java.util.Arrays; +import java.util.stream.Collectors; + +import org.w3c.dom.Document; +import org.w3c.dom.Element; + +/** + * Simple DOM based config file writer. + * + * This class can init/load existing ozone-site.xml fragments and append + * new entries and write to the file system. Review comment: Should this class build ozone-default.xml or ozone-site.xml? I was thinking the compilation process builds ozone-default.xml and user can define ozone-site.xml by using a tool like GenConfig. I know we have checked in some empty ozone-site.xml, just a question. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 233058) Time Spent: 1h (was: 50m) > Generate default configuration fragments based on annotations > - > > Key: HDDS-1469 > URL: https://issues.apache.org/jira/browse/HDDS-1469 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Elek, Marton >Assignee: Elek, Marton >Priority: Major > Labels: pull-request-available > Time Spent: 1h > Remaining Estimate: 0h > > See the design doc in the parent jira for more details. > In this jira I introduce a new annotation processor which can generate > ozone-default.xml fragments based on the annotations which are introduced by > HDDS-1468. > The ozone-default-generated.xml fragments can be used directly by the > OzoneConfiguration as I added a small code to the constructor to check ALL > the available ozone-default-generated.xml files and add them to the available > resources. > With this approach we don't need to edit ozone-default.xml as all the > configuration can be defined in java code. > As a side effect each service will see only the available configuration keys > and values based on the classpath. (If the ozone-default-generated.xml file > of OzoneManager is not on the classpath of the SCM, SCM doesn't see the > available configs.) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1469) Generate default configuration fragments based on annotations
[
https://issues.apache.org/jira/browse/HDDS-1469?focusedWorklogId=233064=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-233064
]
ASF GitHub Bot logged work on HDDS-1469:
Author: ASF GitHub Bot
Created on: 25/Apr/19 19:34
Start Date: 25/Apr/19 19:34
Worklog Time Spent: 10m
Work Description: anuengineer commented on pull request #773: HDDS-1469.
Generate default configuration fragments based on annotations
URL: https://github.com/apache/hadoop/pull/773#discussion_r278699642
##
File path:
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/ReplicationManager.java
##
@@ -754,4 +747,64 @@ private InflightAction(final DatanodeDetails datanode,
this.time = time;
}
}
+
+ /**
+ * Configuration used by the Replication Manager.
+ */
+ @ConfigGroup(prefix = "hdds.scm.replication")
+ public static class ReplicationManagerConfiguration {
+/**
+ * The frequency in which ReplicationMonitor thread should run.
+ */
+private long interval = 5 * 60 * 1000;
+
+/**
+ * Timeout for container replication & deletion command issued by
+ * ReplicationManager.
+ */
+private long eventTimeout = 10 * 60 * 1000;
+
+@Config(key = "thread.interval",
+type = ConfigType.TIME,
+defaultValue = "3s",
+tags = {SCM, OZONE},
+description = "When a heartbeat from the data node arrives on SCM, "
++ "It is queued for processing with the time stamp of when the "
Review comment:
@nandakumar131 Do we still use this Key ? or is this some code changed but
we forgot to remove the config value case?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 233064)
Time Spent: 1.5h (was: 1h 20m)
> Generate default configuration fragments based on annotations
> -
>
> Key: HDDS-1469
> URL: https://issues.apache.org/jira/browse/HDDS-1469
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
>Reporter: Elek, Marton
>Assignee: Elek, Marton
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> See the design doc in the parent jira for more details.
> In this jira I introduce a new annotation processor which can generate
> ozone-default.xml fragments based on the annotations which are introduced by
> HDDS-1468.
> The ozone-default-generated.xml fragments can be used directly by the
> OzoneConfiguration as I added a small code to the constructor to check ALL
> the available ozone-default-generated.xml files and add them to the available
> resources.
> With this approach we don't need to edit ozone-default.xml as all the
> configuration can be defined in java code.
> As a side effect each service will see only the available configuration keys
> and values based on the classpath. (If the ozone-default-generated.xml file
> of OzoneManager is not on the classpath of the SCM, SCM doesn't see the
> available configs.)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1469) Generate default configuration fragments based on annotations
[
https://issues.apache.org/jira/browse/HDDS-1469?focusedWorklogId=233062=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-233062
]
ASF GitHub Bot logged work on HDDS-1469:
Author: ASF GitHub Bot
Created on: 25/Apr/19 19:34
Start Date: 25/Apr/19 19:34
Worklog Time Spent: 10m
Work Description: anuengineer commented on pull request #773: HDDS-1469.
Generate default configuration fragments based on annotations
URL: https://github.com/apache/hadoop/pull/773#discussion_r278697984
##
File path:
hadoop-hdds/config/src/main/java/org/apache/hadoop/hdds/conf/ConfigFileGenerator.java
##
@@ -0,0 +1,115 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hdds.conf;
+
+import javax.annotation.processing.AbstractProcessor;
+import javax.annotation.processing.Filer;
+import javax.annotation.processing.RoundEnvironment;
+import javax.annotation.processing.SupportedAnnotationTypes;
+import javax.lang.model.element.Element;
+import javax.lang.model.element.ElementKind;
+import javax.lang.model.element.TypeElement;
+import javax.tools.Diagnostic.Kind;
+import javax.tools.FileObject;
+import javax.tools.StandardLocation;
+import java.io.FileNotFoundException;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStreamWriter;
+import java.io.Writer;
+import java.nio.charset.StandardCharsets;
+import java.util.Set;
+
+/**
+ * Annotation processor to generate ozone-site-generated fragments from
+ * ozone-site.xml.
+ */
+@SupportedAnnotationTypes("org.apache.hadoop.hdds.conf.ConfigGroup")
+public class ConfigFileGenerator extends AbstractProcessor {
+
+ public static final String OUTPUT_FILE_NAME = "ozone-default-generated.xml";
+
+ @Override
+ public boolean process(Set annotations,
+ RoundEnvironment roundEnv) {
+if (roundEnv.processingOver()) {
+ return false;
+}
+
+Filer filer = processingEnv.getFiler();
+System.out.println("round");
Review comment:
Debug?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 233062)
Time Spent: 1h 10m (was: 1h)
> Generate default configuration fragments based on annotations
> -
>
> Key: HDDS-1469
> URL: https://issues.apache.org/jira/browse/HDDS-1469
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
>Reporter: Elek, Marton
>Assignee: Elek, Marton
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> See the design doc in the parent jira for more details.
> In this jira I introduce a new annotation processor which can generate
> ozone-default.xml fragments based on the annotations which are introduced by
> HDDS-1468.
> The ozone-default-generated.xml fragments can be used directly by the
> OzoneConfiguration as I added a small code to the constructor to check ALL
> the available ozone-default-generated.xml files and add them to the available
> resources.
> With this approach we don't need to edit ozone-default.xml as all the
> configuration can be defined in java code.
> As a side effect each service will see only the available configuration keys
> and values based on the classpath. (If the ozone-default-generated.xml file
> of OzoneManager is not on the classpath of the SCM, SCM doesn't see the
> available configs.)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1469) Generate default configuration fragments based on annotations
[
https://issues.apache.org/jira/browse/HDDS-1469?focusedWorklogId=233059=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-233059
]
ASF GitHub Bot logged work on HDDS-1469:
Author: ASF GitHub Bot
Created on: 25/Apr/19 19:34
Start Date: 25/Apr/19 19:34
Worklog Time Spent: 10m
Work Description: anuengineer commented on pull request #773: HDDS-1469.
Generate default configuration fragments based on annotations
URL: https://github.com/apache/hadoop/pull/773#discussion_r278694151
##
File path:
hadoop-hdds/common/src/test/java/org/apache/hadoop/hdds/conf/SimpleConfiguration.java
##
@@ -0,0 +1,83 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hdds.conf;
+
+import java.util.concurrent.TimeUnit;
+
+/**
+ * Example configuration to test the configuration injection.
+ */
+@ConfigGroup(prefix = "ozone.scm.client")
+public class SimpleConfiguration {
+
+ private String clientAddress;
+
+ private String bindHost;
+
+ private boolean enabled;
+
+ private int port = 1234;
+
+ private long waitTime = 1;
+
+ @Config(key = "address", defaultValue = "localhost")
+ public void setClientAddress(String clientAddress) {
+this.clientAddress = clientAddress;
+ }
+
+ @Config(key = "bind.host", defaultValue = "0.0.0.0")
+ public void setBindHost(String bindHost) {
+this.bindHost = bindHost;
+ }
+
+ @Config(key = "enabled", defaultValue = "true")
+ public void setEnabled(boolean enabled) {
+this.enabled = enabled;
Review comment:
nit: what does client. enabled mean?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 233059)
Time Spent: 1h (was: 50m)
> Generate default configuration fragments based on annotations
> -
>
> Key: HDDS-1469
> URL: https://issues.apache.org/jira/browse/HDDS-1469
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
>Reporter: Elek, Marton
>Assignee: Elek, Marton
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h
> Remaining Estimate: 0h
>
> See the design doc in the parent jira for more details.
> In this jira I introduce a new annotation processor which can generate
> ozone-default.xml fragments based on the annotations which are introduced by
> HDDS-1468.
> The ozone-default-generated.xml fragments can be used directly by the
> OzoneConfiguration as I added a small code to the constructor to check ALL
> the available ozone-default-generated.xml files and add them to the available
> resources.
> With this approach we don't need to edit ozone-default.xml as all the
> configuration can be defined in java code.
> As a side effect each service will see only the available configuration keys
> and values based on the classpath. (If the ozone-default-generated.xml file
> of OzoneManager is not on the classpath of the SCM, SCM doesn't see the
> available configs.)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1469) Generate default configuration fragments based on annotations
[
https://issues.apache.org/jira/browse/HDDS-1469?focusedWorklogId=233043=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-233043
]
ASF GitHub Bot logged work on HDDS-1469:
Author: ASF GitHub Bot
Created on: 25/Apr/19 19:09
Start Date: 25/Apr/19 19:09
Worklog Time Spent: 10m
Work Description: anuengineer commented on pull request #773: HDDS-1469.
Generate default configuration fragments based on annotations
URL: https://github.com/apache/hadoop/pull/773#discussion_r278691904
##
File path:
hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/conf/OzoneConfiguration.java
##
@@ -161,14 +285,14 @@ public static void activate() {
Configuration.addDefaultResource("hdfs-default.xml");
Configuration.addDefaultResource("hdfs-site.xml");
Configuration.addDefaultResource("ozone-default.xml");
-Configuration.addDefaultResource("ozone-site.xml");
Review comment:
Shouldn't we still allow this over-ride?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 233043)
Time Spent: 40m (was: 0.5h)
> Generate default configuration fragments based on annotations
> -
>
> Key: HDDS-1469
> URL: https://issues.apache.org/jira/browse/HDDS-1469
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
>Reporter: Elek, Marton
>Assignee: Elek, Marton
>Priority: Major
> Labels: pull-request-available
> Time Spent: 40m
> Remaining Estimate: 0h
>
> See the design doc in the parent jira for more details.
> In this jira I introduce a new annotation processor which can generate
> ozone-default.xml fragments based on the annotations which are introduced by
> HDDS-1468.
> The ozone-default-generated.xml fragments can be used directly by the
> OzoneConfiguration as I added a small code to the constructor to check ALL
> the available ozone-default-generated.xml files and add them to the available
> resources.
> With this approach we don't need to edit ozone-default.xml as all the
> configuration can be defined in java code.
> As a side effect each service will see only the available configuration keys
> and values based on the classpath. (If the ozone-default-generated.xml file
> of OzoneManager is not on the classpath of the SCM, SCM doesn't see the
> available configs.)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1469) Generate default configuration fragments based on annotations
[
https://issues.apache.org/jira/browse/HDDS-1469?focusedWorklogId=233044=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-233044
]
ASF GitHub Bot logged work on HDDS-1469:
Author: ASF GitHub Bot
Created on: 25/Apr/19 19:09
Start Date: 25/Apr/19 19:09
Worklog Time Spent: 10m
Work Description: anuengineer commented on pull request #773: HDDS-1469.
Generate default configuration fragments based on annotations
URL: https://github.com/apache/hadoop/pull/773#discussion_r278691904
##
File path:
hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/conf/OzoneConfiguration.java
##
@@ -161,14 +285,14 @@ public static void activate() {
Configuration.addDefaultResource("hdfs-default.xml");
Configuration.addDefaultResource("hdfs-site.xml");
Configuration.addDefaultResource("ozone-default.xml");
-Configuration.addDefaultResource("ozone-site.xml");
Review comment:
Shouldn't we still allow this over-ride?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 233044)
Time Spent: 50m (was: 40m)
> Generate default configuration fragments based on annotations
> -
>
> Key: HDDS-1469
> URL: https://issues.apache.org/jira/browse/HDDS-1469
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
>Reporter: Elek, Marton
>Assignee: Elek, Marton
>Priority: Major
> Labels: pull-request-available
> Time Spent: 50m
> Remaining Estimate: 0h
>
> See the design doc in the parent jira for more details.
> In this jira I introduce a new annotation processor which can generate
> ozone-default.xml fragments based on the annotations which are introduced by
> HDDS-1468.
> The ozone-default-generated.xml fragments can be used directly by the
> OzoneConfiguration as I added a small code to the constructor to check ALL
> the available ozone-default-generated.xml files and add them to the available
> resources.
> With this approach we don't need to edit ozone-default.xml as all the
> configuration can be defined in java code.
> As a side effect each service will see only the available configuration keys
> and values based on the classpath. (If the ozone-default-generated.xml file
> of OzoneManager is not on the classpath of the SCM, SCM doesn't see the
> available configs.)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1469) Generate default configuration fragments based on annotations
[ https://issues.apache.org/jira/browse/HDDS-1469?focusedWorklogId=232858=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-232858 ] ASF GitHub Bot logged work on HDDS-1469: Author: ASF GitHub Bot Created on: 25/Apr/19 13:30 Start Date: 25/Apr/19 13:30 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on issue #773: HDDS-1469. Generate default configuration fragments based on annotations URL: https://github.com/apache/hadoop/pull/773#issuecomment-486673145 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 0 | Docker mode activated. | | -1 | patch | 8 | https://github.com/apache/hadoop/pull/773 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. | | Subsystem | Report/Notes | |--:|:-| | GITHUB PR | https://github.com/apache/hadoop/pull/773 | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-773/1/console | | Powered by | Apache Yetus 0.9.0 http://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 232858) Time Spent: 0.5h (was: 20m) > Generate default configuration fragments based on annotations > - > > Key: HDDS-1469 > URL: https://issues.apache.org/jira/browse/HDDS-1469 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Elek, Marton >Assignee: Elek, Marton >Priority: Major > Labels: pull-request-available > Time Spent: 0.5h > Remaining Estimate: 0h > > See the design doc in the parent jira for more details. > In this jira I introduce a new annotation processor which can generate > ozone-default.xml fragments based on the annotations which are introduced by > HDDS-1468. > The ozone-default-generated.xml fragments can be used directly by the > OzoneConfiguration as I added a small code to the constructor to check ALL > the available ozone-default-generated.xml files and add them to the available > resources. > With this approach we don't need to edit ozone-default.xml as all the > configuration can be defined in java code. > As a side effect each service will see only the available configuration keys > and values based on the classpath. (If the ozone-default-generated.xml file > of OzoneManager is not on the classpath of the SCM, SCM doesn't see the > available configs.) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1469) Generate default configuration fragments based on annotations
[ https://issues.apache.org/jira/browse/HDDS-1469?focusedWorklogId=232857=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-232857 ] ASF GitHub Bot logged work on HDDS-1469: Author: ASF GitHub Bot Created on: 25/Apr/19 13:28 Start Date: 25/Apr/19 13:28 Worklog Time Spent: 10m Work Description: elek commented on issue #773: HDDS-1469. Generate default configuration fragments based on annotations URL: https://github.com/apache/hadoop/pull/773#issuecomment-486672703 /cc @anuengineer This is the annotation processor. With a small modification in the OzoneConfiguration (to load all the generated fragments) we don't need to merge all the generated config files to one big ozone-default.xml This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 232857) Time Spent: 20m (was: 10m) > Generate default configuration fragments based on annotations > - > > Key: HDDS-1469 > URL: https://issues.apache.org/jira/browse/HDDS-1469 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Elek, Marton >Assignee: Elek, Marton >Priority: Major > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > > See the design doc in the parent jira for more details. > In this jira I introduce a new annotation processor which can generate > ozone-default.xml fragments based on the annotations which are introduced by > HDDS-1468. > The ozone-default-generated.xml fragments can be used directly by the > OzoneConfiguration as I added a small code to the constructor to check ALL > the available ozone-default-generated.xml files and add them to the available > resources. > With this approach we don't need to edit ozone-default.xml as all the > configuration can be defined in java code. > As a side effect each service will see only the available configuration keys > and values based on the classpath. (If the ozone-default-generated.xml file > of OzoneManager is not on the classpath of the SCM, SCM doesn't see the > available configs.) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1469) Generate default configuration fragments based on annotations
[ https://issues.apache.org/jira/browse/HDDS-1469?focusedWorklogId=232855=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-232855 ] ASF GitHub Bot logged work on HDDS-1469: Author: ASF GitHub Bot Created on: 25/Apr/19 13:27 Start Date: 25/Apr/19 13:27 Worklog Time Spent: 10m Work Description: elek commented on pull request #773: HDDS-1469. Generate default configuration fragments based on annotations URL: https://github.com/apache/hadoop/pull/773 See the design doc in the parent jira for more details. In this jira I introduce a new annotation processor which can generate ozone-default.xml fragments based on the annotations which are introduced by HDDS-1468. The ozone-default-generated.xml fragments can be used directly by the OzoneConfiguration as I added a small code to the constructor to check ALL the available ozone-default-generated.xml files and add them to the available resources. With this approach we don't need to edit ozone-default.xml as all the configuration can be defined in java code. As a side effect each service will see only the available configuration keys and values based on the classpath. (If the ozone-default-generated.xml file of OzoneManager is not on the classpath of the SCM, SCM doesn't see the available configs.) See: https://issues.apache.org/jira/browse/HDDS-1469 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 232855) Time Spent: 10m Remaining Estimate: 0h > Generate default configuration fragments based on annotations > - > > Key: HDDS-1469 > URL: https://issues.apache.org/jira/browse/HDDS-1469 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Elek, Marton >Assignee: Elek, Marton >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > See the design doc in the parent jira for more details. > In this jira I introduce a new annotation processor which can generate > ozone-default.xml fragments based on the annotations which are introduced by > HDDS-1468. > The ozone-default-generated.xml fragments can be used directly by the > OzoneConfiguration as I added a small code to the constructor to check ALL > the available ozone-default-generated.xml files and add them to the available > resources. > With this approach we don't need to edit ozone-default.xml as all the > configuration can be defined in java code. > As a side effect each service will see only the available configuration keys > and values based on the classpath. (If the ozone-default-generated.xml file > of OzoneManager is not on the classpath of the SCM, SCM doesn't see the > available configs.) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
