[GitHub] jmark99 commented on a change in pull request #337: ACCUMULO-4732 No APIs to configure iterators or locality groups for new tables
jmark99 commented on a change in pull request #337: ACCUMULO-4732 No APIs to configure iterators or locality groups for new tables URL: https://github.com/apache/accumulo/pull/337#discussion_r156669518 ## File path: core/src/main/java/org/apache/accumulo/core/client/admin/NewTableConfiguration.java ## @@ -146,4 +163,116 @@ public NewTableConfiguration enableSummarization(SummarizerConfiguration... conf summarizerProps = tmp; return this; } + + /** + * Configures a table's locality groups prior to initial table creation. + * + * Allows locality groups to be set prior to table creation. Additional calls to this method prior to table creation will overwrite previous locality group + * mappings. + * + * @param groups + * mapping of locality group names to column families in the locality group + * + * @since 2.0.0 + */ + public void setLocalityGroups(Mapgroups) { +// ensure locality groups do not overlap +LocalityGroupUtil.ensureNonOverlappingGroups(groups); +localityProps = new HashMap<>(); +for (Entry entry : groups.entrySet()) { + Set colFams = entry.getValue(); + String value = LocalityGroupUtil.encodeColumnFamilies(colFams); + localityProps.put(Property.TABLE_LOCALITY_GROUP_PREFIX + entry.getKey(), value); +} +// localityProps.put(Property.TABLE_LOCALITY_GROUPS.getKey(), Joiner.on(",").join(groups.keySet())); +localityProps.put(Property.TABLE_LOCALITY_GROUPS.getKey(), groups.keySet().stream().collect(Collectors.joining(","))); +// localityProps.put(Property.TABLE_LOCALITY_GROUPS.getKey(), Stream.of(groups.keySet().collect(Collectors.joining(",")); Review comment: Just an oversight. Thanks for the catch. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] bfach10 commented on a change in pull request #336: ACCUMULO-4755 Custom serialization for AbstractId types
bfach10 commented on a change in pull request #336: ACCUMULO-4755 Custom serialization for AbstractId types URL: https://github.com/apache/accumulo/pull/336#discussion_r156671887 ## File path: core/src/main/java/org/apache/accumulo/core/client/impl/AbstractId.java ## @@ -39,7 +39,7 @@ protected AbstractId(final String canonical) { /** * The canonical ID */ - public final String canonicalID() { + public String canonicalID() { Review comment: Definitely an oversight on my part, fixed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] jmark99 commented on a change in pull request #337: ACCUMULO-4732 No APIs to configure iterators or locality groups for new tables
jmark99 commented on a change in pull request #337: ACCUMULO-4732 No APIs to configure iterators or locality groups for new tables URL: https://github.com/apache/accumulo/pull/337#discussion_r156690169 ## File path: test/src/main/java/org/apache/accumulo/test/NewConfigurationTestIT.java ## @@ -0,0 +1,473 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.accumulo.test; + +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertFalse; + +import java.util.Collections; +import java.util.EnumSet; +import java.util.HashMap; +import java.util.Map; +import java.util.Map.Entry; +import java.util.Set; +import java.util.TreeMap; + +import org.apache.accumulo.core.client.AccumuloException; +import org.apache.accumulo.core.client.AccumuloSecurityException; +import org.apache.accumulo.core.client.Connector; +import org.apache.accumulo.core.client.IteratorSetting; +import org.apache.accumulo.core.client.TableExistsException; +import org.apache.accumulo.core.client.TableNotFoundException; +import org.apache.accumulo.core.client.admin.NewTableConfiguration; +import org.apache.accumulo.core.iterators.IteratorUtil.IteratorScope; +import org.apache.accumulo.harness.SharedMiniClusterBase; +import org.apache.hadoop.io.Text; +import org.junit.AfterClass; +import org.junit.Assert; +import org.junit.BeforeClass; +import org.junit.Test; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import com.google.common.collect.ImmutableSet; + +public class NewConfigurationTestIT extends SharedMiniClusterBase { + + private static final Logger log = LoggerFactory.getLogger(NewConfigurationTestIT.class); + + @Override + protected int defaultTimeoutSeconds() { +return 30; + } + + @BeforeClass + public static void setup() throws Exception { +SharedMiniClusterBase.startMiniCluster(); + } + + @AfterClass + public static void teardown() throws Exception { +SharedMiniClusterBase.stopMiniCluster(); + } + + /** + * Test that setting properties more than once overwrites the previous property settings. + */ + @Test + public void testSetPropertiesOverwriteOlderProperties() throws AccumuloSecurityException, AccumuloException, TableExistsException, TableNotFoundException { +Connector conn = getConnector(); +String tableName = getUniqueNames(2)[0]; +NewTableConfiguration ntc = new NewTableConfiguration(); +MapinitialProps = new HashMap<>(); +initialProps.put("prop1", "val1"); +initialProps.put("prop2", "val2"); +ntc.setProperties(initialProps); +// Create a new set of properties and set them with setProperties +Map updatedProps = new HashMap<>(); +updatedProps.put("newerprop1", "newerval1"); +updatedProps.put("newerprop2", "newerval2"); +ntc.setProperties(updatedProps); +conn.tableOperations().create(tableName, ntc); +// verify +Map props = ntc.getProperties(); +assertEquals(props.get("newerprop1"), "newerval1"); +assertEquals(props.get("newerprop2"), "newerval2"); +assertFalse(props.keySet().contains("prop1")); +assertFalse(props.keySet().contains("prop2")); + } + + /** + * Verify that you cannot have overlapping locality groups. + * + * Attempt to set a locality group with overlapping groups. This test should throw an IllegalArgumentException indicating that groups overlap. + */ + @Test(expected = IllegalArgumentException.class) + public void testOverlappingGroupsFail() throws AccumuloSecurityException, AccumuloException, TableExistsException { +Connector conn = getConnector(); +String tableName = getUniqueNames(2)[0]; + +NewTableConfiguration ntc = new NewTableConfiguration(); +Map lgroups = new HashMap<>(); +lgroups.put("lg1", ImmutableSet.of(new Text("colFamA"), new Text("colFamB"))); +lgroups.put("lg2", ImmutableSet.of(new Text("colFamC"), new Text("colFamB"))); +ntc.setLocalityGroups(lgroups); +conn.tableOperations().create(tableName, ntc); + } + + /** + * Test simplest case of setting locality groups at table creation. + */ + @Test + public void testSimpleLocalityGroupCreation() throws
Accumulo-Pull-Requests - Build # 895 - Still Failing
The Apache Jenkins build system has built Accumulo-Pull-Requests (build #895) Status: Still Failing Check console output at https://builds.apache.org/job/Accumulo-Pull-Requests/895/ to view the results.
[GitHub] jmark99 commented on a change in pull request #337: ACCUMULO-4732 No APIs to configure iterators or locality groups for new tables
jmark99 commented on a change in pull request #337: ACCUMULO-4732 No APIs to configure iterators or locality groups for new tables URL: https://github.com/apache/accumulo/pull/337#discussion_r156669339 ## File path: core/src/main/java/org/apache/accumulo/core/client/admin/NewTableConfiguration.java ## @@ -146,4 +163,116 @@ public NewTableConfiguration enableSummarization(SummarizerConfiguration... conf summarizerProps = tmp; return this; } + + /** + * Configures a table's locality groups prior to initial table creation. + * + * Allows locality groups to be set prior to table creation. Additional calls to this method prior to table creation will overwrite previous locality group + * mappings. + * + * @param groups + * mapping of locality group names to column families in the locality group + * + * @since 2.0.0 + */ Review comment: Added @see tag This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] jmark99 commented on a change in pull request #337: ACCUMULO-4732 No APIs to configure iterators or locality groups for new tables
jmark99 commented on a change in pull request #337: ACCUMULO-4732 No APIs to configure iterators or locality groups for new tables URL: https://github.com/apache/accumulo/pull/337#discussion_r156671315 ## File path: core/src/main/java/org/apache/accumulo/core/client/admin/NewTableConfiguration.java ## @@ -146,4 +163,116 @@ public NewTableConfiguration enableSummarization(SummarizerConfiguration... conf summarizerProps = tmp; return this; } + + /** + * Configures a table's locality groups prior to initial table creation. + * + * Allows locality groups to be set prior to table creation. Additional calls to this method prior to table creation will overwrite previous locality group + * mappings. + * + * @param groups + * mapping of locality group names to column families in the locality group + * + * @since 2.0.0 + */ + public void setLocalityGroups(Mapgroups) { Review comment: @keith-turner I went back and forth initially and at one point considered having two versions, one returning this and one returning void. I eventually went with void to match the return type of the TableOperations version. I can modify to return 'this' instead though. Would you suggest that the attachIterator methods return 'this' as well? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] jmark99 commented on a change in pull request #337: ACCUMULO-4732 No APIs to configure iterators or locality groups for new tables
jmark99 commented on a change in pull request #337: ACCUMULO-4732 No APIs to configure iterators or locality groups for new tables URL: https://github.com/apache/accumulo/pull/337#discussion_r156669625 ## File path: core/src/main/java/org/apache/accumulo/core/client/admin/NewTableConfiguration.java ## @@ -146,4 +163,116 @@ public NewTableConfiguration enableSummarization(SummarizerConfiguration... conf summarizerProps = tmp; return this; } + + /** + * Configures a table's locality groups prior to initial table creation. + * + * Allows locality groups to be set prior to table creation. Additional calls to this method prior to table creation will overwrite previous locality group + * mappings. + * + * @param groups + * mapping of locality group names to column families in the locality group + * + * @since 2.0.0 + */ + public void setLocalityGroups(Mapgroups) { +// ensure locality groups do not overlap +LocalityGroupUtil.ensureNonOverlappingGroups(groups); +localityProps = new HashMap<>(); +for (Entry entry : groups.entrySet()) { + Set colFams = entry.getValue(); + String value = LocalityGroupUtil.encodeColumnFamilies(colFams); + localityProps.put(Property.TABLE_LOCALITY_GROUP_PREFIX + entry.getKey(), value); +} +// localityProps.put(Property.TABLE_LOCALITY_GROUPS.getKey(), Joiner.on(",").join(groups.keySet())); +localityProps.put(Property.TABLE_LOCALITY_GROUPS.getKey(), groups.keySet().stream().collect(Collectors.joining(","))); +// localityProps.put(Property.TABLE_LOCALITY_GROUPS.getKey(), Stream.of(groups.keySet().collect(Collectors.joining(",")); +// Stream.of(groups.keySet()).collect(joining(",")); + } + + /** + * Configure iterator settings for a table prior to its creation. + * + * Additional calls to this method before table creation will overwrite previous iterator settings. + * + * @param setting + * object specifying the properties of the iterator + * @throws AccumuloSecurityException + * thrown if the user does not have the ability to set properties on the table + * @throws AccumuloException + * if a general error occurs + * @throws TableNotFoundException + * if the table does not exist + * + * @since 2.0.0 + */ + public void attachIterator(IteratorSetting setting) throws AccumuloException, TableNotFoundException { +attachIterator(setting, EnumSet.allOf(IteratorScope.class)); + } + + /** + * Configure iterator settings for a table prior to its creation. + * + * @param setting + * object specifying the properties of the iterator + * @param scopes + * enumerated set of iterator scopes + * @throws AccumuloException + * if a general error occurs + * @throws AccumuloSecurityException Review comment: Removed. Leftover from some previous incarnation of the code. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] jmark99 commented on a change in pull request #337: ACCUMULO-4732 No APIs to configure iterators or locality groups for new tables
jmark99 commented on a change in pull request #337: ACCUMULO-4732 No APIs to configure iterators or locality groups for new tables URL: https://github.com/apache/accumulo/pull/337#discussion_r156690169 ## File path: test/src/main/java/org/apache/accumulo/test/NewConfigurationTestIT.java ## @@ -0,0 +1,473 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.accumulo.test; + +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertFalse; + +import java.util.Collections; +import java.util.EnumSet; +import java.util.HashMap; +import java.util.Map; +import java.util.Map.Entry; +import java.util.Set; +import java.util.TreeMap; + +import org.apache.accumulo.core.client.AccumuloException; +import org.apache.accumulo.core.client.AccumuloSecurityException; +import org.apache.accumulo.core.client.Connector; +import org.apache.accumulo.core.client.IteratorSetting; +import org.apache.accumulo.core.client.TableExistsException; +import org.apache.accumulo.core.client.TableNotFoundException; +import org.apache.accumulo.core.client.admin.NewTableConfiguration; +import org.apache.accumulo.core.iterators.IteratorUtil.IteratorScope; +import org.apache.accumulo.harness.SharedMiniClusterBase; +import org.apache.hadoop.io.Text; +import org.junit.AfterClass; +import org.junit.Assert; +import org.junit.BeforeClass; +import org.junit.Test; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import com.google.common.collect.ImmutableSet; + +public class NewConfigurationTestIT extends SharedMiniClusterBase { + + private static final Logger log = LoggerFactory.getLogger(NewConfigurationTestIT.class); + + @Override + protected int defaultTimeoutSeconds() { +return 30; + } + + @BeforeClass + public static void setup() throws Exception { +SharedMiniClusterBase.startMiniCluster(); + } + + @AfterClass + public static void teardown() throws Exception { +SharedMiniClusterBase.stopMiniCluster(); + } + + /** + * Test that setting properties more than once overwrites the previous property settings. + */ + @Test + public void testSetPropertiesOverwriteOlderProperties() throws AccumuloSecurityException, AccumuloException, TableExistsException, TableNotFoundException { +Connector conn = getConnector(); +String tableName = getUniqueNames(2)[0]; +NewTableConfiguration ntc = new NewTableConfiguration(); +MapinitialProps = new HashMap<>(); +initialProps.put("prop1", "val1"); +initialProps.put("prop2", "val2"); +ntc.setProperties(initialProps); +// Create a new set of properties and set them with setProperties +Map updatedProps = new HashMap<>(); +updatedProps.put("newerprop1", "newerval1"); +updatedProps.put("newerprop2", "newerval2"); +ntc.setProperties(updatedProps); +conn.tableOperations().create(tableName, ntc); +// verify +Map props = ntc.getProperties(); +assertEquals(props.get("newerprop1"), "newerval1"); +assertEquals(props.get("newerprop2"), "newerval2"); +assertFalse(props.keySet().contains("prop1")); +assertFalse(props.keySet().contains("prop2")); + } + + /** + * Verify that you cannot have overlapping locality groups. + * + * Attempt to set a locality group with overlapping groups. This test should throw an IllegalArgumentException indicating that groups overlap. + */ + @Test(expected = IllegalArgumentException.class) + public void testOverlappingGroupsFail() throws AccumuloSecurityException, AccumuloException, TableExistsException { +Connector conn = getConnector(); +String tableName = getUniqueNames(2)[0]; + +NewTableConfiguration ntc = new NewTableConfiguration(); +Map lgroups = new HashMap<>(); +lgroups.put("lg1", ImmutableSet.of(new Text("colFamA"), new Text("colFamB"))); +lgroups.put("lg2", ImmutableSet.of(new Text("colFamC"), new Text("colFamB"))); +ntc.setLocalityGroups(lgroups); +conn.tableOperations().create(tableName, ntc); + } + + /** + * Test simplest case of setting locality groups at table creation. + */ + @Test + public void testSimpleLocalityGroupCreation() throws
[GitHub] jmark99 commented on a change in pull request #337: ACCUMULO-4732 No APIs to configure iterators or locality groups for new tables
jmark99 commented on a change in pull request #337: ACCUMULO-4732 No APIs to configure iterators or locality groups for new tables URL: https://github.com/apache/accumulo/pull/337#discussion_r156679381 ## File path: test/src/main/java/org/apache/accumulo/test/NewConfigurationTestIT.java ## @@ -0,0 +1,473 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.accumulo.test; + +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertFalse; + +import java.util.Collections; +import java.util.EnumSet; +import java.util.HashMap; +import java.util.Map; +import java.util.Map.Entry; +import java.util.Set; +import java.util.TreeMap; + +import org.apache.accumulo.core.client.AccumuloException; +import org.apache.accumulo.core.client.AccumuloSecurityException; +import org.apache.accumulo.core.client.Connector; +import org.apache.accumulo.core.client.IteratorSetting; +import org.apache.accumulo.core.client.TableExistsException; +import org.apache.accumulo.core.client.TableNotFoundException; +import org.apache.accumulo.core.client.admin.NewTableConfiguration; +import org.apache.accumulo.core.iterators.IteratorUtil.IteratorScope; +import org.apache.accumulo.harness.SharedMiniClusterBase; +import org.apache.hadoop.io.Text; +import org.junit.AfterClass; +import org.junit.Assert; +import org.junit.BeforeClass; +import org.junit.Test; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import com.google.common.collect.ImmutableSet; + +public class NewConfigurationTestIT extends SharedMiniClusterBase { + + private static final Logger log = LoggerFactory.getLogger(NewConfigurationTestIT.class); + + @Override + protected int defaultTimeoutSeconds() { +return 30; + } + + @BeforeClass + public static void setup() throws Exception { +SharedMiniClusterBase.startMiniCluster(); + } + + @AfterClass + public static void teardown() throws Exception { +SharedMiniClusterBase.stopMiniCluster(); + } + + /** + * Test that setting properties more than once overwrites the previous property settings. + */ + @Test + public void testSetPropertiesOverwriteOlderProperties() throws AccumuloSecurityException, AccumuloException, TableExistsException, TableNotFoundException { +Connector conn = getConnector(); +String tableName = getUniqueNames(2)[0]; +NewTableConfiguration ntc = new NewTableConfiguration(); +MapinitialProps = new HashMap<>(); +initialProps.put("prop1", "val1"); +initialProps.put("prop2", "val2"); +ntc.setProperties(initialProps); +// Create a new set of properties and set them with setProperties +Map updatedProps = new HashMap<>(); +updatedProps.put("newerprop1", "newerval1"); +updatedProps.put("newerprop2", "newerval2"); +ntc.setProperties(updatedProps); +conn.tableOperations().create(tableName, ntc); +// verify +Map props = ntc.getProperties(); +assertEquals(props.get("newerprop1"), "newerval1"); +assertEquals(props.get("newerprop2"), "newerval2"); +assertFalse(props.keySet().contains("prop1")); +assertFalse(props.keySet().contains("prop2")); + } + + /** + * Verify that you cannot have overlapping locality groups. + * + * Attempt to set a locality group with overlapping groups. This test should throw an IllegalArgumentException indicating that groups overlap. + */ + @Test(expected = IllegalArgumentException.class) + public void testOverlappingGroupsFail() throws AccumuloSecurityException, AccumuloException, TableExistsException { +Connector conn = getConnector(); +String tableName = getUniqueNames(2)[0]; + +NewTableConfiguration ntc = new NewTableConfiguration(); +Map lgroups = new HashMap<>(); +lgroups.put("lg1", ImmutableSet.of(new Text("colFamA"), new Text("colFamB"))); +lgroups.put("lg2", ImmutableSet.of(new Text("colFamC"), new Text("colFamB"))); +ntc.setLocalityGroups(lgroups); +conn.tableOperations().create(tableName, ntc); Review comment: deleted. This is an automated message from the Apache Git Service. To
Accumulo-Pull-Requests - Build # 892 - Still Failing
The Apache Jenkins build system has built Accumulo-Pull-Requests (build #892) Status: Still Failing Check console output at https://builds.apache.org/job/Accumulo-Pull-Requests/892/ to view the results.
[GitHub] ctubbsii commented on a change in pull request #336: ACCUMULO-4755 Custom serialization for AbstractId types
ctubbsii commented on a change in pull request #336: ACCUMULO-4755 Custom serialization for AbstractId types URL: https://github.com/apache/accumulo/pull/336#discussion_r156852510 ## File path: core/src/main/java/org/apache/accumulo/core/client/impl/AbstractId.java ## @@ -39,7 +39,7 @@ protected AbstractId(final String canonical) { /** * The canonical ID */ - public final String canonicalID() { + public String canonicalID() { Review comment: Ah, good. I was worried that this was required for the serialization to work for some reason. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] ctubbsii commented on a change in pull request #336: ACCUMULO-4755 Custom serialization for AbstractId types
ctubbsii commented on a change in pull request #336: ACCUMULO-4755 Custom serialization for AbstractId types URL: https://github.com/apache/accumulo/pull/336#discussion_r156853581 ## File path: core/src/main/java/org/apache/accumulo/core/client/impl/Namespace.java ## @@ -35,6 +37,7 @@ * Uses an internal cache and private constructor for storing a WeakReference of every Namespace.ID. Therefore, a Namespace.ID can't be instantiated outside * this class and is accessed by calling Namespace.ID.{@link #of(String)}. */ + @XmlJavaTypeAdapter(JaxbAbstractIdSerializer.class) Review comment: Too bad `@XmlJavaTypeAdapter` is not marked with `@Inherited`. Then, you could just annotate `AbstractId` instead of both `Table.ID` and `Namespace.ID`. It may be possible to register the serializer directly, using a method call in the monitor code instead of annotations. This would also save from adding the compile-time dependency to the core module. But, this is almost certainly not worth it. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] ctubbsii commented on a change in pull request #338: ACCUMULO-4763 Avoid use of 'file' in property descriptions
ctubbsii commented on a change in pull request #338: ACCUMULO-4763 Avoid use of 'file' in property descriptions URL: https://github.com/apache/accumulo/pull/338#discussion_r156847267 ## File path: core/src/main/java/org/apache/accumulo/core/conf/Property.java ## @@ -336,13 +336,13 @@ + "scan is switched to that local file. We can not switch to the minor compacted file because it may have been modified by iterators. The file " + "dumped to the local dir is an exact copy of what was in memory."), TSERV_BULK_PROCESS_THREADS("tserver.bulk.process.threads", "1", PropertyType.COUNT, - "The master will task a tablet server with pre-processing a bulk file prior to assigning it to the appropriate tablet servers. This configuration" + "The master will task a tablet server with pre-processing a bulk import file prior to assigning it to the appropriate tablet servers. This configuration" Review comment: Do we want to specify "bulk import RFile" here (and possibly in other places), since RFile is the file type for bulk-imported files, or is "bulk import file" sufficient? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] ctubbsii commented on a change in pull request #338: ACCUMULO-4763 Avoid use of 'file' in property descriptions
ctubbsii commented on a change in pull request #338: ACCUMULO-4763 Avoid use of 'file' in property descriptions URL: https://github.com/apache/accumulo/pull/338#discussion_r156847071 ## File path: core/src/main/java/org/apache/accumulo/core/conf/Property.java ## @@ -235,11 +235,11 @@ MASTER_RECOVERY_DELAY("master.recovery.delay", "10s", PropertyType.TIMEDURATION, "When a tablet server's lock is deleted, it takes time for it to completely quit. This delay gives it time before log recoveries begin."), MASTER_LEASE_RECOVERY_WAITING_PERIOD("master.lease.recovery.interval", "5s", PropertyType.TIMEDURATION, - "The amount of time to wait after requesting a WAL file to be recovered"), + "The amount of time to wait after requesting a write-ahead log to be recovered"), MASTER_WALOG_CLOSER_IMPLEMETATION("master.walog.closer.implementation", "org.apache.accumulo.server.master.recovery.HadoopLogCloser", PropertyType.CLASSNAME, - "A class that implements a mechansim to steal write access to a file"), + "A class that implements a mechanism to steal write access to a write-ahead log"), MASTER_FATE_THREADPOOL_SIZE("master.fate.threadpool.size", "4", PropertyType.COUNT, - "The number of threads used to run FAult-Tolerant Executions. These are primarily table operations like merge."), + "The number of threads used to run Fault-Tolerant Executions. These are primarily table operations like merge."), Review comment: The capitalization wasn't a typo. It was highlighting where the name "FATE" comes from. If we're not going to highlight that, then none of it should be capitalized. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
Accumulo-Pull-Requests - Build # 899 - Failure
The Apache Jenkins build system has built Accumulo-Pull-Requests (build #899) Status: Failure Check console output at https://builds.apache.org/job/Accumulo-Pull-Requests/899/ to view the results.
[GitHub] jmark99 commented on a change in pull request #337: ACCUMULO-4732 No APIs to configure iterators or locality groups for new tables
jmark99 commented on a change in pull request #337: ACCUMULO-4732 No APIs to configure iterators or locality groups for new tables URL: https://github.com/apache/accumulo/pull/337#discussion_r156814785 ## File path: core/src/main/java/org/apache/accumulo/core/client/admin/NewTableConfiguration.java ## @@ -146,4 +163,116 @@ public NewTableConfiguration enableSummarization(SummarizerConfiguration... conf summarizerProps = tmp; return this; } + + /** + * Configures a table's locality groups prior to initial table creation. + * + * Allows locality groups to be set prior to table creation. Additional calls to this method prior to table creation will overwrite previous locality group + * mappings. + * + * @param groups + * mapping of locality group names to column families in the locality group + * + * @since 2.0.0 + */ + public void setLocalityGroups(Mapgroups) { +// ensure locality groups do not overlap +LocalityGroupUtil.ensureNonOverlappingGroups(groups); +localityProps = new HashMap<>(); +for (Entry entry : groups.entrySet()) { + Set colFams = entry.getValue(); + String value = LocalityGroupUtil.encodeColumnFamilies(colFams); + localityProps.put(Property.TABLE_LOCALITY_GROUP_PREFIX + entry.getKey(), value); +} +// localityProps.put(Property.TABLE_LOCALITY_GROUPS.getKey(), Joiner.on(",").join(groups.keySet())); +localityProps.put(Property.TABLE_LOCALITY_GROUPS.getKey(), groups.keySet().stream().collect(Collectors.joining(","))); +// localityProps.put(Property.TABLE_LOCALITY_GROUPS.getKey(), Stream.of(groups.keySet().collect(Collectors.joining(",")); +// Stream.of(groups.keySet()).collect(joining(",")); + } + + /** + * Configure iterator settings for a table prior to its creation. + * + * Additional calls to this method before table creation will overwrite previous iterator settings. + * + * @param setting + * object specifying the properties of the iterator + * @throws AccumuloSecurityException + * thrown if the user does not have the ability to set properties on the table + * @throws AccumuloException + * if a general error occurs + * @throws TableNotFoundException + * if the table does not exist + * + * @since 2.0.0 + */ + public void attachIterator(IteratorSetting setting) throws AccumuloException, TableNotFoundException { +attachIterator(setting, EnumSet.allOf(IteratorScope.class)); + } + + /** + * Configure iterator settings for a table prior to its creation. + * + * @param setting + * object specifying the properties of the iterator + * @param scopes + * enumerated set of iterator scopes + * @throws AccumuloException + * if a general error occurs + * @throws AccumuloSecurityException + * thrown if the user does not have the ability to set properties on the table + * @throws TableNotFoundException + * if the table does not exist + * + * @since 2.0.0 + */ + public void attachIterator(IteratorSetting setting, EnumSet scopes) throws AccumuloException, TableNotFoundException { +checkArgument(setting != null, "setting is null"); +checkArgument(scopes != null, "scopes is null"); +if (iteratorProps.isEmpty()) { + iteratorProps = new HashMap<>(); +} +checkIteratorConflicts(setting, scopes); +for (IteratorScope scope : scopes) { + String root = String.format("%s%s.%s", Property.TABLE_ITERATOR_PREFIX, scope.name().toLowerCase(), setting.getName()); + for (Entry prop : setting.getOptions().entrySet()) { +iteratorProps.put(root + ".opt." + prop.getKey(), prop.getValue()); + } + iteratorProps.put(root, setting.getPriority() + "," + setting.getIteratorClass()); +} + } + + private void checkIteratorConflicts(IteratorSetting setting, EnumSet scopes) throws AccumuloException, TableNotFoundException { +checkArgument(setting != null, "setting is null"); Review comment: Created static version of checkIteratorConflicts and refactored existing version of method within TableOptionsHelper to make use of static version. Used static version to check conflicts in new NewTableConfiguration methods. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (ACCUMULO-4763) Avoid use of ambiguous term 'file' in property descriptions
[ https://issues.apache.org/jira/browse/ACCUMULO-4763?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated ACCUMULO-4763: - Labels: pull-request-available (was: ) > Avoid use of ambiguous term 'file' in property descriptions > --- > > Key: ACCUMULO-4763 > URL: https://issues.apache.org/jira/browse/ACCUMULO-4763 > Project: Accumulo > Issue Type: Task > Components: core >Affects Versions: 2.0.0 > Environment: In the configuration property descriptions, 'file' is > used in place of 'RFile' or 'write-ahead log'. Properties would be easier to > understand if the descriptions were more explicit. >Reporter: Mike Walch >Assignee: Mike Walch >Priority: Minor > Labels: pull-request-available > Fix For: 2.0.0 > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[GitHub] mikewalch opened a new pull request #338: ACCUMULO-4763 Avoid use of 'file' in property descriptions
mikewalch opened a new pull request #338: ACCUMULO-4763 Avoid use of 'file' in property descriptions URL: https://github.com/apache/accumulo/pull/338 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
Accumulo-Pull-Requests - Build # 896 - Still Failing
The Apache Jenkins build system has built Accumulo-Pull-Requests (build #896) Status: Still Failing Check console output at https://builds.apache.org/job/Accumulo-Pull-Requests/896/ to view the results.
[GitHub] jmark99 commented on a change in pull request #337: ACCUMULO-4732 No APIs to configure iterators or locality groups for new tables
jmark99 commented on a change in pull request #337: ACCUMULO-4732 No APIs to configure iterators or locality groups for new tables URL: https://github.com/apache/accumulo/pull/337#discussion_r156721002 ## File path: test/src/main/java/org/apache/accumulo/test/NewConfigurationTestIT.java ## @@ -0,0 +1,473 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.accumulo.test; + +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertFalse; + +import java.util.Collections; +import java.util.EnumSet; +import java.util.HashMap; +import java.util.Map; +import java.util.Map.Entry; +import java.util.Set; +import java.util.TreeMap; + +import org.apache.accumulo.core.client.AccumuloException; +import org.apache.accumulo.core.client.AccumuloSecurityException; +import org.apache.accumulo.core.client.Connector; +import org.apache.accumulo.core.client.IteratorSetting; +import org.apache.accumulo.core.client.TableExistsException; +import org.apache.accumulo.core.client.TableNotFoundException; +import org.apache.accumulo.core.client.admin.NewTableConfiguration; +import org.apache.accumulo.core.iterators.IteratorUtil.IteratorScope; +import org.apache.accumulo.harness.SharedMiniClusterBase; +import org.apache.hadoop.io.Text; +import org.junit.AfterClass; +import org.junit.Assert; +import org.junit.BeforeClass; +import org.junit.Test; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import com.google.common.collect.ImmutableSet; + +public class NewConfigurationTestIT extends SharedMiniClusterBase { + + private static final Logger log = LoggerFactory.getLogger(NewConfigurationTestIT.class); + + @Override + protected int defaultTimeoutSeconds() { +return 30; + } + + @BeforeClass + public static void setup() throws Exception { +SharedMiniClusterBase.startMiniCluster(); + } + + @AfterClass + public static void teardown() throws Exception { +SharedMiniClusterBase.stopMiniCluster(); + } + + /** + * Test that setting properties more than once overwrites the previous property settings. + */ + @Test + public void testSetPropertiesOverwriteOlderProperties() throws AccumuloSecurityException, AccumuloException, TableExistsException, TableNotFoundException { +Connector conn = getConnector(); +String tableName = getUniqueNames(2)[0]; +NewTableConfiguration ntc = new NewTableConfiguration(); +MapinitialProps = new HashMap<>(); +initialProps.put("prop1", "val1"); +initialProps.put("prop2", "val2"); +ntc.setProperties(initialProps); +// Create a new set of properties and set them with setProperties +Map updatedProps = new HashMap<>(); +updatedProps.put("newerprop1", "newerval1"); +updatedProps.put("newerprop2", "newerval2"); +ntc.setProperties(updatedProps); +conn.tableOperations().create(tableName, ntc); +// verify +Map props = ntc.getProperties(); +assertEquals(props.get("newerprop1"), "newerval1"); +assertEquals(props.get("newerprop2"), "newerval2"); +assertFalse(props.keySet().contains("prop1")); +assertFalse(props.keySet().contains("prop2")); + } + + /** + * Verify that you cannot have overlapping locality groups. + * + * Attempt to set a locality group with overlapping groups. This test should throw an IllegalArgumentException indicating that groups overlap. + */ + @Test(expected = IllegalArgumentException.class) + public void testOverlappingGroupsFail() throws AccumuloSecurityException, AccumuloException, TableExistsException { +Connector conn = getConnector(); +String tableName = getUniqueNames(2)[0]; + +NewTableConfiguration ntc = new NewTableConfiguration(); +Map lgroups = new HashMap<>(); +lgroups.put("lg1", ImmutableSet.of(new Text("colFamA"), new Text("colFamB"))); +lgroups.put("lg2", ImmutableSet.of(new Text("colFamC"), new Text("colFamB"))); +ntc.setLocalityGroups(lgroups); +conn.tableOperations().create(tableName, ntc); + } + + /** + * Test simplest case of setting locality groups at table creation. + */ + @Test + public void testSimpleLocalityGroupCreation() throws
[GitHub] asfgit commented on issue #338: ACCUMULO-4763 Avoid use of 'file' in property descriptions
asfgit commented on issue #338: ACCUMULO-4763 Avoid use of 'file' in property descriptions URL: https://github.com/apache/accumulo/pull/338#issuecomment-351468279 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Created] (ACCUMULO-4763) Avoid use of ambiguous term 'file' in property descriptions
Mike Walch created ACCUMULO-4763: Summary: Avoid use of ambiguous term 'file' in property descriptions Key: ACCUMULO-4763 URL: https://issues.apache.org/jira/browse/ACCUMULO-4763 Project: Accumulo Issue Type: Task Components: core Affects Versions: 2.0.0 Environment: In the configuration property descriptions, 'file' is used in place of 'RFile' or 'write-ahead log'. Properties would be easier to understand if the descriptions were more explicit. Reporter: Mike Walch Assignee: Mike Walch Priority: Minor Fix For: 2.0.0 -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[GitHub] mikewalch commented on issue #53: Created post announcing Accumulo tour
mikewalch commented on issue #53: Created post announcing Accumulo tour URL: https://github.com/apache/accumulo-website/pull/53#issuecomment-351447638 @milleruntime, I added a teaser in eee053b This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] mikewalch closed pull request #53: Created post announcing Accumulo tour
mikewalch closed pull request #53: Created post announcing Accumulo tour URL: https://github.com/apache/accumulo-website/pull/53 This is a PR merged from a forked repository. As GitHub hides the original diff on merge, it is displayed below for the sake of provenance: As this is a foreign pull request (from a fork), the diff is supplied below (as it won't show otherwise due to GitHub magic): diff --git a/_posts/blog/2017-12-12-take-the-accumulo-tour.md b/_posts/blog/2017-12-12-take-the-accumulo-tour.md new file mode 100644 index ..6dab228d --- /dev/null +++ b/_posts/blog/2017-12-12-take-the-accumulo-tour.md @@ -0,0 +1,23 @@ +--- +title: "Take the Accumulo Tour!" +--- + +Apache Accumulo now has a [hands-on tour][tour] that introduces users to key Accumulo concepts +by completing a series of programming exercises. Don't worry! Solutions are provided if +you get stuck. + +The tour has a [getting started][gs] page that helps you set up your development environment and +clone a repository with template code. After you are set up, the tour has exercises that cover the +following Accumulo concepts: + + * [Writing and Reading from Accumulo](/tour/basic-read-write/) + * [Data Model](/tour/data-model/) + * [Authorizations](/tour/authorizations/) + * [Ranges & Splits](/tour/ranges-splits/) + * [Batch Scanner](/tour/batch-scanner/) + * [Conditional Writer](/tour/conditional-writer/) + +More exercises (such as one on Accumulo Iterators) will be added in the future so check for updates. + +[tour]: /tour/ +[gs]: /tour/getting-started/ diff --git a/tour/index.md b/tour/index.md index 2aac4f2d..4e9224a1 100644 --- a/tour/index.md +++ b/tour/index.md @@ -24,5 +24,5 @@ or [create an issue][issue]. 1. [{{ link_to_page.title }}]({{ doc_url }}) {% endfor %} -[mlist]: /mailing_list/ +[mlist]: /contact-us/#mailing-lists [issue]: https://issues.apache.org/jira/projects/ACCUMULO This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] mikewalch commented on issue #338: ACCUMULO-4763 Avoid use of 'file' in property descriptions
mikewalch commented on issue #338: ACCUMULO-4763 Avoid use of 'file' in property descriptions URL: https://github.com/apache/accumulo/pull/338#issuecomment-351477256 ok to test This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] jmark99 commented on a change in pull request #337: ACCUMULO-4732 No APIs to configure iterators or locality groups for new tables
jmark99 commented on a change in pull request #337: ACCUMULO-4732 No APIs to configure iterators or locality groups for new tables URL: https://github.com/apache/accumulo/pull/337#discussion_r156672610 ## File path: core/src/main/java/org/apache/accumulo/core/client/admin/NewTableConfiguration.java ## @@ -146,4 +163,116 @@ public NewTableConfiguration enableSummarization(SummarizerConfiguration... conf summarizerProps = tmp; return this; } + + /** + * Configures a table's locality groups prior to initial table creation. + * + * Allows locality groups to be set prior to table creation. Additional calls to this method prior to table creation will overwrite previous locality group + * mappings. + * + * @param groups + * mapping of locality group names to column families in the locality group + * + * @since 2.0.0 + */ + public void setLocalityGroups(Mapgroups) { +// ensure locality groups do not overlap +LocalityGroupUtil.ensureNonOverlappingGroups(groups); +localityProps = new HashMap<>(); +for (Entry entry : groups.entrySet()) { + Set colFams = entry.getValue(); + String value = LocalityGroupUtil.encodeColumnFamilies(colFams); + localityProps.put(Property.TABLE_LOCALITY_GROUP_PREFIX + entry.getKey(), value); +} +// localityProps.put(Property.TABLE_LOCALITY_GROUPS.getKey(), Joiner.on(",").join(groups.keySet())); +localityProps.put(Property.TABLE_LOCALITY_GROUPS.getKey(), groups.keySet().stream().collect(Collectors.joining(","))); +// localityProps.put(Property.TABLE_LOCALITY_GROUPS.getKey(), Stream.of(groups.keySet().collect(Collectors.joining(",")); +// Stream.of(groups.keySet()).collect(joining(",")); + } + + /** + * Configure iterator settings for a table prior to its creation. + * + * Additional calls to this method before table creation will overwrite previous iterator settings. + * + * @param setting + * object specifying the properties of the iterator + * @throws AccumuloSecurityException + * thrown if the user does not have the ability to set properties on the table + * @throws AccumuloException + * if a general error occurs + * @throws TableNotFoundException Review comment: You are correct. It initially had been propagated up from a utility method. Forgot to remove when that utility method was modified. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] keith-turner commented on a change in pull request #337: ACCUMULO-4732 No APIs to configure iterators or locality groups for new tables
keith-turner commented on a change in pull request #337: ACCUMULO-4732 No APIs to configure iterators or locality groups for new tables URL: https://github.com/apache/accumulo/pull/337#discussion_r156675129 ## File path: core/src/main/java/org/apache/accumulo/core/client/admin/NewTableConfiguration.java ## @@ -146,4 +163,116 @@ public NewTableConfiguration enableSummarization(SummarizerConfiguration... conf summarizerProps = tmp; return this; } + + /** + * Configures a table's locality groups prior to initial table creation. + * + * Allows locality groups to be set prior to table creation. Additional calls to this method prior to table creation will overwrite previous locality group + * mappings. + * + * @param groups + * mapping of locality group names to column families in the locality group + * + * @since 2.0.0 + */ + public void setLocalityGroups(Mapgroups) { Review comment: Yeah I think both should return this. IMO its better to match the norms in this class rather than what tableoperations does. Also it allows method calls the be chained. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
Accumulo-Pull-Requests - Build # 894 - Failure
The Apache Jenkins build system has built Accumulo-Pull-Requests (build #894) Status: Failure Check console output at https://builds.apache.org/job/Accumulo-Pull-Requests/894/ to view the results.
Accumulo-Pull-Requests - Build # 897 - Fixed
The Apache Jenkins build system has built Accumulo-Pull-Requests (build #897) Status: Fixed Check console output at https://builds.apache.org/job/Accumulo-Pull-Requests/897/ to view the results.
[GitHub] jmark99 commented on a change in pull request #337: ACCUMULO-4732 No APIs to configure iterators or locality groups for new tables
jmark99 commented on a change in pull request #337: ACCUMULO-4732 No APIs to configure iterators or locality groups for new tables URL: https://github.com/apache/accumulo/pull/337#discussion_r156779479 ## File path: core/src/main/java/org/apache/accumulo/core/client/admin/NewTableConfiguration.java ## @@ -146,4 +163,116 @@ public NewTableConfiguration enableSummarization(SummarizerConfiguration... conf summarizerProps = tmp; return this; } + + /** + * Configures a table's locality groups prior to initial table creation. + * + * Allows locality groups to be set prior to table creation. Additional calls to this method prior to table creation will overwrite previous locality group + * mappings. + * + * @param groups + * mapping of locality group names to column families in the locality group + * + * @since 2.0.0 + */ + public void setLocalityGroups(Mapgroups) { +// ensure locality groups do not overlap +LocalityGroupUtil.ensureNonOverlappingGroups(groups); +localityProps = new HashMap<>(); +for (Entry entry : groups.entrySet()) { + Set colFams = entry.getValue(); + String value = LocalityGroupUtil.encodeColumnFamilies(colFams); + localityProps.put(Property.TABLE_LOCALITY_GROUP_PREFIX + entry.getKey(), value); +} +// localityProps.put(Property.TABLE_LOCALITY_GROUPS.getKey(), Joiner.on(",").join(groups.keySet())); +localityProps.put(Property.TABLE_LOCALITY_GROUPS.getKey(), groups.keySet().stream().collect(Collectors.joining(","))); +// localityProps.put(Property.TABLE_LOCALITY_GROUPS.getKey(), Stream.of(groups.keySet().collect(Collectors.joining(",")); +// Stream.of(groups.keySet()).collect(joining(",")); + } + + /** + * Configure iterator settings for a table prior to its creation. + * + * Additional calls to this method before table creation will overwrite previous iterator settings. + * + * @param setting + * object specifying the properties of the iterator + * @throws AccumuloSecurityException + * thrown if the user does not have the ability to set properties on the table + * @throws AccumuloException + * if a general error occurs + * @throws TableNotFoundException + * if the table does not exist + * + * @since 2.0.0 + */ + public void attachIterator(IteratorSetting setting) throws AccumuloException, TableNotFoundException { +attachIterator(setting, EnumSet.allOf(IteratorScope.class)); + } + + /** + * Configure iterator settings for a table prior to its creation. + * + * @param setting + * object specifying the properties of the iterator + * @param scopes + * enumerated set of iterator scopes + * @throws AccumuloException + * if a general error occurs + * @throws AccumuloSecurityException + * thrown if the user does not have the ability to set properties on the table + * @throws TableNotFoundException + * if the table does not exist + * + * @since 2.0.0 + */ + public void attachIterator(IteratorSetting setting, EnumSet scopes) throws AccumuloException, TableNotFoundException { +checkArgument(setting != null, "setting is null"); +checkArgument(scopes != null, "scopes is null"); +if (iteratorProps.isEmpty()) { + iteratorProps = new HashMap<>(); +} +checkIteratorConflicts(setting, scopes); +for (IteratorScope scope : scopes) { + String root = String.format("%s%s.%s", Property.TABLE_ITERATOR_PREFIX, scope.name().toLowerCase(), setting.getName()); + for (Entry prop : setting.getOptions().entrySet()) { +iteratorProps.put(root + ".opt." + prop.getKey(), prop.getValue()); + } + iteratorProps.put(root, setting.getPriority() + "," + setting.getIteratorClass()); +} + } + + private void checkIteratorConflicts(IteratorSetting setting, EnumSet scopes) throws AccumuloException, TableNotFoundException { +checkArgument(setting != null, "setting is null"); Review comment: @keith-turner I have a question concerning this. It's strightforward enough to create the new method and refactor the old. Perhaps I'm not completely versed on its usage yet, but wouldn't calling the method from NewTableConfiguration require instantiation of the TableOperationsImpl object? When I was playing around with it, this required a ClientConfiguration object which then required an Instance object,etc. It seemed to get messy attempting the call. Am I missing something obvious? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] mikewalch commented on issue #338: ACCUMULO-4763 Avoid use of 'file' in property descriptions
mikewalch commented on issue #338: ACCUMULO-4763 Avoid use of 'file' in property descriptions URL: https://github.com/apache/accumulo/pull/338#issuecomment-351519258 @milleruntime, this doesn't fix a bug so I am not too keen on putting it in 1.8 or 1.7 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Created] (ACCUMULO-4764) Move static html from JS to templates
Michael Miller created ACCUMULO-4764: Summary: Move static html from JS to templates Key: ACCUMULO-4764 URL: https://issues.apache.org/jira/browse/ACCUMULO-4764 Project: Accumulo Issue Type: Improvement Components: monitor Affects Versions: 2.0.0 Reporter: Michael Miller Assignee: Michael Miller Fix For: 2.0.0 The new Monitor has too much code embedded in the Javascript. A lot of it is just static html code that should be created in the freemarker templates before the JS is loaded. For example, all of the table column descriptions used for tool tips are loaded into one massive JS Arrary. Moving the static code to the templates will help greatly with maintenance, specifically keeping external Javascript libraries up to date. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (ACCUMULO-4764) Move static html from JS to templates
[ https://issues.apache.org/jira/browse/ACCUMULO-4764?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Miller updated ACCUMULO-4764: - Description: The new Monitor has too much code embedded in the Javascript. A lot of it is just static html code that should be created in the freemarker templates before the JS is loaded. For example, all of the table column descriptions used for tool tips are loaded into [one massive JS Arrary|https://github.com/apache/accumulo/blob/master/server/monitor/src/main/resources/org/apache/accumulo/monitor/resources/js/global.js]. Moving the static code to the templates will help greatly with maintenance, specifically keeping external Javascript libraries up to date. (was: The new Monitor has too much code embedded in the Javascript. A lot of it is just static html code that should be created in the freemarker templates before the JS is loaded. For example, all of the table column descriptions used for tool tips are loaded into one massive JS Arrary. Moving the static code to the templates will help greatly with maintenance, specifically keeping external Javascript libraries up to date.) > Move static html from JS to templates > - > > Key: ACCUMULO-4764 > URL: https://issues.apache.org/jira/browse/ACCUMULO-4764 > Project: Accumulo > Issue Type: Improvement > Components: monitor >Affects Versions: 2.0.0 >Reporter: Michael Miller >Assignee: Michael Miller > Fix For: 2.0.0 > > > The new Monitor has too much code embedded in the Javascript. A lot of it is > just static html code that should be created in the freemarker templates > before the JS is loaded. For example, all of the table column descriptions > used for tool tips are loaded into [one massive JS > Arrary|https://github.com/apache/accumulo/blob/master/server/monitor/src/main/resources/org/apache/accumulo/monitor/resources/js/global.js]. > Moving the static code to the templates will help greatly with maintenance, > specifically keeping external Javascript libraries up to date. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[GitHub] keith-turner commented on a change in pull request #337: ACCUMULO-4732 No APIs to configure iterators or locality groups for new tables
keith-turner commented on a change in pull request #337: ACCUMULO-4732 No APIs to configure iterators or locality groups for new tables URL: https://github.com/apache/accumulo/pull/337#discussion_r156799615 ## File path: core/src/main/java/org/apache/accumulo/core/client/admin/NewTableConfiguration.java ## @@ -146,4 +163,116 @@ public NewTableConfiguration enableSummarization(SummarizerConfiguration... conf summarizerProps = tmp; return this; } + + /** + * Configures a table's locality groups prior to initial table creation. + * + * Allows locality groups to be set prior to table creation. Additional calls to this method prior to table creation will overwrite previous locality group + * mappings. + * + * @param groups + * mapping of locality group names to column families in the locality group + * + * @since 2.0.0 + */ + public void setLocalityGroups(Mapgroups) { +// ensure locality groups do not overlap +LocalityGroupUtil.ensureNonOverlappingGroups(groups); +localityProps = new HashMap<>(); +for (Entry entry : groups.entrySet()) { + Set colFams = entry.getValue(); + String value = LocalityGroupUtil.encodeColumnFamilies(colFams); + localityProps.put(Property.TABLE_LOCALITY_GROUP_PREFIX + entry.getKey(), value); +} +// localityProps.put(Property.TABLE_LOCALITY_GROUPS.getKey(), Joiner.on(",").join(groups.keySet())); +localityProps.put(Property.TABLE_LOCALITY_GROUPS.getKey(), groups.keySet().stream().collect(Collectors.joining(","))); +// localityProps.put(Property.TABLE_LOCALITY_GROUPS.getKey(), Stream.of(groups.keySet().collect(Collectors.joining(",")); +// Stream.of(groups.keySet()).collect(joining(",")); + } + + /** + * Configure iterator settings for a table prior to its creation. + * + * Additional calls to this method before table creation will overwrite previous iterator settings. + * + * @param setting + * object specifying the properties of the iterator + * @throws AccumuloSecurityException + * thrown if the user does not have the ability to set properties on the table + * @throws AccumuloException + * if a general error occurs + * @throws TableNotFoundException + * if the table does not exist + * + * @since 2.0.0 + */ + public void attachIterator(IteratorSetting setting) throws AccumuloException, TableNotFoundException { +attachIterator(setting, EnumSet.allOf(IteratorScope.class)); + } + + /** + * Configure iterator settings for a table prior to its creation. + * + * @param setting + * object specifying the properties of the iterator + * @param scopes + * enumerated set of iterator scopes + * @throws AccumuloException + * if a general error occurs + * @throws AccumuloSecurityException + * thrown if the user does not have the ability to set properties on the table + * @throws TableNotFoundException + * if the table does not exist + * + * @since 2.0.0 + */ + public void attachIterator(IteratorSetting setting, EnumSet scopes) throws AccumuloException, TableNotFoundException { +checkArgument(setting != null, "setting is null"); +checkArgument(scopes != null, "scopes is null"); +if (iteratorProps.isEmpty()) { + iteratorProps = new HashMap<>(); +} +checkIteratorConflicts(setting, scopes); +for (IteratorScope scope : scopes) { + String root = String.format("%s%s.%s", Property.TABLE_ITERATOR_PREFIX, scope.name().toLowerCase(), setting.getName()); + for (Entry prop : setting.getOptions().entrySet()) { +iteratorProps.put(root + ".opt." + prop.getKey(), prop.getValue()); + } + iteratorProps.put(root, setting.getPriority() + "," + setting.getIteratorClass()); +} + } + + private void checkIteratorConflicts(IteratorSetting setting, EnumSet scopes) throws AccumuloException, TableNotFoundException { +checkArgument(setting != null, "setting is null"); Review comment: Can you make the new method in TableOpHelper static? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] jmark99 commented on a change in pull request #337: ACCUMULO-4732 No APIs to configure iterators or locality groups for new tables
jmark99 commented on a change in pull request #337: ACCUMULO-4732 No APIs to configure iterators or locality groups for new tables URL: https://github.com/apache/accumulo/pull/337#discussion_r156800156 ## File path: core/src/main/java/org/apache/accumulo/core/client/admin/NewTableConfiguration.java ## @@ -146,4 +163,116 @@ public NewTableConfiguration enableSummarization(SummarizerConfiguration... conf summarizerProps = tmp; return this; } + + /** + * Configures a table's locality groups prior to initial table creation. + * + * Allows locality groups to be set prior to table creation. Additional calls to this method prior to table creation will overwrite previous locality group + * mappings. + * + * @param groups + * mapping of locality group names to column families in the locality group + * + * @since 2.0.0 + */ + public void setLocalityGroups(Mapgroups) { +// ensure locality groups do not overlap +LocalityGroupUtil.ensureNonOverlappingGroups(groups); +localityProps = new HashMap<>(); +for (Entry entry : groups.entrySet()) { + Set colFams = entry.getValue(); + String value = LocalityGroupUtil.encodeColumnFamilies(colFams); + localityProps.put(Property.TABLE_LOCALITY_GROUP_PREFIX + entry.getKey(), value); +} +// localityProps.put(Property.TABLE_LOCALITY_GROUPS.getKey(), Joiner.on(",").join(groups.keySet())); +localityProps.put(Property.TABLE_LOCALITY_GROUPS.getKey(), groups.keySet().stream().collect(Collectors.joining(","))); +// localityProps.put(Property.TABLE_LOCALITY_GROUPS.getKey(), Stream.of(groups.keySet().collect(Collectors.joining(",")); +// Stream.of(groups.keySet()).collect(joining(",")); + } + + /** + * Configure iterator settings for a table prior to its creation. + * + * Additional calls to this method before table creation will overwrite previous iterator settings. + * + * @param setting + * object specifying the properties of the iterator + * @throws AccumuloSecurityException + * thrown if the user does not have the ability to set properties on the table + * @throws AccumuloException + * if a general error occurs + * @throws TableNotFoundException + * if the table does not exist + * + * @since 2.0.0 + */ + public void attachIterator(IteratorSetting setting) throws AccumuloException, TableNotFoundException { +attachIterator(setting, EnumSet.allOf(IteratorScope.class)); + } + + /** + * Configure iterator settings for a table prior to its creation. + * + * @param setting + * object specifying the properties of the iterator + * @param scopes + * enumerated set of iterator scopes + * @throws AccumuloException + * if a general error occurs + * @throws AccumuloSecurityException + * thrown if the user does not have the ability to set properties on the table + * @throws TableNotFoundException + * if the table does not exist + * + * @since 2.0.0 + */ + public void attachIterator(IteratorSetting setting, EnumSet scopes) throws AccumuloException, TableNotFoundException { +checkArgument(setting != null, "setting is null"); +checkArgument(scopes != null, "scopes is null"); +if (iteratorProps.isEmpty()) { + iteratorProps = new HashMap<>(); +} +checkIteratorConflicts(setting, scopes); +for (IteratorScope scope : scopes) { + String root = String.format("%s%s.%s", Property.TABLE_ITERATOR_PREFIX, scope.name().toLowerCase(), setting.getName()); + for (Entry prop : setting.getOptions().entrySet()) { +iteratorProps.put(root + ".opt." + prop.getKey(), prop.getValue()); + } + iteratorProps.put(root, setting.getPriority() + "," + setting.getIteratorClass()); +} + } + + private void checkIteratorConflicts(IteratorSetting setting, EnumSet scopes) throws AccumuloException, TableNotFoundException { +checkArgument(setting != null, "setting is null"); Review comment: I'll give it a shot and see how that works. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] jmark99 commented on a change in pull request #337: ACCUMULO-4732 No APIs to configure iterators or locality groups for new tables
jmark99 commented on a change in pull request #337: ACCUMULO-4732 No APIs to configure iterators or locality groups for new tables URL: https://github.com/apache/accumulo/pull/337#discussion_r156795806 ## File path: core/src/main/java/org/apache/accumulo/core/client/admin/NewTableConfiguration.java ## @@ -146,4 +163,116 @@ public NewTableConfiguration enableSummarization(SummarizerConfiguration... conf summarizerProps = tmp; return this; } + + /** + * Configures a table's locality groups prior to initial table creation. + * + * Allows locality groups to be set prior to table creation. Additional calls to this method prior to table creation will overwrite previous locality group + * mappings. + * + * @param groups + * mapping of locality group names to column families in the locality group + * + * @since 2.0.0 + */ + public void setLocalityGroups(Mapgroups) { Review comment: Updated the methods to return 'this' and created new test to verify behavior. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] jmark99 commented on a change in pull request #337: ACCUMULO-4732 No APIs to configure iterators or locality groups for new tables
jmark99 commented on a change in pull request #337: ACCUMULO-4732 No APIs to configure iterators or locality groups for new tables URL: https://github.com/apache/accumulo/pull/337#discussion_r156796817 ## File path: core/src/main/java/org/apache/accumulo/core/client/admin/NewTableConfiguration.java ## @@ -146,4 +163,116 @@ public NewTableConfiguration enableSummarization(SummarizerConfiguration... conf summarizerProps = tmp; return this; } + + /** + * Configures a table's locality groups prior to initial table creation. + * + * Allows locality groups to be set prior to table creation. Additional calls to this method prior to table creation will overwrite previous locality group + * mappings. + * + * @param groups + * mapping of locality group names to column families in the locality group + * + * @since 2.0.0 + */ + public void setLocalityGroups(Mapgroups) { +// ensure locality groups do not overlap +LocalityGroupUtil.ensureNonOverlappingGroups(groups); +localityProps = new HashMap<>(); +for (Entry entry : groups.entrySet()) { + Set colFams = entry.getValue(); + String value = LocalityGroupUtil.encodeColumnFamilies(colFams); + localityProps.put(Property.TABLE_LOCALITY_GROUP_PREFIX + entry.getKey(), value); +} +// localityProps.put(Property.TABLE_LOCALITY_GROUPS.getKey(), Joiner.on(",").join(groups.keySet())); +localityProps.put(Property.TABLE_LOCALITY_GROUPS.getKey(), groups.keySet().stream().collect(Collectors.joining(","))); +// localityProps.put(Property.TABLE_LOCALITY_GROUPS.getKey(), Stream.of(groups.keySet().collect(Collectors.joining(",")); +// Stream.of(groups.keySet()).collect(joining(",")); + } + + /** + * Configure iterator settings for a table prior to its creation. + * + * Additional calls to this method before table creation will overwrite previous iterator settings. + * + * @param setting + * object specifying the properties of the iterator + * @throws AccumuloSecurityException + * thrown if the user does not have the ability to set properties on the table + * @throws AccumuloException + * if a general error occurs + * @throws TableNotFoundException + * if the table does not exist + * + * @since 2.0.0 + */ + public void attachIterator(IteratorSetting setting) throws AccumuloException, TableNotFoundException { +attachIterator(setting, EnumSet.allOf(IteratorScope.class)); + } + + /** + * Configure iterator settings for a table prior to its creation. + * + * @param setting + * object specifying the properties of the iterator + * @param scopes + * enumerated set of iterator scopes + * @throws AccumuloException + * if a general error occurs + * @throws AccumuloSecurityException + * thrown if the user does not have the ability to set properties on the table + * @throws TableNotFoundException + * if the table does not exist + * + * @since 2.0.0 + */ + public void attachIterator(IteratorSetting setting, EnumSet scopes) throws AccumuloException, TableNotFoundException { +checkArgument(setting != null, "setting is null"); +checkArgument(scopes != null, "scopes is null"); +if (iteratorProps.isEmpty()) { + iteratorProps = new HashMap<>(); +} +checkIteratorConflicts(setting, scopes); +for (IteratorScope scope : scopes) { + String root = String.format("%s%s.%s", Property.TABLE_ITERATOR_PREFIX, scope.name().toLowerCase(), setting.getName()); + for (Entry prop : setting.getOptions().entrySet()) { +iteratorProps.put(root + ".opt." + prop.getKey(), prop.getValue()); + } + iteratorProps.put(root, setting.getPriority() + "," + setting.getIteratorClass()); +} + } + + private void checkIteratorConflicts(IteratorSetting setting, EnumSet scopes) throws AccumuloException, TableNotFoundException { +checkArgument(setting != null, "setting is null"); Review comment: There is at least one more very similar instance of checkIteratorConflicts for namespaces as well. Would it make sense to create a new ticket to consolidate the various permutations of this method and perhaps place the new versions into the iteratorUtil class where it could more easily be called by the various classes that need that check? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services