[GitHub] [incubator-hudi] vinothchandar commented on a change in pull request #1159: [HUDI-479] Eliminate or Minimize use of Guava if possible
vinothchandar commented on a change in pull request #1159: [HUDI-479] Eliminate
or Minimize use of Guava if possible
URL: https://github.com/apache/incubator-hudi/pull/1159#discussion_r399312059
##
File path:
hudi-common/src/main/java/org/apache/hudi/common/util/FileIOUtils.java
##
@@ -91,4 +94,29 @@ public static void writeStringToFile(String str, String
filePath) throws IOExcep
out.flush();
out.close();
}
+
+ /**
+ * Closes a {@link Closeable}, with control over whether an {@code
IOException} may be thrown.
+ * @param closeable the {@code Closeable} object to be closed, or null,
+ * in which case this method does nothing.
+ * @param swallowIOException if true, don't propagate IO exceptions thrown
by the {@code close} methods.
+ *
+ * @throws IOException if {@code swallowIOException} is false and {@code
close} throws an {@code IOException}.
+ */
+ public static void close(@Nullable Closeable closeable, boolean
swallowIOException)
+ throws IOException {
+if (closeable == null) {
+ return;
+}
+try {
+ closeable.close();
+} catch (IOException e) {
+ if (!swallowIOException) {
+throw e;
+ }
+}
+ }
+
+ /** Maximum loop count when creating temp directories. */
+ private static final int TEMP_DIR_ATTEMPTS = 1;
Review comment:
we don't need this anymore correct?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-hudi] vinothchandar commented on a change in pull request #1159: [HUDI-479] Eliminate or Minimize use of Guava if possible
vinothchandar commented on a change in pull request #1159: [HUDI-479] Eliminate
or Minimize use of Guava if possible
URL: https://github.com/apache/incubator-hudi/pull/1159#discussion_r398713100
##
File path:
hudi-common/src/main/java/org/apache/hudi/common/util/CollectionUtils.java
##
@@ -0,0 +1,132 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.util;
+
+import java.util.Collections;
+import java.util.Iterator;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+public class CollectionUtils {
+ /**
+ * Determines whether two iterators contain equal elements in the same
order. More specifically,
+ * this method returns {@code true} if {@code iterator1} and {@code
iterator2} contain the same
+ * number of elements and every element of {@code iterator1} is equal to the
corresponding element
+ * of {@code iterator2}.
+ *
+ * Note that this will modify the supplied iterators, since they will
have been advanced some
+ * number of elements forward.
+ */
+ public static boolean elementsEqual(Iterator iterator1, Iterator
iterator2) {
+while (iterator1.hasNext()) {
+ if (!iterator2.hasNext()) {
+return false;
+ }
+ Object o1 = iterator1.next();
+ Object o2 = iterator2.next();
+ if (!Objects.equals(o1, o2)) {
+return false;
+ }
+}
+return !iterator2.hasNext();
+ }
+
+ @SafeVarargs
+ public static Set createSetFromElements(final T... elements) {
+return Stream.of(elements).collect(Collectors.toSet());
+ }
+
+ public static Map createImmutableMap(final K key, final V value)
{
+return Collections.unmodifiableMap(Collections.singletonMap(key, value));
+ }
+
+ @SafeVarargs
+ public static List createImmutableList(final T... elements) {
+return
Collections.unmodifiableList(Stream.of(elements).collect(Collectors.toList()));
+ }
+
+ public static Map createImmutableMap(final Map map) {
+return Collections.unmodifiableMap(map);
+ }
+
+ @SafeVarargs
+ public static Set createImmutableSet(final T... elements) {
+return
Collections.unmodifiableSet(Stream.of(elements).collect(Collectors.toSet()));
+ }
+
+ public static Set createImmutableSet(final Set set) {
+return Collections.unmodifiableSet(set);
+ }
+
+ public static List createImmutableList(final List list) {
+return Collections.unmodifiableList(list);
+ }
+
+ private static Object[] checkElementsNotNull(Object... array) {
+return checkElementsNotNull(array, array.length);
+ }
+
+ private static Object[] checkElementsNotNull(Object[] array, int length) {
+for (int i = 0; i < length; i++) {
+ checkElementNotNull(array[i], i);
+}
+return array;
+ }
+
+ private static Object checkElementNotNull(Object element, int index) {
+if (element == null) {
+ throw new NullPointerException("at index " + index);
+}
+return element;
+ }
+
+ public static class Maps {
Review comment:
https://issues.apache.org/jira/browse/HUDI-737 filed this
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-hudi] vinothchandar commented on a change in pull request #1159: [HUDI-479] Eliminate or Minimize use of Guava if possible
vinothchandar commented on a change in pull request #1159: [HUDI-479] Eliminate
or Minimize use of Guava if possible
URL: https://github.com/apache/incubator-hudi/pull/1159#discussion_r39816
##
File path:
hudi-common/src/main/java/org/apache/hudi/common/util/CollectionUtils.java
##
@@ -0,0 +1,132 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.util;
+
+import java.util.Collections;
+import java.util.Iterator;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+public class CollectionUtils {
+ /**
+ * Determines whether two iterators contain equal elements in the same
order. More specifically,
+ * this method returns {@code true} if {@code iterator1} and {@code
iterator2} contain the same
+ * number of elements and every element of {@code iterator1} is equal to the
corresponding element
+ * of {@code iterator2}.
+ *
+ * Note that this will modify the supplied iterators, since they will
have been advanced some
+ * number of elements forward.
+ */
+ public static boolean elementsEqual(Iterator iterator1, Iterator
iterator2) {
+while (iterator1.hasNext()) {
+ if (!iterator2.hasNext()) {
+return false;
+ }
+ Object o1 = iterator1.next();
+ Object o2 = iterator2.next();
+ if (!Objects.equals(o1, o2)) {
+return false;
+ }
+}
+return !iterator2.hasNext();
+ }
+
+ @SafeVarargs
+ public static Set createSetFromElements(final T... elements) {
+return Stream.of(elements).collect(Collectors.toSet());
+ }
+
+ public static Map createImmutableMap(final K key, final V value)
{
+return Collections.unmodifiableMap(Collections.singletonMap(key, value));
+ }
+
+ @SafeVarargs
+ public static List createImmutableList(final T... elements) {
+return
Collections.unmodifiableList(Stream.of(elements).collect(Collectors.toList()));
+ }
+
+ public static Map createImmutableMap(final Map map) {
+return Collections.unmodifiableMap(map);
+ }
+
+ @SafeVarargs
+ public static Set createImmutableSet(final T... elements) {
+return
Collections.unmodifiableSet(Stream.of(elements).collect(Collectors.toSet()));
+ }
+
+ public static Set createImmutableSet(final Set set) {
+return Collections.unmodifiableSet(set);
+ }
+
+ public static List createImmutableList(final List list) {
+return Collections.unmodifiableList(list);
+ }
+
+ private static Object[] checkElementsNotNull(Object... array) {
+return checkElementsNotNull(array, array.length);
+ }
+
+ private static Object[] checkElementsNotNull(Object[] array, int length) {
+for (int i = 0; i < length; i++) {
+ checkElementNotNull(array[i], i);
+}
+return array;
+ }
+
+ private static Object checkElementNotNull(Object element, int index) {
+if (element == null) {
+ throw new NullPointerException("at index " + index);
+}
+return element;
+ }
+
+ public static class Maps {
Review comment:
we can file a follow up JIRA and deal with it later it as well
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-hudi] vinothchandar commented on a change in pull request #1159: [HUDI-479] Eliminate or Minimize use of Guava if possible
vinothchandar commented on a change in pull request #1159: [HUDI-479] Eliminate
or Minimize use of Guava if possible
URL: https://github.com/apache/incubator-hudi/pull/1159#discussion_r398323808
##
File path:
hudi-common/src/main/java/org/apache/hudi/common/util/FileIOUtils.java
##
@@ -91,4 +94,55 @@ public static void writeStringToFile(String str, String
filePath) throws IOExcep
out.flush();
out.close();
}
+
+ /**
+ * Closes a {@link Closeable}, with control over whether an {@code
IOException} may be thrown.
+ * @param closeable the {@code Closeable} object to be closed, or null,
+ * in which case this method does nothing.
+ * @param swallowIOException if true, don't propagate IO exceptions thrown
by the {@code close} methods.
+ *
+ * @throws IOException if {@code swallowIOException} is false and {@code
close} throws an {@code IOException}.
+ */
+ public static void close(@Nullable Closeable closeable, boolean
swallowIOException)
+ throws IOException {
+if (closeable == null) {
+ return;
+}
+try {
+ closeable.close();
+} catch (IOException e) {
+ if (!swallowIOException) {
+throw e;
+ }
+}
+ }
+
+ /** Maximum loop count when creating temp directories. */
+ private static final int TEMP_DIR_ATTEMPTS = 1;
+
+ /**
+ * Create a Temporary Directory.
+ * @return {@code File}
+ */
+ public static File createTempDir() {
Review comment:
this is mostly used by tests, right? I think we should just replace those
with Junit temporary folder or use `Files.createTempDirectory()` from jdk here
instead of creating it by hand.. This code and the retry loops can be
simplified to 1 line, java will set up hooks to delete the directories on exit
etc, if we do that..
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-hudi] vinothchandar commented on a change in pull request #1159: [HUDI-479] Eliminate or Minimize use of Guava if possible
vinothchandar commented on a change in pull request #1159: [HUDI-479] Eliminate
or Minimize use of Guava if possible
URL: https://github.com/apache/incubator-hudi/pull/1159#discussion_r398318223
##
File path:
hudi-cli/src/test/java/org/apache/hudi/cli/common/HoodieTestCommitMetadataGenerator.java
##
@@ -95,10 +96,9 @@ public static HoodieCommitMetadata
generateCommitMetadata(String basePath) throw
HoodieTestUtils.createNewDataFile(basePath,
DEFAULT_FIRST_PARTITION_PATH, "000");
String file1P1C0 =
HoodieTestUtils.createNewDataFile(basePath,
DEFAULT_SECOND_PARTITION_PATH, "000");
-return generateCommitMetadata(new ImmutableMap.Builder()
- .put(DEFAULT_FIRST_PARTITION_PATH, new
ImmutableList.Builder<>().add(file1P0C0).build())
- .put(DEFAULT_SECOND_PARTITION_PATH, new
ImmutableList.Builder<>().add(file1P1C0).build())
- .build());
+return generateCommitMetadata(new Maps.MapBuilder>()
Review comment:
this can just be replaced with a in-place Map construction syntax
```
Map doubleBraceMap = new HashMap() {{
put("key1", "value1");
put("key2", "value2");
}};
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-hudi] vinothchandar commented on a change in pull request #1159: [HUDI-479] Eliminate or Minimize use of Guava if possible
vinothchandar commented on a change in pull request #1159: [HUDI-479] Eliminate
or Minimize use of Guava if possible
URL: https://github.com/apache/incubator-hudi/pull/1159#discussion_r398320467
##
File path:
hudi-common/src/main/java/org/apache/hudi/common/util/CleanerUtils.java
##
@@ -28,19 +28,19 @@
import org.apache.hudi.common.versioning.clean.CleanV1MigrationHandler;
import org.apache.hudi.common.versioning.clean.CleanV2MigrationHandler;
-import com.google.common.collect.ImmutableMap;
-
import java.io.IOException;
import java.util.List;
+import static org.apache.hudi.common.util.CollectionUtils.Maps;
+
public class CleanerUtils {
public static final Integer CLEAN_METADATA_VERSION_1 =
CleanV1MigrationHandler.VERSION;
public static final Integer CLEAN_METADATA_VERSION_2 =
CleanV2MigrationHandler.VERSION;
public static final Integer LATEST_CLEAN_METADATA_VERSION =
CLEAN_METADATA_VERSION_2;
public static HoodieCleanMetadata convertCleanMetadata(HoodieTableMetaClient
metaClient,
String startCleanTime, Option durationInMs, List
cleanStats) {
-ImmutableMap.Builder
partitionMetadataBuilder = ImmutableMap.builder();
+Maps.MapBuilder
partitionMetadataBuilder = new Maps.MapBuilder<>();
Review comment:
this probably can be just replaced by a concrete map impl in place right?
this sort of usage does not really warrant the builder pattern IMO.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-hudi] vinothchandar commented on a change in pull request #1159: [HUDI-479] Eliminate or Minimize use of Guava if possible
vinothchandar commented on a change in pull request #1159: [HUDI-479] Eliminate
or Minimize use of Guava if possible
URL: https://github.com/apache/incubator-hudi/pull/1159#discussion_r398321039
##
File path:
hudi-common/src/main/java/org/apache/hudi/common/util/CollectionUtils.java
##
@@ -0,0 +1,132 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.util;
+
+import java.util.Collections;
+import java.util.Iterator;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+public class CollectionUtils {
+ /**
+ * Determines whether two iterators contain equal elements in the same
order. More specifically,
+ * this method returns {@code true} if {@code iterator1} and {@code
iterator2} contain the same
+ * number of elements and every element of {@code iterator1} is equal to the
corresponding element
+ * of {@code iterator2}.
+ *
+ * Note that this will modify the supplied iterators, since they will
have been advanced some
+ * number of elements forward.
+ */
+ public static boolean elementsEqual(Iterator iterator1, Iterator
iterator2) {
+while (iterator1.hasNext()) {
+ if (!iterator2.hasNext()) {
+return false;
+ }
+ Object o1 = iterator1.next();
+ Object o2 = iterator2.next();
+ if (!Objects.equals(o1, o2)) {
+return false;
+ }
+}
+return !iterator2.hasNext();
+ }
+
+ @SafeVarargs
+ public static Set createSetFromElements(final T... elements) {
Review comment:
rename to just `createSet`?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-hudi] vinothchandar commented on a change in pull request #1159: [HUDI-479] Eliminate or Minimize use of Guava if possible
vinothchandar commented on a change in pull request #1159: [HUDI-479] Eliminate
or Minimize use of Guava if possible
URL: https://github.com/apache/incubator-hudi/pull/1159#discussion_r398322800
##
File path:
hudi-common/src/main/java/org/apache/hudi/common/util/CollectionUtils.java
##
@@ -0,0 +1,132 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.util;
+
+import java.util.Collections;
+import java.util.Iterator;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+public class CollectionUtils {
+ /**
+ * Determines whether two iterators contain equal elements in the same
order. More specifically,
+ * this method returns {@code true} if {@code iterator1} and {@code
iterator2} contain the same
+ * number of elements and every element of {@code iterator1} is equal to the
corresponding element
+ * of {@code iterator2}.
+ *
+ * Note that this will modify the supplied iterators, since they will
have been advanced some
+ * number of elements forward.
+ */
+ public static boolean elementsEqual(Iterator iterator1, Iterator
iterator2) {
+while (iterator1.hasNext()) {
+ if (!iterator2.hasNext()) {
+return false;
+ }
+ Object o1 = iterator1.next();
+ Object o2 = iterator2.next();
+ if (!Objects.equals(o1, o2)) {
+return false;
+ }
+}
+return !iterator2.hasNext();
+ }
+
+ @SafeVarargs
+ public static Set createSetFromElements(final T... elements) {
+return Stream.of(elements).collect(Collectors.toSet());
+ }
+
+ public static Map createImmutableMap(final K key, final V value)
{
+return Collections.unmodifiableMap(Collections.singletonMap(key, value));
+ }
+
+ @SafeVarargs
+ public static List createImmutableList(final T... elements) {
+return
Collections.unmodifiableList(Stream.of(elements).collect(Collectors.toList()));
+ }
+
+ public static Map createImmutableMap(final Map map) {
+return Collections.unmodifiableMap(map);
+ }
+
+ @SafeVarargs
+ public static Set createImmutableSet(final T... elements) {
+return
Collections.unmodifiableSet(Stream.of(elements).collect(Collectors.toSet()));
+ }
+
+ public static Set createImmutableSet(final Set set) {
+return Collections.unmodifiableSet(set);
+ }
+
+ public static List createImmutableList(final List list) {
+return Collections.unmodifiableList(list);
+ }
+
+ private static Object[] checkElementsNotNull(Object... array) {
+return checkElementsNotNull(array, array.length);
+ }
+
+ private static Object[] checkElementsNotNull(Object[] array, int length) {
+for (int i = 0; i < length; i++) {
+ checkElementNotNull(array[i], i);
+}
+return array;
+ }
+
+ private static Object checkElementNotNull(Object element, int index) {
+if (element == null) {
+ throw new NullPointerException("at index " + index);
+}
+return element;
+ }
+
+ public static class Maps {
Review comment:
I am wondering if we can eliminate the need for this class and the builder
class, but replacing with plain hashmaps inline?
- in places where a static HashMap is needed we can use the map
initialization syntax I pasted above.
Don't feel very strongly about this,, but thinking if we can avoid builder
patterns where it actually is not used that way.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-hudi] vinothchandar commented on a change in pull request #1159: [HUDI-479] Eliminate or Minimize use of Guava if possible
vinothchandar commented on a change in pull request #1159: [HUDI-479] Eliminate
or Minimize use of Guava if possible
URL: https://github.com/apache/incubator-hudi/pull/1159#discussion_r398322962
##
File path:
hudi-common/src/main/java/org/apache/hudi/common/util/FileIOUtils.java
##
@@ -91,4 +94,55 @@ public static void writeStringToFile(String str, String
filePath) throws IOExcep
out.flush();
out.close();
}
+
+ /**
+ * Closes a {@link Closeable}, with control over whether an {@code
IOException} may be thrown.
Review comment:
did we reuse this code from somewhere?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-hudi] vinothchandar commented on a change in pull request #1159: [HUDI-479] Eliminate or Minimize use of Guava if possible
vinothchandar commented on a change in pull request #1159: [HUDI-479] Eliminate
or Minimize use of Guava if possible
URL: https://github.com/apache/incubator-hudi/pull/1159#discussion_r398325200
##
File path:
hudi-common/src/main/java/org/apache/hudi/common/util/ReflectionUtils.java
##
@@ -90,16 +99,58 @@ public static Object loadClass(String clazz, Object...
constructorArgs) {
}
/**
- * Return stream of top level class names in the same class path as
passed-in class.
- *
- * @param clazz
+ * Scans all classes accessible from the context class loader
+ * which belong to the given package and subpackages.
+ *
+ * @param clazz class
+ * @return Stream of Class names in package
*/
- public static Stream getTopLevelClassesInClasspath(Class clazz) {
+ public static Stream getTopLevelClassesInClasspath(Class clazz) {
Review comment:
this method is just used by the dummy bundle main classes right.. Ideally,
like to simplify this down the line. if you agree, we can raise a JIRA?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-hudi] vinothchandar commented on a change in pull request #1159: [HUDI-479] Eliminate or Minimize use of Guava if possible
vinothchandar commented on a change in pull request #1159: [HUDI-479] Eliminate
or Minimize use of Guava if possible
URL: https://github.com/apache/incubator-hudi/pull/1159#discussion_r398321995
##
File path:
hudi-common/src/main/java/org/apache/hudi/common/util/CollectionUtils.java
##
@@ -0,0 +1,132 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.util;
+
+import java.util.Collections;
+import java.util.Iterator;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+public class CollectionUtils {
+ /**
+ * Determines whether two iterators contain equal elements in the same
order. More specifically,
+ * this method returns {@code true} if {@code iterator1} and {@code
iterator2} contain the same
+ * number of elements and every element of {@code iterator1} is equal to the
corresponding element
+ * of {@code iterator2}.
+ *
+ * Note that this will modify the supplied iterators, since they will
have been advanced some
+ * number of elements forward.
+ */
+ public static boolean elementsEqual(Iterator iterator1, Iterator
iterator2) {
+while (iterator1.hasNext()) {
+ if (!iterator2.hasNext()) {
+return false;
+ }
+ Object o1 = iterator1.next();
+ Object o2 = iterator2.next();
+ if (!Objects.equals(o1, o2)) {
+return false;
+ }
+}
+return !iterator2.hasNext();
+ }
+
+ @SafeVarargs
+ public static Set createSetFromElements(final T... elements) {
+return Stream.of(elements).collect(Collectors.toSet());
+ }
+
+ public static Map createImmutableMap(final K key, final V value)
{
+return Collections.unmodifiableMap(Collections.singletonMap(key, value));
+ }
+
+ @SafeVarargs
+ public static List createImmutableList(final T... elements) {
+return
Collections.unmodifiableList(Stream.of(elements).collect(Collectors.toList()));
+ }
+
+ public static Map createImmutableMap(final Map map) {
+return Collections.unmodifiableMap(map);
+ }
+
+ @SafeVarargs
+ public static Set createImmutableSet(final T... elements) {
+return
Collections.unmodifiableSet(Stream.of(elements).collect(Collectors.toSet()));
+ }
+
+ public static Set createImmutableSet(final Set set) {
+return Collections.unmodifiableSet(set);
+ }
+
+ public static List createImmutableList(final List list) {
+return Collections.unmodifiableList(list);
+ }
+
+ private static Object[] checkElementsNotNull(Object... array) {
+return checkElementsNotNull(array, array.length);
+ }
+
+ private static Object[] checkElementsNotNull(Object[] array, int length) {
+for (int i = 0; i < length; i++) {
+ checkElementNotNull(array[i], i);
+}
+return array;
+ }
+
+ private static Object checkElementNotNull(Object element, int index) {
+if (element == null) {
Review comment:
just call Objects.requireNonNull(obj, string) ?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-hudi] vinothchandar commented on a change in pull request #1159: [HUDI-479] Eliminate or Minimize use of Guava if possible
vinothchandar commented on a change in pull request #1159: [HUDI-479] Eliminate
or Minimize use of Guava if possible
URL: https://github.com/apache/incubator-hudi/pull/1159#discussion_r398321209
##
File path:
hudi-common/src/main/java/org/apache/hudi/common/util/CollectionUtils.java
##
@@ -0,0 +1,132 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.util;
+
+import java.util.Collections;
+import java.util.Iterator;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+public class CollectionUtils {
+ /**
+ * Determines whether two iterators contain equal elements in the same
order. More specifically,
+ * this method returns {@code true} if {@code iterator1} and {@code
iterator2} contain the same
+ * number of elements and every element of {@code iterator1} is equal to the
corresponding element
+ * of {@code iterator2}.
+ *
+ * Note that this will modify the supplied iterators, since they will
have been advanced some
+ * number of elements forward.
+ */
+ public static boolean elementsEqual(Iterator iterator1, Iterator
iterator2) {
+while (iterator1.hasNext()) {
+ if (!iterator2.hasNext()) {
+return false;
+ }
+ Object o1 = iterator1.next();
+ Object o2 = iterator2.next();
+ if (!Objects.equals(o1, o2)) {
+return false;
+ }
+}
+return !iterator2.hasNext();
+ }
+
+ @SafeVarargs
+ public static Set createSetFromElements(final T... elements) {
+return Stream.of(elements).collect(Collectors.toSet());
+ }
+
+ public static Map createImmutableMap(final K key, final V value)
{
+return Collections.unmodifiableMap(Collections.singletonMap(key, value));
+ }
+
+ @SafeVarargs
+ public static List createImmutableList(final T... elements) {
+return
Collections.unmodifiableList(Stream.of(elements).collect(Collectors.toList()));
+ }
+
+ public static Map createImmutableMap(final Map map) {
+return Collections.unmodifiableMap(map);
+ }
+
+ @SafeVarargs
+ public static Set createImmutableSet(final T... elements) {
+return
Collections.unmodifiableSet(Stream.of(elements).collect(Collectors.toSet()));
Review comment:
reuse and call createSet() from above?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-hudi] vinothchandar commented on a change in pull request #1159: [HUDI-479] Eliminate or Minimize use of Guava if possible
vinothchandar commented on a change in pull request #1159: [HUDI-479] Eliminate
or Minimize use of Guava if possible
URL: https://github.com/apache/incubator-hudi/pull/1159#discussion_r398323238
##
File path:
hudi-common/src/main/java/org/apache/hudi/common/util/FileIOUtils.java
##
@@ -91,4 +94,55 @@ public static void writeStringToFile(String str, String
filePath) throws IOExcep
out.flush();
out.close();
}
+
+ /**
+ * Closes a {@link Closeable}, with control over whether an {@code
IOException} may be thrown.
+ * @param closeable the {@code Closeable} object to be closed, or null,
+ * in which case this method does nothing.
+ * @param swallowIOException if true, don't propagate IO exceptions thrown
by the {@code close} methods.
+ *
+ * @throws IOException if {@code swallowIOException} is false and {@code
close} throws an {@code IOException}.
+ */
+ public static void close(@Nullable Closeable closeable, boolean
swallowIOException)
+ throws IOException {
+if (closeable == null) {
+ return;
+}
+try {
+ closeable.close();
Review comment:
just the following in the try block?
```
if (closeable != null) {
closeable.close();
}
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-hudi] vinothchandar commented on a change in pull request #1159: [HUDI-479] Eliminate or Minimize use of Guava if possible
vinothchandar commented on a change in pull request #1159: [HUDI-479] Eliminate or Minimize use of Guava if possible URL: https://github.com/apache/incubator-hudi/pull/1159#discussion_r398323935 ## File path: hudi-common/src/main/java/org/apache/hudi/common/util/ReflectionUtils.java ## @@ -21,21 +21,30 @@ import org.apache.hudi.common.model.HoodieRecordPayload; import org.apache.hudi.exception.HoodieException; -import com.google.common.reflect.ClassPath; -import com.google.common.reflect.ClassPath.ClassInfo; +import org.slf4j.Logger; Review comment: I think we use log4j directly? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [incubator-hudi] vinothchandar commented on a change in pull request #1159: [HUDI-479] Eliminate or Minimize use of Guava if possible
vinothchandar commented on a change in pull request #1159: [HUDI-479] Eliminate
or Minimize use of Guava if possible
URL: https://github.com/apache/incubator-hudi/pull/1159#discussion_r398318223
##
File path:
hudi-cli/src/test/java/org/apache/hudi/cli/common/HoodieTestCommitMetadataGenerator.java
##
@@ -95,10 +96,9 @@ public static HoodieCommitMetadata
generateCommitMetadata(String basePath) throw
HoodieTestUtils.createNewDataFile(basePath,
DEFAULT_FIRST_PARTITION_PATH, "000");
String file1P1C0 =
HoodieTestUtils.createNewDataFile(basePath,
DEFAULT_SECOND_PARTITION_PATH, "000");
-return generateCommitMetadata(new ImmutableMap.Builder()
- .put(DEFAULT_FIRST_PARTITION_PATH, new
ImmutableList.Builder<>().add(file1P0C0).build())
- .put(DEFAULT_SECOND_PARTITION_PATH, new
ImmutableList.Builder<>().add(file1P1C0).build())
- .build());
+return generateCommitMetadata(new Maps.MapBuilder>()
Review comment:
this can just be replaced with a in-place Map construction syntax ?
```
Map doubleBraceMap = new HashMap() {{
put("key1", "value1");
put("key2", "value2");
}};
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-hudi] vinothchandar commented on a change in pull request #1159: [HUDI-479] Eliminate or Minimize use of Guava if possible
vinothchandar commented on a change in pull request #1159: [HUDI-479] Eliminate
or Minimize use of Guava if possible
URL: https://github.com/apache/incubator-hudi/pull/1159#discussion_r362295605
##
File path:
hudi-client/src/main/java/org/apache/hudi/table/RollbackExecutor.java
##
@@ -217,7 +216,7 @@ private HoodieRollbackStat
mergeRollbackStat(HoodieRollbackStat stat1, HoodieRol
private Map generateHeader(String commit) {
// generate metadata
-Map header = Maps.newHashMap();
+Map header = new HashMap<>();
Review comment:
good call. Still moving this in some form to a common class would be good.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-hudi] vinothchandar commented on a change in pull request #1159: [HUDI-479] Eliminate or Minimize use of Guava if possible
vinothchandar commented on a change in pull request #1159: [HUDI-479] Eliminate
or Minimize use of Guava if possible
URL: https://github.com/apache/incubator-hudi/pull/1159#discussion_r362271984
##
File path:
hudi-client/src/main/java/org/apache/hudi/index/bloom/BloomIndexFileInfo.java
##
@@ -78,23 +78,21 @@ public boolean equals(Object o) {
}
BloomIndexFileInfo that = (BloomIndexFileInfo) o;
-return Objects.equal(that.fileId, fileId) &&
Objects.equal(that.minRecordKey, minRecordKey)
-&& Objects.equal(that.maxRecordKey, maxRecordKey);
+return Objects.equals(that.fileId, fileId) &&
Objects.equals(that.minRecordKey, minRecordKey)
+&& Objects.equals(that.maxRecordKey, maxRecordKey);
}
@Override
public int hashCode() {
-return Objects.hashCode(fileId, minRecordKey, maxRecordKey);
+return Objects.hash(fileId, minRecordKey, maxRecordKey);
}
@Override
public String toString() {
-final StringBuilder sb = new StringBuilder("BloomIndexFileInfo {");
-sb.append(" fileId=").append(fileId);
-sb.append(" minRecordKey=").append(minRecordKey);
-sb.append(" maxRecordKey=").append(maxRecordKey);
-sb.append('}');
-return sb.toString();
+return "BloomIndexFileInfo {" + " fileId=" + fileId
Review comment:
why this change? StringBuilder reuses allocation right?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-hudi] vinothchandar commented on a change in pull request #1159: [HUDI-479] Eliminate or Minimize use of Guava if possible
vinothchandar commented on a change in pull request #1159: [HUDI-479] Eliminate or Minimize use of Guava if possible URL: https://github.com/apache/incubator-hudi/pull/1159#discussion_r362274816 ## File path: hudi-integ-test/pom.xml ## @@ -117,7 +117,7 @@ com.google.guava guava - 20.0 + 15.0 Review comment: why the downgrade? this should be fine for the integ test , if its passing consistently right This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [incubator-hudi] vinothchandar commented on a change in pull request #1159: [HUDI-479] Eliminate or Minimize use of Guava if possible
vinothchandar commented on a change in pull request #1159: [HUDI-479] Eliminate or Minimize use of Guava if possible URL: https://github.com/apache/incubator-hudi/pull/1159#discussion_r362274938 ## File path: pom.xml ## @@ -67,9 +67,9 @@ -2.6 +3.2.0 Review comment: does the guava change need these version changes? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [incubator-hudi] vinothchandar commented on a change in pull request #1159: [HUDI-479] Eliminate or Minimize use of Guava if possible
vinothchandar commented on a change in pull request #1159: [HUDI-479] Eliminate
or Minimize use of Guava if possible
URL: https://github.com/apache/incubator-hudi/pull/1159#discussion_r362274606
##
File path:
hudi-common/src/test/java/org/apache/hudi/common/table/TestHoodieTableMetaClient.java
##
@@ -120,7 +120,7 @@ public void checkArchiveCommitTimeline() throws
IOException {
HoodieInstant instant2 = new HoodieInstant(false,
HoodieTimeline.COMMIT_ACTION, "2");
HoodieInstant instant3 = new HoodieInstant(false,
HoodieTimeline.COMMIT_ACTION, "3");
-assertEquals(Lists.newArrayList(instant1, instant2, instant3),
+assertEquals(Stream.of(instant1, instant2,
instant3).collect(Collectors.toList()),
Review comment:
Arrays.asList() instead?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-hudi] vinothchandar commented on a change in pull request #1159: [HUDI-479] Eliminate or Minimize use of Guava if possible
vinothchandar commented on a change in pull request #1159: [HUDI-479] Eliminate
or Minimize use of Guava if possible
URL: https://github.com/apache/incubator-hudi/pull/1159#discussion_r362273413
##
File path:
hudi-common/src/main/java/org/apache/hudi/common/model/HoodieKey.java
##
@@ -58,20 +57,18 @@ public boolean equals(Object o) {
return false;
}
HoodieKey otherKey = (HoodieKey) o;
-return Objects.equal(recordKey, otherKey.recordKey) &&
Objects.equal(partitionPath, otherKey.partitionPath);
+return Objects.equals(recordKey, otherKey.recordKey) &&
Objects.equals(partitionPath, otherKey.partitionPath);
}
@Override
public int hashCode() {
-return Objects.hashCode(recordKey, partitionPath);
+return Objects.hash(recordKey, partitionPath);
}
@Override
public String toString() {
-final StringBuilder sb = new StringBuilder("HoodieKey {");
Review comment:
same comment on string builder.. If we print the key in fast path (for
debugging), the extra allocations would matter?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-hudi] vinothchandar commented on a change in pull request #1159: [HUDI-479] Eliminate or Minimize use of Guava if possible
vinothchandar commented on a change in pull request #1159: [HUDI-479] Eliminate
or Minimize use of Guava if possible
URL: https://github.com/apache/incubator-hudi/pull/1159#discussion_r362273016
##
File path:
hudi-client/src/main/java/org/apache/hudi/io/compact/HoodieRealtimeTableCompactor.java
##
@@ -125,7 +125,7 @@
config.getCompactionReverseLogReadEnabled(),
config.getMaxDFSStreamBufferSize(),
config.getSpillableMapBasePath());
if (!scanner.iterator().hasNext()) {
- return Lists.newArrayList();
+ return new ArrayList<>();
Review comment:
how about Collections.emptyList() for new list creations?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-hudi] vinothchandar commented on a change in pull request #1159: [HUDI-479] Eliminate or Minimize use of Guava if possible
vinothchandar commented on a change in pull request #1159: [HUDI-479] Eliminate
or Minimize use of Guava if possible
URL: https://github.com/apache/incubator-hudi/pull/1159#discussion_r362273153
##
File path:
hudi-client/src/main/java/org/apache/hudi/table/RollbackExecutor.java
##
@@ -217,7 +216,7 @@ private HoodieRollbackStat
mergeRollbackStat(HoodieRollbackStat stat1, HoodieRol
private Map generateHeader(String commit) {
// generate metadata
-Map header = Maps.newHashMap();
+Map header = new HashMap<>();
Review comment:
Collections.emptyMap() works? Similarly, if we can pull this into our own
`CollectionUtils` class, that would be useful to change later on?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-hudi] vinothchandar commented on a change in pull request #1159: [HUDI-479] Eliminate or Minimize use of Guava if possible
vinothchandar commented on a change in pull request #1159: [HUDI-479] Eliminate or Minimize use of Guava if possible URL: https://github.com/apache/incubator-hudi/pull/1159#discussion_r362272832 ## File path: hudi-client/src/main/java/org/apache/hudi/io/compact/HoodieRealtimeTableCompactor.java ## @@ -112,8 +112,8 @@ // loaded and load it using CompositeAvroLogReader // Since a DeltaCommit is not defined yet, reading all the records. revisit this soon. String maxInstantTime = metaClient - .getActiveTimeline().getTimelineOfActions(Sets.newHashSet(HoodieTimeline.COMMIT_ACTION, -HoodieTimeline.ROLLBACK_ACTION, HoodieTimeline.DELTA_COMMIT_ACTION)) + .getActiveTimeline().getTimelineOfActions(Stream.of(HoodieTimeline.COMMIT_ACTION, Review comment: Throw the set instantiation into a common `CollectionUtils#setOf()` in hudi-common ? This way we can change the implementation underneath later on if needed? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
