dlmarion commented on a change in pull request #2221:
URL: https://github.com/apache/accumulo/pull/2221#discussion_r682864525
##########
File path:
core/src/main/java/org/apache/accumulo/core/file/rfile/bcfile/Compression.java
##########
@@ -250,33 +277,71 @@ public void initializeDefaultCodec() {
*/
@Override
protected CompressionCodec createNewCodec(final int bufferSize) {
- DefaultCodec myCodec = new DefaultCodec();
- Configuration myConf = new Configuration(conf);
- // only use the buffersize if > 0, otherwise we'll use
- // the default defined within the codec
- if (bufferSize > 0)
- myConf.setInt(BUFFER_SIZE_OPT, bufferSize);
- myCodec.setConf(myConf);
- return myCodec;
+ if (USE_QAT) {
+ String extClazz =
+ (conf.get(CONF_QAT_CLASS) == null ?
System.getProperty(CONF_QAT_CLASS) : null);
+ String clazz = (extClazz != null) ? extClazz : DEFAULT_QAT_CLASS;
+ try {
+ LOG.info("Trying to load qat codec class: " + clazz);
+
+ Configuration myConf = new Configuration(conf);
+ // only use the buffersize if > 0, otherwise we'll use
+ // the default defined within the codec
+ if (bufferSize > 0)
+ myConf.setInt(QAT_BUFFER_SIZE_OPT, bufferSize);
+
+ CompressionCodec c =
+ (CompressionCodec)
ReflectionUtils.newInstance(Class.forName(clazz), myConf);
+ if (c instanceof Configurable) {
+ ((Configurable) c).setConf(myConf);
+ }
+ return c;
+ } catch (ClassNotFoundException e) {
+ LOG.error("Unable to create QAT codec", e);
+ }
+ return null;
Review comment:
I was thinking that if someone specified `-Duse.qat`, then they were
intending on using the device and expected it to work. The fact that it was not
working in this case would be an error as maybe they have a configuration
issue. Falling back to the default implementation and logging it is not
honoring the users intent.
##########
File path:
core/src/main/java/org/apache/accumulo/core/file/rfile/bcfile/Compression.java
##########
@@ -250,33 +277,71 @@ public void initializeDefaultCodec() {
*/
@Override
protected CompressionCodec createNewCodec(final int bufferSize) {
- DefaultCodec myCodec = new DefaultCodec();
- Configuration myConf = new Configuration(conf);
- // only use the buffersize if > 0, otherwise we'll use
- // the default defined within the codec
- if (bufferSize > 0)
- myConf.setInt(BUFFER_SIZE_OPT, bufferSize);
- myCodec.setConf(myConf);
- return myCodec;
+ if (USE_QAT) {
+ String extClazz =
+ (conf.get(CONF_QAT_CLASS) == null ?
System.getProperty(CONF_QAT_CLASS) : null);
+ String clazz = (extClazz != null) ? extClazz : DEFAULT_QAT_CLASS;
+ try {
+ LOG.info("Trying to load qat codec class: " + clazz);
+
+ Configuration myConf = new Configuration(conf);
+ // only use the buffersize if > 0, otherwise we'll use
+ // the default defined within the codec
+ if (bufferSize > 0)
+ myConf.setInt(QAT_BUFFER_SIZE_OPT, bufferSize);
+
+ CompressionCodec c =
+ (CompressionCodec)
ReflectionUtils.newInstance(Class.forName(clazz), myConf);
Review comment:
I just copied what the other implementations were doing. For Hadoop to
be able to use the Codec it has to be on the CLASSPATH.
##########
File path:
core/src/main/java/org/apache/accumulo/core/file/rfile/bcfile/Compression.java
##########
@@ -250,33 +277,71 @@ public void initializeDefaultCodec() {
*/
@Override
protected CompressionCodec createNewCodec(final int bufferSize) {
- DefaultCodec myCodec = new DefaultCodec();
- Configuration myConf = new Configuration(conf);
- // only use the buffersize if > 0, otherwise we'll use
- // the default defined within the codec
- if (bufferSize > 0)
- myConf.setInt(BUFFER_SIZE_OPT, bufferSize);
- myCodec.setConf(myConf);
- return myCodec;
+ if (USE_QAT) {
+ String extClazz =
+ (conf.get(CONF_QAT_CLASS) == null ?
System.getProperty(CONF_QAT_CLASS) : null);
+ String clazz = (extClazz != null) ? extClazz : DEFAULT_QAT_CLASS;
Review comment:
I was just doing what the others do here (mostly copy/paste). LZO does
it
[here](https://github.com/apache/accumulo/blob/1.10/core/src/main/java/org/apache/accumulo/core/file/rfile/bcfile/Compression.java#L159)
and Snappy does it
[here](https://github.com/apache/accumulo/blob/1.10/core/src/main/java/org/apache/accumulo/core/file/rfile/bcfile/Compression.java#L387)
##########
File path:
core/src/main/java/org/apache/accumulo/core/file/rfile/bcfile/Compression.java
##########
@@ -250,33 +277,71 @@ public void initializeDefaultCodec() {
*/
@Override
protected CompressionCodec createNewCodec(final int bufferSize) {
- DefaultCodec myCodec = new DefaultCodec();
- Configuration myConf = new Configuration(conf);
- // only use the buffersize if > 0, otherwise we'll use
- // the default defined within the codec
- if (bufferSize > 0)
- myConf.setInt(BUFFER_SIZE_OPT, bufferSize);
- myCodec.setConf(myConf);
- return myCodec;
+ if (USE_QAT) {
+ String extClazz =
+ (conf.get(CONF_QAT_CLASS) == null ?
System.getProperty(CONF_QAT_CLASS) : null);
+ String clazz = (extClazz != null) ? extClazz : DEFAULT_QAT_CLASS;
+ try {
+ LOG.info("Trying to load qat codec class: " + clazz);
+
+ Configuration myConf = new Configuration(conf);
+ // only use the buffersize if > 0, otherwise we'll use
+ // the default defined within the codec
+ if (bufferSize > 0)
+ myConf.setInt(QAT_BUFFER_SIZE_OPT, bufferSize);
+
+ CompressionCodec c =
+ (CompressionCodec)
ReflectionUtils.newInstance(Class.forName(clazz), myConf);
+ if (c instanceof Configurable) {
+ ((Configurable) c).setConf(myConf);
+ }
+ return c;
+ } catch (ClassNotFoundException e) {
+ LOG.error("Unable to create QAT codec", e);
+ }
+ return null;
Review comment:
I'd rather fail fast, rather than fail silently. From a deployment
perspective, if I configure something to run a specific way, then I want it to
do that or fail so I can fix it. It's not fun or easy to diagnose why something
isn't working the way you expect it to. If you specify `-Xmx8g` for the JVM,
you expect it to use up to that much memory. If the JVM can't allocate that
much memory it doesn't fall back to allocating whatever it can. I'd rather
throw an error here and move on.
##########
File path:
core/src/main/java/org/apache/accumulo/core/file/rfile/bcfile/Compression.java
##########
@@ -250,33 +277,71 @@ public void initializeDefaultCodec() {
*/
@Override
protected CompressionCodec createNewCodec(final int bufferSize) {
- DefaultCodec myCodec = new DefaultCodec();
- Configuration myConf = new Configuration(conf);
- // only use the buffersize if > 0, otherwise we'll use
- // the default defined within the codec
- if (bufferSize > 0)
- myConf.setInt(BUFFER_SIZE_OPT, bufferSize);
- myCodec.setConf(myConf);
- return myCodec;
+ if (USE_QAT) {
+ String extClazz =
+ (conf.get(CONF_QAT_CLASS) == null ?
System.getProperty(CONF_QAT_CLASS) : null);
+ String clazz = (extClazz != null) ? extClazz : DEFAULT_QAT_CLASS;
+ try {
+ LOG.info("Trying to load qat codec class: " + clazz);
+
+ Configuration myConf = new Configuration(conf);
+ // only use the buffersize if > 0, otherwise we'll use
+ // the default defined within the codec
+ if (bufferSize > 0)
+ myConf.setInt(QAT_BUFFER_SIZE_OPT, bufferSize);
+
+ CompressionCodec c =
+ (CompressionCodec)
ReflectionUtils.newInstance(Class.forName(clazz), myConf);
+ if (c instanceof Configurable) {
+ ((Configurable) c).setConf(myConf);
+ }
+ return c;
+ } catch (ClassNotFoundException e) {
Review comment:
ReflectionUtils is public. The newInstance method actually catches all
Exceptions and throws a RuntimeException. The CNFE exception here is being
thrown from the Class.forName() call.
```
@InterfaceAudience.Public
@InterfaceStability.Evolving
public class ReflectionUtils
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
