ephraimbuddy commented on a change in pull request #14105: URL: https://github.com/apache/airflow/pull/14105#discussion_r583844482
########## File path: airflow/providers/google/leveldb/hooks/leveldb.py ########## @@ -0,0 +1,208 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +"""Hook for Level DB""" +from typing import Callable, List, Optional + +import plyvel +from plyvel import DB + +from airflow.exceptions import AirflowException +from airflow.hooks.base import BaseHook + + +class LevelDBHookException(AirflowException): + """Exception specific for LevelDB""" + + +class LevelDBHook(BaseHook): + """ + Plyvel Wrapper to Interact With LevelDB Database + `LevelDB Connection Documentation <https://plyvel.readthedocs.io/en/latest/>`__ + """ + + conn_name_attr = 'leveldb_conn_id' + default_conn_name = 'leveldb_default' + conn_type = 'leveldb' + hook_name = 'LevelDB' + + def __init__(self, leveldb_conn_id: str = default_conn_name): + super().__init__() + self.leveldb_conn_id = leveldb_conn_id + self.connection = self.get_connection(leveldb_conn_id) + self.db = None + + # pylint: disable=too-many-arguments + # Fifteen is reasonable in this case. + + def get_conn( + self, + name: str = '/tmp/testdb/', + create_if_missing: bool = False, + error_if_exists: bool = False, + paranoid_checks: bool = None, + write_buffer_size: bool = None, + max_open_files: int = None, + lru_cache_size: int = None, + block_size: int = None, + block_restart_interval: int = None, + max_file_size: bool = None, + compression: str = 'snappy', + bloom_filter_bits: int = 0, + comparator: Callable = None, + comparator_name: bytes = None, Review comment: What do you think about having `**kwargs` here and removing all except `name` and `create_if_missing`? Then in operator code, you pass the two args(`name`, `create_if_missing`) and another called `create_db_extra_options`. That way, if extra arguments are added to plyvel.DB we will not have to change our code. Users will add it in `create_db_extra_options` and we pass it straight to `plyvel.DB`. This is how your operator code would look like: ``` class LevelDBOperator(BaseOperator): """ Execute command in LevelDB .. seealso:: For more information on how to use this operator, take a look at the guide: :ref:`howto/operator:LevelDBOperator` """ # pylint: disable=too-many-instance-attributes # Twenty is reasonable in this case. @apply_defaults def __init__( self, *, command: str, key: bytes, value: bytes = None, keys: List[bytes] = None, values: List[bytes] = None, leveldb_conn_id: str = 'leveldb_default', name: str = '/tmp/testdb/', create_if_missing: bool = False, create_db_extra_options: Optional[Dict[str, Any]]=None **kwargs, ) -> None: super().__init__(**kwargs) self.command = command self.key = self.validate_and_convert_param(key) self.value = self.validate_and_convert_param(value) self.keys = self.validate_and_convert_param_list(keys) self.values = self.validate_and_convert_param_list(values) self.leveldb_conn_id = leveldb_conn_id self.name = name self.create_if_missing = create_if_missing self.create_db_extra_options = create_db_extra_options or {} ``` And this `get_conn` would be like: ``` def get_conn( self, name: str = '/tmp/testdb/', create_if_missing: bool = False, **kwargs ): if self.db is not None: return self.db self.db = plyvel.DB( name=name, create_if_missing=create_if_missing, **kwargs ) ``` In Operator side when you get the hook, you do : ``` leveldb_hook.get_conn( name=self.name, create_if_missing=self.create_if_missing, **create_db_extra_options) ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
