dramaticlly commented on a change in pull request #4262: URL: https://github.com/apache/iceberg/pull/4262#discussion_r820992590
########## File path: python/src/iceberg/expression/literals.py ########## @@ -0,0 +1,741 @@ +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +import datetime +import uuid +from decimal import ROUND_HALF_UP, Decimal +from functools import singledispatch + +try: + from functools import singledispatchmethod +except ImportError: + from singledispatch import singledispatchmethod # type: ignore + +import pytz + +from iceberg.types import ( + BinaryType, + BooleanType, + DateType, + DecimalType, + DoubleType, + FixedType, + FloatType, + IntegerType, + LongType, + Singleton, + StringType, + TimestampType, + TimestamptzType, + TimeType, + UUIDType, +) + +JAVA_MAX_INT = 2147483647 Review comment: I was looking at https://iceberg.apache.org/spec/ and did NOT find any explicit mentioning of it, I guess this comes from `Integer.MAX_VALUE` so maybe we can keep them as is? ########## File path: python/.coveragerc ########## @@ -20,5 +20,5 @@ skip_empty = true [run] omit = - # omit this single file - src/iceberg/literals.py + src/iceberg/expression/literals.py Review comment: @jun-he , I understand that from logical perspective, it make sense to have consistent standard where all merged codes are conforming to the same requirement on code coverage. But in this specific Literal cases, I found many places of coverage gap comes from base base such as - BaseLiterals <img width="767" alt="image" src="https://user-images.githubusercontent.com/5961173/157098041-e3ded4f6-466f-4b04-b159-91b982b36819.png";> - ComparableLiterals <img width="644" alt="image" src="https://user-images.githubusercontent.com/5961173/157098129-5d7c1d13-3349-477a-8ac6-0f87119c10f3.png";> Which if we remove the code for those, then it does not make sense for rest of the code to check in. So there's no very clean way of separating them out. What we can do here is, either ask for more code coverage for internal method of base class, to merge it within same PR, or allow for some temporary exclusion and assign the specific issue and come back in a separate PR. ########## File path: python/tests/expression/conftest.py ########## @@ -0,0 +1,32 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +import pytest + +from iceberg.types import DecimalType + + Review comment: yes Jun, that's my intention. This conftest is used for expression unit tests ########## File path: python/src/iceberg/expression/literals.py ########## @@ -0,0 +1,741 @@ +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +import datetime +import uuid +from decimal import ROUND_HALF_UP, Decimal +from functools import singledispatch + +try: + from functools import singledispatchmethod +except ImportError: + from singledispatch import singledispatchmethod # type: ignore + +import pytz + +from iceberg.types import ( + BinaryType, + BooleanType, + DateType, + DecimalType, + DoubleType, + FixedType, + FloatType, + IntegerType, + LongType, + Singleton, + StringType, + TimestampType, + TimestamptzType, + TimeType, + UUIDType, +) + +JAVA_MAX_INT = 2147483647 +JAVA_MIN_INT = -2147483648 +JAVA_MAX_FLOAT = 3.4028235e38 +JAVA_MIN_FLOAT = -3.4028235e38 +EPOCH = datetime.datetime.utcfromtimestamp(0) +EPOCH_DAY = EPOCH.date() + +""" +Iceberg literal is wrapper class used in expressions, which return unbound predicates +It's being organized as below +Literal +|-- AboveMax +|-- BelowMin +|-- BaseLiteral + |-- StringLiteral + |-- FixedLiteral + |-- BinaryLiteral + |-- ComparableLiteral + |-- BooleanLiteral + |-- IntegerLiteral + |-- LongLiteral + |-- FloatLiteral + |-- DoubleLiteral + |-- DateLiteral + |-- TimeLiteral + |-- TimestampLiteral + |-- DecimalLiteral + |-- UUIDLiteral +""" + + +class Literal: + def to(self, type_var): + raise NotImplementedError() + + def to_byte_buffer(self): + raise NotImplementedError() + + +@singledispatch +def of(value): + """ + A generic Literal factory to construct an iceberg Literal based on python primitive data type + using dynamic overloading + + Args: + value(python primitive type): the value to be associated with literal + + Example: + import iceberg.expressions.literals + >>> iceberg.expressions.literals.of(1) + IntegerLiteral(1) + """ + raise TypeError(f"Unimplemented Type Literal for value: {value}") + + [email protected](bool) +def _of(value): + return BooleanLiteral(value) + + [email protected](int) # type: ignore[no-redef] +def _of(value): + """ + Upgrade to long if python int is outside the JAVA_MIN_INT and JAVA_MAX_INT + """ + if value < JAVA_MIN_INT or value > JAVA_MAX_INT: + return LongLiteral(value) + return IntegerLiteral(value) + + [email protected](float) # type: ignore[no-redef] +def _of(value): + """ + Upgrade to double if python float is outside the JAVA_MIN_FLOAT and JAVA_MAX_FLOAT + """ + if value < JAVA_MIN_FLOAT or value > JAVA_MAX_FLOAT: + return DoubleLiteral(value) + return FloatLiteral(value) + + [email protected](str) # type: ignore[no-redef] +def _of(value): + return StringLiteral(value) + + [email protected](uuid.UUID) # type: ignore[no-redef] +def _of(value): + return UUIDLiteral(value) + + [email protected](bytes) # type: ignore[no-redef] +def _of(value): + return FixedLiteral(value) + + [email protected](bytearray) # type: ignore[no-redef] +def _of(value): + return BinaryLiteral(value) + + [email protected](Decimal) # type: ignore[no-redef] +def _of(value): + return DecimalLiteral(value) + + +class BaseLiteral(Literal): + """Base literal which has a value and can be converted between types""" + + def __init__(self, repr_string: str, value): Review comment: for `value` here it would be typed of `Any` as well, let me know if you want me to add it ########## File path: python/src/iceberg/expression/literals.py ########## @@ -0,0 +1,742 @@ +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +import datetime +import sys +import uuid +from decimal import ROUND_HALF_UP, Decimal +from functools import singledispatch + +if sys.version_info >= (3, 8): + from functools import singledispatchmethod +else: + from singledispatch import singledispatchmethod + +import pytz + +from iceberg.types import ( + BinaryType, + BooleanType, + DateType, + DecimalType, + DoubleType, + FixedType, + FloatType, + IntegerType, + LongType, + Singleton, + StringType, + TimestampType, + TimestamptzType, + TimeType, + UUIDType, +) + +JAVA_MAX_INT = 2147483647 +JAVA_MIN_INT = -2147483648 +JAVA_MAX_FLOAT = 3.4028235e38 +JAVA_MIN_FLOAT = -3.4028235e38 +EPOCH = datetime.datetime.utcfromtimestamp(0) +EPOCH_DAY = EPOCH.date() + +""" +Iceberg literal is wrapper class used in expressions, which return unbound predicates +It's being organized as below +Literal +|-- AboveMax +|-- BelowMin +|-- BaseLiteral + |-- StringLiteral + |-- FixedLiteral + |-- BinaryLiteral + |-- ComparableLiteral + |-- BooleanLiteral + |-- IntegerLiteral + |-- LongLiteral + |-- FloatLiteral + |-- DoubleLiteral + |-- DateLiteral + |-- TimeLiteral + |-- TimestampLiteral + |-- DecimalLiteral + |-- UUIDLiteral +""" + + +class Literal: + def to(self, type_var): + raise NotImplementedError() + + def to_byte_buffer(self): + raise NotImplementedError() + + +@singledispatch +def of(value): + """ + A generic Literal factory to construct an iceberg Literal based on python primitive data type + using dynamic overloading + + Args: + value(python primitive type): the value to be associated with literal + + Example: + import iceberg.expressions.literals + >>> iceberg.expressions.literals.of(1) + IntegerLiteral(1) + """ + raise TypeError(f"Unimplemented Type Literal for value: {value}") + + [email protected](bool) +def _of(value): Review comment: there's type annotation on the decorator above, do you think if it's sufficient? or I can add for `value` if you prefer ########## File path: python/src/iceberg/expression/literals.py ########## @@ -0,0 +1,742 @@ +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +import datetime +import sys +import uuid +from decimal import ROUND_HALF_UP, Decimal +from functools import singledispatch + +if sys.version_info >= (3, 8): + from functools import singledispatchmethod +else: + from singledispatch import singledispatchmethod + +import pytz + +from iceberg.types import ( + BinaryType, + BooleanType, + DateType, + DecimalType, + DoubleType, + FixedType, + FloatType, + IntegerType, + LongType, + Singleton, + StringType, + TimestampType, + TimestamptzType, + TimeType, + UUIDType, +) + +JAVA_MAX_INT = 2147483647 +JAVA_MIN_INT = -2147483648 +JAVA_MAX_FLOAT = 3.4028235e38 +JAVA_MIN_FLOAT = -3.4028235e38 +EPOCH = datetime.datetime.utcfromtimestamp(0) +EPOCH_DAY = EPOCH.date() + +""" +Iceberg literal is wrapper class used in expressions, which return unbound predicates +It's being organized as below +Literal +|-- AboveMax +|-- BelowMin +|-- BaseLiteral + |-- StringLiteral + |-- FixedLiteral + |-- BinaryLiteral + |-- ComparableLiteral + |-- BooleanLiteral + |-- IntegerLiteral + |-- LongLiteral + |-- FloatLiteral + |-- DoubleLiteral + |-- DateLiteral + |-- TimeLiteral + |-- TimestampLiteral + |-- DecimalLiteral + |-- UUIDLiteral +""" + + +class Literal: + def to(self, type_var): Review comment: I guess if we need , the `type_var` would be of `Any` type here as it's the base method and type_var can be unrestricted and concrete Literals below will have better defined method for allowed type conversion. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
![https://user-images.githubusercontent.com/5961173/157098041-e3ded4f6-466f-4b04-b159-91b982b36819.png"](https://user-images.githubusercontent.com/5961173/157098041-e3ded4f6-466f-4b04-b159-91b982b36819.png"){kind=link}
![https://user-images.githubusercontent.com/5961173/157098129-5d7c1d13-3349-477a-8ac6-0f87119c10f3.png"](https://user-images.githubusercontent.com/5961173/157098129-5d7c1d13-3349-477a-8ac6-0f87119c10f3.png"){kind=link}