[jira] Commented: (PIG-1595) casting relation to scalar- problem with handling of data from non PigStorage loaders
[ https://issues.apache.org/jira/browse/PIG-1595?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12906932#action_12906932 ] Daniel Dai commented on PIG-1595: - +1 for the test failure fix. casting relation to scalar- problem with handling of data from non PigStorage loaders - Key: PIG-1595 URL: https://issues.apache.org/jira/browse/PIG-1595 Project: Pig Issue Type: Bug Reporter: Thejas M Nair Assignee: Thejas M Nair Fix For: 0.8.0 Attachments: PIG-1595.1.patch, PIG-1595.2.patch If load functions that don't follow the same bytearray format as PigStorage for other supported datatypes, or those that don't implement the LoadCaster interface are used in 'casting relation to scalar' (PIG-1434), it can cause the query to fail or create incorrect results. The root cause of the problem is that there is a real dependency between the ReadScalars udf that returns the scalar value and the LogicalOperator that acts as its input. But the logicalplan does not capture this dependency. So in SchemaResetter visitor used by the optimizer, the order in which schema is reset and evaluated does not take this into consideration. If the schema of the input LogicalOperator does not get evaluated before the ReadScalar udf, the resutltype of ReadScalar udf becomes bytearray. POUserFunc will convert the input to bytearray using ' new DataByteArray(inp.toString().getBytes())'. But this bytearray encoding of other supported types might not be same for the LoadFunction associated with the column, and that can result in problems. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1595) casting relation to scalar- problem with handling of data from non PigStorage loaders
[ https://issues.apache.org/jira/browse/PIG-1595?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12906952#action_12906952 ] Thejas M Nair commented on PIG-1595: PIG-1595.2.patch committed to trunk and 0.8 branch. casting relation to scalar- problem with handling of data from non PigStorage loaders - Key: PIG-1595 URL: https://issues.apache.org/jira/browse/PIG-1595 Project: Pig Issue Type: Bug Reporter: Thejas M Nair Assignee: Thejas M Nair Fix For: 0.8.0 Attachments: PIG-1595.1.patch, PIG-1595.2.patch If load functions that don't follow the same bytearray format as PigStorage for other supported datatypes, or those that don't implement the LoadCaster interface are used in 'casting relation to scalar' (PIG-1434), it can cause the query to fail or create incorrect results. The root cause of the problem is that there is a real dependency between the ReadScalars udf that returns the scalar value and the LogicalOperator that acts as its input. But the logicalplan does not capture this dependency. So in SchemaResetter visitor used by the optimizer, the order in which schema is reset and evaluated does not take this into consideration. If the schema of the input LogicalOperator does not get evaluated before the ReadScalar udf, the resutltype of ReadScalar udf becomes bytearray. POUserFunc will convert the input to bytearray using ' new DataByteArray(inp.toString().getBytes())'. But this bytearray encoding of other supported types might not be same for the LoadFunction associated with the column, and that can result in problems. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1595) casting relation to scalar- problem with handling of data from non PigStorage loaders
[ https://issues.apache.org/jira/browse/PIG-1595?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12906322#action_12906322 ] Daniel Dai commented on PIG-1595: - Patch break TestScalarAliases.testScalarErrMultipleRowsInInput. Comment out TestScalarAliases.testScalarErrMultipleRowsInInput temporarily. casting relation to scalar- problem with handling of data from non PigStorage loaders - Key: PIG-1595 URL: https://issues.apache.org/jira/browse/PIG-1595 Project: Pig Issue Type: Bug Reporter: Thejas M Nair Assignee: Thejas M Nair Fix For: 0.8.0 Attachments: PIG-1595.1.patch If load functions that don't follow the same bytearray format as PigStorage for other supported datatypes, or those that don't implement the LoadCaster interface are used in 'casting relation to scalar' (PIG-1434), it can cause the query to fail or create incorrect results. The root cause of the problem is that there is a real dependency between the ReadScalars udf that returns the scalar value and the LogicalOperator that acts as its input. But the logicalplan does not capture this dependency. So in SchemaResetter visitor used by the optimizer, the order in which schema is reset and evaluated does not take this into consideration. If the schema of the input LogicalOperator does not get evaluated before the ReadScalar udf, the resutltype of ReadScalar udf becomes bytearray. POUserFunc will convert the input to bytearray using ' new DataByteArray(inp.toString().getBytes())'. But this bytearray encoding of other supported types might not be same for the LoadFunction associated with the column, and that can result in problems. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
