[jira] [Commented] (AVRO-1682) Support for bigdecimal type
[ https://issues.apache.org/jira/browse/AVRO-1682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14590807#comment-14590807 ] Ryan Blue commented on AVRO-1682: - Hi [~jacopo.moronato], thanks for reporting the issue. This actually isn't too surprising because we just added support for decimals in AVRO-1497. The next Avro release, 1.7.8, should include the Avro-side support and then libraries like Pig can start integrating with it. That can't happen quite yet since the release isn't out, but the Avro side is complete and just waiting for 1.7.8. I think the next step is to report this to the Pig community so they can add the integration that is needed in that project. Would you mind opening an issue for the Pig project? Then I'll link this one to it and close this since it's done! Thanks! Support for bigdecimal type --- Key: AVRO-1682 URL: https://issues.apache.org/jira/browse/AVRO-1682 Project: Avro Issue Type: Wish Affects Versions: 1.7.6 Environment: Apache Pig version 0.12.0-cdh5.4.2 Reporter: Jacopo Moronato Priority: Minor I'm using a pig script to: 1) load a semicolon-separate file 2) convert some chararray fields to bigdecimal 3) store into avro file. Point 3 fails raising this exception: at org.apache.pig.newplan.logical.rules.InputOutputFileValidator$InputOutputFileVisitor.visit(InputOutputFileValidator.java:75) at org.apache.pig.newplan.logical.relational.LOStore.accept(LOStore.java:66) at org.apache.pig.newplan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:64) at org.apache.pig.newplan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:66) at org.apache.pig.newplan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:66) at org.apache.pig.newplan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:66) at org.apache.pig.newplan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:66) at org.apache.pig.newplan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:66) at org.apache.pig.newplan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:66) at org.apache.pig.newplan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:66) at org.apache.pig.newplan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:66) at org.apache.pig.newplan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:66) at org.apache.pig.newplan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:66) at org.apache.pig.newplan.DepthFirstWalker.walk(DepthFirstWalker.java:53) at org.apache.pig.newplan.PlanVisitor.visit(PlanVisitor.java:52) at org.apache.pig.newplan.logical.rules.InputOutputFileValidator.validate(InputOutputFileValidator.java:45) at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.compile(HExecutionEngine.java:311) at org.apache.pig.PigServer.compilePp(PigServer.java:1392) at org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1317) at org.apache.pig.PigServer.execute(PigServer.java:1309) at org.apache.pig.PigServer.executeBatch(PigServer.java:387) at org.apache.pig.PigServer.executeBatch(PigServer.java:365) at org.apache.pig.tools.grunt.GruntParser.executeBatch(GruntParser.java:140) at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:202) at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:173) at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:84) at org.apache.pig.Main.run(Main.java:478) at org.apache.pig.PigRunner.run(PigRunner.java:49) at org.apache.oozie.action.hadoop.PigMain.runPigJob(PigMain.java:285) at org.apache.oozie.action.hadoop.PigMain.run(PigMain.java:228) at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:46) at org.apache.oozie.action.hadoop.PigMain.main(PigMain.java:76) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:228) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343) at org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runSubtask(LocalContainerLauncher.java:370) at org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runTask(LocalContainerLauncher.java:295) at org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.access$200(LocalContainerLauncher.java:181) at org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler$1.run(LocalContainerLauncher.java:224) at
[jira] [Updated] (AVRO-1685) Allow specifying sync in DataFileWriter.create
[ https://issues.apache.org/jira/browse/AVRO-1685?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sehrope Sarkuni updated AVRO-1685: -- Attachment: AVRO-1685.patch Allow specifying sync in DataFileWriter.create -- Key: AVRO-1685 URL: https://issues.apache.org/jira/browse/AVRO-1685 Project: Avro Issue Type: Improvement Components: java Reporter: Sehrope Sarkuni Priority: Minor Attachments: AVRO-1685.patch Currently DataFileWriter generates a random 16-byte sync each time a new file is created. This means that even if you write the exact same data in a new file writer, the file itself will be slightly different (specifically the sync will be different). I'd like to be able to generate the exact same file multiple times. To do so, I need a way to specify the 16-byte sync. I've created a patch that adds this functionality by adding an overload of the create() that takes a byte[] array as the third parameter. If the byte array is null then a random sync is generated using the same internal static generateSync() method as before. If it's not null then the length is checked and it's used as the sync. The other two overloads of create(...) have been modified to call the three parameter version with a null sync. The patch includes three additional tests to check the error cases (invalid length) and verify that generating the same file twice results in the same byte array (i.e. exact match). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (AVRO-1685) Allow specifying sync in DataFileWriter.create
Sehrope Sarkuni created AVRO-1685: - Summary: Allow specifying sync in DataFileWriter.create Key: AVRO-1685 URL: https://issues.apache.org/jira/browse/AVRO-1685 Project: Avro Issue Type: Improvement Components: java Reporter: Sehrope Sarkuni Priority: Minor Currently DataFileWriter generates a random 16-byte sync each time a new file is created. This means that even if you write the exact same data in a new file writer, the file itself will be slightly different (specifically the sync will be different). I'd like to be able to generate the exact same file multiple times. To do so, I need a way to specify the 16-byte sync. I've created a patch that adds this functionality by adding an overload of the create() that takes a byte[] array as the third parameter. If the byte array is null then a random sync is generated using the same internal static generateSync() method as before. If it's not null then the length is checked and it's used as the sync. The other two overloads of create(...) have been modified to call the three parameter version with a null sync. The patch includes three additional tests to check the error cases (invalid length) and verify that generating the same file twice results in the same byte array (i.e. exact match). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (AVRO-1684) Add date, time, and timestamp to specific object model classes
[ https://issues.apache.org/jira/browse/AVRO-1684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14590748#comment-14590748 ] ASF GitHub Bot commented on AVRO-1684: -- GitHub user rdblue opened a pull request: https://github.com/apache/avro/pull/37 AVRO-1684: Add support for date, time, and timestamp to specific This is based on AVRO-1672, which adds the date, time, and timestamp conversions. The specific IDL has been updated with 3 new field types: date, time_ms, and timestamp_ms. These will translate to fields that use the appropriate logical types that were added in AVRO-1672. The compiler now includes logic to add date, time, and timestamp fields using Joda time objects, and the data reader and writer support classes with those objects. In addition, there is a test to validate that classes compiled from schemas with logical types before this patch still work. Field conversions have been added to SpecificRecordBase via `getConversion(int)` and `getConversion(String fieldName)`. You can merge this pull request into a Git repository by running: $ git pull https://github.com/rdblue/avro AVRO-1684-specific-date-time Alternatively you can review and apply these changes as the patch at: https://github.com/apache/avro/pull/37.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #37 commit 6a046d455bc521bd1da5a5f4ad64d4cb1a75a697 Author: Ryan Blue b...@apache.org Date: 2015-05-28T22:46:44Z AVRO-1672: Add time logical types and conversions. This uses Joda classes to represent the new logical types: * date = LocalDate * time-millis = LocalTime * timestamp-millis = DateTime Joda is an optional dependency and will only be loaded if the conversions in org.apache.avro.data.TimeConversions are referenced. By default, no logical type conversions are enabled so there is no additional runtime dependency for existing applications. commit 09b3dbebbf06bd71ccd18039e00d278226e4d2e3 Author: Ryan Blue b...@apache.org Date: 2015-05-28T23:59:40Z AVRO-1684: Add time types to the specific compiler. This adds a dependency on Joda time for both the compiler and the compiled classes. When generating Java classes, any conversion that is registered with the compiler's SpecificData instance will be used. commit bead1610648cc7c0cb63a4b19e0d503b31f1e043 Author: Ryan Blue b...@apache.org Date: 2015-06-16T20:08:46Z AVRO-1672: Add tests for specific date/time types. commit 8754a2ba8ed4675b98bb42bf60043acdebb81326 Author: Ryan Blue b...@apache.org Date: 2015-06-17T19:20:48Z AVRO-1684: Make specific time support compatible with existing classes. Add date, time, and timestamp to specific object model classes -- Key: AVRO-1684 URL: https://issues.apache.org/jira/browse/AVRO-1684 Project: Avro Issue Type: New Feature Components: java Affects Versions: 1.7.7 Reporter: Ryan Blue Fix For: 1.7.8 AVRO-1672 adds conversions for date, time, and timestamp. These should be available to specific classes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] avro pull request: AVRO-1684: Add support for date, time, and time...
GitHub user rdblue opened a pull request: https://github.com/apache/avro/pull/37 AVRO-1684: Add support for date, time, and timestamp to specific This is based on AVRO-1672, which adds the date, time, and timestamp conversions. The specific IDL has been updated with 3 new field types: date, time_ms, and timestamp_ms. These will translate to fields that use the appropriate logical types that were added in AVRO-1672. The compiler now includes logic to add date, time, and timestamp fields using Joda time objects, and the data reader and writer support classes with those objects. In addition, there is a test to validate that classes compiled from schemas with logical types before this patch still work. Field conversions have been added to SpecificRecordBase via `getConversion(int)` and `getConversion(String fieldName)`. You can merge this pull request into a Git repository by running: $ git pull https://github.com/rdblue/avro AVRO-1684-specific-date-time Alternatively you can review and apply these changes as the patch at: https://github.com/apache/avro/pull/37.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #37 commit 6a046d455bc521bd1da5a5f4ad64d4cb1a75a697 Author: Ryan Blue b...@apache.org Date: 2015-05-28T22:46:44Z AVRO-1672: Add time logical types and conversions. This uses Joda classes to represent the new logical types: * date = LocalDate * time-millis = LocalTime * timestamp-millis = DateTime Joda is an optional dependency and will only be loaded if the conversions in org.apache.avro.data.TimeConversions are referenced. By default, no logical type conversions are enabled so there is no additional runtime dependency for existing applications. commit 09b3dbebbf06bd71ccd18039e00d278226e4d2e3 Author: Ryan Blue b...@apache.org Date: 2015-05-28T23:59:40Z AVRO-1684: Add time types to the specific compiler. This adds a dependency on Joda time for both the compiler and the compiled classes. When generating Java classes, any conversion that is registered with the compiler's SpecificData instance will be used. commit bead1610648cc7c0cb63a4b19e0d503b31f1e043 Author: Ryan Blue b...@apache.org Date: 2015-06-16T20:08:46Z AVRO-1672: Add tests for specific date/time types. commit 8754a2ba8ed4675b98bb42bf60043acdebb81326 Author: Ryan Blue b...@apache.org Date: 2015-06-17T19:20:48Z AVRO-1684: Make specific time support compatible with existing classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Created] (AVRO-1684) Add date, time, and timestamp to specific object model classes
Ryan Blue created AVRO-1684: --- Summary: Add date, time, and timestamp to specific object model classes Key: AVRO-1684 URL: https://issues.apache.org/jira/browse/AVRO-1684 Project: Avro Issue Type: New Feature Components: java Affects Versions: 1.7.7 Reporter: Ryan Blue Fix For: 1.7.8 AVRO-1672 adds conversions for date, time, and timestamp. These should be available to specific classes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
